justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252989.1@dunegpschedd02.fnal.gov

Jobsub ID252989.1@dunegpschedd02.fnal.gov
Workflow ID10765
Stage ID1
User namehiguera@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 11:38:24
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-11-26 15:19:31
From worker nodeHostnamewn75
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 11:39:57
Input fileshd-protodune:pdhd_prod_beam__238266_8_1_20251106T055437Z_gen_g4_IonScintPDExt.root_255828_165_1_20251123T220843Z_PDInt.root
JobscriptExit code0
Real time3h (12914s)
CPU time2h (10590s = 82%)
Max RSS bytes4397772800 (4194 MiB)
Outputting started2025-11-26 15:15:13
Output files
Finished2025-11-26 15:19:31
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

8627 traces tagged "wiener"
FrameSaver: q=2.97355e+06 n=1268867 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8727 XUs and 11014 XVs -> 96 XUVs
C:0 T:1 77034 XUs and 83796 XVs -> 2482 XUVs
C:0 T:2 24882 XUs and 14304 XVs -> 212 XUVs
C:0 T:5 26708 XUs and 29773 XVs -> 632 XUVs
C:0 T:6 53266 XUs and 21666 XVs -> 503 XUVs
3925 XUVs total
2032 collection wire objects
3925 potential space points
Neighbour search...
92831 tests to find 40272 neighbours
Iterating with no regularization...
Begin: 3.25875e+07
0 3.04571e+07
1 3.03264e+07
2 3.03172e+07
Now with regularization...
Begin: 2.92425e+07
0 2.92394e+07
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
26-Nov-2025 12:14:58 -03  Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root"
26-Nov-2025 12:14:58 -03  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            189.313       224.837       298.74        214.031       32.4336        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00220319     0.0095641     0.0210385     0.010786     0.00588263       10     
reco:opdec:Deconvolution                              1.1087        1.54651       2.08328       1.55363      0.234778        10     
reco:ophitspe:OpHitFinderDeco                        0.311253      0.342636      0.381811      0.343781      0.0173218       10     
reco:opflash:OpFlashFinder                           0.113521      0.166452      0.263435      0.163961      0.0388297       10     
reco:opslicer:OpSlicer                               0.342767      0.507208      0.754365      0.494296      0.121533        10     
reco:wclsdatahd:WireCellToolkit                       67.3996       69.7015       72.2475       69.8062       1.40148        10     
reco:gaushit:GausHitFinder                           0.967788       1.53407       2.37487       1.43098      0.440285        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000315012   0.000504534   0.000751268   0.000516039   0.000128295      10     
reco:reco3d:SpacePointSolver                         0.902958       1.66821       2.54268       1.58415      0.466442        10     
reco:hitpdune:DisambigFromSpacePoints                 0.88145       1.42485       1.97697       1.39409      0.305325        10     
reco:pandora:StandardPandora                          28.0482       40.643        52.0775       40.7299       8.86501        10     
reco:pandoraWriter:StandardPandora                   0.513491      0.758318       1.16224      0.777263      0.203631        10     
reco:pandoraTrack:LArPandoraTrackCreation              1.922        3.49886       6.08199       3.39525       1.29935        10     
reco:pandoraShower:LArPandoraShowerCreation           2.3504        3.88127       7.46405       3.46235       1.44606        10     
reco:pandoracalo:Calorimetry                          1.01225       1.85563       3.15252       1.78229      0.704104        10     
reco:pandoracalonosce:Calorimetry                    0.999059       1.6959        2.89341       1.57799      0.589725        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.71003       3.58925       7.33776       3.13236       1.53387        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         2.27717       3.2676        6.38075       3.00539       1.17548        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0709911     0.109854      0.169144      0.112835      0.0283039       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.055418       0.10661       0.16339      0.109869      0.0303086       10     
reco:emtrkmichelid:EmTrackMichelId                    47.1315       53.4674       71.5871       50.7078       7.31459        10     
reco:crttag:SingleCRTMatchingProducer               0.00772231     0.0100328     0.0143894    0.00967198    0.00196703       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00642913    0.00819465     0.0109393     0.0078615    0.00129922       10     
reco:pandora2Track:LArPandoraTrackCreation            5.94978       9.63895       17.1888       9.24734       3.26495        10     
reco:pandora2calo:ShowerCalorimetry                   2.0201        3.33856       6.82598       2.77262       1.41795        10     
reco:pandora2calonosce:ShowerCalorimetry              1.79514       3.22556       6.90634       2.95663       1.42308        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0530046     0.111995       0.16838      0.118137      0.0291673       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0542425     0.108757      0.162101      0.108562      0.0276623       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.70299       4.10197       7.26442       3.89516       1.23466        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.52556       4.25059       8.25941       3.74034       1.61822        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.8957        4.22623       7.67844       4.10374       1.55704        10     
[art]:TriggerResults:TriggerResultInserter          2.5921e-05    5.1208e-05    0.000147756   3.9246e-05    3.41815e-05      10     
end_path:out1:RootOutput                             4.72e-06     8.9585e-06    3.2001e-05    6.4055e-06    7.7388e-06       10     
end_path:out1:RootOutput(write)                       3.97308       5.86325       7.50803       5.82976      0.974812        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6557.71 MB
  Peak resident set size usage (VmHWM): 4322.07 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
    arbjson = json.load(injson)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root.temp.json ====
.:
total 6652684
-rw-r--r-- 1 nobody nobody      43218 Nov 26 12:15 jobscript.log
-rw-r--r-- 1 nobody nobody        204 Nov 26 12:15 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody          0 Nov 26 12:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody         31 Nov 26 12:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody         25 Nov 26 12:15 all-input-dids.txt
-rw-r--r-- 1 nobody nobody        519 Nov 26 12:14 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody       3574 Nov 26 12:14 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody     204800 Nov 26 12:14 mem.db
-rw-r--r-- 1 nobody nobody      36864 Nov 26 12:14 time.db
-rw-r--r-- 1 nobody nobody 6755936028 Nov 26 12:14 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252989_1_1_20251126T114004Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody   56096060 Nov 26 12:13 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:38 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody          0 Nov 26 11:34 debugprod.log
-rw-r--r-- 1 nobody nobody        519 Nov 26 11:34 detsim_single_protodunehd_hist.root
justIN time: 2026-02-04 07:18:55 UTC       justIN version: 01.06.00