Jobsub ID 258379.6@dunegpschedd01.fnal.gov
| Jobsub ID | 258379.6@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 05:51:08 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 09:33:14 |
| From worker node | Hostname | wn123 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 05:52:43 |
| Input files | hd-protodune:pdhd_prod_beam__238203_66_1_20251106T042209Z_gen_g4_IonScintPDExt.root_255155_52_1_20251121T231958Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 3h (12966s) |
| CPU time | 2h (9327s = 71%) |
| Max RSS bytes | 4315734016 (4115 MiB) |
| Outputting started | 2025-11-26 09:28:50 |
| Output files | |
| Finished | 2025-11-26 09:33:12 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
ks
wclsFrameSaver: saving 68978 traces tagged "gauss"
FrameSaver: q=1.99898e+06 n=1001542 tag=gauss
wclsFrameSaver: saving 89334 traces tagged "wiener"
FrameSaver: q=2.50645e+06 n=967496 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9075 XUs and 9073 XVs -> 29 XUVs
C:0 T:1 37288 XUs and 41416 XVs -> 1290 XUVs
C:0 T:2 14554 XUs and 10198 XVs -> 871 XUVs
C:0 T:5 17213 XUs and 34335 XVs -> 1375 XUVs
C:0 T:6 55300 XUs and 13389 XVs -> 250 XUVs
3815 XUVs total
1485 collection wire objects
3815 potential space points
Neighbour search...
114423 tests to find 71280 neighbours
Iterating with no regularization...
Begin: 3.35195e+07
0 3.19471e+07
1 3.18304e+07
2 3.18098e+07
Now with regularization...
Begin: 2.98272e+07
0 2.98214e+07
26-Nov-2025 06:28:40 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root"
26-Nov-2025 06:28:40 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 159.556 239.21 357.206 225.823 65.6624 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00159295 0.00930689 0.0170682 0.00983204 0.00476338 10
reco:opdec:Deconvolution 1.06175 1.42696 1.98794 1.42212 0.247741 10
reco:ophitspe:OpHitFinderDeco 0.170882 0.326762 0.412453 0.334956 0.0590005 10
reco:opflash:OpFlashFinder 0.0982149 0.150018 0.282057 0.142747 0.0520578 10
reco:opslicer:OpSlicer 0.271144 0.456533 0.656217 0.449058 0.104691 10
reco:wclsdatahd:WireCellToolkit 61.4186 67.3977 69.9375 68.8792 2.75906 10
reco:gaushit:GausHitFinder 1.08007 1.54902 2.22263 1.50516 0.336238 10
reco:nhitsfilter:NumberOfHitsFilter 0.000331756 0.000469249 0.000670873 0.000461733 0.000108573 10
reco:reco3d:SpacePointSolver 0.949346 2.11017 4.05331 1.64459 1.08463 10
reco:hitpdune:DisambigFromSpacePoints 0.618136 1.62843 2.82178 1.64661 0.683071 10
reco:pandora:StandardPandora 18.9807 63.1853 181.845 48.3017 49.0525 10
reco:pandoraWriter:StandardPandora 0.408613 0.639564 0.92136 0.667613 0.178657 10
reco:pandoraTrack:LArPandoraTrackCreation 1.34975 3.41921 5.52541 3.18882 1.23601 10
reco:pandoraShower:LArPandoraShowerCreation 1.55657 3.15135 5.46922 3.13547 1.1508 10
reco:pandoracalo:Calorimetry 0.80875 1.78203 2.67845 1.63766 0.659067 10
reco:pandoracalonosce:Calorimetry 0.790335 1.76036 2.64569 1.6214 0.639596 10
reco:pandoraShowercalo:ShowerCalorimetry 1.59035 3.32258 6.08561 3.29738 1.33413 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.56768 3.16605 5.71274 3.05389 1.24926 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0811069 0.113833 0.157588 0.111412 0.0216303 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0811418 0.113684 0.162621 0.110146 0.0227765 10
reco:emtrkmichelid:EmTrackMichelId 39.0894 52.1492 71.5185 52.3945 9.81111 10
reco:crttag:SingleCRTMatchingProducer 0.00656858 0.00891521 0.0114726 0.00895436 0.00137189 10
reco:crtreco:TwoCRTMatchingProducer 0.00594072 0.00778618 0.0106848 0.00772982 0.00126011 10
reco:pandora2Track:LArPandoraTrackCreation 4.37549 8.19858 15.5552 7.66403 3.25679 10
reco:pandora2calo:ShowerCalorimetry 1.48251 2.7915 5.06441 2.67695 1.10937 10
reco:pandora2calonosce:ShowerCalorimetry 1.4601 2.70567 4.02252 2.63896 0.937987 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0816447 0.111125 0.161345 0.112405 0.0211502 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0828766 0.111267 0.160485 0.111071 0.0202211 10
reco:pandora2Shower:LArPandoraShowerCreation 1.93306 3.58975 6.0019 3.39064 1.28282 10
reco:pandora2Showercalo:ShowerCalorimetry 2.04766 3.90955 6.70851 3.56018 1.47177 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.99695 3.89207 6.57213 3.50997 1.51543 10
[art]:TriggerResults:TriggerResultInserter 2.7841e-05 4.22671e-05 9.9845e-05 3.3827e-05 2.18427e-05 10
end_path:out1:RootOutput 5e-06 7.8834e-06 2.7922e-05 5.72e-06 6.68702e-06 10
end_path:out1:RootOutput(write) 4.7224 5.86934 8.23758 5.78256 0.97899 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6196.55 MB
Peak resident set size usage (VmHWM): 3993.08 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root.temp.json ====
.:
total 6242324
-rw-r--r-- 1 nobody nobody 43773 Nov 26 06:28 jobscript.log
-rw-r--r-- 1 nobody nobody 203 Nov 26 06:28 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 06:28 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 06:28 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 06:28 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 06:28 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 06:28 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 06:28 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 06:28 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 06:28 time.db
-rw-r--r-- 1 nobody nobody 6335137606 Nov 26 06:28 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_6_1_20251126T055249Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 56687750 Nov 26 06:27 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:49 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 05:43 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 05:43 detsim_single_protodunehd_hist.root 