Jobsub ID 252748.33@dunegpschedd02.fnal.gov
| Jobsub ID | 252748.33@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:41:11 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 08:32:41 |
| From worker node | Hostname | ccwcondor0005 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:42:21 |
| Input files | hd-protodune:pdhd_prod_beam__238276_127_1_20251106T060914Z_gen_g4_IonScintPDExt.root_255163_76_1_20251122T001437Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6587s) |
| CPU time | 1h (5996s = 91%) |
| Max RSS bytes | 7399112704 (7056 MiB) |
| Outputting started | 2025-11-26 08:32:09 |
| Output files | |
| Finished | 2025-11-26 08:32:41 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
0188 traces tagged "wiener"
FrameSaver: q=3.38404e+06 n=1348629 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 15702 XUs and 10957 XVs -> 195 XUVs
C:0 T:1 94839 XUs and 76146 XVs -> 4910 XUVs
C:0 T:2 23828 XUs and 28271 XVs -> 389 XUVs
C:0 T:3 65 XUs and 36 XVs -> 1 XUVs
C:0 T:4 11 XUs and 12 XVs -> 0 XUVs
C:0 T:5 58267 XUs and 55019 XVs -> 2580 XUVs
C:0 T:6 46776 XUs and 53298 XVs -> 1697 XUVs
C:0 T:7 10 XUs and 26 XVs -> 1 XUVs
9773 XUVs total
3767 collection wire objects
9773 potential space points
Neighbour search...
670591 tests to find 240586 neighbours
Iterating with no regularization...
Begin: 4.09541e+07
0 3.58073e+07
1 3.53475e+07
2 3.52939e+07
3 3.52791e+07
Now with regularization...
Begin: 3.32353e+07
0 3.32254e+07
26-Nov-2025 09:31:57 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root"
26-Nov-2025 09:31:57 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 96.4587 148.389 253.906 145.178 44.5281 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00127005 0.0105061 0.0282374 0.0104279 0.0072176 10
reco:opdec:Deconvolution 0.724863 0.928427 1.10751 0.977312 0.121064 10
reco:ophitspe:OpHitFinderDeco 0.146482 0.190818 0.218421 0.19343 0.0202682 10
reco:opflash:OpFlashFinder 0.0765564 0.106916 0.143395 0.108403 0.0215005 10
reco:opslicer:OpSlicer 0.232967 0.310996 0.385185 0.316885 0.0535033 10
reco:wclsdatahd:WireCellToolkit 42.0516 47.1208 56.7441 46.5882 3.91131 10
reco:gaushit:GausHitFinder 0.617614 0.962589 1.21363 0.999616 0.186633 10
reco:nhitsfilter:NumberOfHitsFilter 0.000239464 0.000355434 0.000654965 0.000324923 0.000108085 10
reco:reco3d:SpacePointSolver 0.31772 1.34817 2.95122 1.42351 0.734347 10
reco:hitpdune:DisambigFromSpacePoints 0.387024 0.840941 1.41576 0.916594 0.308082 10
reco:pandora:StandardPandora 12.8701 36.8727 119.031 27.8848 29.6445 10
reco:pandoraWriter:StandardPandora 0.196109 0.373348 0.506743 0.363503 0.0894463 10
reco:pandoraTrack:LArPandoraTrackCreation 1.26759 3.08328 4.94132 2.91469 1.24997 10
reco:pandoraShower:LArPandoraShowerCreation 0.951815 2.92784 4.38176 3.11127 1.08454 10
reco:pandoracalo:Calorimetry 0.706337 1.48558 2.59966 1.41733 0.60161 10
reco:pandoracalonosce:Calorimetry 0.655578 1.40257 2.4009 1.4302 0.521804 10
reco:pandoraShowercalo:ShowerCalorimetry 1.0522 2.75279 4.82242 2.90117 1.07328 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.20606 2.63089 5.27082 2.72861 1.05707 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0467015 0.0687352 0.084682 0.0701158 0.00982029 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0475378 0.0671228 0.0846642 0.0694351 0.00982601 10
reco:emtrkmichelid:EmTrackMichelId 16.5839 22.1145 29.1026 22.7993 3.83829 10
reco:crttag:SingleCRTMatchingProducer 0.00502815 0.00785792 0.0103914 0.00824989 0.00200546 10
reco:crtreco:TwoCRTMatchingProducer 0.00402047 0.00624239 0.00852042 0.00630019 0.00154832 10
reco:pandora2Track:LArPandoraTrackCreation 3.51023 6.73973 8.89203 7.10925 1.89996 10
reco:pandora2calo:ShowerCalorimetry 1.09867 2.37378 3.89475 2.29672 0.825706 10
reco:pandora2calonosce:ShowerCalorimetry 1.10195 2.21307 3.4442 2.23063 0.756744 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0440865 0.0641685 0.0874542 0.0641475 0.0157412 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.04184 0.0621042 0.0827498 0.0633316 0.0125858 10
reco:pandora2Shower:LArPandoraShowerCreation 1.36858 2.69231 3.74772 2.88339 0.78101 10
reco:pandora2Showercalo:ShowerCalorimetry 1.48616 2.99383 4.49206 3.15471 0.979209 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.41252 2.87521 4.19784 3.07617 0.960482 10
[art]:TriggerResults:TriggerResultInserter 3.5905e-05 6.47605e-05 0.000168326 5.75125e-05 3.58653e-05 10
end_path:out1:RootOutput 8.352e-06 1.35255e-05 4.0221e-05 1.01105e-05 9.10372e-06 10
end_path:out1:RootOutput(write) 1.98738 2.54768 2.92151 2.5753 0.322024 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6961.84 MB
Peak resident set size usage (VmHWM): 4408.32 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root.temp.json ====
.:
total 6415988
-rw-r--r-- 1 dune001 lbno 41053 Nov 26 09:32 jobscript.log
-rw-r--r-- 1 dune001 lbno 204 Nov 26 09:32 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:32 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:32 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 09:32 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 09:32 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:31 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 09:31 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 09:31 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 09:31 time.db
-rw-r--r-- 1 dune001 lbno 6513174507 Nov 26 09:31 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_33_1_20251126T064227Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 56483040 Nov 26 09:30 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:08 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:05 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:05 detsim_single_protodunehd_hist.root 