Jobsub ID 258577.9@dunegpschedd01.fnal.gov
| Jobsub ID | 258577.9@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 10:27:24 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce01 |
| Last heartbeat | 2025-11-26 12:16:16 |
| From worker node | Hostname | ccwcondor0635 |
| cpuinfo | AMD EPYC 7453 28-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 10:28:40 |
| Input files | hd-protodune:pdhd_prod_beam__243932_18_1_20251106T051557Z_gen_g4_IonScintPDExt.root_250168_72_1_20251123T203117Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6426s) |
| CPU time | 1h (6167s = 95%) |
| Max RSS bytes | 6658805760 (6350 MiB) |
| Outputting started | 2025-11-26 12:15:48 |
| Output files | |
| Finished | 2025-11-26 12:16:16 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9711 XUs and 7061 XVs -> 127 XUVs
C:0 T:1 77023 XUs and 57933 XVs -> 4227 XUVs
C:0 T:2 10494 XUs and 17019 XVs -> 301 XUVs
C:0 T:3 5 XUs and 7 XVs -> 2 XUVs
C:0 T:4 17 XUs and 48 XVs -> 2 XUVs
C:0 T:5 28495 XUs and 54708 XVs -> 13314 XUVs
C:0 T:6 43191 XUs and 99384 XVs -> 952 XUVs
C:0 T:7 26 XUs and 27 XVs -> 0 XUVs
18925 XUVs total
2603 collection wire objects
18925 potential space points
Neighbour search...
4690517 tests to find 2019096 neighbours
Iterating with no regularization...
Begin: 5.8256e+07
0 5.01915e+07
1 4.94686e+07
2 4.93838e+07
3 4.93563e+07
Now with regularization...
Begin: 4.74216e+07
0 4.73874e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 13:15:41 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root"
26-Nov-2025 13:15:41 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 137.909 174.596 207.109 177.764 19.2533 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00146587 0.00888653 0.023032 0.00787701 0.0060098 10
reco:opdec:Deconvolution 0.683965 0.90894 1.08277 0.886184 0.119053 10
reco:ophitspe:OpHitFinderDeco 0.140204 0.207705 0.285391 0.211831 0.0454986 10
reco:opflash:OpFlashFinder 0.0700449 0.0941791 0.127774 0.0928075 0.0177251 10
reco:opslicer:OpSlicer 0.227549 0.288449 0.366094 0.288535 0.0426943 10
reco:wclsdatahd:WireCellToolkit 49.5358 52.0988 54.4897 52.4759 1.74871 10
reco:gaushit:GausHitFinder 0.624259 0.929368 1.1144 0.949333 0.164476 10
reco:nhitsfilter:NumberOfHitsFilter 0.000268371 0.000429235 0.000608456 0.00043355 9.36179e-05 10
reco:reco3d:SpacePointSolver 0.682504 1.54708 2.42954 1.5017 0.580974 10
reco:hitpdune:DisambigFromSpacePoints 0.409405 0.87202 1.29702 0.922143 0.241704 10
reco:pandora:StandardPandora 19.6188 38.2788 67.5351 35.0978 14.4153 10
reco:pandoraWriter:StandardPandora 0.220875 0.347067 0.439522 0.34911 0.0622598 10
reco:pandoraTrack:LArPandoraTrackCreation 2.74033 3.39323 4.35302 3.21359 0.571617 10
reco:pandoraShower:LArPandoraShowerCreation 2.18703 3.55178 4.9771 3.49408 0.887898 10
reco:pandoracalo:Calorimetry 0.930431 1.62656 2.41392 1.56499 0.433789 10
reco:pandoracalonosce:Calorimetry 1.03585 1.76181 2.59146 1.71134 0.394109 10
reco:pandoraShowercalo:ShowerCalorimetry 1.8628 3.09049 4.70994 2.96998 0.91 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.7506 2.85452 4.04437 2.826 0.582716 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.045412 0.0667975 0.0884893 0.0628847 0.0170457 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0356731 0.0614068 0.0887217 0.0586531 0.0172056 10
reco:emtrkmichelid:EmTrackMichelId 32.4722 36.2127 41.2569 36.7542 3.05135 10
reco:crttag:SingleCRTMatchingProducer 0.00796414 0.0108716 0.0142848 0.0111043 0.00178587 10
reco:crtreco:TwoCRTMatchingProducer 0.00669506 0.00966882 0.0134434 0.010044 0.00215707 10
reco:pandora2Track:LArPandoraTrackCreation 6.35539 7.91109 9.83899 8.07585 1.23156 10
reco:pandora2calo:ShowerCalorimetry 1.60321 2.69261 3.37857 2.73639 0.518985 10
reco:pandora2calonosce:ShowerCalorimetry 1.49296 2.60519 3.30539 2.81023 0.561412 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.055816 0.078912 0.0978372 0.0786841 0.0122202 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0517013 0.0789862 0.110618 0.0768797 0.0165595 10
reco:pandora2Shower:LArPandoraShowerCreation 2.69896 3.72873 5.33733 3.6497 0.754038 10
reco:pandora2Showercalo:ShowerCalorimetry 2.49479 3.30134 4.57799 3.31054 0.56406 10
reco:pandora2Showercalonosce:ShowerCalorimetry 2.0397 3.21581 4.37364 3.24838 0.691765 10
[art]:TriggerResults:TriggerResultInserter 4.6908e-05 8.05475e-05 0.000120505 7.7976e-05 2.74139e-05 10
end_path:out1:RootOutput 7.213e-06 1.55066e-05 3.2691e-05 1.1506e-05 7.73776e-06 10
end_path:out1:RootOutput(write) 2.0773 2.53635 2.95132 2.47778 0.292012 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6389.56 MB
Peak resident set size usage (VmHWM): 4167.61 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root.temp.json ====
.:
total 5998668
-rw-r--r-- 1 dune001 lbno 41585 Nov 26 13:15 jobscript.log
-rw-r--r-- 1 dune001 lbno 204 Nov 26 13:15 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 13:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 13:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 13:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 13:15 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 13:15 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 13:15 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 13:15 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 13:15 time.db
-rw-r--r-- 1 dune001 lbno 6090855580 Nov 26 13:15 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258577_9_1_20251126T102846Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 51465830 Nov 26 13:14 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 12:47 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 12:44 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 12:44 detsim_single_protodunehd_hist.root 