Jobsub ID 258379.26@dunegpschedd01.fnal.gov
| Jobsub ID | 258379.26@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 05:51:08 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 07:39:22 |
| From worker node | Hostname | ccwcondor0386 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 05:52:35 |
| Input files | hd-protodune:pdhd_prod_beam__243935_0_1_20251106T052124Z_gen_g4_IonScintPDExt.root_249734_44_1_20251122T173345Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6380s) |
| CPU time | 1h (5860s = 91%) |
| Max RSS bytes | 6995484672 (6671 MiB) |
| Outputting started | 2025-11-26 07:38:56 |
| Output files | |
| Finished | 2025-11-26 07:39:22 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
traces tagged "gauss"
FrameSaver: q=2.05816e+06 n=1095805 tag=gauss
wclsFrameSaver: saving 85181 traces tagged "wiener"
FrameSaver: q=2.53859e+06 n=1058046 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12102 XUs and 6600 XVs -> 69 XUVs
C:0 T:1 44177 XUs and 30264 XVs -> 1255 XUVs
C:0 T:2 41867 XUs and 29345 XVs -> 859 XUVs
C:0 T:5 28784 XUs and 23417 XVs -> 1363 XUVs
C:0 T:6 16102 XUs and 13732 XVs -> 207 XUVs
C:0 T:7 6 XUs and 3 XVs -> 0 XUVs
3753 XUVs total
2171 collection wire objects
3753 potential space points
Neighbour search...
146937 tests to find 54304 neighbours
Iterating with no regularization...
Begin: 3.79291e+07
0 3.43453e+07
1 3.41745e+07
2 3.41606e+07
Now with regularization...
Begin: 3.21319e+07
0 3.21249e+07
26-Nov-2025 08:38:50 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root"
26-Nov-2025 08:38:50 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 108.075 135.573 180.723 130.241 23.4168 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00135823 0.00841934 0.0161113 0.00716438 0.00483079 10
reco:opdec:Deconvolution 0.730509 0.938124 1.06481 0.962092 0.0957435 10
reco:ophitspe:OpHitFinderDeco 0.138536 0.193195 0.22029 0.198321 0.0217198 10
reco:opflash:OpFlashFinder 0.0687723 0.105323 0.143627 0.101392 0.0237699 10
reco:opslicer:OpSlicer 0.201989 0.306352 0.373045 0.30813 0.0502027 10
reco:wclsdatahd:WireCellToolkit 45.973 48.5516 51.6404 48.3696 1.75763 10
reco:gaushit:GausHitFinder 0.672639 0.934584 1.28492 0.891696 0.192004 10
reco:nhitsfilter:NumberOfHitsFilter 0.000176187 0.00031256 0.000532497 0.000290239 9.44436e-05 10
reco:reco3d:SpacePointSolver 0.530283 0.876974 1.22124 0.917405 0.263686 10
reco:hitpdune:DisambigFromSpacePoints 0.492191 0.755336 1.1321 0.757864 0.220496 10
reco:pandora:StandardPandora 13.8993 27.1135 43.6248 21.9257 10.1833 10
reco:pandoraWriter:StandardPandora 0.297036 0.347342 0.436836 0.324394 0.0480237 10
reco:pandoraTrack:LArPandoraTrackCreation 1.20971 2.42689 4.07083 2.04846 0.974703 10
reco:pandoraShower:LArPandoraShowerCreation 1.37111 2.55644 4.64454 2.17628 1.13145 10
reco:pandoracalo:Calorimetry 0.640191 1.25348 2.29097 1.02094 0.546532 10
reco:pandoracalonosce:Calorimetry 0.684895 1.27766 2.45795 0.990198 0.568019 10
reco:pandoraShowercalo:ShowerCalorimetry 1.1851 2.48202 4.06689 2.15792 0.998838 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.1301 2.27068 3.74019 1.98265 0.911613 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0517683 0.0698136 0.0852557 0.0690555 0.0119803 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0528694 0.0667978 0.0830706 0.0672748 0.0100183 10
reco:emtrkmichelid:EmTrackMichelId 17.4369 21.4739 27.1676 21.0714 2.82928 10
reco:crttag:SingleCRTMatchingProducer 0.00544316 0.0083672 0.0125846 0.00885315 0.0020745 10
reco:crtreco:TwoCRTMatchingProducer 0.00422337 0.00616422 0.00959793 0.00597432 0.00146894 10
reco:pandora2Track:LArPandoraTrackCreation 4.06309 6.54177 11.6768 5.81176 2.30907 10
reco:pandora2calo:ShowerCalorimetry 1.22377 2.17985 4.47076 1.82332 0.98714 10
reco:pandora2calonosce:ShowerCalorimetry 1.19751 2.24335 4.27068 1.92925 0.973741 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0538775 0.0713568 0.0928747 0.0697981 0.0115342 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0526793 0.0671017 0.0869542 0.0649873 0.0110288 10
reco:pandora2Shower:LArPandoraShowerCreation 1.61474 2.45148 3.72939 2.20888 0.750694 10
reco:pandora2Showercalo:ShowerCalorimetry 1.55647 2.66874 4.47167 2.32988 1.03088 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.47436 2.60098 4.58799 2.3574 1.02737 10
[art]:TriggerResults:TriggerResultInserter 4.3977e-05 6.06269e-05 0.000107934 5.72315e-05 1.74153e-05 10
end_path:out1:RootOutput 1.2349e-05 1.52912e-05 3.1327e-05 1.3841e-05 5.37967e-06 10
end_path:out1:RootOutput(write) 2.23662 2.53198 3.09343 2.4566 0.281263 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6842.91 MB
Peak resident set size usage (VmHWM): 4329.24 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root.temp.json ====
.:
total 6393344
-rw-r--r-- 1 dune001 lbno 40590 Nov 26 08:38 jobscript.log
-rw-r--r-- 1 dune001 lbno 185 Nov 26 08:38 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 08:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 08:38 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:38 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 08:38 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 08:38 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 08:38 time.db
-rw-r--r-- 1 dune001 lbno 6491358212 Nov 26 08:38 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_26_1_20251126T055242Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 55115080 Nov 26 08:38 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:17 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:14 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:14 detsim_single_protodunehd_hist.root 