Jobsub ID 252748.32@dunegpschedd02.fnal.gov
| Jobsub ID | 252748.32@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:41:11 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 08:47:05 |
| From worker node | Hostname | ccwcondor0697 |
| cpuinfo | AMD EPYC 7302 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:42:17 |
| Input files | hd-protodune:pdhd_prod_beam__238303_16_1_20251106T064415Z_gen_g4_IonScintPDExt.root_249560_23_1_20251122T075410Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7458s) |
| CPU time | 1h (6599s = 88%) |
| Max RSS bytes | 6422523904 (6124 MiB) |
| Outputting started | 2025-11-26 08:46:36 |
| Output files | |
| Finished | 2025-11-26 08:47:05 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
2363e+06 n=1342438 tag=gauss
wclsFrameSaver: saving 104627 traces tagged "wiener"
FrameSaver: q=3.54644e+06 n=1301672 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 17006 XUs and 13677 XVs -> 182 XUVs
C:0 T:1 125282 XUs and 105579 XVs -> 3586 XUVs
C:0 T:2 48742 XUs and 23159 XVs -> 523 XUVs
C:0 T:4 28 XUs and 14 XVs -> 0 XUVs
C:0 T:5 20758 XUs and 22641 XVs -> 621 XUVs
C:0 T:6 27162 XUs and 38214 XVs -> 433 XUVs
C:0 T:7 12 XUs and 15 XVs -> 0 XUVs
5345 XUVs total
2906 collection wire objects
5345 potential space points
Neighbour search...
91815 tests to find 43862 neighbours
Iterating with no regularization...
Begin: 5.97348e+07
0 5.65266e+07
1 5.63168e+07
2 5.6306e+07
Now with regularization...
Begin: 5.44322e+07
0 5.4429e+07
26-Nov-2025 09:46:29 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root"
26-Nov-2025 09:46:29 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 125.261 189.006 231.86 189.025 28.827 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00151872 0.00874946 0.0137156 0.00946919 0.00371725 10
reco:opdec:Deconvolution 0.840638 1.01588 1.27743 1.01596 0.13509 10
reco:ophitspe:OpHitFinderDeco 0.202527 0.218354 0.233357 0.216421 0.0107953 10
reco:opflash:OpFlashFinder 0.0563174 0.0869139 0.123761 0.0926326 0.0199631 10
reco:opslicer:OpSlicer 0.236908 0.311283 0.41467 0.301806 0.0484159 10
reco:wclsdatahd:WireCellToolkit 50.8993 62.5692 68.9208 63.3908 4.57834 10
reco:gaushit:GausHitFinder 0.674847 1.02276 1.29017 1.06692 0.17019 10
reco:nhitsfilter:NumberOfHitsFilter 0.000408852 0.000549126 0.000876405 0.00052932 0.000125173 10
reco:reco3d:SpacePointSolver 0.632485 1.34738 2.33097 1.16774 0.494837 10
reco:hitpdune:DisambigFromSpacePoints 0.328628 0.858669 1.28236 0.826074 0.261503 10
reco:pandora:StandardPandora 16.6227 40.7265 74.031 41.042 14.7779 10
reco:pandoraWriter:StandardPandora 0.265149 0.373153 0.452825 0.378798 0.057959 10
reco:pandoraTrack:LArPandoraTrackCreation 1.33673 2.8465 4.81056 2.72295 1.0023 10
reco:pandoraShower:LArPandoraShowerCreation 0.973761 2.5768 4.56406 3.00211 1.13279 10
reco:pandoracalo:Calorimetry 0.813368 1.32108 2.21725 1.32004 0.393126 10
reco:pandoracalonosce:Calorimetry 0.738343 1.28737 2.2892 1.27451 0.415601 10
reco:pandoraShowercalo:ShowerCalorimetry 1.05952 2.46343 4.27335 2.66571 1.07776 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.963547 2.29456 3.72169 2.63554 0.979116 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0491288 0.0799899 0.130099 0.0769006 0.0232854 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0488038 0.0779354 0.120658 0.0745991 0.0210394 10
reco:emtrkmichelid:EmTrackMichelId 25.3611 45.1218 59.4622 45.8925 10.7762 10
reco:crttag:SingleCRTMatchingProducer 0.00558973 0.00745468 0.00936709 0.00781391 0.00104952 10
reco:crtreco:TwoCRTMatchingProducer 0.00480986 0.00617781 0.00721012 0.00648345 0.000728198 10
reco:pandora2Track:LArPandoraTrackCreation 2.91838 6.60563 9.84021 6.71319 2.16204 10
reco:pandora2calo:ShowerCalorimetry 0.916172 2.15316 3.67695 2.34651 0.926954 10
reco:pandora2calonosce:ShowerCalorimetry 0.829038 2.06014 3.69288 2.34594 0.873363 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0552143 0.0761918 0.0969446 0.0743603 0.0130772 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0525187 0.0700238 0.0955175 0.071484 0.0146612 10
reco:pandora2Shower:LArPandoraShowerCreation 1.12657 2.78733 4.62728 3.03487 1.03515 10
reco:pandora2Showercalo:ShowerCalorimetry 1.22814 2.84582 5.08942 3.06723 1.20131 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.27716 2.8416 4.91515 3.07776 1.12281 10
[art]:TriggerResults:TriggerResultInserter 3.6509e-05 6.35831e-05 0.000145174 5.53795e-05 3.05561e-05 10
end_path:out1:RootOutput 9.308e-06 1.36808e-05 4.0978e-05 1.07955e-05 9.14028e-06 10
end_path:out1:RootOutput(write) 2.09172 2.77648 3.35029 2.95832 0.396956 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6175.55 MB
Peak resident set size usage (VmHWM): 3973.22 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root.temp.json ====
.:
total 5861300
-rw-r--r-- 1 dune001 lbno 43199 Nov 26 09:46 jobscript.log
-rw-r--r-- 1 dune001 lbno 203 Nov 26 09:46 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 09:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 09:46 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:46 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 09:46 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 09:46 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 09:46 time.db
-rw-r--r-- 1 dune001 lbno 5949552827 Nov 26 09:46 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252748_32_1_20251126T064223Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 52105150 Nov 26 09:44 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:16 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:13 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:12 detsim_single_protodunehd_hist.root 