Jobsub ID 252787.16@dunegpschedd02.fnal.gov
| Jobsub ID | 252787.16@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 07:51:14 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 09:32:19 |
| From worker node | Hostname | ccwcondor0366 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 07:52:48 |
| Input files | hd-protodune:pdhd_prod_beam__238443_10_1_20251106T085454Z_gen_g4_IonScintPDExt.root_250166_161_1_20251123T202716Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5946s) |
| CPU time | 1h (5447s = 91%) |
| Max RSS bytes | 6822486016 (6506 MiB) |
| Outputting started | 2025-11-26 09:31:55 |
| Output files | |
| Finished | 2025-11-26 09:32:19 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
263854 tag=gauss
wclsFrameSaver: saving 99187 traces tagged "wiener"
FrameSaver: q=2.98334e+06 n=1194893 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9519 XUs and 9441 XVs -> 68 XUVs
C:0 T:1 52070 XUs and 70251 XVs -> 3817 XUVs
C:0 T:2 27346 XUs and 20308 XVs -> 368 XUVs
C:0 T:4 13 XUs and 27 XVs -> 7 XUVs
C:0 T:5 17082 XUs and 23532 XVs -> 716 XUVs
C:0 T:6 34441 XUs and 42764 XVs -> 2541 XUVs
C:0 T:7 1 XUs and 11 XVs -> 1 XUVs
7518 XUVs total
2710 collection wire objects
7518 potential space points
Neighbour search...
435260 tests to find 260172 neighbours
Iterating with no regularization...
Begin: 1.90312e+07
0 1.73184e+07
1 1.71854e+07
2 1.71648e+07
3 1.71582e+07
Now with regularization...
Begin: 1.53503e+07
0 1.53446e+07
26-Nov-2025 10:31:50 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root"
26-Nov-2025 10:31:50 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 92.4609 130.508 179.624 116.855 28.34 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000828765 0.0087433 0.016816 0.0092519 0.00542827 10
reco:opdec:Deconvolution 0.631652 0.757162 0.990088 0.734351 0.116014 10
reco:ophitspe:OpHitFinderDeco 0.112463 0.164135 0.214444 0.168925 0.039828 10
reco:opflash:OpFlashFinder 0.0651825 0.0839667 0.105684 0.0871271 0.0131635 10
reco:opslicer:OpSlicer 0.172588 0.242531 0.354831 0.238362 0.0532247 10
reco:wclsdatahd:WireCellToolkit 41.7564 43.5846 47.0615 43.377 1.55523 10
reco:gaushit:GausHitFinder 0.614635 0.814337 1.13646 0.838346 0.155045 10
reco:nhitsfilter:NumberOfHitsFilter 0.00018496 0.000420554 0.00116772 0.000327035 0.0002615 10
reco:reco3d:SpacePointSolver 0.542754 1.38849 3.38317 1.20254 0.752832 10
reco:hitpdune:DisambigFromSpacePoints 0.332396 0.644623 0.974419 0.674662 0.17566 10
reco:pandora:StandardPandora 12.5442 29.1739 74.6019 22.3208 18.2454 10
reco:pandoraWriter:StandardPandora 0.229647 0.305937 0.416241 0.296741 0.055298 10
reco:pandoraTrack:LArPandoraTrackCreation 1.5801 2.48629 4.54134 2.05445 0.981511 10
reco:pandoraShower:LArPandoraShowerCreation 1.08865 2.64484 5.63553 2.07579 1.51355 10
reco:pandoracalo:Calorimetry 0.645743 1.23613 2.3721 0.945578 0.562023 10
reco:pandoracalonosce:Calorimetry 0.645705 1.2007 2.24985 1.04116 0.459416 10
reco:pandoraShowercalo:ShowerCalorimetry 1.14121 2.28835 4.12239 1.76688 1.04219 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.23596 2.22112 4.30942 1.89442 1.0267 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0444051 0.0618445 0.0739574 0.063958 0.00791293 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0460298 0.0593803 0.0685057 0.0584407 0.00707072 10
reco:emtrkmichelid:EmTrackMichelId 16.3187 21.2767 26.4304 21.4316 2.74152 10
reco:crttag:SingleCRTMatchingProducer 0.0051415 0.00796463 0.00969043 0.00823086 0.00133606 10
reco:crtreco:TwoCRTMatchingProducer 0.00388629 0.00584686 0.00756525 0.00594412 0.000982119 10
reco:pandora2Track:LArPandoraTrackCreation 2.91996 6.34978 12.6336 6.24167 2.99296 10
reco:pandora2calo:ShowerCalorimetry 0.83681 1.86327 3.20458 2.03456 0.826164 10
reco:pandora2calonosce:ShowerCalorimetry 0.785178 1.73028 2.91051 1.89259 0.706084 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0339173 0.0541216 0.084601 0.0494478 0.0153428 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0332261 0.0517494 0.0757733 0.0490925 0.0149031 10
reco:pandora2Shower:LArPandoraShowerCreation 1.0296 2.4081 4.66157 2.36081 1.06051 10
reco:pandora2Showercalo:ShowerCalorimetry 1.1577 2.56714 4.83429 2.29023 1.10716 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.14137 2.40392 4.50077 2.31183 1.03594 10
[art]:TriggerResults:TriggerResultInserter 2.7321e-05 4.64492e-05 8.3446e-05 4.42275e-05 1.49762e-05 10
end_path:out1:RootOutput 6.02e-06 1.10569e-05 2.6641e-05 9.535e-06 5.59557e-06 10
end_path:out1:RootOutput(write) 2.03935 2.24039 2.69571 2.18181 0.199539 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6768.15 MB
Peak resident set size usage (VmHWM): 4280.37 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root.temp.json ====
.:
total 6175152
-rw-r--r-- 1 dune001 lbno 64122 Nov 26 10:31 jobscript.log
-rw-r--r-- 1 dune001 lbno 204 Nov 26 10:31 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:31 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:31 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 10:31 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 10:31 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 10:31 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 10:31 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 10:31 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 10:31 time.db
-rw-r--r-- 1 dune001 lbno 6267187154 Nov 26 10:31 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252787_16_1_20251126T075252Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 55834040 Nov 26 10:30 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:10 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 10:08 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 10:08 detsim_single_protodunehd_hist.root 