Jobsub ID 258404.1@dunegpschedd01.fnal.gov
| Jobsub ID | 258404.1@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:33:09 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 08:36:27 |
| From worker node | Hostname | ccwcondor0722 |
| cpuinfo | AMD EPYC 7302 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:34:32 |
| Input files | hd-protodune:pdhd_prod_beam__238179_9_1_20251106T040015Z_gen_g4_IonScintPDExt.root_250157_114_1_20251123T200403Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7288s) |
| CPU time | 1h (6955s = 95%) |
| Max RSS bytes | 7997796352 (7627 MiB) |
| Outputting started | 2025-11-26 08:36:00 |
| Output files | |
| Finished | 2025-11-26 08:36:27 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
er: q=2.39859e+06 n=1161255 tag=gauss
wclsFrameSaver: saving 92889 traces tagged "wiener"
FrameSaver: q=3.08931e+06 n=1121105 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9657 XUs and 10276 XVs -> 113 XUVs
C:0 T:1 74283 XUs and 76900 XVs -> 4523 XUVs
C:0 T:2 34016 XUs and 32206 XVs -> 1528 XUVs
C:0 T:3 41 XUs and 31 XVs -> 7 XUVs
C:0 T:5 11355 XUs and 13956 XVs -> 409 XUVs
C:0 T:6 15511 XUs and 24671 XVs -> 1575 XUVs
C:0 T:7 11 XUs and 13 XVs -> 0 XUVs
8155 XUVs total
2740 collection wire objects
8155 potential space points
Neighbour search...
464025 tests to find 162034 neighbours
Iterating with no regularization...
Begin: 6.75236e+07
0 6.05804e+07
1 5.98371e+07
2 5.97937e+07
Now with regularization...
Begin: 5.77256e+07
0 5.77116e+07
26-Nov-2025 09:35:55 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root"
26-Nov-2025 09:35:55 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 137.127 179.949 251.94 162.088 38.0774 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00186351 0.00714634 0.0130933 0.00706226 0.00337736 10
reco:opdec:Deconvolution 0.747423 1.07885 1.29811 1.10065 0.165007 10
reco:ophitspe:OpHitFinderDeco 0.149638 0.219126 0.243017 0.228381 0.0267219 10
reco:opflash:OpFlashFinder 0.079384 0.11621 0.158207 0.111288 0.025378 10
reco:opslicer:OpSlicer 0.246742 0.348429 0.421537 0.357815 0.0563804 10
reco:wclsdatahd:WireCellToolkit 55.6003 59.6709 64.048 60.0237 2.88272 10
reco:gaushit:GausHitFinder 0.641936 0.991173 1.3675 0.897029 0.236896 10
reco:nhitsfilter:NumberOfHitsFilter 0.000398772 0.000565577 0.00105371 0.000472897 0.000183033 10
reco:reco3d:SpacePointSolver 0.810856 1.52693 2.78584 1.42552 0.590502 10
reco:hitpdune:DisambigFromSpacePoints 0.552841 0.995019 1.77654 0.825877 0.398676 10
reco:pandora:StandardPandora 15.2041 29.0263 45.0138 27.2681 10.7423 10
reco:pandoraWriter:StandardPandora 0.226 0.363248 0.550589 0.345805 0.104888 10
reco:pandoraTrack:LArPandoraTrackCreation 0.977917 2.64936 4.35068 2.56023 1.14538 10
reco:pandoraShower:LArPandoraShowerCreation 1.21777 2.80698 5.30241 2.15434 1.44991 10
reco:pandoracalo:Calorimetry 0.697657 1.20891 1.84816 1.17199 0.43143 10
reco:pandoracalonosce:Calorimetry 0.585097 1.16402 1.67661 1.08275 0.374724 10
reco:pandoraShowercalo:ShowerCalorimetry 1.30157 2.57918 4.30519 1.968 1.19947 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.30275 2.63158 4.96637 1.88808 1.30762 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0609156 0.0835383 0.130385 0.0747972 0.0212633 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0589001 0.0776756 0.12238 0.068811 0.0206263 10
reco:emtrkmichelid:EmTrackMichelId 33.0055 47.9765 69.7772 43.7063 11.3002 10
reco:crttag:SingleCRTMatchingProducer 0.00537201 0.0086029 0.0125931 0.00843424 0.00223716 10
reco:crtreco:TwoCRTMatchingProducer 0.00461123 0.00718403 0.0105772 0.006712 0.0016442 10
reco:pandora2Track:LArPandoraTrackCreation 3.80001 7.26504 13.5934 5.65543 3.25505 10
reco:pandora2calo:ShowerCalorimetry 1.27031 2.47948 4.48613 1.69179 1.25616 10
reco:pandora2calonosce:ShowerCalorimetry 1.20922 2.38515 4.15017 1.79628 1.09653 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0668822 0.0840139 0.10914 0.0757423 0.0157356 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0655745 0.0799358 0.104846 0.0746651 0.0129365 10
reco:pandora2Shower:LArPandoraShowerCreation 1.65518 2.99263 5.41192 2.71213 1.21008 10
reco:pandora2Showercalo:ShowerCalorimetry 1.67814 3.08638 5.71379 2.51695 1.29689 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.66592 3.02506 5.75628 2.37474 1.36245 10
[art]:TriggerResults:TriggerResultInserter 2.8493e-05 4.85724e-05 0.000133031 4.0772e-05 2.86758e-05 10
end_path:out1:RootOutput 5.059e-06 1.15418e-05 3.693e-05 9.2425e-06 8.5806e-06 10
end_path:out1:RootOutput(write) 2.24534 2.8662 4.16086 2.65638 0.600374 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6422.52 MB
Peak resident set size usage (VmHWM): 4211.88 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root.temp.json ====
.:
total 6914008
-rw-r--r-- 1 dune001 lbno 41763 Nov 26 09:36 jobscript.log
-rw-r--r-- 1 dune001 lbno 205 Nov 26 09:36 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:36 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:36 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 09:36 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 09:35 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:35 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 09:35 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 09:35 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 09:35 time.db
-rw-r--r-- 1 dune001 lbno 7022422037 Nov 26 09:35 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258404_1_1_20251126T063437Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 57206290 Nov 26 09:34 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:08 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 09:04 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 09:03 detsim_single_protodunehd_hist.root 