Jobsub ID 258379.12@dunegpschedd01.fnal.gov
| Jobsub ID | 258379.12@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 05:51:08 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 07:38:08 |
| From worker node | Hostname | ccwcondor0386 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 05:52:57 |
| Input files | hd-protodune:pdhd_prod_beam__238838_49_1_20251106T222804Z_gen_g4_IonScintPDExt.root_255822_103_1_20251123T212253Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6280s) |
| CPU time | 1h (6070s = 96%) |
| Max RSS bytes | 7197741056 (6864 MiB) |
| Outputting started | 2025-11-26 07:37:38 |
| Output files | |
| Finished | 2025-11-26 07:38:08 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
meSaver: saving 102029 traces tagged "wiener"
FrameSaver: q=3.53696e+06 n=1291858 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14497 XUs and 9146 XVs -> 151 XUVs
C:0 T:1 122787 XUs and 74778 XVs -> 3444 XUVs
C:0 T:2 37116 XUs and 31959 XVs -> 931 XUVs
C:0 T:3 44 XUs and 25 XVs -> 0 XUVs
C:0 T:4 63 XUs and 18 XVs -> 4 XUVs
C:0 T:5 45692 XUs and 42913 XVs -> 2171 XUVs
C:0 T:6 29624 XUs and 23884 XVs -> 567 XUVs
C:0 T:7 5 XUs and 14 XVs -> 0 XUVs
7268 XUVs total
2951 collection wire objects
7268 potential space points
Neighbour search...
199864 tests to find 104648 neighbours
Iterating with no regularization...
Begin: 4.5616e+07
0 4.12057e+07
1 4.09555e+07
2 4.0931e+07
Now with regularization...
Begin: 3.91953e+07
0 3.91885e+07
26-Nov-2025 08:37:32 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root"
26-Nov-2025 08:37:32 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 110.274 143.921 174.699 146.053 19.6257 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00119266 0.00861804 0.0141845 0.0100498 0.00426987 10
reco:opdec:Deconvolution 0.89645 1.01932 1.30062 0.996861 0.103523 10
reco:ophitspe:OpHitFinderDeco 0.194087 0.212762 0.235969 0.210891 0.0111051 10
reco:opflash:OpFlashFinder 0.101758 0.118748 0.15393 0.116369 0.0144835 10
reco:opslicer:OpSlicer 0.294212 0.346445 0.40051 0.34934 0.0311926 10
reco:wclsdatahd:WireCellToolkit 45.1368 46.1744 47.3716 46.0472 0.737932 10
reco:gaushit:GausHitFinder 0.78288 1.01067 1.14473 1.03174 0.114204 10
reco:nhitsfilter:NumberOfHitsFilter 0.00027067 0.000398782 0.000722985 0.000352614 0.000132956 10
reco:reco3d:SpacePointSolver 0.559231 1.06748 1.69076 1.02092 0.378104 10
reco:hitpdune:DisambigFromSpacePoints 0.479895 0.875096 1.22666 0.889378 0.22314 10
reco:pandora:StandardPandora 18.2476 26.5001 35.556 26.8332 5.97452 10
reco:pandoraWriter:StandardPandora 0.284779 0.391595 0.476661 0.400644 0.0464855 10
reco:pandoraTrack:LArPandoraTrackCreation 1.52866 3.64547 5.68279 3.28177 1.24094 10
reco:pandoraShower:LArPandoraShowerCreation 1.94026 3.5153 4.99928 3.46895 0.972488 10
reco:pandoracalo:Calorimetry 0.736993 1.54994 2.60428 1.39819 0.514464 10
reco:pandoracalonosce:Calorimetry 0.745931 1.57155 2.4526 1.44161 0.477592 10
reco:pandoraShowercalo:ShowerCalorimetry 1.92756 3.34378 4.92151 3.49058 1.00667 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.67007 3.0466 4.82177 3.13072 0.970075 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0538721 0.0757207 0.101529 0.0762593 0.0140546 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0538644 0.0717796 0.0959226 0.0722756 0.0131586 10
reco:emtrkmichelid:EmTrackMichelId 18.6472 22.82 27.8272 22.984 2.78096 10
reco:crttag:SingleCRTMatchingProducer 0.00547785 0.0102513 0.0153723 0.0101023 0.00353537 10
reco:crtreco:TwoCRTMatchingProducer 0.00407309 0.00779489 0.0146203 0.00704151 0.0030868 10
reco:pandora2Track:LArPandoraTrackCreation 4.03296 8.21886 11.1471 9.34252 2.40205 10
reco:pandora2calo:ShowerCalorimetry 1.17948 2.80003 4.19106 3.11814 0.884237 10
reco:pandora2calonosce:ShowerCalorimetry 1.25478 2.62389 3.77714 2.80952 0.738239 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0505685 0.0749694 0.0908329 0.0788716 0.0143377 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0490271 0.0715807 0.0897049 0.0732166 0.0143493 10
reco:pandora2Shower:LArPandoraShowerCreation 1.53096 3.24845 4.35998 3.6167 0.91662 10
reco:pandora2Showercalo:ShowerCalorimetry 1.59115 3.39935 5.01501 3.35428 1.07012 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.45833 3.26782 4.87579 3.30648 1.09135 10
[art]:TriggerResults:TriggerResultInserter 3.1457e-05 6.18508e-05 0.00013228 5.4127e-05 2.74062e-05 10
end_path:out1:RootOutput 7.2e-06 1.33351e-05 3.5634e-05 1.09815e-05 7.61356e-06 10
end_path:out1:RootOutput(write) 2.21348 2.62863 3.23537 2.64444 0.309002 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7056.96 MB
Peak resident set size usage (VmHWM): 4536.11 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root.temp.json ====
.:
total 6677460
-rw-r--r-- 1 dune001 lbno 43419 Nov 26 08:37 jobscript.log
-rw-r--r-- 1 dune001 lbno 206 Nov 26 08:37 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:37 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:37 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 08:37 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 08:37 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:37 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 08:37 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 08:37 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 08:37 time.db
-rw-r--r-- 1 dune001 lbno 6780369702 Nov 26 08:37 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258379_12_1_20251126T055305Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 57035130 Nov 26 08:36 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:14 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:12 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:11 detsim_single_protodunehd_hist.root 