Jobsub ID 258181.94@dunegpschedd01.fnal.gov
| Jobsub ID | 258181.94@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 03:02:03 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 07:30:26 |
| From worker node | Hostname | n262.dur.scotgrid.ac.uk |
| cpuinfo | AMD EPYC 9534 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 03:03:37 |
| Input files | hd-protodune:pdhd_prod_beam__238390_68_1_20251106T073529Z_gen_g4_IonScintPDExt.root_250132_27_1_20251123T181249Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 4h (15963s) |
| CPU time | 1h (4953s = 31%) |
| Max RSS bytes | 6983340032 (6659 MiB) |
| Outputting started | 2025-11-26 07:29:41 |
| Output files | |
| Finished | 2025-11-26 07:30:26 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
5 XUVs
C:0 T:3 11 XUs and 16 XVs -> 0 XUVs
C:0 T:4 30 XUs and 50 XVs -> 3 XUVs
C:0 T:5 33129 XUs and 32352 XVs -> 1460 XUVs
C:0 T:6 8886 XUs and 6403 XVs -> 567 XUVs
9277 XUVs total
3002 collection wire objects
9277 potential space points
Neighbour search...
701381 tests to find 407820 neighbours
Iterating with no regularization...
Begin: 4.44891e+07
0 4.07812e+07
1 4.0396e+07
2 4.03409e+07
3 4.03251e+07
Now with regularization...
Begin: 3.81833e+07
0 3.81705e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 07:29:35 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root"
26-Nov-2025 07:29:35 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 75.1322 105.246 148.808 105.789 20.9439 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00107273 0.00776866 0.01286 0.00891074 0.00400905 10
reco:opdec:Deconvolution 0.600636 0.722996 0.831311 0.721817 0.0802668 10
reco:ophitspe:OpHitFinderDeco 0.106459 0.116231 0.134251 0.114475 0.00776196 10
reco:opflash:OpFlashFinder 0.0665925 0.0885983 0.117304 0.0880844 0.0159438 10
reco:opslicer:OpSlicer 0.165068 0.212042 0.26725 0.215362 0.0361953 10
reco:wclsdatahd:WireCellToolkit 34.3688 36.8153 39.2221 36.8822 1.3944 10
reco:gaushit:GausHitFinder 0.46048 0.598363 0.755707 0.603942 0.0989491 10
reco:nhitsfilter:NumberOfHitsFilter 0.000244071 0.000431118 0.000879753 0.000430961 0.000183467 10
reco:reco3d:SpacePointSolver 0.415269 0.947996 1.74056 0.975061 0.351785 10
reco:hitpdune:DisambigFromSpacePoints 0.293018 0.558841 0.857201 0.553789 0.171499 10
reco:pandora:StandardPandora 8.34884 20.4496 41.22 18.9783 8.46828 10
reco:pandoraWriter:StandardPandora 0.189222 0.265636 0.339911 0.276475 0.0499954 10
reco:pandoraTrack:LArPandoraTrackCreation 0.989147 2.22101 4.5028 1.93534 1.0379 10
reco:pandoraShower:LArPandoraShowerCreation 0.783021 1.94558 3.98904 1.86475 0.926336 10
reco:pandoracalo:Calorimetry 0.448259 0.981985 1.73651 0.942526 0.364458 10
reco:pandoracalonosce:Calorimetry 0.504753 1.00438 1.74256 0.952149 0.364546 10
reco:pandoraShowercalo:ShowerCalorimetry 0.745806 1.87778 3.46874 1.75773 0.849865 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.717358 1.81077 3.06367 1.8761 0.760073 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0287212 0.0462805 0.0582808 0.0512885 0.0102611 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0284074 0.0441307 0.0535648 0.0477347 0.00840142 10
reco:emtrkmichelid:EmTrackMichelId 14.6541 18.558 22.4432 18.7939 2.5108 10
reco:crttag:SingleCRTMatchingProducer 0.00324761 0.00562358 0.00751563 0.00557511 0.0014683 10
reco:crtreco:TwoCRTMatchingProducer 0.00259923 0.00403102 0.00577329 0.00424571 0.000918653 10
reco:pandora2Track:LArPandoraTrackCreation 2.31815 5.06698 10.1261 5.02459 2.41578 10
reco:pandora2calo:ShowerCalorimetry 0.6765 1.52008 2.62796 1.57518 0.682626 10
reco:pandora2calonosce:ShowerCalorimetry 0.659589 1.47578 2.60206 1.48658 0.651142 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0287747 0.0422793 0.0494531 0.0458243 0.00733143 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0290219 0.0410937 0.0485964 0.0433667 0.00659065 10
reco:pandora2Shower:LArPandoraShowerCreation 0.951357 1.83436 2.85978 1.93293 0.66934 10
reco:pandora2Showercalo:ShowerCalorimetry 0.990891 1.99536 3.43899 1.99102 0.808636 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.950811 1.92723 3.26298 1.8976 0.764892 10
[art]:TriggerResults:TriggerResultInserter 2.15e-05 3.88932e-05 8.686e-05 3.49705e-05 1.73502e-05 10
end_path:out1:RootOutput 5.04e-06 1.03281e-05 1.663e-05 9.7955e-06 3.39385e-06 10
end_path:out1:RootOutput(write) 1.50854 1.88341 2.28582 1.92099 0.239442 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6963.3 MB
Peak resident set size usage (VmHWM): 4531.83 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root.temp.json ====
.:
total 6540284
-rw-r--r--. 1 dune004 dune 40974 Nov 26 07:29 jobscript.log
-rw-r--r--. 1 dune004 dune 203 Nov 26 07:29 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 07:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 07:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 07:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 07:29 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 07:29 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 07:29 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 07:29 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 07:29 time.db
-rw-r--r--. 1 dune004 dune 6638733811 Nov 26 07:29 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258181_94_1_20251126T030343Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 58180450 Nov 26 07:28 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 07:12 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 07:10 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 07:10 detsim_single_protodunehd_hist.root 