Jobsub ID 258022.53@dunegpschedd01.fnal.gov
| Jobsub ID | 258022.53@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 01:07:56 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 04:03:45 |
| From worker node | Hostname | n264.dur.scotgrid.ac.uk |
| cpuinfo | AMD EPYC 9534 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 01:12:17 |
| Input files | hd-protodune:pdhd_prod_beam__238211_149_1_20251106T044422Z_gen_g4_IonScintPDExt.root_255822_143_1_20251123T212423Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (10199s) |
| CPU time | 1h (4151s = 40%) |
| Max RSS bytes | 6587662336 (6282 MiB) |
| Outputting started | 2025-11-26 04:02:17 |
| Output files | |
| Finished | 2025-11-26 04:03:45 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
ver: saving 77211 traces tagged "gauss"
FrameSaver: q=2.46823e+06 n=1205026 tag=gauss
wclsFrameSaver: saving 97418 traces tagged "wiener"
FrameSaver: q=3.19685e+06 n=1169095 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 15994 XUs and 13904 XVs -> 112 XUVs
C:0 T:1 81305 XUs and 79473 XVs -> 2549 XUVs
C:0 T:2 19624 XUs and 19801 XVs -> 1489 XUVs
C:0 T:5 13279 XUs and 13806 XVs -> 875 XUVs
C:0 T:6 41677 XUs and 21136 XVs -> 570 XUVs
5595 XUVs total
2701 collection wire objects
5595 potential space points
Neighbour search...
118175 tests to find 65810 neighbours
Iterating with no regularization...
Begin: 4.46267e+07
0 4.22601e+07
1 4.20686e+07
2 4.20469e+07
Now with regularization...
Begin: 4.04883e+07
0 4.04821e+07
26-Nov-2025 04:02:03 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root"
26-Nov-2025 04:02:03 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 73.9509 97.1909 138.221 95.7605 16.0285 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000553913 0.00788318 0.0192045 0.00598794 0.00521473 10
reco:opdec:Deconvolution 0.632511 0.762981 1.11507 0.727693 0.149015 10
reco:ophitspe:OpHitFinderDeco 0.112132 0.134942 0.165122 0.129591 0.0197111 10
reco:opflash:OpFlashFinder 0.067645 0.0918817 0.14909 0.0844588 0.0239732 10
reco:opslicer:OpSlicer 0.167527 0.24355 0.395449 0.221637 0.0784303 10
reco:wclsdatahd:WireCellToolkit 36.5474 38.9843 45.2565 37.4729 2.77087 10
reco:gaushit:GausHitFinder 0.484588 0.619925 0.865578 0.58775 0.105318 10
reco:nhitsfilter:NumberOfHitsFilter 0.000171101 0.000253709 0.000465932 0.000208216 9.33931e-05 10
reco:reco3d:SpacePointSolver 0.347049 0.798828 2.02408 0.67553 0.478211 10
reco:hitpdune:DisambigFromSpacePoints 0.307013 0.490496 0.785925 0.439158 0.143241 10
reco:pandora:StandardPandora 10.5874 15.1967 26.8211 12.8499 5.54254 10
reco:pandoraWriter:StandardPandora 0.177902 0.238875 0.310084 0.228402 0.0375112 10
reco:pandoraTrack:LArPandoraTrackCreation 0.807658 1.73892 2.92808 1.72452 0.561022 10
reco:pandoraShower:LArPandoraShowerCreation 0.633916 1.57643 2.8185 1.60101 0.554946 10
reco:pandoracalo:Calorimetry 0.451718 0.775532 1.28007 0.724043 0.223944 10
reco:pandoracalonosce:Calorimetry 0.44052 0.766931 1.03398 0.74144 0.193945 10
reco:pandoraShowercalo:ShowerCalorimetry 0.67262 1.47095 2.20434 1.4708 0.455146 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.666319 1.40132 2.21078 1.30683 0.43746 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0287926 0.0429037 0.0592866 0.0435099 0.00850074 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0276642 0.0432512 0.0611722 0.0448977 0.00876272 10
reco:emtrkmichelid:EmTrackMichelId 12.8301 18.1407 24.6589 18.1737 2.87939 10
reco:crttag:SingleCRTMatchingProducer 0.00366204 0.0050013 0.00772422 0.00439009 0.00132701 10
reco:crtreco:TwoCRTMatchingProducer 0.00269306 0.00386205 0.00569787 0.00363445 0.000889241 10
reco:pandora2Track:LArPandoraTrackCreation 1.92354 3.88919 7.67616 3.67997 1.42893 10
reco:pandora2calo:ShowerCalorimetry 0.604536 1.33694 3.2531 1.16393 0.702635 10
reco:pandora2calonosce:ShowerCalorimetry 0.600256 1.26652 2.36052 1.09043 0.474938 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0277423 0.0425299 0.0607516 0.0406848 0.00898832 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.029366 0.0401321 0.046255 0.041785 0.00563689 10
reco:pandora2Shower:LArPandoraShowerCreation 0.777207 1.65859 4.02355 1.51803 0.834589 10
reco:pandora2Showercalo:ShowerCalorimetry 0.823408 1.74653 3.18535 1.64577 0.614309 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.845337 1.62912 2.90074 1.53858 0.542586 10
[art]:TriggerResults:TriggerResultInserter 2.251e-05 3.2444e-05 7.374e-05 2.8465e-05 1.45016e-05 10
end_path:out1:RootOutput 6.4e-06 8.427e-06 1.9871e-05 7.16e-06 3.86178e-06 10
end_path:out1:RootOutput(write) 1.46535 1.89484 2.85401 1.85624 0.349409 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6809.4 MB
Peak resident set size usage (VmHWM): 4324.93 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root.temp.json ====
.:
total 6069740
-rw-r--r--. 1 dune004 dune 40800 Nov 26 04:02 jobscript.log
-rw-r--r--. 1 dune004 dune 195 Nov 26 04:02 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:02 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:02 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 04:02 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 04:02 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 04:02 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 04:02 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 04:02 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 04:02 time.db
-rw-r--r--. 1 dune004 dune 6160304372 Nov 26 04:02 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_53_1_20251126T011229Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 54776280 Nov 26 04:01 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:46 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:44 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 03:44 detsim_single_protodunehd_hist.root 