Jobsub ID 252436.12@dunegpschedd02.fnal.gov
| Jobsub ID | 252436.12@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 01:49:57 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 06:11:16 |
| From worker node | Hostname | n252.dur.scotgrid.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 01:55:34 |
| Input files | hd-protodune:pdhd_prod_beam__243931_97_1_20251106T051229Z_gen_g4_IonScintPDExt.root_255161_100_1_20251122T001115Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 4h (15312s) |
| CPU time | 1h (5447s = 35%) |
| Max RSS bytes | 7142563840 (6811 MiB) |
| Outputting started | 2025-11-26 06:10:47 |
| Output files | |
| Finished | 2025-11-26 06:11:16 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
gauss"
FrameSaver: q=2.58979e+06 n=1300842 tag=gauss
wclsFrameSaver: saving 103128 traces tagged "wiener"
FrameSaver: q=3.25423e+06 n=1251204 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8992 XUs and 11758 XVs -> 75 XUVs
C:0 T:1 40703 XUs and 79831 XVs -> 2249 XUVs
C:0 T:2 54213 XUs and 42967 XVs -> 2447 XUVs
C:0 T:4 2 XUs and 6 XVs -> 0 XUVs
C:0 T:5 15938 XUs and 18264 XVs -> 768 XUVs
C:0 T:6 23870 XUs and 23166 XVs -> 616 XUVs
6155 XUVs total
2913 collection wire objects
6155 potential space points
Neighbour search...
168043 tests to find 93004 neighbours
Iterating with no regularization...
Begin: 4.81353e+07
0 4.34314e+07
1 4.31835e+07
2 4.3145e+07
Now with regularization...
Begin: 4.04725e+07
0 4.04594e+07
26-Nov-2025 06:10:39 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root"
26-Nov-2025 06:10:39 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 108.767 160.257 294.657 150.447 53.8828 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00176728 0.00999711 0.0321316 0.00884995 0.00796507 10
reco:opdec:Deconvolution 0.603051 0.785838 0.990867 0.777085 0.136431 10
reco:ophitspe:OpHitFinderDeco 0.133143 0.148073 0.201788 0.141996 0.018852 10
reco:opflash:OpFlashFinder 0.0462909 0.0975106 0.165153 0.100033 0.0331702 10
reco:opslicer:OpSlicer 0.158244 0.225201 0.281108 0.236473 0.0449042 10
reco:wclsdatahd:WireCellToolkit 46.0183 47.7704 50.8049 47.8211 1.30585 10
reco:gaushit:GausHitFinder 0.551948 0.77925 1.25815 0.743219 0.210901 10
reco:nhitsfilter:NumberOfHitsFilter 0.000650911 0.00107229 0.00203212 0.000953583 0.000366105 10
reco:reco3d:SpacePointSolver 0.611509 0.973844 1.59663 0.897899 0.262728 10
reco:hitpdune:DisambigFromSpacePoints 0.386772 0.715734 1.61838 0.604485 0.351311 10
reco:pandora:StandardPandora 12.5766 27.5427 83.3405 23.3473 19.3301 10
reco:pandoraWriter:StandardPandora 0.302766 0.399824 0.608066 0.364449 0.0955854 10
reco:pandoraTrack:LArPandoraTrackCreation 1.68743 4.05164 8.64971 3.70878 2.01413 10
reco:pandoraShower:LArPandoraShowerCreation 1.24123 3.80272 10.5351 3.36948 2.75436 10
reco:pandoracalo:Calorimetry 0.66052 1.58642 3.97662 1.39451 0.931043 10
reco:pandoracalonosce:Calorimetry 0.63275 1.45109 3.39969 1.16195 0.8285 10
reco:pandoraShowercalo:ShowerCalorimetry 0.941202 3.16654 8.22666 2.57766 2.393 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.937533 2.95311 7.81486 2.35321 2.19697 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0371187 0.061341 0.0899776 0.0619208 0.0166527 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0388056 0.0567295 0.0828182 0.0553458 0.0141528 10
reco:emtrkmichelid:EmTrackMichelId 25.9817 36.3002 52.2316 36.2366 7.9246 10
reco:crttag:SingleCRTMatchingProducer 0.00638478 0.00924086 0.0166923 0.0086075 0.00280301 10
reco:crtreco:TwoCRTMatchingProducer 0.00468594 0.00703166 0.0161662 0.00604849 0.00320128 10
reco:pandora2Track:LArPandoraTrackCreation 3.47099 10.4715 24.1504 9.63035 6.81973 10
reco:pandora2calo:ShowerCalorimetry 0.816715 2.43747 6.08528 1.92563 1.76529 10
reco:pandora2calonosce:ShowerCalorimetry 0.831819 2.55748 5.93056 1.84223 1.92769 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0373893 0.05928 0.0926198 0.0589787 0.0177609 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0365513 0.0531366 0.0795428 0.0551041 0.0142553 10
reco:pandora2Shower:LArPandoraShowerCreation 1.30015 3.34587 7.27645 3.13274 1.97108 10
reco:pandora2Showercalo:ShowerCalorimetry 1.15143 3.0141 6.85829 2.57443 1.83303 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.30302 3.06826 7.68734 2.48823 2.0758 10
[art]:TriggerResults:TriggerResultInserter 4.4565e-05 0.000197105 0.00145856 5.7033e-05 0.000420617 10
end_path:out1:RootOutput 1.1101e-05 1.78807e-05 3.8242e-05 1.52885e-05 7.74066e-06 10
end_path:out1:RootOutput(write) 1.63877 2.10655 2.76268 2.08327 0.367937 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6251.14 MB
Peak resident set size usage (VmHWM): 4049.55 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root.temp.json ====
.:
total 6480112
-rw-r--r--. 1 dune004 dune 41317 Nov 26 06:10 jobscript.log
-rw-r--r--. 1 dune004 dune 204 Nov 26 06:10 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:10 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:10 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 06:10 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 06:10 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 06:10 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 06:10 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 06:10 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 06:10 time.db
-rw-r--r--. 1 dune004 dune 6577428589 Nov 26 06:10 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252436_12_1_20251126T015549Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 57874320 Nov 26 06:09 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:44 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:42 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 05:41 detsim_single_protodunehd_hist.root 