Jobsub ID 258022.51@dunegpschedd01.fnal.gov
| Jobsub ID | 258022.51@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 01:07:56 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 03:04:43 |
| From worker node | Hostname | n264.dur.scotgrid.ac.uk |
| cpuinfo | AMD EPYC 9534 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 01:12:59 |
| Input files | hd-protodune:pdhd_prod_beam__238396_30_1_20251106T074602Z_gen_g4_IonScintPDExt.root_255881_31_1_20251123T231636Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6665s) |
| CPU time | 1h (4497s = 67%) |
| Max RSS bytes | 7483162624 (7136 MiB) |
| Outputting started | 2025-11-26 03:04:06 |
| Output files | |
| Finished | 2025-11-26 03:04:43 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
ameSaver: saving 75676 traces tagged "gauss"
FrameSaver: q=2.22931e+06 n=1148697 tag=gauss
wclsFrameSaver: saving 97427 traces tagged "wiener"
FrameSaver: q=2.89151e+06 n=1116080 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12505 XUs and 10136 XVs -> 113 XUVs
C:0 T:1 69618 XUs and 65050 XVs -> 1793 XUVs
C:0 T:2 54962 XUs and 56383 XVs -> 1539 XUVs
C:0 T:5 8863 XUs and 9576 XVs -> 270 XUVs
C:0 T:6 45677 XUs and 26345 XVs -> 721 XUVs
4436 XUVs total
2593 collection wire objects
4436 potential space points
Neighbour search...
70316 tests to find 38034 neighbours
Iterating with no regularization...
Begin: 4.15026e+07
0 3.92223e+07
1 3.91067e+07
2 3.91e+07
Now with regularization...
Begin: 3.72784e+07
0 3.72762e+07
26-Nov-2025 03:04:01 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root"
26-Nov-2025 03:04:01 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 86.922 98.6634 121.093 94.6458 11.7721 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0026115 0.00796257 0.0144543 0.00862604 0.00359472 10
reco:opdec:Deconvolution 0.574547 0.736158 0.954454 0.677457 0.130099 10
reco:ophitspe:OpHitFinderDeco 0.105348 0.129086 0.23116 0.120776 0.0350422 10
reco:opflash:OpFlashFinder 0.0640427 0.102898 0.173131 0.092897 0.0331571 10
reco:opslicer:OpSlicer 0.153565 0.22835 0.339783 0.222639 0.0581007 10
reco:wclsdatahd:WireCellToolkit 34.4162 36.1473 39.3501 36.2898 1.56041 10
reco:gaushit:GausHitFinder 0.474648 0.60863 0.96021 0.575097 0.12991 10
reco:nhitsfilter:NumberOfHitsFilter 0.000135111 0.000749177 0.00542745 0.000215246 0.00156092 10
reco:reco3d:SpacePointSolver 0.495069 0.747241 0.981042 0.734104 0.151049 10
reco:hitpdune:DisambigFromSpacePoints 0.307619 0.481101 0.67367 0.478129 0.115523 10
reco:pandora:StandardPandora 13.6076 19.0472 26.5958 17.2572 4.17788 10
reco:pandoraWriter:StandardPandora 0.161707 0.241376 0.288285 0.237229 0.0382152 10
reco:pandoraTrack:LArPandoraTrackCreation 0.842179 1.66749 2.11297 1.73654 0.391197 10
reco:pandoraShower:LArPandoraShowerCreation 0.91143 1.70895 2.49098 1.88983 0.494421 10
reco:pandoracalo:Calorimetry 0.414735 0.788267 1.12198 0.790835 0.202477 10
reco:pandoracalonosce:Calorimetry 0.405017 0.816253 1.23066 0.78443 0.245358 10
reco:pandoraShowercalo:ShowerCalorimetry 0.768736 1.62303 2.88908 1.47011 0.627366 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.681406 1.54951 2.99968 1.38118 0.642615 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0287637 0.0404072 0.0546492 0.0397842 0.00643423 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0288402 0.0399677 0.0527515 0.0393373 0.00598896 10
reco:emtrkmichelid:EmTrackMichelId 13.3685 17.0796 21.3531 16.8279 1.92464 10
reco:crttag:SingleCRTMatchingProducer 0.00307162 0.00455663 0.00618727 0.00426291 0.000870952 10
reco:crtreco:TwoCRTMatchingProducer 0.00241414 0.00342051 0.00414012 0.00344496 0.000436894 10
reco:pandora2Track:LArPandoraTrackCreation 1.75877 4.53662 6.9119 4.28019 1.58802 10
reco:pandora2calo:ShowerCalorimetry 0.570976 1.53428 2.92337 1.39688 0.672591 10
reco:pandora2calonosce:ShowerCalorimetry 0.557658 1.49977 2.81055 1.37255 0.678063 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0274164 0.0403215 0.0513218 0.0405985 0.00693195 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0273053 0.0402473 0.0493201 0.0407426 0.00633444 10
reco:pandora2Shower:LArPandoraShowerCreation 0.657443 1.71839 2.71775 1.8303 0.579731 10
reco:pandora2Showercalo:ShowerCalorimetry 0.748146 1.76106 3.11646 1.68562 0.638218 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.721232 1.75516 3.44614 1.63144 0.721654 10
[art]:TriggerResults:TriggerResultInserter 1.882e-05 2.81871e-05 6.443e-05 2.37555e-05 1.27664e-05 10
end_path:out1:RootOutput 3.52e-06 7.6752e-06 1.815e-05 5.47e-06 5.32602e-06 10
end_path:out1:RootOutput(write) 1.43273 1.83035 2.14577 1.81264 0.199567 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6949.71 MB
Peak resident set size usage (VmHWM): 4425.28 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root.temp.json ====
.:
total 6614704
-rw-r--r--. 1 dune004 dune 40801 Nov 26 03:04 jobscript.log
-rw-r--r--. 1 dune004 dune 185 Nov 26 03:04 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:04 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:04 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 03:04 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 03:04 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 03:04 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 03:04 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 03:04 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 03:04 time.db
-rw-r--r--. 1 dune004 dune 6717663995 Nov 26 03:04 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258022_51_1_20251126T011309Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 55467300 Nov 26 03:03 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 02:47 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 02:45 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 02:45 detsim_single_protodunehd_hist.root 