Jobsub ID 252498.130@dunegpschedd02.fnal.gov
| Jobsub ID | 252498.130@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 02:31:59 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 06:59:28 |
| From worker node | Hostname | n262.dur.scotgrid.ac.uk |
| cpuinfo | AMD EPYC 9534 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 02:34:30 |
| Input files | hd-protodune:pdhd_prod_beam__238276_152_1_20251106T061011Z_gen_g4_IonScintPDExt.root_255533_173_1_20251122T225502Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 4h (15860s) |
| CPU time | 1h (4874s = 30%) |
| Max RSS bytes | 7598882816 (7246 MiB) |
| Outputting started | 2025-11-26 06:58:51 |
| Output files | |
| Finished | 2025-11-26 06:59:28 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
ag=gauss
wclsFrameSaver: saving 80596 traces tagged "wiener"
FrameSaver: q=2.27548e+06 n=1012866 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9755 XUs and 5592 XVs -> 45 XUVs
C:0 T:1 42308 XUs and 32941 XVs -> 484 XUVs
C:0 T:2 26890 XUs and 35707 XVs -> 1040 XUVs
C:0 T:3 9 XUs and 29 XVs -> 0 XUVs
C:0 T:4 21 XUs and 11 XVs -> 0 XUVs
C:0 T:5 15949 XUs and 9833 XVs -> 334 XUVs
C:0 T:6 17349 XUs and 14904 XVs -> 384 XUVs
2287 XUVs total
1577 collection wire objects
2287 potential space points
Neighbour search...
28729 tests to find 14948 neighbours
Iterating with no regularization...
Begin: 2.24426e+07
0 2.17746e+07
1 2.17476e+07
2 2.17465e+07
Now with regularization...
Begin: 2.03224e+07
0 2.03218e+07
26-Nov-2025 06:58:44 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root"
26-Nov-2025 06:58:44 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 74.3837 107.089 165.62 95.8063 30.363 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00110081 0.00733158 0.0115521 0.00823599 0.00336363 10
reco:opdec:Deconvolution 0.594421 0.808487 0.96031 0.833777 0.116442 10
reco:ophitspe:OpHitFinderDeco 0.119047 0.142279 0.244452 0.127166 0.0369157 10
reco:opflash:OpFlashFinder 0.0643872 0.122815 0.205683 0.107643 0.0439321 10
reco:opslicer:OpSlicer 0.146949 0.272094 0.438654 0.248388 0.0797079 10
reco:wclsdatahd:WireCellToolkit 35.5126 38.3164 41.0666 38.7879 1.55332 10
reco:gaushit:GausHitFinder 0.499373 0.625277 0.951757 0.533237 0.166104 10
reco:nhitsfilter:NumberOfHitsFilter 0.000235551 0.000436557 0.000661503 0.000411302 0.000126725 10
reco:reco3d:SpacePointSolver 0.309286 0.896362 1.8071 0.978432 0.467034 10
reco:hitpdune:DisambigFromSpacePoints 0.273363 0.506088 0.947757 0.413116 0.23395 10
reco:pandora:StandardPandora 9.56683 19.8957 36.9185 16.5245 9.48809 10
reco:pandoraWriter:StandardPandora 0.177951 0.266226 0.424512 0.238661 0.082786 10
reco:pandoraTrack:LArPandoraTrackCreation 0.917382 2.12087 4.10169 1.72122 1.06872 10
reco:pandoraShower:LArPandoraShowerCreation 0.809734 1.99493 3.99583 1.20744 1.17744 10
reco:pandoracalo:Calorimetry 0.429232 0.986178 2.18324 0.780972 0.524032 10
reco:pandoracalonosce:Calorimetry 0.425348 0.914419 1.74131 0.751371 0.42231 10
reco:pandoraShowercalo:ShowerCalorimetry 0.714818 1.85962 4.54541 1.30434 1.25963 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.730497 1.72584 4.01091 1.06976 1.13578 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0329137 0.0456838 0.0716269 0.0388349 0.0137157 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0330311 0.0456391 0.0694531 0.0405 0.0130599 10
reco:emtrkmichelid:EmTrackMichelId 14.1102 18.7259 26.7168 17.2689 4.05104 10
reco:crttag:SingleCRTMatchingProducer 0.00389135 0.00539515 0.00929983 0.0047698 0.001637 10
reco:crtreco:TwoCRTMatchingProducer 0.00292198 0.00463904 0.0103341 0.00376721 0.00212384 10
reco:pandora2Track:LArPandoraTrackCreation 2.00649 5.31392 12.641 3.37241 3.5123 10
reco:pandora2calo:ShowerCalorimetry 0.62265 1.58675 3.82096 0.970468 1.10668 10
reco:pandora2calonosce:ShowerCalorimetry 0.598657 1.57165 3.66067 0.955608 1.0893 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0292814 0.0441124 0.0633986 0.0449672 0.0112869 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0316374 0.0443478 0.061257 0.0420423 0.010765 10
reco:pandora2Shower:LArPandoraShowerCreation 0.788315 2.03241 4.27375 1.40875 1.1696 10
reco:pandora2Showercalo:ShowerCalorimetry 0.822276 2.04837 4.79172 1.32832 1.34528 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.816445 1.9842 4.56701 1.29846 1.2959 10
[art]:TriggerResults:TriggerResultInserter 2.9631e-05 4.00981e-05 7.592e-05 3.623e-05 1.26122e-05 10
end_path:out1:RootOutput 4.51e-06 9.6301e-06 2.374e-05 8.565e-06 4.90806e-06 10
end_path:out1:RootOutput(write) 1.62 1.99971 2.81814 1.83693 0.387923 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7310.07 MB
Peak resident set size usage (VmHWM): 4778.94 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root.temp.json ====
.:
total 6691608
-rw-r--r--. 1 dune004 dune 41353 Nov 26 06:58 jobscript.log
-rw-r--r--. 1 dune004 dune 205 Nov 26 06:58 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 06:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 06:58 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 06:58 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 06:58 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 06:58 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 06:58 time.db
-rw-r--r--. 1 dune004 dune 6795047850 Nov 26 06:58 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252498_130_1_20251126T023437Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 56828660 Nov 26 06:58 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:41 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:39 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 06:39 detsim_single_protodunehd_hist.root 