Jobsub ID 258000.27@dunegpschedd01.fnal.gov
| Jobsub ID | 258000.27@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 00:51:55 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 02:48:43 |
| From worker node | Hostname | n209.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 00:53:17 |
| Input files | hd-protodune:pdhd_prod_beam__238093_129_1_20251106T000319Z_gen_g4_IonScintPDExt.root_255824_162_1_20251123T215004Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6900s) |
| CPU time | 1h (5666s = 82%) |
| Max RSS bytes | 8286887936 (7902 MiB) |
| Outputting started | 2025-11-26 02:48:18 |
| Output files | |
| Finished | 2025-11-26 02:48:43 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
: q=3.00616e+06 n=1141340 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11659 XUs and 8620 XVs -> 146 XUVs
C:0 T:1 137808 XUs and 91846 XVs -> 4727 XUVs
C:0 T:2 12767 XUs and 12651 XVs -> 969 XUVs
C:0 T:3 4 XUs and 11 XVs -> 3 XUVs
C:0 T:4 24 XUs and 24 XVs -> 3 XUVs
C:0 T:5 29677 XUs and 23464 XVs -> 792 XUVs
C:0 T:6 18994 XUs and 26090 XVs -> 644 XUVs
7284 XUVs total
2827 collection wire objects
7284 potential space points
Neighbour search...
270936 tests to find 116044 neighbours
Iterating with no regularization...
Begin: 3.10174e+07
0 2.781e+07
1 2.75943e+07
2 2.75728e+07
Now with regularization...
Begin: 2.59008e+07
0 2.58952e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 02:48:12 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root"
26-Nov-2025 02:48:12 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 144.024 191.541 339.652 171.308 53.4076 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000738161 0.0103773 0.0318897 0.00967799 0.00789775 10
reco:opdec:Deconvolution 0.995232 1.29457 1.77511 1.29668 0.191586 10
reco:ophitspe:OpHitFinderDeco 0.236924 0.242395 0.258196 0.240643 0.00600984 10
reco:opflash:OpFlashFinder 0.106082 0.149379 0.29139 0.134351 0.0508769 10
reco:opslicer:OpSlicer 0.197511 0.334553 0.484659 0.326859 0.0727838 10
reco:wclsdatahd:WireCellToolkit 64.6877 66.9152 73.656 65.5364 3.05104 10
reco:gaushit:GausHitFinder 0.897691 1.08609 1.44766 1.00961 0.160238 10
reco:nhitsfilter:NumberOfHitsFilter 0.000412765 0.000512178 0.000668738 0.000481723 8.49682e-05 10
reco:reco3d:SpacePointSolver 0.719337 1.60898 3.48259 1.4788 0.739641 10
reco:hitpdune:DisambigFromSpacePoints 0.835673 1.15968 2.4083 0.954785 0.457626 10
reco:pandora:StandardPandora 19.1451 31.7943 69.4668 28.017 14.4544 10
reco:pandoraWriter:StandardPandora 0.291636 0.358737 0.517499 0.340868 0.0595948 10
reco:pandoraTrack:LArPandoraTrackCreation 2.68188 5.21126 12.9565 4.06164 2.80219 10
reco:pandoraShower:LArPandoraShowerCreation 2.88112 5.39448 13.3723 4.21893 2.99523 10
reco:pandoracalo:Calorimetry 1.17816 2.11776 5.46156 1.61551 1.20065 10
reco:pandoracalonosce:Calorimetry 1.10722 2.07092 5.33389 1.59895 1.18138 10
reco:pandoraShowercalo:ShowerCalorimetry 2.2732 4.2809 10.96 3.31666 2.52547 10
reco:pandoraShowercalonosce:ShowerCalorimetry 2.27642 3.881 9.93136 2.98236 2.26675 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0617139 0.0771274 0.115104 0.071299 0.0153133 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.060923 0.0760348 0.113187 0.0715074 0.0150401 10
reco:emtrkmichelid:EmTrackMichelId 21.9969 27.3515 38.1252 25.7186 4.40871 10
reco:crttag:SingleCRTMatchingProducer 0.0101057 0.0135574 0.021044 0.0119948 0.00321626 10
reco:crtreco:TwoCRTMatchingProducer 0.0075386 0.0108917 0.0185604 0.00923274 0.00354459 10
reco:pandora2Track:LArPandoraTrackCreation 6.56587 12.5744 31.0772 9.78154 6.94563 10
reco:pandora2calo:ShowerCalorimetry 1.87638 3.50211 9.21263 2.65388 2.20912 10
reco:pandora2calonosce:ShowerCalorimetry 1.77305 3.28514 8.68899 2.49308 2.0668 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0576261 0.0730578 0.110078 0.0677183 0.0155172 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0575325 0.0734642 0.109612 0.0684817 0.0151209 10
reco:pandora2Shower:LArPandoraShowerCreation 2.20284 4.90787 12.5388 3.94057 2.84137 10
reco:pandora2Showercalo:ShowerCalorimetry 2.27232 4.44247 12.0992 3.32375 2.84785 10
reco:pandora2Showercalonosce:ShowerCalorimetry 2.11749 4.24475 11.2374 3.31365 2.61538 10
[art]:TriggerResults:TriggerResultInserter 2.8488e-05 3.84869e-05 7.2181e-05 3.36245e-05 1.26749e-05 10
end_path:out1:RootOutput 7.135e-06 8.8718e-06 1.9224e-05 7.849e-06 3.46226e-06 10
end_path:out1:RootOutput(write) 2.30799 2.84996 4.16436 2.67287 0.49252 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6928.4 MB
Peak resident set size usage (VmHWM): 4511.99 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root.temp.json ====
.:
total 6561624
-rw-r--r--. 1 dune004 dune 47103 Nov 26 02:48 jobscript.log
-rw-r--r--. 1 dune004 dune 177 Nov 26 02:48 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 02:48 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 02:48 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 02:48 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 02:48 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 02:48 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 02:48 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 200704 Nov 26 02:48 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 02:48 time.db
-rw-r--r--. 1 dune004 dune 6661125951 Nov 26 02:48 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258000_27_1_20251126T005321Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 57645850 Nov 26 02:47 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 02:17 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 02:14 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 02:13 detsim_single_protodunehd_hist.root 