Jobsub ID 252489.153@dunegpschedd02.fnal.gov
| Jobsub ID | 252489.153@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 02:27:59 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 05:27:56 |
| From worker node | Hostname | n179.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 02:30:48 |
| Input files | hd-protodune:pdhd_prod_beam__243778_6_1_20251105T233524Z_gen_g4_IonScintPDExt.root_249562_20_1_20251122T081607Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (10606s) |
| CPU time | 1h (5299s = 49%) |
| Max RSS bytes | 6062665728 (5781 MiB) |
| Outputting started | 2025-11-26 05:27:35 |
| Output files | |
| Finished | 2025-11-26 05:27:56 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
saving 87581 traces tagged "wiener"
FrameSaver: q=2.72088e+06 n=1052855 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9171 XUs and 11558 XVs -> 169 XUVs
C:0 T:1 91204 XUs and 111072 XVs -> 4732 XUVs
C:0 T:2 7796 XUs and 5315 XVs -> 81 XUVs
C:0 T:5 49644 XUs and 53152 XVs -> 1573 XUVs
C:0 T:6 15279 XUs and 22034 XVs -> 400 XUVs
6955 XUVs total
2583 collection wire objects
6955 potential space points
Neighbour search...
467431 tests to find 146776 neighbours
Iterating with no regularization...
Begin: 3.90994e+07
0 3.27342e+07
1 3.20373e+07
2 3.19903e+07
3 3.19836e+07
Now with regularization...
Begin: 3.0841e+07
0 3.08366e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 05:27:30 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root"
26-Nov-2025 05:27:30 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 138.276 181.328 265.386 175.291 35.9894 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00105454 0.0106132 0.0364669 0.00943574 0.00929555 10
reco:opdec:Deconvolution 0.956141 1.15265 1.57192 1.14134 0.182241 10
reco:ophitspe:OpHitFinderDeco 0.229063 0.242056 0.257542 0.241922 0.00943883 10
reco:opflash:OpFlashFinder 0.0695673 0.113513 0.183832 0.104808 0.0344736 10
reco:opslicer:OpSlicer 0.208474 0.289583 0.420028 0.28615 0.0580744 10
reco:wclsdatahd:WireCellToolkit 62.0208 67.6934 70.4721 68.188 2.52713 10
reco:gaushit:GausHitFinder 0.83999 1.01799 1.31228 0.985216 0.142985 10
reco:nhitsfilter:NumberOfHitsFilter 0.00038533 0.000481598 0.000772864 0.000441677 0.00011631 10
reco:reco3d:SpacePointSolver 0.686207 1.58966 3.15283 1.31173 0.661887 10
reco:hitpdune:DisambigFromSpacePoints 0.631671 0.990499 1.51997 0.963404 0.285517 10
reco:pandora:StandardPandora 16.3844 36.0004 91.873 29.1748 20.3874 10
reco:pandoraWriter:StandardPandora 0.277127 0.329673 0.424714 0.325499 0.0503248 10
reco:pandoraTrack:LArPandoraTrackCreation 2.40779 4.24519 6.16728 3.88928 1.25707 10
reco:pandoraShower:LArPandoraShowerCreation 1.70904 3.8111 6.86337 3.26274 1.65993 10
reco:pandoracalo:Calorimetry 0.916016 1.67014 2.62628 1.48481 0.525704 10
reco:pandoracalonosce:Calorimetry 1.08071 1.6818 2.60576 1.39826 0.586337 10
reco:pandoraShowercalo:ShowerCalorimetry 1.35168 3.13552 6.07661 2.71691 1.44479 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.28301 2.83383 4.96332 2.49495 1.26171 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0495512 0.0670762 0.0932632 0.0653234 0.013253 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0511941 0.0671105 0.0926389 0.0648399 0.0129603 10
reco:emtrkmichelid:EmTrackMichelId 20.2077 25.0835 31.82 24.6677 4.03813 10
reco:crttag:SingleCRTMatchingProducer 0.00830246 0.0129272 0.0198454 0.012583 0.00319116 10
reco:crtreco:TwoCRTMatchingProducer 0.0054522 0.0115573 0.0177435 0.0110606 0.00361078 10
reco:pandora2Track:LArPandoraTrackCreation 6.25861 10.0947 14.9145 9.22173 3.29584 10
reco:pandora2calo:ShowerCalorimetry 1.25116 2.65475 4.9373 2.26568 1.24634 10
reco:pandora2calonosce:ShowerCalorimetry 1.19231 2.60173 5.39591 2.01787 1.34794 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0502566 0.0660449 0.0882554 0.0656222 0.0121554 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.048641 0.0655563 0.0869537 0.0647497 0.0126217 10
reco:pandora2Shower:LArPandoraShowerCreation 2.20704 3.9888 6.50867 3.63313 1.44153 10
reco:pandora2Showercalo:ShowerCalorimetry 1.82418 3.59833 6.56911 3.01404 1.57792 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.98858 3.50203 6.73696 2.8164 1.59994 10
[art]:TriggerResults:TriggerResultInserter 2.5895e-05 3.44948e-05 7.7034e-05 2.962e-05 1.44474e-05 10
end_path:out1:RootOutput 3.755e-06 6.6276e-06 1.9516e-05 5.735e-06 4.377e-06 10
end_path:out1:RootOutput(write) 2.11469 2.56656 3.40891 2.51088 0.413696 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6785.63 MB
Peak resident set size usage (VmHWM): 4269.11 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root.temp.json ====
.:
total 6147056
-rw-r--r--. 1 dune004 dune 46174 Nov 26 05:27 jobscript.log
-rw-r--r--. 1 dune004 dune 202 Nov 26 05:27 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:27 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:27 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 05:27 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 05:27 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 05:27 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 05:27 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 200704 Nov 26 05:27 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 05:27 time.db
-rw-r--r--. 1 dune004 dune 6240396544 Nov 26 05:27 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252489_153_1_20251126T023051Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 53854260 Nov 26 05:26 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:58 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:55 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 04:55 detsim_single_protodunehd_hist.root 