Jobsub ID 252393.0@dunegpschedd02.fnal.gov
| Jobsub ID | 252393.0@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 01:21:56 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 05:18:41 |
| From worker node | Hostname | n250.dur.scotgrid.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 01:27:00 |
| Input files | hd-protodune:pdhd_prod_beam__238225_78_1_20251106T051548Z_gen_g4_IonScintPDExt.root_255828_98_1_20251123T220535Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 3h (13866s) |
| CPU time | 1h (4165s = 30%) |
| Max RSS bytes | 5727670272 (5462 MiB) |
| Outputting started | 2025-11-26 05:18:08 |
| Output files | |
| Finished | 2025-11-26 05:18:41 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12674 XUs and 12403 XVs -> 119 XUVs
C:0 T:1 64068 XUs and 55336 XVs -> 1874 XUVs
C:0 T:2 38238 XUs and 72610 XVs -> 2118 XUVs
C:0 T:4 11 XUs and 27 XVs -> 0 XUVs
C:0 T:5 18802 XUs and 20726 XVs -> 593 XUVs
C:0 T:6 40386 XUs and 29831 XVs -> 993 XUVs
5697 XUVs total
2792 collection wire objects
5697 potential space points
Neighbour search...
193703 tests to find 97064 neighbours
Iterating with no regularization...
Begin: 4.94076e+07
0 4.48295e+07
1 4.45477e+07
2 4.45222e+07
Now with regularization...
Begin: 4.16155e+07
0 4.16048e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 05:18:01 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root"
26-Nov-2025 05:18:01 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 86.1621 111.636 203.18 104.671 32.4411 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00209085 0.00487863 0.0081353 0.00514879 0.00184746 10
reco:opdec:Deconvolution 0.562063 0.682428 0.838177 0.686308 0.0809553 10
reco:ophitspe:OpHitFinderDeco 0.125721 0.137238 0.183646 0.131925 0.0158627 10
reco:opflash:OpFlashFinder 0.054054 0.0721208 0.0906437 0.069875 0.0103401 10
reco:opslicer:OpSlicer 0.114547 0.175983 0.233142 0.177702 0.033514 10
reco:wclsdatahd:WireCellToolkit 41.9477 43.2816 48.5091 42.9336 1.80125 10
reco:gaushit:GausHitFinder 0.419561 0.573806 0.798491 0.564403 0.108389 10
reco:nhitsfilter:NumberOfHitsFilter 0.000262235 0.000544192 0.000874942 0.00054046 0.000143488 10
reco:reco3d:SpacePointSolver 0.357968 0.825885 2.33452 0.717408 0.532478 10
reco:hitpdune:DisambigFromSpacePoints 0.254219 0.465852 0.898365 0.402636 0.20003 10
reco:pandora:StandardPandora 7.97469 16.5984 37.5417 16.174 7.63617 10
reco:pandoraWriter:StandardPandora 0.195779 0.253427 0.388634 0.243872 0.0565249 10
reco:pandoraTrack:LArPandoraTrackCreation 0.580207 1.84848 6.38538 1.30531 1.58222 10
reco:pandoraShower:LArPandoraShowerCreation 0.479001 1.65317 6.60831 1.13092 1.69591 10
reco:pandoracalo:Calorimetry 0.345699 0.850432 2.36734 0.690527 0.546377 10
reco:pandoracalonosce:Calorimetry 0.327693 0.839039 2.32462 0.659776 0.53829 10
reco:pandoraShowercalo:ShowerCalorimetry 0.504182 1.35932 4.15047 1.08385 1.00325 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.495519 1.33698 4.63825 1.01252 1.14626 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0291289 0.0399761 0.0534023 0.0388291 0.00745622 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0296923 0.0395778 0.049652 0.0395532 0.00643294 10
reco:emtrkmichelid:EmTrackMichelId 19.7052 26.0156 33.219 25.6688 4.37831 10
reco:crttag:SingleCRTMatchingProducer 0.00357333 0.0068674 0.013882 0.00612439 0.0030918 10
reco:crtreco:TwoCRTMatchingProducer 0.00261473 0.00543404 0.0144974 0.00419773 0.00340633 10
reco:pandora2Track:LArPandoraTrackCreation 1.62165 4.19342 15.3953 3.2305 3.84406 10
reco:pandora2calo:ShowerCalorimetry 0.452759 1.33122 5.1097 0.963444 1.29669 10
reco:pandora2calonosce:ShowerCalorimetry 0.451952 1.31015 5.10622 0.910201 1.30083 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0298398 0.0401123 0.0517839 0.0411091 0.00713036 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0283213 0.0387943 0.0478767 0.0400929 0.0060979 10
reco:pandora2Shower:LArPandoraShowerCreation 0.633149 1.9178 8.25527 1.30914 2.14543 10
reco:pandora2Showercalo:ShowerCalorimetry 0.701886 1.95372 7.74964 1.31705 1.9791 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.661318 1.86218 7.20087 1.28545 1.8267 10
[art]:TriggerResults:TriggerResultInserter 3.5516e-05 6.05441e-05 0.000210618 4.19235e-05 5.04513e-05 10
end_path:out1:RootOutput 5.531e-06 1.53601e-05 5.4153e-05 1.1762e-05 1.31251e-05 10
end_path:out1:RootOutput(write) 1.41175 1.77462 2.33311 1.71344 0.266536 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5874.42 MB
Peak resident set size usage (VmHWM): 3682.28 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root.temp.json ====
.:
total 5174696
-rw-r--r--. 1 dune004 dune 40461 Nov 26 05:18 jobscript.log
-rw-r--r--. 1 dune004 dune 204 Nov 26 05:18 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:18 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:18 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 05:18 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 05:18 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 05:18 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 05:18 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 05:18 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 05:18 time.db
-rw-r--r--. 1 dune004 dune 5251645572 Nov 26 05:18 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252393_0_1_20251126T012708Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 46914030 Nov 26 05:17 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:00 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 04:57 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 04:57 detsim_single_protodunehd_hist.root 