Jobsub ID 252408.172@dunegpschedd02.fnal.gov
| Jobsub ID | 252408.172@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 01:29:57 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 03:46:11 |
| From worker node | Hostname | n167.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 01:39:19 |
| Input files | hd-protodune:pdhd_prod_beam__238316_123_1_20251106T070515Z_gen_g4_IonScintPDExt.root_249479_21_1_20251121T231452Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 2h (7585s) |
| CPU time | 1h (5020s = 66%) |
| Max RSS bytes | 7040389120 (6714 MiB) |
| Outputting started | 2025-11-26 03:45:45 |
| Output files | |
| Finished | 2025-11-26 03:46:11 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
9e+06 n=1227751 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14333 XUs and 10941 XVs -> 93 XUVs
C:0 T:1 72337 XUs and 67257 XVs -> 1201 XUVs
C:0 T:2 10544 XUs and 19507 XVs -> 1092 XUVs
C:0 T:3 0 XUs and 2 XVs -> 0 XUVs
C:0 T:4 13 XUs and 7 XVs -> 0 XUVs
C:0 T:5 48989 XUs and 23204 XVs -> 901 XUVs
C:0 T:6 20079 XUs and 19290 XVs -> 441 XUVs
3728 XUVs total
2175 collection wire objects
3728 potential space points
Neighbour search...
53312 tests to find 31170 neighbours
Iterating with no regularization...
Begin: 5.94563e+07
0 5.79515e+07
1 5.78875e+07
2 5.78814e+07
Now with regularization...
Begin: 5.53961e+07
0 5.53937e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 03:45:40 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root"
26-Nov-2025 03:45:40 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 113.701 172.695 226.31 178.094 35.5847 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00104418 0.0123616 0.0641073 0.00671872 0.0175474 10
reco:opdec:Deconvolution 0.976796 1.05227 1.22377 1.03063 0.0722882 10
reco:ophitspe:OpHitFinderDeco 0.22545 0.237676 0.285465 0.232785 0.0168025 10
reco:opflash:OpFlashFinder 0.0724224 0.093846 0.116068 0.093835 0.0129585 10
reco:opslicer:OpSlicer 0.20665 0.250278 0.307531 0.251813 0.03115 10
reco:wclsdatahd:WireCellToolkit 59.6426 62.4583 65.4051 62.6571 1.99136 10
reco:gaushit:GausHitFinder 0.657705 0.995538 1.37446 0.990358 0.186144 10
reco:nhitsfilter:NumberOfHitsFilter 0.000348973 0.000528421 0.000675323 0.000539244 8.83469e-05 10
reco:reco3d:SpacePointSolver 0.470097 1.59128 3.00922 1.36925 0.951964 10
reco:hitpdune:DisambigFromSpacePoints 0.365165 0.94314 1.68189 0.977215 0.38998 10
reco:pandora:StandardPandora 11.9753 32.8201 65.7851 30.0855 15.8551 10
reco:pandoraWriter:StandardPandora 0.213953 0.323861 0.405616 0.340481 0.0542092 10
reco:pandoraTrack:LArPandoraTrackCreation 2.02332 4.13212 6.94062 3.85507 1.49661 10
reco:pandoraShower:LArPandoraShowerCreation 1.15705 4.07461 6.74238 4.16265 1.73458 10
reco:pandoracalo:Calorimetry 0.86781 1.66068 2.97307 1.523 0.576411 10
reco:pandoracalonosce:Calorimetry 0.921737 1.65157 3.01581 1.48708 0.581312 10
reco:pandoraShowercalo:ShowerCalorimetry 1.09293 3.17703 5.12641 3.1683 1.24088 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.976655 3.03573 5.04236 3.02988 1.22512 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0383166 0.0645092 0.0790307 0.0658685 0.0111246 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.038871 0.0639134 0.0793163 0.0650047 0.0109776 10
reco:emtrkmichelid:EmTrackMichelId 16.8417 25.4184 31.0644 25.9965 4.05081 10
reco:crttag:SingleCRTMatchingProducer 0.00614326 0.0117643 0.013889 0.012346 0.00218325 10
reco:crtreco:TwoCRTMatchingProducer 0.00444917 0.0102988 0.0132933 0.0102394 0.00248937 10
reco:pandora2Track:LArPandoraTrackCreation 3.50593 9.87771 14.9081 10.5864 3.54205 10
reco:pandora2calo:ShowerCalorimetry 0.902827 2.63675 4.38043 2.62637 1.05246 10
reco:pandora2calonosce:ShowerCalorimetry 0.93135 2.61378 4.56081 2.58351 1.07262 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0373191 0.0607014 0.074938 0.0617652 0.0103562 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0392753 0.0605568 0.0757439 0.0605281 0.00961927 10
reco:pandora2Shower:LArPandoraShowerCreation 1.46066 3.83691 5.77446 4.04208 1.42101 10
reco:pandora2Showercalo:ShowerCalorimetry 1.36907 3.58254 6.07387 3.58378 1.39487 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.33427 3.42934 5.60943 3.48336 1.27394 10
[art]:TriggerResults:TriggerResultInserter 2.8278e-05 3.86187e-05 9.7668e-05 3.1532e-05 1.99577e-05 10
end_path:out1:RootOutput 5.545e-06 8.3542e-06 2.4225e-05 6.655e-06 5.31181e-06 10
end_path:out1:RootOutput(write) 1.7398 2.37588 2.7308 2.41559 0.294242 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6673.47 MB
Peak resident set size usage (VmHWM): 4156.23 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root.temp.json ====
.:
total 5894328
-rw-r--r--. 1 dune004 dune 42476 Nov 26 03:45 jobscript.log
-rw-r--r--. 1 dune004 dune 204 Nov 26 03:45 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:45 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:45 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 03:45 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 03:45 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 03:45 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 03:45 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 03:45 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 03:45 time.db
-rw-r--r--. 1 dune004 dune 5982837341 Nov 26 03:45 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252408_172_1_20251126T013924Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 52619620 Nov 26 03:44 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:17 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 03:14 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 03:14 detsim_single_protodunehd_hist.root 