Jobsub ID 258263.68@dunegpschedd01.fnal.gov
| Jobsub ID | 258263.68@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 03:59:03 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 05:46:10 |
| From worker node | Hostname | n187.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 04:00:24 |
| Input files | hd-protodune:pdhd_prod_beam__244153_84_1_20251106T090644Z_gen_g4_IonScintPDExt.root_255780_20_1_20251123T175103Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6321s) |
| CPU time | 1h (5220s = 82%) |
| Max RSS bytes | 7136518144 (6805 MiB) |
| Outputting started | 2025-11-26 05:45:47 |
| Output files | |
| Finished | 2025-11-26 05:46:10 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
38633 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11414 XUs and 7350 XVs -> 71 XUVs
C:0 T:1 62344 XUs and 40924 XVs -> 1485 XUVs
C:0 T:2 18246 XUs and 24165 XVs -> 1191 XUVs
C:0 T:3 12 XUs and 15 XVs -> 0 XUVs
C:0 T:5 15555 XUs and 33223 XVs -> 796 XUVs
C:0 T:6 52997 XUs and 63030 XVs -> 5699 XUVs
C:0 T:7 100 XUs and 166 XVs -> 67 XUVs
9309 XUVs total
2767 collection wire objects
9309 potential space points
Neighbour search...
747415 tests to find 414626 neighbours
Iterating with no regularization...
Begin: 5.35751e+07
0 4.80665e+07
1 4.76734e+07
2 4.76104e+07
3 4.75899e+07
Now with regularization...
Begin: 4.5125e+07
0 4.51087e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 05:45:43 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root"
26-Nov-2025 05:45:43 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 119.052 178.371 244.399 177.039 40.2317 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00116203 0.0115152 0.0351462 0.00839095 0.00915385 10
reco:opdec:Deconvolution 1.01663 1.20558 1.6358 1.17662 0.189471 10
reco:ophitspe:OpHitFinderDeco 0.234013 0.251136 0.272517 0.25007 0.0128416 10
reco:opflash:OpFlashFinder 0.0807726 0.128245 0.216582 0.11859 0.0389853 10
reco:opslicer:OpSlicer 0.215288 0.299794 0.461104 0.282259 0.0715243 10
reco:wclsdatahd:WireCellToolkit 59.2526 63.284 69.7981 63.065 3.33169 10
reco:gaushit:GausHitFinder 0.694282 1.04153 1.38254 1.06098 0.197031 10
reco:nhitsfilter:NumberOfHitsFilter 0.000357083 0.000513935 0.000812596 0.000497283 0.000121975 10
reco:reco3d:SpacePointSolver 0.440246 1.26531 2.76786 1.24316 0.609213 10
reco:hitpdune:DisambigFromSpacePoints 0.366755 0.947266 1.35236 0.912966 0.330175 10
reco:pandora:StandardPandora 15.3366 30.6366 61.6439 25.6994 13.5289 10
reco:pandoraWriter:StandardPandora 0.228504 0.335102 0.412119 0.330022 0.0561733 10
reco:pandoraTrack:LArPandoraTrackCreation 2.06247 4.80407 8.17703 4.69691 1.85165 10
reco:pandoraShower:LArPandoraShowerCreation 1.21332 4.54166 7.5617 4.73243 1.98717 10
reco:pandoracalo:Calorimetry 0.781285 1.9024 3.30543 1.77928 0.739382 10
reco:pandoracalonosce:Calorimetry 0.813893 1.9208 3.5819 1.70537 0.781093 10
reco:pandoraShowercalo:ShowerCalorimetry 1.07087 3.66201 6.77539 3.62916 1.75388 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.05002 3.52984 6.23653 3.58646 1.6476 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0425006 0.0687083 0.0895369 0.067431 0.0146401 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0422851 0.0678597 0.0890952 0.0671581 0.0139821 10
reco:emtrkmichelid:EmTrackMichelId 17.1308 25.1296 31.5223 24.5669 4.40464 10
reco:crttag:SingleCRTMatchingProducer 0.00795491 0.01292 0.0184979 0.0120168 0.00303682 10
reco:crtreco:TwoCRTMatchingProducer 0.0063319 0.0113658 0.0180247 0.0108132 0.00333124 10
reco:pandora2Track:LArPandoraTrackCreation 3.98958 11.4491 19.6063 10.867 5.11559 10
reco:pandora2calo:ShowerCalorimetry 0.883884 3.12754 5.40068 3.10525 1.4893 10
reco:pandora2calonosce:ShowerCalorimetry 0.862129 3.02637 5.32968 3.06073 1.42254 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.040749 0.0661054 0.0841323 0.0666838 0.0135956 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0418157 0.0658323 0.0867817 0.0647634 0.013473 10
reco:pandora2Shower:LArPandoraShowerCreation 1.4402 4.67358 9.33412 4.03204 2.32929 10
reco:pandora2Showercalo:ShowerCalorimetry 1.33456 4.25085 8.18572 3.85437 2.11581 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.27452 3.92743 7.47454 3.62062 1.91896 10
[art]:TriggerResults:TriggerResultInserter 2.3359e-05 3.49976e-05 8.859e-05 2.94895e-05 1.80858e-05 10
end_path:out1:RootOutput 5.726e-06 8.7025e-06 2.5092e-05 7.2135e-06 5.50273e-06 10
end_path:out1:RootOutput(write) 1.86377 2.58528 3.36454 2.50487 0.427605 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6593.94 MB
Peak resident set size usage (VmHWM): 4159.82 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root.temp.json ====
.:
total 6113008
-rw-r--r--. 1 dune004 dune 42642 Nov 26 05:45 jobscript.log
-rw-r--r--. 1 dune004 dune 175 Nov 26 05:45 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:45 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:45 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 05:45 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 05:45 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 05:45 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 05:45 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 05:45 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 05:45 time.db
-rw-r--r--. 1 dune004 dune 6205448601 Nov 26 05:45 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258263_68_1_20251126T040028Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 53939510 Nov 26 05:43 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:17 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 05:14 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 05:14 detsim_single_protodunehd_hist.root 