Jobsub ID 252371.57@dunegpschedd02.fnal.gov
| Jobsub ID | 252371.57@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 01:05:55 |
| Site | UK_Durham |
| Entry | DUNE_UK_SGridDurham_ce3 |
| Last heartbeat | 2025-11-26 06:40:19 |
| From worker node | Hostname | n161.dur.scotgrid.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 01:08:49 |
| Input files | hd-protodune:pdhd_prod_beam__238271_152_1_20251106T060437Z_gen_g4_IonScintPDExt.root_255757_108_1_20251123T161247Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 5h (19856s) |
| CPU time | 1h (5372s = 27%) |
| Max RSS bytes | 6879604736 (6560 MiB) |
| Outputting started | 2025-11-26 06:39:47 |
| Output files | |
| Finished | 2025-11-26 06:40:19 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10581 XUs and 8072 XVs -> 102 XUVs
C:0 T:1 62527 XUs and 45850 XVs -> 1562 XUVs
C:0 T:2 4114 XUs and 6015 XVs -> 120 XUVs
C:0 T:3 6 XUs and 6 XVs -> 0 XUVs
C:0 T:4 2 XUs and 7 XVs -> 0 XUVs
C:0 T:5 16349 XUs and 18386 XVs -> 914 XUVs
C:0 T:6 10902 XUs and 12647 XVs -> 461 XUVs
C:0 T:7 7 XUs and 7 XVs -> 0 XUVs
3159 XUVs total
2049 collection wire objects
3159 potential space points
Neighbour search...
56807 tests to find 24826 neighbours
Iterating with no regularization...
Begin: 2.13343e+07
0 1.967e+07
1 1.9571e+07
2 1.95645e+07
Now with regularization...
Begin: 1.83795e+07
0 1.83779e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 06:39:42 GMT Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root"
26-Nov-2025 06:39:42 GMT Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 99.5319 175.379 320.475 130.998 83.968 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00137035 0.00670183 0.00942738 0.00764921 0.00260542 10
reco:opdec:Deconvolution 0.700776 1.07842 1.52068 1.07845 0.27303 10
reco:ophitspe:OpHitFinderDeco 0.168117 0.215161 0.26959 0.206618 0.039123 10
reco:opflash:OpFlashFinder 0.0474334 0.104946 0.192133 0.101992 0.0417136 10
reco:opslicer:OpSlicer 0.157855 0.268157 0.429819 0.244636 0.0820048 10
reco:wclsdatahd:WireCellToolkit 47.9104 57.2767 74.3051 52.6266 9.58948 10
reco:gaushit:GausHitFinder 0.618266 0.938581 1.60779 0.82525 0.319071 10
reco:nhitsfilter:NumberOfHitsFilter 0.000333215 0.000525556 0.000913905 0.000472027 0.000176489 10
reco:reco3d:SpacePointSolver 0.43298 1.23001 2.84827 0.951125 0.752133 10
reco:hitpdune:DisambigFromSpacePoints 0.42759 1.05068 2.22601 0.841892 0.624833 10
reco:pandora:StandardPandora 13.3005 36.4561 107.024 22.3216 30.6448 10
reco:pandoraWriter:StandardPandora 0.205337 0.324966 0.546916 0.288316 0.112316 10
reco:pandoraTrack:LArPandoraTrackCreation 1.53748 4.20235 10.285 2.51248 3.1913 10
reco:pandoraShower:LArPandoraShowerCreation 0.934683 4.33633 14.1902 2.52638 4.09966 10
reco:pandoracalo:Calorimetry 0.698039 1.84545 4.57132 1.10947 1.35017 10
reco:pandoracalonosce:Calorimetry 0.708393 1.82839 4.5977 1.09324 1.36598 10
reco:pandoraShowercalo:ShowerCalorimetry 0.87167 3.79576 12.7148 2.41525 3.58274 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.833106 3.63119 11.9246 2.29387 3.37351 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0378887 0.0614087 0.118443 0.0550671 0.0244365 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0387602 0.0614958 0.116908 0.0545367 0.0232682 10
reco:emtrkmichelid:EmTrackMichelId 17.2496 24.8237 38.9296 21.5608 7.37341 10
reco:crttag:SingleCRTMatchingProducer 0.00698384 0.011963 0.0262551 0.00937359 0.00574781 10
reco:crtreco:TwoCRTMatchingProducer 0.00462474 0.00937152 0.0254945 0.00637616 0.0064307 10
reco:pandora2Track:LArPandoraTrackCreation 3.10676 10.5034 32.3695 6.38808 9.33387 10
reco:pandora2calo:ShowerCalorimetry 0.723152 3.14099 10.4874 2.0153 2.97127 10
reco:pandora2calonosce:ShowerCalorimetry 0.700093 3.05406 9.94755 2.03408 2.82451 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0366789 0.0602477 0.116477 0.0566218 0.0237318 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0371514 0.0603484 0.112631 0.0577605 0.0222044 10
reco:pandora2Shower:LArPandoraShowerCreation 1.20121 4.39397 13.5007 2.84945 3.92246 10
reco:pandora2Showercalo:ShowerCalorimetry 1.14754 4.12673 12.6881 2.84523 3.65227 10
reco:pandora2Showercalonosce:ShowerCalorimetry 1.11951 3.963 12.1212 2.7492 3.50748 10
[art]:TriggerResults:TriggerResultInserter 2.8692e-05 4.20251e-05 0.000117692 3.17625e-05 2.57615e-05 10
end_path:out1:RootOutput 3.352e-06 7.9896e-06 2.8807e-05 5.8075e-06 7.01481e-06 10
end_path:out1:RootOutput(write) 1.64761 2.36603 3.97074 2.10286 0.753235 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6727.92 MB
Peak resident set size usage (VmHWM): 4206.38 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root.temp.json ====
.:
total 6191316
-rw-r--r--. 1 dune004 dune 40922 Nov 26 06:39 jobscript.log
-rw-r--r--. 1 dune004 dune 206 Nov 26 06:39 justin-processed-pfns.txt
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:39 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root.json
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:39 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root.temp.json
-rw-r--r--. 1 dune004 dune 31 Nov 26 06:39 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root.ext.json
-rw-r--r--. 1 dune004 dune 25 Nov 26 06:39 all-input-dids.txt
-rw-r--r--. 1 dune004 dune 519 Nov 26 06:39 reco_protodunehd_hist.root
-rw-r--r--. 1 dune004 dune 3574 Nov 26 06:39 Pandora_Geometry.xml
-rw-r--r--. 1 dune004 dune 204800 Nov 26 06:39 mem.db
-rw-r--r--. 1 dune004 dune 36864 Nov 26 06:39 time.db
-rw-r--r--. 1 dune004 dune 6282971706 Nov 26 06:39 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252371_57_1_20251126T010854Z_detsim_reco1.root
-rw-r--r--. 1 dune004 dune 56607340 Nov 26 06:39 Pandora_Events.pndr
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:13 ProtoDUNE.log
-rw-r--r--. 1 dune004 dune 0 Nov 26 06:08 debugprod.log
-rw-r--r--. 1 dune004 dune 519 Nov 26 06:08 detsim_single_protodunehd_hist.root 