Jobsub ID 258416.30@dunegpschedd01.fnal.gov
| Jobsub ID | 258416.30@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:47:10 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 11:27:01 |
| From worker node | Hostname | wn117 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:49:01 |
| Input files | hd-protodune:pdhd_prod_beam__238215_115_1_20251106T050320Z_gen_g4_IonScintPDExt.root_249556_3_1_20251122T073937Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 4h (16404s) |
| CPU time | 3h (11357s = 69%) |
| Max RSS bytes | 4184285184 (3990 MiB) |
| Outputting started | 2025-11-26 11:22:27 |
| Output files | |
| Finished | 2025-11-26 11:27:01 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
r
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8981 XUs and 6759 XVs -> 84 XUVs
C:0 T:1 55513 XUs and 52173 XVs -> 1566 XUVs
C:0 T:2 34264 XUs and 29973 XVs -> 842 XUVs
C:0 T:3 13 XUs and 15 XVs -> 0 XUVs
C:0 T:4 19 XUs and 45 XVs -> 0 XUVs
C:0 T:5 56086 XUs and 46236 XVs -> 1927 XUVs
C:0 T:6 50017 XUs and 32143 XVs -> 1153 XUVs
C:0 T:7 5 XUs and 5 XVs -> 1 XUVs
5573 XUVs total
2837 collection wire objects
5573 potential space points
Neighbour search...
162967 tests to find 88118 neighbours
Iterating with no regularization...
Begin: 3.28589e+07
0 3.02113e+07
1 2.99975e+07
2 2.99832e+07
Now with regularization...
Begin: 2.78879e+07
0 2.78833e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 08:22:10 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root"
26-Nov-2025 08:22:10 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 261.226 422.813 631.648 380.819 120.595 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00162594 0.0201533 0.0597658 0.0179666 0.0159723 10
reco:opdec:Deconvolution 1.02889 1.35165 1.63208 1.35916 0.14142 10
reco:ophitspe:OpHitFinderDeco 0.19332 0.375357 0.768718 0.326244 0.163999 10
reco:opflash:OpFlashFinder 0.111114 0.160241 0.267805 0.139898 0.0448141 10
reco:opslicer:OpSlicer 0.276281 0.430355 0.519802 0.457735 0.0822844 10
reco:wclsdatahd:WireCellToolkit 79.9955 125.785 445.513 87.3796 107.114 10
reco:gaushit:GausHitFinder 1.2367 3.27071 15.5321 1.83637 4.11 10
reco:nhitsfilter:NumberOfHitsFilter 0.00159483 0.00322553 0.00973406 0.00233493 0.0023256 10
reco:reco3d:SpacePointSolver 1.26933 3.03032 4.77147 2.99283 1.14647 10
reco:hitpdune:DisambigFromSpacePoints 0.702582 1.94539 4.30943 1.70855 1.00901 10
reco:pandora:StandardPandora 29.1992 76.4878 186.12 66.1978 42.6461 10
reco:pandoraWriter:StandardPandora 0.648688 0.963432 1.173 1.01548 0.180297 10
reco:pandoraTrack:LArPandoraTrackCreation 4.23425 11.9223 23.7031 11.1872 5.36699 10
reco:pandoraShower:LArPandoraShowerCreation 4.922 10.9083 30.1027 9.20403 6.85587 10
reco:pandoracalo:Calorimetry 1.90876 5.27577 11.5683 4.27541 2.94685 10
reco:pandoracalonosce:Calorimetry 2.02469 4.87699 9.12546 3.95976 2.54172 10
reco:pandoraShowercalo:ShowerCalorimetry 3.52441 9.42375 19.6461 7.59818 5.18561 10
reco:pandoraShowercalonosce:ShowerCalorimetry 3.39442 8.57884 18.4403 7.24838 4.54797 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.111275 0.146625 0.244086 0.142701 0.0369372 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.109064 0.14248 0.202505 0.146923 0.0260953 10
reco:emtrkmichelid:EmTrackMichelId 63.9384 77.3958 95.7615 75.3102 10.8531 10
reco:crttag:SingleCRTMatchingProducer 0.0127361 0.0217891 0.0340838 0.0189443 0.00776916 10
reco:crtreco:TwoCRTMatchingProducer 0.0130006 0.0192146 0.0308705 0.0156268 0.00646533 10
reco:pandora2Track:LArPandoraTrackCreation 15.4987 26.0787 47.8719 20.1057 11.0158 10
reco:pandora2calo:ShowerCalorimetry 3.7363 8.47126 17.9667 6.44546 4.68133 10
reco:pandora2calonosce:ShowerCalorimetry 3.92768 8.41226 19.4919 5.82834 5.42218 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0828762 0.14545 0.212942 0.142246 0.0347016 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.117919 0.152977 0.212362 0.150005 0.0277804 10
reco:pandora2Shower:LArPandoraShowerCreation 6.22013 11.3485 25.8194 8.52674 6.05136 10
reco:pandora2Showercalo:ShowerCalorimetry 5.28392 10.7904 26.1984 8.05608 6.54131 10
reco:pandora2Showercalonosce:ShowerCalorimetry 4.25945 9.97653 24.5797 7.57465 6.23285 10
[art]:TriggerResults:TriggerResultInserter 4.6522e-05 0.00010567 0.000208918 7.85425e-05 6.2952e-05 10
end_path:out1:RootOutput 7.49e-06 1.52727e-05 4.4002e-05 1.18655e-05 1.03458e-05 10
end_path:out1:RootOutput(write) 3.46622 4.45008 6.16325 4.53265 0.775661 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6654.63 MB
Peak resident set size usage (VmHWM): 4129.9 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root.temp.json ====
.:
total 6492304
-rw-r--r-- 1 nobody nobody 43963 Nov 26 08:22 jobscript.log
-rw-r--r-- 1 nobody nobody 203 Nov 26 08:22 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 08:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 08:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 08:22 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 08:22 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 08:22 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 08:22 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 08:22 time.db
-rw-r--r-- 1 nobody nobody 6586571316 Nov 26 08:22 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258416_30_1_20251126T064909Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 61234600 Nov 26 08:16 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 07:14 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 07:08 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 07:07 detsim_single_protodunehd_hist.root 