Jobsub ID 258580.8@dunegpschedd01.fnal.gov
| Jobsub ID | 258580.8@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 10:29:25 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2025-11-26 17:31:41 |
| From worker node | Hostname | wn114 |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 10:31:27 |
| Input files | hd-protodune:pdhd_prod_beam__238227_112_1_20251106T052217Z_gen_g4_IonScintPDExt.root_255549_104_1_20251122T235610Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 6h (24912s) |
| CPU time | 3h (13927s = 55%) |
| Max RSS bytes | 4176650240 (3983 MiB) |
| Outputting started | 2025-11-26 17:26:41 |
| Output files | |
| Finished | 2025-11-26 17:31:41 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
inding XUV coincidences...
C:0 T:0 6926 XUs and 10176 XVs -> 41 XUVs
C:0 T:1 45844 XUs and 62862 XVs -> 1303 XUVs
C:0 T:2 19833 XUs and 70425 XVs -> 538 XUVs
C:0 T:3 10 XUs and 17 XVs -> 0 XUVs
C:0 T:4 11 XUs and 7 XVs -> 0 XUVs
C:0 T:5 9148 XUs and 12605 XVs -> 342 XUVs
C:0 T:6 12937 XUs and 19082 XVs -> 332 XUVs
2556 XUVs total
1719 collection wire objects
2556 potential space points
Neighbour search...
37698 tests to find 17634 neighbours
Iterating with no regularization...
Begin: 4.02358e+07
0 3.89611e+07
1 3.89023e+07
2 3.88978e+07
Now with regularization...
Begin: 3.7412e+07
0 3.74105e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 14:26:07 -03 Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root"
26-Nov-2025 14:26:07 -03 Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 326.805 538.666 966.418 473.153 190.266 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00669141 0.0249639 0.0829494 0.0203549 0.0199824 10
reco:opdec:Deconvolution 1.30484 2.014 3.36448 1.88365 0.587802 10
reco:ophitspe:OpHitFinderDeco 0.416602 1.04006 1.80392 0.977198 0.413754 10
reco:opflash:OpFlashFinder 0.0986817 0.170866 0.260414 0.158052 0.043514 10
reco:opslicer:OpSlicer 0.284576 0.57953 1.53512 0.518829 0.33819 10
reco:wclsdatahd:WireCellToolkit 98.3249 162.577 215.798 163.125 30.2322 10
reco:gaushit:GausHitFinder 1.08064 2.56322 10.7516 1.63938 2.76736 10
reco:nhitsfilter:NumberOfHitsFilter 0.00197891 0.00524214 0.0138838 0.00450887 0.00362716 10
reco:reco3d:SpacePointSolver 0.920569 4.33071 17.0772 2.8856 4.50152 10
reco:hitpdune:DisambigFromSpacePoints 0.548088 1.48366 2.76365 1.4021 0.652764 10
reco:pandora:StandardPandora 24.1989 57.9713 150.232 48.7063 36.2524 10
reco:pandoraWriter:StandardPandora 0.560033 1.0621 1.89163 0.97627 0.417061 10
reco:pandoraTrack:LArPandoraTrackCreation 2.78649 14.9771 61.8633 11.861 16.1462 10
reco:pandoraShower:LArPandoraShowerCreation 1.89288 15.2839 49.4527 11.2083 13.0143 10
reco:pandoracalo:Calorimetry 1.23816 4.59913 16.074 3.38638 3.93824 10
reco:pandoracalonosce:Calorimetry 1.23126 4.70953 16.8779 3.65185 4.19169 10
reco:pandoraShowercalo:ShowerCalorimetry 1.55537 10.5198 34.4691 8.5221 8.72599 10
reco:pandoraShowercalonosce:ShowerCalorimetry 1.4034 10.2896 22.2685 7.72633 7.09789 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0748548 0.196439 0.702215 0.143694 0.171673 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0734205 0.171417 0.418218 0.143042 0.0891321 10
reco:emtrkmichelid:EmTrackMichelId 85.5168 136.433 189.213 129.553 31.12 10
reco:crttag:SingleCRTMatchingProducer 0.0176995 0.0437173 0.120045 0.0260956 0.0328277 10
reco:crtreco:TwoCRTMatchingProducer 0.0163101 0.0616638 0.18813 0.0333834 0.0546142 10
reco:pandora2Track:LArPandoraTrackCreation 12.4211 38.958 78.6057 35.2338 21.904 10
reco:pandora2calo:ShowerCalorimetry 1.69767 9.42213 23.2612 7.28756 6.7158 10
reco:pandora2calonosce:ShowerCalorimetry 1.8651 10.3588 29.2733 7.22147 7.9747 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0555558 0.179422 0.542624 0.153277 0.130014 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0570287 0.200683 0.785584 0.148042 0.19983 10
reco:pandora2Shower:LArPandoraShowerCreation 4.56237 16.158 50.2983 12.7905 12.7614 10
reco:pandora2Showercalo:ShowerCalorimetry 2.31763 12.0847 34.9158 7.60535 9.63112 10
reco:pandora2Showercalonosce:ShowerCalorimetry 3.98377 14.1953 57.3423 8.46807 15.1918 10
[art]:TriggerResults:TriggerResultInserter 4.9182e-05 8.66071e-05 0.00026406 6.5747e-05 6.1219e-05 10
end_path:out1:RootOutput 1.023e-05 1.87597e-05 6.3182e-05 1.34855e-05 1.50399e-05 10
end_path:out1:RootOutput(write) 3.88156 5.45445 12.9138 4.54096 2.54008 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6345.61 MB
Peak resident set size usage (VmHWM): 4061.64 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root.temp.json ====
.:
total 6244756
-rw-r--r-- 1 nobody nobody 41918 Nov 26 14:26 jobscript.log
-rw-r--r-- 1 nobody nobody 206 Nov 26 14:26 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root.json
-rw-r--r-- 1 nobody nobody 0 Nov 26 14:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 nobody nobody 31 Nov 26 14:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 nobody nobody 25 Nov 26 14:26 all-input-dids.txt
-rw-r--r-- 1 nobody nobody 519 Nov 26 14:26 reco_protodunehd_hist.root
-rw-r--r-- 1 nobody nobody 3574 Nov 26 14:26 Pandora_Geometry.xml
-rw-r--r-- 1 nobody nobody 204800 Nov 26 14:26 mem.db
-rw-r--r-- 1 nobody nobody 36864 Nov 26 14:26 time.db
-rw-r--r-- 1 nobody nobody 6339168575 Nov 26 14:26 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258580_8_1_20251126T103137Z_detsim_reco1.root
-rw-r--r-- 1 nobody nobody 55148300 Nov 26 14:22 Pandora_Events.pndr
-rw-r--r-- 1 nobody nobody 0 Nov 26 13:01 ProtoDUNE.log
-rw-r--r-- 1 nobody nobody 0 Nov 26 12:49 debugprod.log
-rw-r--r-- 1 nobody nobody 519 Nov 26 12:49 detsim_single_protodunehd_hist.root 