Jobsub ID 258588.0@dunegpschedd01.fnal.gov
| Jobsub ID | 258588.0@dunegpschedd01.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 10:33:24 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce04 |
| Last heartbeat | 2025-11-26 12:15:15 |
| From worker node | Hostname | ccwcondor0381 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 10:34:25 |
| Input files | hd-protodune:pdhd_prod_beam__243918_49_1_20251106T043834Z_gen_g4_IonScintPDExt.root_255820_90_1_20251123T205832Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (6023s) |
| CPU time | 1h (5763s = 95%) |
| Max RSS bytes | 6897213440 (6577 MiB) |
| Outputting started | 2025-11-26 12:14:50 |
| Output files | |
| Finished | 2025-11-26 12:15:15 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
clsFrameSaver: saving 79239 traces tagged "wiener"
FrameSaver: q=2.64726e+06 n=982799 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12636 XUs and 9400 XVs -> 218 XUVs
C:0 T:1 131096 XUs and 98983 XVs -> 3520 XUVs
C:0 T:2 14249 XUs and 10210 XVs -> 405 XUVs
C:0 T:4 11 XUs and 61 XVs -> 2 XUVs
C:0 T:5 18982 XUs and 15216 XVs -> 1729 XUVs
C:0 T:6 6770 XUs and 3058 XVs -> 105 XUVs
5979 XUVs total
2498 collection wire objects
5979 potential space points
Neighbour search...
434243 tests to find 209044 neighbours
Iterating with no regularization...
Begin: 5.62552e+07
0 5.19478e+07
1 5.17403e+07
2 5.17215e+07
Now with regularization...
Begin: 4.98457e+07
0 4.98333e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 13:14:43 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root"
26-Nov-2025 13:14:43 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 110.246 160.359 253.634 152.322 42.0126 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00135314 0.010403 0.0204939 0.010683 0.00554416 10
reco:opdec:Deconvolution 0.792331 0.949437 1.08518 0.964453 0.0938296 10
reco:ophitspe:OpHitFinderDeco 0.189728 0.209863 0.244145 0.209237 0.0139481 10
reco:opflash:OpFlashFinder 0.0696369 0.10143 0.133185 0.101684 0.0239413 10
reco:opslicer:OpSlicer 0.225318 0.282155 0.318749 0.283301 0.0273813 10
reco:wclsdatahd:WireCellToolkit 43.0112 48.8241 50.9317 50.1764 2.74189 10
reco:gaushit:GausHitFinder 0.784298 0.979713 1.23499 0.979182 0.156107 10
reco:nhitsfilter:NumberOfHitsFilter 0.000255688 0.000495564 0.000773634 0.000515227 0.000190383 10
reco:reco3d:SpacePointSolver 0.574583 1.97844 5.39351 1.50129 1.3835 10
reco:hitpdune:DisambigFromSpacePoints 0.358085 0.859056 1.68185 0.754687 0.393396 10
reco:pandora:StandardPandora 12.1977 31.5856 78.024 26.553 17.2934 10
reco:pandoraWriter:StandardPandora 0.287944 0.385809 0.518806 0.364155 0.0777978 10
reco:pandoraTrack:LArPandoraTrackCreation 1.9151 4.52736 9.26897 3.71072 2.2419 10
reco:pandoraShower:LArPandoraShowerCreation 1.21326 4.33659 8.18483 3.98797 2.15501 10
reco:pandoracalo:Calorimetry 0.877963 2.04215 3.80984 1.93389 0.895074 10
reco:pandoracalonosce:Calorimetry 0.854904 2.01815 3.66673 1.91396 0.913887 10
reco:pandoraShowercalo:ShowerCalorimetry 1.0108 4.21045 8.3901 3.76393 2.26675 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.717104 3.71293 6.89482 3.13378 1.95148 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.0436611 0.0746319 0.0961364 0.0774473 0.018206 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0411719 0.070779 0.0954785 0.0735103 0.0175033 10
reco:emtrkmichelid:EmTrackMichelId 17.369 23.2362 29.6564 23.3545 4.03804 10
reco:crttag:SingleCRTMatchingProducer 0.00528103 0.0112287 0.0236869 0.0106566 0.00488423 10
reco:crtreco:TwoCRTMatchingProducer 0.00414979 0.00894937 0.01554 0.00935268 0.00365651 10
reco:pandora2Track:LArPandoraTrackCreation 2.30064 9.61205 15.5771 10.1534 3.79091 10
reco:pandora2calo:ShowerCalorimetry 0.605267 3.06443 4.8465 3.41022 1.37595 10
reco:pandora2calonosce:ShowerCalorimetry 0.567456 3.10015 5.63367 2.99212 1.54312 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0412997 0.0785031 0.10386 0.0809447 0.0201914 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0399083 0.0722752 0.09876 0.0736386 0.0180242 10
reco:pandora2Shower:LArPandoraShowerCreation 1.01649 3.62715 5.65537 3.60411 1.49041 10
reco:pandora2Showercalo:ShowerCalorimetry 0.886502 3.73753 6.87036 3.01852 1.9869 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.960841 3.86229 8.04413 2.92569 2.27781 10
[art]:TriggerResults:TriggerResultInserter 4.3085e-05 6.70929e-05 0.000165862 5.28655e-05 3.52803e-05 10
end_path:out1:RootOutput 9.474e-06 1.52502e-05 5.26e-05 1.0661e-05 1.25467e-05 10
end_path:out1:RootOutput(write) 2.01349 2.54358 3.24793 2.50575 0.399446 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6633.93 MB
Peak resident set size usage (VmHWM): 4130.3 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root.temp.json ====
.:
total 6079480
-rw-r--r-- 1 dune001 lbno 50877 Nov 26 13:14 jobscript.log
-rw-r--r-- 1 dune001 lbno 205 Nov 26 13:14 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 13:14 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 13:14 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 13:14 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 13:14 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 13:14 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 13:14 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 13:14 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 13:14 time.db
-rw-r--r-- 1 dune001 lbno 6170634756 Nov 26 13:14 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_258588_0_1_20251126T103432Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 54429670 Nov 26 13:14 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 12:50 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 12:46 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 12:46 detsim_single_protodunehd_hist.root 