Jobsub ID 252736.23@dunegpschedd02.fnal.gov
| Jobsub ID | 252736.23@dunegpschedd02.fnal.gov |
| Workflow ID | 10765 |
| Stage ID | 1 |
| User name | higuera@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-11-26 06:27:09 |
| Site | FR_CCIN2P3 |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 |
| Last heartbeat | 2025-11-26 07:51:28 |
| From worker node | Hostname | ccwcondor0366 |
| cpuinfo | AMD EPYC 9334 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 106200 (29 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-11-26 06:29:26 |
| Input files | hd-protodune:pdhd_prod_beam__244219_54_1_20251106T101719Z_gen_g4_IonScintPDExt.root_249487_29_1_20251121T235114Z_PDInt.root
|
| Jobscript | Exit code | 0 |
| Real time | 1h (4896s) |
| CPU time | 1h (4592s = 93%) |
| Max RSS bytes | 6041931776 (5762 MiB) |
| Outputting started | 2025-11-26 07:51:03 |
| Output files | |
| Finished | 2025-11-26 07:51:28 |
| List job events (HTCondor job logs unavailable) |
Jobscript log (last 10,000 characters)
wiener"
FrameSaver: q=2.52488e+06 n=996922 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12831 XUs and 9042 XVs -> 86 XUVs
C:0 T:1 71139 XUs and 55371 XVs -> 2793 XUVs
C:0 T:2 14626 XUs and 25484 XVs -> 2665 XUVs
C:0 T:3 1 XUs and 4 XVs -> 0 XUVs
C:0 T:4 16 XUs and 26 XVs -> 3 XUVs
C:0 T:5 19263 XUs and 14726 XVs -> 642 XUVs
C:0 T:6 7962 XUs and 8149 XVs -> 776 XUVs
6965 XUVs total
2910 collection wire objects
6965 potential space points
Neighbour search...
330875 tests to find 143266 neighbours
Iterating with no regularization...
Begin: 3.13487e+07
0 2.76942e+07
1 2.74536e+07
2 2.74327e+07
Now with regularization...
Begin: 2.50894e+07
0 2.5082e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
26-Nov-2025 08:50:57 CET Closed output file "pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root"
26-Nov-2025 08:50:57 CET Closed input file "temp_detsim.root"
======================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
======================================================================================================================================
Full event 82.8601 108.878 130.657 107.787 11.8844 10
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0013304 0.0076918 0.0141523 0.00886673 0.00393392 10
reco:opdec:Deconvolution 0.575099 0.76023 1.02475 0.735759 0.133964 10
reco:ophitspe:OpHitFinderDeco 0.109109 0.173884 0.236541 0.190914 0.0417842 10
reco:opflash:OpFlashFinder 0.0520333 0.0789078 0.10668 0.0860271 0.0191454 10
reco:opslicer:OpSlicer 0.157219 0.207186 0.274752 0.201686 0.0414388 10
reco:wclsdatahd:WireCellToolkit 40.4847 43.507 47.8516 42.8581 2.49531 10
reco:gaushit:GausHitFinder 0.542262 0.765891 0.995553 0.775566 0.126135 10
reco:nhitsfilter:NumberOfHitsFilter 0.000205131 0.000289447 0.000469521 0.000276384 7.32152e-05 10
reco:reco3d:SpacePointSolver 0.555897 0.964815 1.94852 0.735626 0.43196 10
reco:hitpdune:DisambigFromSpacePoints 0.436389 0.605635 0.908848 0.548195 0.149342 10
reco:pandora:StandardPandora 10.0851 19.8252 27.211 19.8132 4.17506 10
reco:pandoraWriter:StandardPandora 0.227905 0.289984 0.404255 0.273919 0.0543271 10
reco:pandoraTrack:LArPandoraTrackCreation 0.818611 1.81165 3.14855 1.91386 0.657703 10
reco:pandoraShower:LArPandoraShowerCreation 0.797386 1.64977 3.37488 1.36867 0.69471 10
reco:pandoracalo:Calorimetry 0.550854 0.856013 1.64431 0.720737 0.317487 10
reco:pandoracalonosce:Calorimetry 0.650773 0.88631 1.44692 0.82128 0.245853 10
reco:pandoraShowercalo:ShowerCalorimetry 0.987105 1.54416 3.59984 1.21 0.760386 10
reco:pandoraShowercalonosce:ShowerCalorimetry 0.883069 1.58428 3.27932 1.23553 0.734089 10
reco:pandoraGnocchiCalo:GnocchiCalorimetry 0.030344 0.0551739 0.0727502 0.055556 0.0134602 10
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0316029 0.0535361 0.0669286 0.0530317 0.0104824 10
reco:emtrkmichelid:EmTrackMichelId 15.1941 18.6554 23.6626 18.2586 2.37002 10
reco:crttag:SingleCRTMatchingProducer 0.00608824 0.00819988 0.0120083 0.0076293 0.00179767 10
reco:crtreco:TwoCRTMatchingProducer 0.00395168 0.00537152 0.0089782 0.00468413 0.00158024 10
reco:pandora2Track:LArPandoraTrackCreation 2.63307 4.43785 7.98981 4.07623 1.47848 10
reco:pandora2calo:ShowerCalorimetry 0.773016 1.40859 2.69027 1.32299 0.552506 10
reco:pandora2calonosce:ShowerCalorimetry 0.818664 1.29991 2.52571 1.10078 0.504782 10
reco:pandora2GnocchiCalo:GnocchiCalorimetry 0.0317631 0.0515224 0.0758222 0.0509154 0.0141043 10
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry 0.0302189 0.0498478 0.0730592 0.0510256 0.0137433 10
reco:pandora2Shower:LArPandoraShowerCreation 0.973601 1.7453 3.00802 1.591 0.548012 10
reco:pandora2Showercalo:ShowerCalorimetry 1.02554 1.74923 3.09577 1.55664 0.597834 10
reco:pandora2Showercalonosce:ShowerCalorimetry 0.914276 1.63885 2.93961 1.47318 0.519092 10
[art]:TriggerResults:TriggerResultInserter 2.5338e-05 4.05952e-05 8.577e-05 3.67505e-05 1.60298e-05 10
end_path:out1:RootOutput 9.434e-06 1.17436e-05 2.2133e-05 1.06255e-05 3.5737e-06 10
end_path:out1:RootOutput(write) 1.7216 2.02944 2.348 2.04529 0.198547 10
======================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 6555.36 MB
Peak resident set size usage (VmHWM): 3969.38 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
Traceback (most recent call last):
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 434, in <module>
main()
File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_10_04d00/bin/extractor_prod.py", line 379, in main
arbjson = json.load(injson)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 1 (char 454)
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root.ext.json ====
Error loading input json file.
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root.ext.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root.json ====
==== Start pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root.temp.json ====
==== End pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root.temp.json ====
.:
total 5687804
-rw-r--r-- 1 dune001 lbno 40847 Nov 26 08:51 jobscript.log
-rw-r--r-- 1 dune001 lbno 206 Nov 26 08:51 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:51 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root.json
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:51 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root.temp.json
-rw-r--r-- 1 dune001 lbno 31 Nov 26 08:51 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root.ext.json
-rw-r--r-- 1 dune001 lbno 25 Nov 26 08:50 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:50 reco_protodunehd_hist.root
-rw-r--r-- 1 dune001 lbno 3574 Nov 26 08:50 Pandora_Geometry.xml
-rw-r--r-- 1 dune001 lbno 204800 Nov 26 08:50 mem.db
-rw-r--r-- 1 dune001 lbno 36864 Nov 26 08:50 time.db
-rw-r--r-- 1 dune001 lbno 5772687121 Nov 26 08:50 pdhd_prod_beam__gen_g4_IonScintPDExt_PDInt_252736_23_1_20251126T062930Z_detsim_reco1.root
-rw-r--r-- 1 dune001 lbno 51313480 Nov 26 08:50 Pandora_Events.pndr
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:33 ProtoDUNE.log
-rw-r--r-- 1 dune001 lbno 0 Nov 26 08:31 debugprod.log
-rw-r--r-- 1 dune001 lbno 519 Nov 26 08:31 detsim_single_protodunehd_hist.root 