Jobsub ID 22831.94@dunegpschedd01.fnal.gov

Jobsub ID	22831.94@dunegpschedd01.fnal.gov
Workflow ID	728
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-10 14:24:25
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-08-10 18:16:44
From worker node	Hostname	dunegli-6561904-0-fnpc19112.fnal.gov
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-10 16:06:27
Input files	fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco.root
Jobscript	Exit code	0
	Real time	2h (7775s)
	CPU time	2h (7699s = 99%)
	Max RSS bytes	1668386816 (1591 MiB)
Outputting started	2025-08-10 18:16:03
Output files	fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root
Finished	2025-08-10 18:16:44
Saved logs	justin-logs:22831.94-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 11 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 6 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 4 hits
...
Generated 1168 OpFlashes
Begin processing the 49th record. run: 251 subRun: 0 event: 494199 at 10-Aug-2025 18:11:37 UTC
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 2
Printing 2D view candidate length: 2
Printing background CaloHitList length: 524
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
*Electron identified*
*Electron identified*
Printing U view candidate length: 2
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 2
Printing background CaloHitList length: 13
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
34496
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 42596!
Genrating OpFlash 0 with 4 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 10 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 5 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 6 hits
...
Generated 1249 OpFlashes
Begin processing the 50th record. run: 251 subRun: 0 event: 494200 at 10-Aug-2025 18:13:49 UTC
*Signal Pixel identified*
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 1
Printing 2D view candidate length: 1
Printing background CaloHitList length: 509
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 21
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
33541
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 40789!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 6 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 8 hits
Genrating OpFlash 7 with 7 hits
Genrating OpFlash 8 with 4 hits
Genrating OpFlash 9 with 5 hits
...
Generated 1165 OpFlashes
10-Aug-2025 18:16:00 UTC  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root"
10-Aug-2025 18:16:00 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/0b/a1/prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              127.072       153.685       229.537       149.276       22.9238        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00154489    0.00669581     0.063988     0.00242347     0.0104458       50     
reco:linecluster:LineCluster                           0.0136688     0.0416011     0.127549      0.0268825     0.0348593       50     
reco:trajcluster:TrajCluster                           0.0027882    0.00500308     0.0192048    0.00424964     0.0029154       50     
reco:pandora:StandardPandora                            51.1918       69.1227       103.321       68.9915       12.8713        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000112251   0.000235758   0.00250929    0.000190696   0.000326857      50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000149211   0.000212476    0.0014953    0.000180097   0.000184894      50     
reco:pandoracalo:Calorimetry                          9.9161e-05    0.000142016   0.000736797   0.000126621   8.69193e-05      50     
reco:pandorapid:Chi2ParticleID                        2.7481e-05    4.13666e-05   0.000385443   3.22605e-05   4.94692e-05      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00141301    0.00388044    0.00796747     0.003669     0.00153839       50     
reco:pmtracktccalo:Calorimetry                        6.1211e-05    0.000176585   0.00144954    0.000152121   0.000195421      50     
reco:pmtracktcpid:Chi2ParticleID                      2.2841e-05    5.62301e-05   0.000405564   4.88805e-05   5.25276e-05      50     
reco:emtrkmichelid:EmTrackMichelId                      2.91246       3.82666       5.74779       3.63659      0.685759        50     
reco:opdec:Deconvolution                                8.92137       9.9371        12.4663       9.61924      0.946697        50     
reco:ophitspe:OpHitFinderDeco                           57.5257       66.4833       107.133       61.1975       13.0104        50     
reco:opflash:OpFlashFinder                              3.3509        4.03311       4.88025       3.99017       0.36834        50     
reco:solarflash:SolarOpFlash                           0.118764      0.142263      0.214143      0.132848      0.0244013       50     
reco:rns:RandomNumberSaver                             3.596e-05    5.05656e-05   0.000330503   4.43205e-05   4.03832e-05      50     
[art]:TriggerResults:TriggerResultInserter             1.298e-05    1.74716e-05    5.037e-05     1.607e-05    5.64235e-06      50     
end_path:out1:RootOutput                                 5e-06      8.05104e-06    2.568e-05     7.59e-06     2.94447e-06      50     
end_path:out1:RootOutput(write)                        0.0709076     0.0809486     0.105508      0.0784304    0.00867045       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3140.3 MB
  Peak resident set size usage (VmHWM): 1668.39 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root.ext.json
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root.json
prodbackground_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250802T200601Z_gen_009884_supernova_g4_halfActiveVol_detsim_reco_20250810T160631Z_reco2.root.temp.json
reco2_hist.root