Jobsub ID 22672.93@dunegpschedd01.fnal.gov

Jobsub ID	22672.93@dunegpschedd01.fnal.gov
Workflow ID	704
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-09 23:58:40
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce02
Last heartbeat	2025-08-10 06:47:20
From worker node	Hostname	dune001-7612097.0-host-172-16-114-102.nubes.stfc.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-10 01:30:12
Input files	fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (18818s)
	CPU time	4h (17071s = 90%)
	Max RSS bytes	1674661888 (1597 MiB)
Outputting started	2025-08-10 06:43:51
Output files	fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco_20250810T013029Z_reco2.root
Finished	2025-08-10 06:47:20
Saved logs	justin-logs:22672.93-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ting OpFlash 0 with 4 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 6 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2159 OpFlashes
Begin processing the 49th record. run: 249 subRun: 0 event: 127199 at 10-Aug-2025 06:31:21 UTC
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 3
Printing V view candidate length: 0
Printing W view candidate length: 2
Printing 2D view candidate length: 5
Printing background CaloHitList length: 712
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 23
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
55211
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71302!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 5 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 7 hits
...
Generated 2215 OpFlashes
Begin processing the 50th record. run: 249 subRun: 0 event: 127200 at 10-Aug-2025 06:37:23 UTC
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 702
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Photon identified*
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 5
Printing 2D view candidate length: 5
Printing background CaloHitList length: 30
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
55325
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 71081!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 3 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 4 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2148 OpFlashes
10-Aug-2025 06:43:45 UTC  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco_20250810T013029Z_reco2.root"
10-Aug-2025 06:43:45 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/a8/b3/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              314.044       359.23        410.384       359.79        15.3342        50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.123896      0.155476      0.281283      0.128605      0.0519303       50     
reco:linecluster:LineCluster                            1.13365       1.21409       1.3523        1.20242      0.0503259       50     
reco:trajcluster:TrajCluster                           0.377055       0.39722      0.536909      0.389086      0.0250676       50     
reco:pandora:StandardPandora                            76.4996       87.7125       130.814       87.1238       7.05847        50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000200783   0.000395258   0.00650849    0.000272065   0.000874038      50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000197474   0.000358136   0.00407346    0.000284751   0.000531828      50     
reco:pandoracalo:Calorimetry                          0.000182133   0.000274609   0.00187885    0.000242035   0.000232992      50     
reco:pandorapid:Chi2ParticleID                        3.9842e-05    7.9562e-05    0.000984381   6.0476e-05    0.000129921      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00301845    0.00504782     0.0184132    0.00467037    0.00223948       50     
reco:pmtracktccalo:Calorimetry                        8.0441e-05    0.000317256   0.00314447    0.000281521   0.000420369      50     
reco:pmtracktcpid:Chi2ParticleID                      3.3181e-05    9.29185e-05   0.00098616    8.14965e-05   0.000130459      50     
reco:emtrkmichelid:EmTrackMichelId                      7.23151       8.46996       10.4059       8.43098      0.633117        50     
reco:opdec:Deconvolution                                19.3358       22.7777       28.7214       22.6392       1.08756        50     
reco:ophitspe:OpHitFinderDeco                           183.162       226.452       256.227       228.073       11.6623        50     
reco:opflash:OpFlashFinder                              9.9078        11.517        13.2819       11.6401      0.708898        50     
reco:solarflash:SolarOpFlash                           0.218195      0.348002      0.409225      0.353575      0.0316305       50     
reco:rns:RandomNumberSaver                            5.2661e-05    8.44807e-05   0.000571981   7.31415e-05   7.03891e-05      50     
[art]:TriggerResults:TriggerResultInserter            1.9379e-05    2.76046e-05   0.000142123   2.44755e-05   1.74067e-05      50     
end_path:out1:RootOutput                                9.9e-06      1.266e-05    3.8271e-05    1.1685e-05    5.29012e-06      50     
end_path:out1:RootOutput(write)                        0.128102      0.177996      0.271136      0.176574      0.0167049       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3217.91 MB
  Peak resident set size usage (VmHWM): 1674.66 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco_20250810T013029Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco_20250810T013029Z_reco2.root
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco_20250810T013029Z_reco2.root.ext.json
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco_20250810T013029Z_reco2.root.json
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T075343Z_gen_002544_supernova_g4_detsim_reco_20250810T013029Z_reco2.root.temp.json
reco2_hist.root