Jobsub ID 16331.80@dunegpschedd02.fnal.gov

Jobsub ID	16331.80@dunegpschedd02.fnal.gov
Workflow ID	704
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-10 13:36:40
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2025-08-11 04:58:39
From worker node	Hostname	heplnc091.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-08-10 13:39:23
Input files	fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco.root
Jobscript	Exit code	0
	Real time	15h (54856s)
	CPU time	14h (50447s = 91%)
	Max RSS bytes	1828073472 (1743 MiB)
Outputting started	2025-08-11 04:53:40
Output files	fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco_20250810T133934Z_reco2.root
Finished	2025-08-11 04:58:39
Saved logs	justin-logs:16331.80-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

6 with 4 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 6 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2240 OpFlashes
Begin processing the 49th record. run: 249 subRun: 0 event: 185099 at 11-Aug-2025 05:20:03 BST
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 0
Printing V view candidate length: 3
Printing W view candidate length: 0
Printing 2D view candidate length: 3
Printing background CaloHitList length: 650
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 45
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  11-Aug-2025 05:25:37 BST run: 249 subRun: 0 event: 185099
Comparing two wires in the same plane: return failure
%MSG
54861
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 70953!
Genrating OpFlash 0 with 5 hits
Genrating OpFlash 1 with 4 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 6 hits
Genrating OpFlash 4 with 6 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 8 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2174 OpFlashes
Begin processing the 50th record. run: 249 subRun: 0 event: 185100 at 11-Aug-2025 05:37:53 BST
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 6
Printing V view candidate length: 7
Printing W view candidate length: 3
Printing 2D view candidate length: 16
Printing background CaloHitList length: 670
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Photon identified*
*Photon identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Photon identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 6
Printing V view candidate length: 5
Printing W view candidate length: 5
Printing 2D view candidate length: 16
Printing background CaloHitList length: 3
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  11-Aug-2025 05:42:45 BST run: 249 subRun: 0 event: 185100
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
54306
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 70384!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 5 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 5 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 5 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2127 OpFlashes
11-Aug-2025 05:53:27 BST  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco_20250810T133934Z_reco2.root"
11-Aug-2025 05:53:27 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/14/c6/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              923.538       1061.43       1123.71       1068.04       39.201         50     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0967309     0.127921      0.239368      0.109343      0.0422495       50     
reco:linecluster:LineCluster                            1.0295        1.08604       1.88231       1.06557      0.116684        50     
reco:trajcluster:TrajCluster                           0.295722      0.309842      0.353227      0.306618      0.0102292       50     
reco:pandora:StandardPandora                            227.241       313.937       332.349       318.417       18.462         50     
reco:pandoraTrack:LArPandoraTrackCreation             0.000448714   0.00131941     0.0174731    0.000753525   0.00281042       50     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000519088   0.00135778     0.0135321    0.00084888    0.00225361       50     
reco:pandoracalo:Calorimetry                          0.000343807   0.000592135   0.00332114    0.000555612   0.000397353      50     
reco:pandorapid:Chi2ParticleID                        0.000118308   0.000224498   0.00183854    0.00019027    0.000233408      50     
reco:pmtracktc:PMAlgTrackMaker                        0.00498834     0.018333      0.0408718     0.0175483    0.00603566       50     
reco:pmtracktccalo:Calorimetry                        0.000241883   0.000826887   0.00700123    0.000547546   0.000931786      50     
reco:pmtracktcpid:Chi2ParticleID                      9.2855e-05    0.000262868   0.00204236    0.000229415   0.000261673      50     
reco:emtrkmichelid:EmTrackMichelId                      21.1225       29.6454       34.7991       29.7566       2.61476        50     
reco:opdec:Deconvolution                                66.9724       80.3879       88.273        80.4979       3.09449        50     
reco:ophitspe:OpHitFinderDeco                           512.594       603.042       650.453       608.226       23.416         50     
reco:opflash:OpFlashFinder                              27.563        30.4549       32.6653       30.5867       1.10632        50     
reco:solarflash:SolarOpFlash                            1.06588       1.7858        2.16691       1.85324      0.266533        50     
reco:rns:RandomNumberSaver                            9.3711e-05    0.000154598   0.000688776   0.000142188    7.964e-05       50     
[art]:TriggerResults:TriggerResultInserter             3.875e-05    6.74522e-05   0.000118291   6.8461e-05    1.4417e-05       50     
end_path:out1:RootOutput                              1.2673e-05    2.01373e-05   6.1324e-05    1.87565e-05   9.13584e-06      50     
end_path:out1:RootOutput(write)                        0.430494      0.629085      0.726378      0.634167      0.0483824       50     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3306.06 MB
  Peak resident set size usage (VmHWM): 1828.07 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco_20250810T133934Z_reco2.root
Extracted metadata
checking file====== fardet-hd:prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco_20250810T133934Z_reco2.root
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco_20250810T133934Z_reco2.root.ext.json
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco_20250810T133934Z_reco2.root.json
prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250802T082432Z_gen_003702_supernova_g4_detsim_reco_20250810T133934Z_reco2.root.temp.json
reco2_hist.root