Jobsub ID 11223.1@dunegpschedd02.fnal.gov

Jobsub ID	11223.1@dunegpschedd02.fnal.gov
Workflow ID	62
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-25 10:03:09
Site	US_Colorado
Entry	CMSHTPC_T3_US_Colorado_heposg01-colorado
Last heartbeat	2025-07-25 14:33:11
From worker node	Hostname	lnxfarm188.colorado.edu
	cpuinfo	Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-07-25 10:04:22
Input files	fardet-hd:prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20241024T092446Z_gen_000265_supernova_g4_detsim_reco.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-07-25 14:33:11
Saved logs	justin-logs:11223.1-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

s
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 5 hits
...
Generated 2258 OpFlashes
Begin processing the 44th record. run: 3809 subRun: 0 event: 13244 at 25-Jul-2025 08:21:51 MDT
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 3
Printing 2D view candidate length: 3
Printing background CaloHitList length: 681
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 12
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
55776
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 72749!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 7 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2287 OpFlashes
Begin processing the 45th record. run: 3809 subRun: 0 event: 13245 at 25-Jul-2025 08:27:23 MDT
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 9
Printing V view candidate length: 7
Printing W view candidate length: 7
Printing 2D view candidate length: 23
Printing background CaloHitList length: 672
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 9
Printing V view candidate length: 7
Printing W view candidate length: 7
Printing 2D view candidate length: 23
Printing background CaloHitList length: 2
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
56006
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 73065!
Genrating OpFlash 0 with 4 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 6 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 7 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 4 hits
...
Generated 2350 OpFlashes
Begin processing the 46th record. run: 3809 subRun: 0 event: 13246 at 25-Jul-2025 08:32:57 MDT
R__unzip: error -3 in inflate (zlib)
R__unzip: error -3 in inflate (zlib)
25-Jul-2025 08:32:59 MDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/3f/29/prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20241024T092446Z_gen_000265_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0270446      344.774       407.662       352.239       53.1781        46     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0253892     0.0417141     0.232435      0.0266294     0.0341089       46     
reco:linecluster:LineCluster                           0.243679      0.255267      0.281305      0.252394      0.0107163       46     
reco:trajcluster:TrajCluster                           0.0792704     0.0838814     0.104132      0.081295     0.00585229       46     
reco:pandora:StandardPandora                            97.7888       116.114       168.764       116.125       11.3937        45     
reco:pandoraTrack:LArPandoraTrackCreation             0.000169892    0.0027305     0.0120023    0.00328457    0.00233188       45     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000185728   0.00099301     0.0121566    0.000759197   0.00176661       45     
reco:pandoracalo:Calorimetry                          0.000121862   0.000292344   0.00360874    0.00021764    0.000504508      45     
reco:pandorapid:Chi2ParticleID                        4.2348e-05    9.83716e-05    0.0014705    7.1592e-05    0.00020745       45     
reco:pmtracktc:PMAlgTrackMaker                        0.00210416     0.0238529     0.189539     0.00917512     0.0322949       45     
reco:pmtracktccalo:Calorimetry                        9.3032e-05    0.000270159   0.000590419   0.000269372   0.00010853       45     
reco:pmtracktcpid:Chi2ParticleID                       3.762e-05    7.45731e-05   0.000126898   7.5738e-05    1.85538e-05      45     
reco:emtrkmichelid:EmTrackMichelId                      9.55167       11.2283       14.115        11.1631      0.783627        45     
reco:opdec:Deconvolution                                23.874        24.3335       25.3867       24.2499      0.320114        45     
reco:ophitspe:OpHitFinderDeco                           180.292       185.112       192.105       184.475       2.59932        45     
reco:opflash:OpFlashFinder                              11.8702       14.6687       17.5471       14.7186       1.90064        45     
reco:solarflash:SolarOpFlash                           0.349456      0.388805      0.423916      0.389221      0.018796        45     
reco:rns:RandomNumberSaver                            2.4021e-05    3.55171e-05   0.000281536   2.9825e-05    3.75297e-05      45     
[art]:TriggerResults:TriggerResultInserter            1.2988e-05    1.49022e-05   4.3568e-05    1.4088e-05    4.45121e-06      45     
end_path:out1:RootOutput                               2.888e-06    3.60327e-06    1.523e-05     3.245e-06    1.78295e-06      45     
end_path:out1:RootOutput(write)                        0.165312      0.179945      0.217784      0.178905     0.00987307       45     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3257.17 MB
  Peak resident set size usage (VmHWM): 1785.07 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 25-Jul-2025 08:32:59 MDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TBasket::ReadBasketBuffers
        fNbytes = 8716810, fKeylen = 119, fObjlen = 19370592, noutot = 0, nout=0, nin=7563989, nbuf=16777215
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module StandardPandora/pandora run: 3809 subRun: 0 event: 13246
    ---- FileReadError END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 45 entries while sim::OpDetDivRecs_opdigi__detsim. has 50 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.