Jobsub ID 17962.1@dunegpschedd01.fnal.gov

Jobsub ID	17962.1@dunegpschedd01.fnal.gov
Workflow ID	62
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-25 18:09:34
Site	US_Colorado
Entry	CMSHTPC_T3_US_Colorado_heposg01-colorado
Last heartbeat	2025-07-25 21:43:05
From worker node	Hostname	lnxfarm343.colorado.edu
	cpuinfo	Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-07-25 18:11:06
Input files	fardet-hd:prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20241024T092446Z_gen_000265_supernova_g4_detsim_reco.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-07-25 21:43:05
Saved logs	justin-logs:17962.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

s
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 5 hits
...
Generated 2258 OpFlashes
Begin processing the 44th record. run: 3809 subRun: 0 event: 13244 at 25-Jul-2025 15:33:44 MDT
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 3
Printing 2D view candidate length: 3
Printing background CaloHitList length: 681
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 12
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
55776
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 72749!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 7 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2287 OpFlashes
Begin processing the 45th record. run: 3809 subRun: 0 event: 13245 at 25-Jul-2025 15:38:18 MDT
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 9
Printing V view candidate length: 7
Printing W view candidate length: 7
Printing 2D view candidate length: 23
Printing background CaloHitList length: 672
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 9
Printing V view candidate length: 7
Printing W view candidate length: 7
Printing 2D view candidate length: 23
Printing background CaloHitList length: 2
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
56006
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 73065!
Genrating OpFlash 0 with 4 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 6 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 7 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 4 hits
...
Generated 2350 OpFlashes
Begin processing the 46th record. run: 3809 subRun: 0 event: 13246 at 25-Jul-2025 15:42:51 MDT
R__unzip: error -3 in inflate (zlib)
R__unzip: error -3 in inflate (zlib)
25-Jul-2025 15:42:54 MDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/3f/29/prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20241024T092446Z_gen_000265_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0261331      264.366       277.427       270.399       39.5508        46     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0254078     0.0362779     0.0766187     0.0264586     0.0153936       46     
reco:linecluster:LineCluster                           0.256831      0.282128      0.317622      0.282317      0.0164363       46     
reco:trajcluster:TrajCluster                           0.0766543     0.0822984     0.140088      0.0778616     0.0111677       46     
reco:pandora:StandardPandora                            56.1015       59.3288       65.5471       59.0869       1.8921         45     
reco:pandoraTrack:LArPandoraTrackCreation             0.000131702   0.00198207    0.00776963     0.0023432    0.00153303       45     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000138932   0.000668312   0.00638419    0.000508038    0.0010569       45     
reco:pandoracalo:Calorimetry                          8.7098e-05     0.0001897     0.0018994    0.000153418   0.000261427      45     
reco:pandorapid:Chi2ParticleID                        2.5766e-05    0.000217877   0.00786401    4.6895e-05    0.00115275       45     
reco:pmtracktc:PMAlgTrackMaker                        0.00139074     0.0132645     0.104933     0.00547988     0.0173481       45     
reco:pmtracktccalo:Calorimetry                        6.1045e-05    0.000177496   0.000380852   0.000178941   6.85666e-05      45     
reco:pmtracktcpid:Chi2ParticleID                      2.2876e-05    4.71166e-05   7.8892e-05    4.8511e-05     1.219e-05       45     
reco:emtrkmichelid:EmTrackMichelId                      6.3463        7.72769       8.52862       7.79146      0.428195        45     
reco:opdec:Deconvolution                                19.5631       20.1602       21.4096       20.0924      0.351521        45     
reco:ophitspe:OpHitFinderDeco                           166.182       170.523       175.439       170.097       2.31908        45     
reco:opflash:OpFlashFinder                              11.0928       11.787        12.7027       11.808        0.38788        45     
reco:solarflash:SolarOpFlash                            0.16939      0.184335      0.200773      0.184628     0.00684098       45     
reco:rns:RandomNumberSaver                            2.0153e-05    2.75232e-05   0.000257594   2.2169e-05    3.47233e-05      45     
[art]:TriggerResults:TriggerResultInserter             8.842e-06    1.04976e-05   3.1896e-05     9.961e-06    3.27103e-06      45     
end_path:out1:RootOutput                               2.265e-06    3.26469e-06   1.5364e-05     2.948e-06    1.86139e-06      45     
end_path:out1:RootOutput(write)                        0.105657      0.110862       0.11738      0.110607     0.00278341       45     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3251.12 MB
  Peak resident set size usage (VmHWM): 1775.84 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 25-Jul-2025 15:42:55 MDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TBasket::ReadBasketBuffers
        fNbytes = 8716810, fKeylen = 119, fObjlen = 19370592, noutot = 0, nout=0, nin=7563989, nbuf=16777215
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module StandardPandora/pandora run: 3809 subRun: 0 event: 13246
    ---- FileReadError END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 45 entries while sim::OpDetDivRecs_opdigi__detsim. has 50 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.