Jobsub ID 17919.0@dunegpschedd01.fnal.gov

Jobsub ID	17919.0@dunegpschedd01.fnal.gov
Workflow ID	62
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-25 14:33:23
Site	US_Colorado
Entry	CMSHTPC_T3_US_Colorado_heposg01-colorado
Last heartbeat	2025-07-25 18:08:52
From worker node	Hostname	lnxfarm331.colorado.edu
	cpuinfo	Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-07-25 14:34:25
Input files	fardet-hd:prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20241024T092446Z_gen_000265_supernova_g4_detsim_reco.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-07-25 18:08:52
Saved logs	justin-logs:17919.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

s
Genrating OpFlash 2 with 5 hits
Genrating OpFlash 3 with 4 hits
Genrating OpFlash 4 with 4 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 5 hits
...
Generated 2258 OpFlashes
Begin processing the 44th record. run: 3809 subRun: 0 event: 13244 at 25-Jul-2025 11:59:29 MDT
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 3
Printing 2D view candidate length: 3
Printing background CaloHitList length: 681
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
Error: A CaloHitList is empty
Printing U view candidate length: 0
Printing V view candidate length: 0
Printing W view candidate length: 0
Printing 2D view candidate length: 0
Printing background CaloHitList length: 12
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
55776
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 72749!
Genrating OpFlash 0 with 3 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 4 hits
Genrating OpFlash 3 with 7 hits
Genrating OpFlash 4 with 3 hits
Genrating OpFlash 5 with 4 hits
Genrating OpFlash 6 with 4 hits
Genrating OpFlash 7 with 4 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 3 hits
...
Generated 2287 OpFlashes
Begin processing the 45th record. run: 3809 subRun: 0 event: 13245 at 25-Jul-2025 12:04:04 MDT
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
*Signal Pixel identified*
Printing U view candidate length: 9
Printing V view candidate length: 7
Printing W view candidate length: 7
Printing 2D view candidate length: 23
Printing background CaloHitList length: 672
Printing New CaloHitList Names: 
SCaloHitListU | SCaloHitListV | SCaloHitListW | SCaloHitList2D | BBCaloHitList
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
*Electron identified*
Printing U view candidate length: 9
Printing V view candidate length: 7
Printing W view candidate length: 7
Printing 2D view candidate length: 23
Printing background CaloHitList length: 2
Printing New CaloHitList Names: 
CaloHitListU | CaloHitListV | CaloHitListW | CaloHitList2D | BCaloHitList
56006
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 73065!
Genrating OpFlash 0 with 4 hits
Genrating OpFlash 1 with 3 hits
Genrating OpFlash 2 with 3 hits
Genrating OpFlash 3 with 3 hits
Genrating OpFlash 4 with 6 hits
Genrating OpFlash 5 with 3 hits
Genrating OpFlash 6 with 7 hits
Genrating OpFlash 7 with 3 hits
Genrating OpFlash 8 with 3 hits
Genrating OpFlash 9 with 4 hits
...
Generated 2350 OpFlashes
Begin processing the 46th record. run: 3809 subRun: 0 event: 13246 at 25-Jul-2025 12:08:39 MDT
R__unzip: error -3 in inflate (zlib)
R__unzip: error -3 in inflate (zlib)
25-Jul-2025 12:08:42 MDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/3f/29/prodmarley_nue_es_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20241024T092446Z_gen_000265_supernova_g4_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0264902      268.077       290.078       272.727       40.2556        46     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0253833     0.0371677     0.0875588     0.0264488     0.0151157       46     
reco:linecluster:LineCluster                           0.257001      0.284839      0.486592      0.280778      0.0330106       46     
reco:trajcluster:TrajCluster                           0.0769429     0.0932508     0.640487      0.0780165     0.0820505       46     
reco:pandora:StandardPandora                            56.5467       62.2635       79.5649       62.3173       3.57205        45     
reco:pandoraTrack:LArPandoraTrackCreation             0.000130669   0.00234206     0.0110722    0.00257021    0.00204927       45     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000142435   0.000649146   0.00660372    0.000519852   0.000976125      45     
reco:pandoracalo:Calorimetry                          8.8775e-05    0.000195828   0.00201962    0.000150379   0.000278428      45     
reco:pandorapid:Chi2ParticleID                        2.7064e-05    6.00376e-05   0.000729413   4.7347e-05    0.000101489      45     
reco:pmtracktc:PMAlgTrackMaker                        0.00143201     0.0135138     0.103099     0.00576694     0.0172424       45     
reco:pmtracktccalo:Calorimetry                        5.9801e-05    0.000180857   0.000390083   0.000189967   6.81096e-05      45     
reco:pmtracktcpid:Chi2ParticleID                      2.2748e-05    4.83934e-05   8.0179e-05    5.1342e-05    1.23919e-05      45     
reco:emtrkmichelid:EmTrackMichelId                      6.72962       7.94476       8.85069       7.95211      0.475184        45     
reco:opdec:Deconvolution                                19.541        20.3303       22.5648       20.1752       0.54225        45     
reco:ophitspe:OpHitFinderDeco                           166.566       170.583       176.765       169.844       2.53592        45     
reco:opflash:OpFlashFinder                              11.2403       12.1817       13.2077       12.2354      0.468582        45     
reco:solarflash:SolarOpFlash                            0.17615      0.185547      0.200167      0.184934     0.00595579       45     
reco:rns:RandomNumberSaver                            1.8324e-05    2.75254e-05   0.000247035   2.2374e-05    3.31422e-05      45     
[art]:TriggerResults:TriggerResultInserter             9.107e-06    1.04822e-05   3.1173e-05     9.853e-06    3.19073e-06      45     
end_path:out1:RootOutput                               2.326e-06    3.24116e-06   1.4287e-05     3.003e-06    1.69611e-06      45     
end_path:out1:RootOutput(write)                         0.10657      0.110926       0.11974      0.110298     0.00291175       45     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3255.48 MB
  Peak resident set size usage (VmHWM): 1778.15 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 25-Jul-2025 12:08:43 MDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TBasket::ReadBasketBuffers
        fNbytes = 8716810, fKeylen = 119, fObjlen = 19370592, noutot = 0, nout=0, nin=7563989, nbuf=16777215
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module StandardPandora/pandora run: 3809 subRun: 0 event: 13246
    ---- FileReadError END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 45 entries while sim::OpDetDivRecs_opdigi__detsim. has 50 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.