justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 249368.2@dunegpschedd01.fnal.gov

Jobsub ID249368.2@dunegpschedd01.fnal.gov
Workflow ID10084
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-13 23:49:17
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce01
Last heartbeat2025-11-14 06:15:07
From worker nodeHostnameccwcondor0362
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-14 06:10:25
Input filesvd-protodune:np02vd_raw_run040486_0000_df-s04-d0_dw_0_20251113T131909.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-14 06:15:07
Saved logsjustin-logs:249368.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

FrameFanin : 0 sec
[07:14:42.032] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.032] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.032] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[07:14:42.033] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[07:14:42.033] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.033] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[07:14:42.033] I [ timer  ] Timer: Total node execution : 0 sec
wclsFrameSaver: saving empty frame to art::Event
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArMaster, STATUS_CODE_NOT_INITIALIZED
[07:14:42.692] D [  main  ] executing 1 apps, thread limit 0:
[07:14:42.692] D [  main  ] executing 1 apps, thread limit 0:
[07:14:42.692] D [  main  ] executing app: "Pgrapher"
[07:14:42.692] D [ pgraph ] <Pgrapher:> executing graph 
[07:14:42.692] D [ pgraph ] executing with 26 nodes
[07:14:42.692] D [ pgraph ] <Pgrapher:> graph execution complete 
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[07:14:42.692] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.693] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.693] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[07:14:42.693] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[07:14:42.693] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[07:14:42.693] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[07:14:42.693] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[07:14:42.693] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.693] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[07:14:42.693] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[07:14:42.693] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[07:14:42.693] I [ timer  ] Timer: Total node execution : 0 sec
wclsFrameSaver: saving empty frame to art::Event
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArMaster, STATUS_CODE_NOT_INITIALIZED
14-Nov-2025 07:14:43 CET  Closed output file "np02vd_raw_run040486_0000_df-s04-d0_dw_0_20251113T131909_reco_stage1_20251114T061443_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        8.31129       8.57332       10.0873       8.44992      0.436775        24     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      4.3215e-05    6.40506e-05   0.000273783   6.10425e-05   1.60408e-05      336    
produce:tpcrawdecoder:PDVDTPCReader               0.56182      0.590521      0.808691      0.585026      0.0300749       336    
produce:triggerrawdecoder:PDVDTriggerReader4     0.0108274     0.0115082     0.016741      0.0113675    0.000700324      336    
produce:pdvddaphne:DAPHNEReaderPDVD             0.000764233   0.00126038    0.00234686    0.00125127    0.00012544       336    
produce:ophit:OpHitFinder                       4.8363e-05    7.38895e-05   0.000701179   6.6801e-05    3.89908e-05      336    
produce:opflash:OpFlashFinderVerticalDrift      2.7381e-05    4.59406e-05   0.000408036   4.1578e-05    2.48924e-05      336    
produce:wclsdatavd:WireCellToolkit              0.000350289   0.000499104   0.000985618   0.000488207   6.29894e-05      336    
produce:gaushit:GausHitFinder                   3.1017e-05    4.7695e-05    0.000525744   4.4362e-05    2.72387e-05      336    
produce:nhitsfilter:NumberOfHitsFilter           8.173e-06    1.27459e-05   0.000174303   1.1633e-05    9.50362e-06      336    
produce:reco3d:SpacePointSolver                 5.5834e-05    8.40089e-05    0.0006624    7.86535e-05   3.57643e-05      336    
produce:hitpdune:DisambigFromSpacePoints        0.00013277    0.000189263   0.00158562    0.000180197   8.04738e-05      336    
produce:pandora:StandardPandora                 0.000421747   0.00481827      1.41357     0.000590962    0.0769684       336    
produce:pandoraTrack:LArPandoraTrackCreation    8.1554e-05    0.000127402    0.0024205    0.000116496   0.000126611      336    
produce:pandoraGnocalo:GnocchiCalorimetry       8.3075e-05    0.000122439   0.00189544    0.000113021   9.96712e-05      336    
[art]:TriggerResults:TriggerResultInserter       8.883e-06    1.37338e-05   0.000187833   1.24885e-05   1.01894e-05      336    
end_path:out1:RootOutput                         2.714e-06    3.83572e-06   2.5058e-05    3.4705e-06    2.3744e-06       336    
end_path:out1:RootOutput(write)                 0.000444782   0.000717871   0.00252397    0.000638864   0.000233011      336    
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3002.58 MB
  Peak resident set size usage (VmHWM): 927.638 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 14-Nov-2025 07:14:43 CET ModuleEndJob
---- EventRangeOverlap BEGIN
  Attempt to merge event ranges that both contain one or more of the same events
   Run: 40486
    SubRun: 1 Event range: [1,2)  vs.
    SubRun: 1 Event range: [1,2)
---- EventRangeOverlap END
---- EventRangeOverlap BEGIN
  Attempt to merge event ranges that both contain one or more of the same events
   Run: 40486
    SubRun: 1 Event range: [1,2)  vs.
    SubRun: 1 Event range: [1,2)
---- EventRangeOverlap END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2026-02-04 09:01:35 UTC       justIN version: 01.06.00