justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 243045.50@dunegpschedd02.fnal.gov

Jobsub ID243045.50@dunegpschedd02.fnal.gov
Workflow ID10084
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-11-13 12:43:22
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce01
Last heartbeat2025-11-14 04:22:14
From worker nodeHostnameccwcondor0379
cpuinfoAMD EPYC 9334 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-14 04:17:30
Input filesvd-protodune:np02vd_raw_run040486_0000_df-s04-d0_dw_0_20251113T131909.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-14 04:22:14
Saved logsjustin-logs:243045.50-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

FrameFanin : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[05:21:48.990] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[05:21:48.990] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:48.990] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[05:21:48.990] I [ timer  ] Timer: Total node execution : 0 sec
wclsFrameSaver: saving empty frame to art::Event
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArMaster, STATUS_CODE_NOT_INITIALIZED
[05:21:49.637] D [  main  ] executing 1 apps, thread limit 0:
[05:21:49.637] D [  main  ] executing 1 apps, thread limit 0:
[05:21:49.637] D [  main  ] executing app: "Pgrapher"
[05:21:49.637] D [ pgraph ] <Pgrapher:> executing graph 
[05:21:49.637] D [ pgraph ] executing with 26 nodes
[05:21:49.637] D [ pgraph ] <Pgrapher:> graph execution complete 
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[05:21:49.637] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[05:21:49.637] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:49.638] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[05:21:49.638] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[05:21:49.638] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[05:21:49.638] I [ timer  ] Timer: Total node execution : 0 sec
wclsFrameSaver: saving empty frame to art::Event
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArMaster, STATUS_CODE_NOT_INITIALIZED
14-Nov-2025 05:21:50 CET  Closed output file "np02vd_raw_run040486_0000_df-s04-d0_dw_0_20251113T131909_reco_stage1_20251114T042150_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        8.14894       8.72958       12.9839       8.34556       1.14535        24     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      4.3906e-05    6.15255e-05   0.000254035   5.90145e-05   1.37053e-05      336    
produce:tpcrawdecoder:PDVDTPCReader              0.551127      0.601897       1.06369      0.578569      0.090427        336    
produce:triggerrawdecoder:PDVDTriggerReader4     0.010564      0.0116642     0.0216431     0.0112226     0.0017379       336    
produce:pdvddaphne:DAPHNEReaderPDVD             0.000679572   0.00117175     0.0020832    0.00119739    0.000143762      336    
produce:ophit:OpHitFinder                       4.3967e-05    6.80828e-05   0.000764861   6.3677e-05    3.97132e-05      336    
produce:opflash:OpFlashFinderVerticalDrift      2.5088e-05    4.12811e-05   0.000440057   3.84885e-05   2.4904e-05       336    
produce:wclsdatavd:WireCellToolkit              0.00036255    0.000521077   0.00290559    0.000506855   0.00017609       336    
produce:gaushit:GausHitFinder                   2.8483e-05    4.78856e-05   0.000580901   4.45775e-05   3.0756e-05       336    
produce:nhitsfilter:NumberOfHitsFilter           7.01e-06     1.26823e-05   0.00020437    1.14375e-05   1.12418e-05      336    
produce:reco3d:SpacePointSolver                 4.8974e-05    7.81871e-05   0.000754574   7.45635e-05   3.95759e-05      336    
produce:hitpdune:DisambigFromSpacePoints        0.000119201   0.000171645   0.00174512    0.000163268   8.89647e-05      336    
produce:pandora:StandardPandora                 0.000391244   0.00450367      1.32055     0.000568438    0.0719032       336    
produce:pandoraTrack:LArPandoraTrackCreation    7.8889e-05    0.000119288   0.00209221    0.000113147   0.000108572      336    
produce:pandoraGnocalo:GnocchiCalorimetry       7.5845e-05    0.000120062   0.00194473    0.00011186    0.000103025      336    
[art]:TriggerResults:TriggerResultInserter       8.844e-06    1.29257e-05   4.9405e-05    1.2284e-05    3.50591e-06      336    
end_path:out1:RootOutput                         3.255e-06    4.03418e-06   2.1042e-05     3.926e-06    1.02331e-06      336    
end_path:out1:RootOutput(write)                 0.000425917   0.000688085   0.00333168    0.00061869    0.000231619      336    
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3020.17 MB
  Peak resident set size usage (VmHWM): 935.215 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 14-Nov-2025 05:21:50 CET ModuleEndJob
---- EventRangeOverlap BEGIN
  Attempt to merge event ranges that both contain one or more of the same events
   Run: 40486
    SubRun: 1 Event range: [1,2)  vs.
    SubRun: 1 Event range: [1,2)
---- EventRangeOverlap END
---- EventRangeOverlap BEGIN
  Attempt to merge event ranges that both contain one or more of the same events
   Run: 40486
    SubRun: 1 Event range: [1,2)  vs.
    SubRun: 1 Event range: [1,2)
---- EventRangeOverlap END
%MSG
Art has completed and will exit with status 1.
Error in reco1
justIN time: 2026-02-04 07:42:39 UTC       justIN version: 01.06.00