justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235526.7@dunegpschedd02.fnal.gov

Jobsub ID235526.7@dunegpschedd02.fnal.gov
Workflow ID9489
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors2
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-30 19:48:49
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce3
Last heartbeat2025-10-30 21:55:25
From worker nodeHostnamen112.dur.scotgrid.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors2
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-30 19:49:29
Input filesvd-protodune:np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422.hdf5
JobscriptExit code0
Real time2h (7244s)
CPU time50m (3053s = 42%)
Max RSS bytes2024153088 (1930 MiB)
Outputting started2025-10-30 21:50:15
Output filesvd-protodune-det-reco:np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422_reco_stage1_20251030T214954_keepup.root
Finished2025-10-30 21:55:25
Saved logsjustin-logs:235526.7-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

raph 
[21:49:52.503] D [ pgraph ] executing with 26 nodes
[21:49:52.503] D [ pgraph ] <Pgrapher:> graph execution complete 
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[21:49:52.503] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::Aux::Resampler : 0 sec
[21:49:52.503] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 0 sec
[21:49:52.503] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[21:49:52.503] I [ timer  ] Timer: Total node execution : 0 sec
wclsFrameSaver: saving empty frame to art::Event
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_17_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArMaster, STATUS_CODE_NOT_INITIALIZED
30-Oct-2025 21:49:54 GMT  Closed output file "np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422_reco_stage1_20251030T214954_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        6.86212       114.196       250.166       153.839       76.8929        59     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      5.9179e-05    0.000109464   0.00032873    0.000103222   4.95222e-05      59     
produce:tpcrawdecoder:PDVDTPCReader               6.22702       59.8369       165.157       73.0177       40.3191        59     
produce:triggerrawdecoder:PDVDTriggerReader4     0.507382      0.613027      0.981695      0.568041      0.108115        59     
produce:pdvddaphne:DAPHNEReaderPDVD             0.000501808   0.000586887   0.00110467    0.000559729   0.000102434      59     
produce:ophit:OpHitFinder                       6.9703e-05    0.000148343   0.000662028   0.000133932   7.51976e-05      59     
produce:opflash:OpFlashFinderVerticalDrift      5.9961e-05    9.76418e-05   0.000374622   8.1985e-05    5.63448e-05      59     
produce:wclsdatavd:WireCellToolkit              0.000563297     38.8248       69.2636       54.0942       27.316         59     
produce:gaushit:GausHitFinder                   5.7919e-05     0.515967       1.15346      0.631747       0.39938        59     
produce:nhitsfilter:NumberOfHitsFilter          2.0345e-05    0.000343414    0.0112893    0.000167151    0.0014406       59     
produce:reco3d:SpacePointSolver                 9.7515e-05      3.75755       12.9548       3.85978       3.16849        59     
produce:hitpdune:DisambigFromSpacePoints        0.000188161    0.0366325     0.113143      0.036273      0.0309031       59     
produce:pandora:StandardPandora                 0.000750053     8.20416       25.5578       6.98584       7.53107        59     
produce:pandoraTrack:LArPandoraTrackCreation    0.000143515     0.24945       1.4096       0.174058      0.298503        59     
produce:pandoraGnocalo:GnocchiCalorimetry       0.000150858    0.0150125     0.0991816     0.0146861     0.015792        59     
[art]:TriggerResults:TriggerResultInserter      1.7384e-05    4.11563e-05    8.232e-05    3.8815e-05    1.75659e-05      59     
end_path:out1:RootOutput                         3.454e-06    7.98236e-06   2.4743e-05     7.739e-06    3.82849e-06      59     
end_path:out1:RootOutput(write)                  0.0128334      2.11827       5.30339       2.71286       1.55978        59     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4068.08 MB
  Peak resident set size usage (VmHWM): 2024.15 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422_reco_stage1_20251030T214954_keepup.root
\tHists: np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422_reco_stage1_20251030T214956_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422_reco_stage1_20251030T214954_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422_reco_stage1_20251030T214954_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1761860997.0,
    "core.end_time": 1761860997.0,
    "core.events": [
      354,
      355,
      356,
      357,
      358,
      359,
      360,
      361,
      362,
      363,
      364,
      365,
      366,
      367,
      368,
      369,
      370,
      371,
      372,
      373,
      374,
      375,
      376,
      377,
      378,
      379,
      380,
      381,
      382,
      383,
      384,
      385,
      386,
      387,
      388,
      389,
      390,
      391,
      392,
      393,
      394,
      395,
      396,
      397,
      398,
      399,
      400,
      401,
      402,
      403,
      404,
      405,
      406,
      407,
      408,
      409,
      410,
      411,
      412
    ],
    "core.event_count": 59,
    "core.first_event_number": 354,
    "core.last_event_number": 412,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40330
    ],
    "core.runs_subruns": [
      4033000001
    ],
    "dune.daq_test": true,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422_reco_stage1_20251030T214956_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_11_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1761860997.0,
    "core.end_time": 1761860997.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      40330
    ],
    "core.runs_subruns": [
      4033000001
    ],
    "dune.daq_test": true,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run040330_0006_df-s04-d0_dw_0_20251030T175422.hdf5"
    }
  ]
}
justIN time: 2025-11-04 01:53:54 UTC       justIN version: 01.05.01