justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 205954.0@dunegpschedd02.fnal.gov

Jobsub ID205954.0@dunegpschedd02.fnal.gov
Workflow ID8011
Stage ID1
User namelwhite86@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-09-24 16:19:14
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_condce_mcore
Last heartbeat2025-09-24 17:12:19
From worker nodeHostnameuct2-c497.mwt2.org
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-24 16:20:07
Input filesfardet-hd:nue_dune10kt_1x2x6_1407_360_20230826T075316Z_gen_g4_detsim_hitreco__20240221T060025Z_reco2.root
JobscriptExit code0
Real time51m (3113s)
CPU time50m (3027s = 97%)
Max RSS bytes1711013888 (1631 MiB)
Outputting started2025-09-24 17:12:01
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/08011/1/001/trackShowerCountingValidation_nue_dune10kt_1x2x6_1407_360_20230826T075316Z_gen_g4_detsim_hitreco__20240221T060025Z_reco2.root
Finished2025-09-24 17:12:19
Saved logsjustin-logs:205954.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

235
Failure in algorithm Alg0100, LArCNNTrackShowerCountingValidation, STATUS_CODE_FAILURE
Begin processing the 53rd record. run: 1407 subRun: 1 event: 36053 at 24-Sep-2025 11:57:09 CDT
Begin processing the 54th record. run: 1407 subRun: 1 event: 36054 at 24-Sep-2025 11:57:19 CDT
Begin processing the 55th record. run: 1407 subRun: 1 event: 36055 at 24-Sep-2025 11:57:24 CDT
Begin processing the 56th record. run: 1407 subRun: 1 event: 36056 at 24-Sep-2025 11:57:30 CDT
Begin processing the 57th record. run: 1407 subRun: 1 event: 36057 at 24-Sep-2025 11:57:34 CDT
Begin processing the 58th record. run: 1407 subRun: 1 event: 36058 at 24-Sep-2025 11:57:41 CDT
Begin processing the 59th record. run: 1407 subRun: 1 event: 36059 at 24-Sep-2025 11:58:04 CDT
Begin processing the 60th record. run: 1407 subRun: 1 event: 36060 at 24-Sep-2025 11:58:09 CDT
Begin processing the 61st record. run: 1407 subRun: 1 event: 36061 at 24-Sep-2025 11:58:15 CDT
Begin processing the 62nd record. run: 1407 subRun: 1 event: 36062 at 24-Sep-2025 11:58:52 CDT
Begin processing the 63rd record. run: 1407 subRun: 1 event: 36063 at 24-Sep-2025 11:58:57 CDT
Begin processing the 64th record. run: 1407 subRun: 1 event: 36064 at 24-Sep-2025 11:59:08 CDT
Begin processing the 65th record. run: 1407 subRun: 1 event: 36065 at 24-Sep-2025 11:59:36 CDT
Begin processing the 66th record. run: 1407 subRun: 1 event: 36066 at 24-Sep-2025 11:59:41 CDT
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraEventClassification/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraEventClassification/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 67th record. run: 1407 subRun: 1 event: 36067 at 24-Sep-2025 11:59:41 CDT
Begin processing the 68th record. run: 1407 subRun: 1 event: 36068 at 24-Sep-2025 11:59:47 CDT
Begin processing the 69th record. run: 1407 subRun: 1 event: 36069 at 24-Sep-2025 11:59:53 CDT
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0100, LArCNNTrackShowerCountingValidation, STATUS_CODE_FAILURE
Begin processing the 70th record. run: 1407 subRun: 1 event: 36070 at 24-Sep-2025 11:59:57 CDT
Begin processing the 71st record. run: 1407 subRun: 1 event: 36071 at 24-Sep-2025 12:00:02 CDT
Begin processing the 72nd record. run: 1407 subRun: 1 event: 36072 at 24-Sep-2025 12:00:08 CDT
Begin processing the 73rd record. run: 1407 subRun: 1 event: 36073 at 24-Sep-2025 12:00:50 CDT
Begin processing the 74th record. run: 1407 subRun: 1 event: 36074 at 24-Sep-2025 12:00:57 CDT
Begin processing the 75th record. run: 1407 subRun: 1 event: 36075 at 24-Sep-2025 12:01:02 CDT
Begin processing the 76th record. run: 1407 subRun: 1 event: 36076 at 24-Sep-2025 12:01:13 CDT
Begin processing the 77th record. run: 1407 subRun: 1 event: 36077 at 24-Sep-2025 12:01:29 CDT
Begin processing the 78th record. run: 1407 subRun: 1 event: 36078 at 24-Sep-2025 12:01:34 CDT
Begin processing the 79th record. run: 1407 subRun: 1 event: 36079 at 24-Sep-2025 12:01:39 CDT
Begin processing the 80th record. run: 1407 subRun: 1 event: 36080 at 24-Sep-2025 12:01:45 CDT
Begin processing the 81st record. run: 1407 subRun: 1 event: 36081 at 24-Sep-2025 12:01:50 CDT
Begin processing the 82nd record. run: 1407 subRun: 1 event: 36082 at 24-Sep-2025 12:01:55 CDT
Begin processing the 83rd record. run: 1407 subRun: 1 event: 36083 at 24-Sep-2025 12:02:00 CDT
Begin processing the 84th record. run: 1407 subRun: 1 event: 36084 at 24-Sep-2025 12:02:06 CDT
Begin processing the 85th record. run: 1407 subRun: 1 event: 36085 at 24-Sep-2025 12:02:11 CDT
Begin processing the 86th record. run: 1407 subRun: 1 event: 36086 at 24-Sep-2025 12:02:18 CDT
Begin processing the 87th record. run: 1407 subRun: 1 event: 36087 at 24-Sep-2025 12:10:38 CDT
Begin processing the 88th record. run: 1407 subRun: 1 event: 36088 at 24-Sep-2025 12:10:43 CDT
Begin processing the 89th record. run: 1407 subRun: 1 event: 36089 at 24-Sep-2025 12:10:57 CDT
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0100, LArCNNTrackShowerCountingValidation, STATUS_CODE_FAILURE
Begin processing the 90th record. run: 1407 subRun: 1 event: 36090 at 24-Sep-2025 12:11:02 CDT
Begin processing the 91st record. run: 1407 subRun: 1 event: 36091 at 24-Sep-2025 12:11:07 CDT
Begin processing the 92nd record. run: 1407 subRun: 1 event: 36092 at 24-Sep-2025 12:11:12 CDT
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0100, LArCNNTrackShowerCountingValidation, STATUS_CODE_FAILURE
Begin processing the 93rd record. run: 1407 subRun: 1 event: 36093 at 24-Sep-2025 12:11:16 CDT
Begin processing the 94th record. run: 1407 subRun: 1 event: 36094 at 24-Sep-2025 12:11:21 CDT
Begin processing the 95th record. run: 1407 subRun: 1 event: 36095 at 24-Sep-2025 12:11:26 CDT
Begin processing the 96th record. run: 1407 subRun: 1 event: 36096 at 24-Sep-2025 12:11:33 CDT
Begin processing the 97th record. run: 1407 subRun: 1 event: 36097 at 24-Sep-2025 12:11:39 CDT
Begin processing the 98th record. run: 1407 subRun: 1 event: 36098 at 24-Sep-2025 12:11:44 CDT
Begin processing the 99th record. run: 1407 subRun: 1 event: 36099 at 24-Sep-2025 12:11:49 CDT
Begin processing the 100th record. run: 1407 subRun: 1 event: 36100 at 24-Sep-2025 12:11:54 CDT
24-Sep-2025 12:11:59 CDT  Closed output file "nue_dune10kt_1x2x6_1407_360_20230826T075316Z_gen_g4_detsim_hitreco__20240221T060025Z_reco2_reco2.root"
24-Sep-2025 12:11:59 CDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/fb/9c/nue_dune10kt_1x2x6_1407_360_20230826T075316Z_gen_g4_detsim_hitreco__20240221T060025Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.073476       29.8212       1738.42       5.23993       178.679        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00664512     0.0250363     0.532683      0.0190879     0.0516413       100    
reco:pandora2:StandardPandora                  0.0614478      29.7717       1738.12       5.20297       178.65         100    
[art]:TriggerResults:TriggerResultInserter    1.4864e-05    5.19532e-05   0.000391123   4.2556e-05    4.0209e-05       100    
end_path:out1:RootOutput                       3.358e-06    8.99671e-06   2.9351e-05    9.0835e-06    3.18406e-06      100    
end_path:out1:RootOutput(write)               0.00181243     0.0240955     0.293856      0.0136669     0.0381981       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2707.64 MB
  Peak resident set size usage (VmHWM): 1711.01 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 1872676
-rw-r--r--. 1 dune osgvo        212 Sep 24 11:20 all-input-dids.txt
-rw-r--r--. 1 dune osgvo          0 Sep 24 11:20 debugprod.log
-rw-r--r--. 1 dune osgvo      46296 Sep 24 12:12 jobscript.log
-rw-r--r--. 1 dune osgvo        182 Sep 24 12:12 justin-processed-pfns.txt
drwxr-xr-x. 4 dune osgvo       4096 Sep 24 11:20 larpandoracontent
-rw-r--r--. 1 dune osgvo 1917526837 Sep 24 12:11 nue_dune10kt_1x2x6_1407_360_20230826T075316Z_gen_g4_detsim_hitreco__20240221T060025Z_reco2_reco2.root
-rw-r--r--. 1 dune osgvo        519 Sep 24 12:11 reco2_hist.root
-rw-r--r--. 1 dune osgvo      12586 Sep 24 12:11 trackShowerCountingValidation_nue_dune10kt_1x2x6_1407_360_20230826T075316Z_gen_g4_detsim_hitreco__20240221T060025Z_reco2.root
justIN time: 2025-11-05 11:14:46 UTC       justIN version: 01.05.01