justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 46184.15@dunegpschedd02.fnal.gov

Jobsub ID46184.15@dunegpschedd02.fnal.gov
Workflow ID3309
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit43200 (12 hours)
Submitted time2025-09-22 19:13:57
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-09-22 19:25:19
From worker nodeHostnamee4064
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-22 19:14:28
Input filesfardet-hd:nu_dune10kt_1x2x6_1122_610_20230828T063355Z_gen_g4_detsim_hitreco__20240223T220723Z_reco2.root
JobscriptExit code0
Real time6m (409s)
CPU time3m (230s = 56%)
Max RSS bytes1433600000 (1367 MiB)
Outputting started2025-09-22 19:21:18
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/splitting_nu_0/fnal/03309/1/003/CheatingKalmanSplittingU_nu_dune10kt_1x2x6_1122_610_20230828T063355Z_gen_g4_detsim_hitreco__20240223T220723Z_reco2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/splitting_nu_0/fnal/03309/1/003/CheatingKalmanSplittingV_nu_dune10kt_1x2x6_1122_610_20230828T063355Z_gen_g4_detsim_hitreco__20240223T220723Z_reco2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/splitting_nu_0/fnal/03309/1/003/CheatingKalmanSplittingW_nu_dune10kt_1x2x6_1122_610_20230828T063355Z_gen_g4_detsim_hitreco__20240223T220723Z_reco2.root
Finished2025-09-22 19:25:19
Saved logsjustin-logs:46184.15-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

mpty ------------------
---------------------------------------------------------
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/74/16/nu_dune10kt_1x2x6_1122_610_20230828T063355Z_gen_g4_detsim_hitreco__20240223T220723Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 45th record. run: 1122 subRun: 1 event: 61045 at 22-Sep-2025 14:18:44 CDT
Begin processing the 46th record. run: 1122 subRun: 1 event: 61046 at 22-Sep-2025 14:18:46 CDT
Begin processing the 47th record. run: 1122 subRun: 1 event: 61047 at 22-Sep-2025 14:18:48 CDT
Begin processing the 48th record. run: 1122 subRun: 1 event: 61048 at 22-Sep-2025 14:18:50 CDT
Begin processing the 49th record. run: 1122 subRun: 1 event: 61049 at 22-Sep-2025 14:18:52 CDT
Begin processing the 50th record. run: 1122 subRun: 1 event: 61050 at 22-Sep-2025 14:18:55 CDT
Begin processing the 51st record. run: 1122 subRun: 1 event: 61051 at 22-Sep-2025 14:18:57 CDT
Begin processing the 52nd record. run: 1122 subRun: 1 event: 61052 at 22-Sep-2025 14:18:59 CDT
Begin processing the 53rd record. run: 1122 subRun: 1 event: 61053 at 22-Sep-2025 14:19:02 CDT
Begin processing the 54th record. run: 1122 subRun: 1 event: 61054 at 22-Sep-2025 14:19:04 CDT
Begin processing the 55th record. run: 1122 subRun: 1 event: 61055 at 22-Sep-2025 14:19:07 CDT
Begin processing the 56th record. run: 1122 subRun: 1 event: 61056 at 22-Sep-2025 14:19:09 CDT
Begin processing the 57th record. run: 1122 subRun: 1 event: 61057 at 22-Sep-2025 14:19:11 CDT
Begin processing the 58th record. run: 1122 subRun: 1 event: 61058 at 22-Sep-2025 14:19:14 CDT
Begin processing the 59th record. run: 1122 subRun: 1 event: 61059 at 22-Sep-2025 14:19:16 CDT
Begin processing the 60th record. run: 1122 subRun: 1 event: 61060 at 22-Sep-2025 14:19:18 CDT
Begin processing the 61st record. run: 1122 subRun: 1 event: 61061 at 22-Sep-2025 14:19:20 CDT
Begin processing the 62nd record. run: 1122 subRun: 1 event: 61062 at 22-Sep-2025 14:19:22 CDT
Begin processing the 63rd record. run: 1122 subRun: 1 event: 61063 at 22-Sep-2025 14:19:28 CDT
Begin processing the 64th record. run: 1122 subRun: 1 event: 61064 at 22-Sep-2025 14:19:30 CDT
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 65th record. run: 1122 subRun: 1 event: 61065 at 22-Sep-2025 14:19:31 CDT
Begin processing the 66th record. run: 1122 subRun: 1 event: 61066 at 22-Sep-2025 14:19:33 CDT
Begin processing the 67th record. run: 1122 subRun: 1 event: 61067 at 22-Sep-2025 14:19:54 CDT
Begin processing the 68th record. run: 1122 subRun: 1 event: 61068 at 22-Sep-2025 14:19:56 CDT
Begin processing the 69th record. run: 1122 subRun: 1 event: 61069 at 22-Sep-2025 14:19:59 CDT
Begin processing the 70th record. run: 1122 subRun: 1 event: 61070 at 22-Sep-2025 14:20:01 CDT
Begin processing the 71st record. run: 1122 subRun: 1 event: 61071 at 22-Sep-2025 14:20:03 CDT
Begin processing the 72nd record. run: 1122 subRun: 1 event: 61072 at 22-Sep-2025 14:20:06 CDT
Begin processing the 73rd record. run: 1122 subRun: 1 event: 61073 at 22-Sep-2025 14:20:08 CDT
Begin processing the 74th record. run: 1122 subRun: 1 event: 61074 at 22-Sep-2025 14:20:10 CDT
Begin processing the 75th record. run: 1122 subRun: 1 event: 61075 at 22-Sep-2025 14:20:13 CDT
Begin processing the 76th record. run: 1122 subRun: 1 event: 61076 at 22-Sep-2025 14:20:15 CDT
Begin processing the 77th record. run: 1122 subRun: 1 event: 61077 at 22-Sep-2025 14:20:17 CDT
Begin processing the 78th record. run: 1122 subRun: 1 event: 61078 at 22-Sep-2025 14:20:19 CDT
Begin processing the 79th record. run: 1122 subRun: 1 event: 61079 at 22-Sep-2025 14:20:22 CDT
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 80th record. run: 1122 subRun: 1 event: 61080 at 22-Sep-2025 14:20:23 CDT
Begin processing the 81st record. run: 1122 subRun: 1 event: 61081 at 22-Sep-2025 14:20:25 CDT
Begin processing the 82nd record. run: 1122 subRun: 1 event: 61082 at 22-Sep-2025 14:20:27 CDT
Begin processing the 83rd record. run: 1122 subRun: 1 event: 61083 at 22-Sep-2025 14:20:30 CDT
Begin processing the 84th record. run: 1122 subRun: 1 event: 61084 at 22-Sep-2025 14:20:32 CDT
Begin processing the 85th record. run: 1122 subRun: 1 event: 61085 at 22-Sep-2025 14:20:35 CDT
Begin processing the 86th record. run: 1122 subRun: 1 event: 61086 at 22-Sep-2025 14:20:38 CDT
Begin processing the 87th record. run: 1122 subRun: 1 event: 61087 at 22-Sep-2025 14:20:40 CDT
Begin processing the 88th record. run: 1122 subRun: 1 event: 61088 at 22-Sep-2025 14:20:43 CDT
Begin processing the 89th record. run: 1122 subRun: 1 event: 61089 at 22-Sep-2025 14:20:46 CDT
Begin processing the 90th record. run: 1122 subRun: 1 event: 61090 at 22-Sep-2025 14:20:49 CDT
Begin processing the 91st record. run: 1122 subRun: 1 event: 61091 at 22-Sep-2025 14:20:52 CDT
Begin processing the 92nd record. run: 1122 subRun: 1 event: 61092 at 22-Sep-2025 14:20:54 CDT
Begin processing the 93rd record. run: 1122 subRun: 1 event: 61093 at 22-Sep-2025 14:20:57 CDT
Begin processing the 94th record. run: 1122 subRun: 1 event: 61094 at 22-Sep-2025 14:21:00 CDT
Begin processing the 95th record. run: 1122 subRun: 1 event: 61095 at 22-Sep-2025 14:21:03 CDT
Begin processing the 96th record. run: 1122 subRun: 1 event: 61096 at 22-Sep-2025 14:21:05 CDT
Begin processing the 97th record. run: 1122 subRun: 1 event: 61097 at 22-Sep-2025 14:21:07 CDT
Begin processing the 98th record. run: 1122 subRun: 1 event: 61098 at 22-Sep-2025 14:21:10 CDT
Begin processing the 99th record. run: 1122 subRun: 1 event: 61099 at 22-Sep-2025 14:21:12 CDT
Begin processing the 100th record. run: 1122 subRun: 1 event: 61100 at 22-Sep-2025 14:21:15 CDT
22-Sep-2025 14:21:17 CDT  Closed output file "nu_dune10kt_1x2x6_1122_610_20230828T063355Z_gen_g4_detsim_hitreco__20240223T220723Z_reco2_reco2.root"
22-Sep-2025 14:21:17 CDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/74/16/nu_dune10kt_1x2x6_1122_610_20230828T063355Z_gen_g4_detsim_hitreco__20240223T220723Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.164882       2.06845       19.4611       1.87169       1.81372        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0347002     0.0408606     0.0545803     0.0407199     0.0016255       100    
reco:pandora:StandardPandora                   0.124192       2.02409       19.4007       1.82594       1.81186        100    
[art]:TriggerResults:TriggerResultInserter     9.808e-06    1.60882e-05    4.784e-05    1.4352e-05    5.86228e-06      100    
end_path:out1:RootOutput                       2.114e-06    4.14152e-06    1.619e-05    4.1275e-06    1.73405e-06      100    
end_path:out1:RootOutput(write)               0.000655553   0.00326409     0.024492     0.00254592     0.0033072       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2422.13 MB
  Peak resident set size usage (VmHWM): 1433.6 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2025-11-05 11:16:47 UTC       justIN version: 01.05.01