justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 247834.0@dunegpschedd02.fnal.gov

Jobsub ID247834.0@dunegpschedd02.fnal.gov
Workflow ID10317
Stage ID1
User nameimawby@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2025-11-17 22:01:01
SiteUS_Wisconsin
EntryHCC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-11-17 22:09:49
From worker nodeHostnamee2600
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-17 22:02:05
Input filesfardet-hd:nu_dune10kt_1x2x6_1431_266_20230828T033343Z_gen_g4_detsim_hitreco__20240222T234956Z_reco2.root
JobscriptExit code0
Real time7m (453s)
CPU time4m (252s = 55%)
Max RSS bytes1398210560 (1333 MiB)
Outputting started2025-11-17 22:09:38
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/splitting_nu_5/fnal/10317/1/001/CheatingKalmanSplittingU_nu_dune10kt_1x2x6_1431_266_20230828T033343Z_gen_g4_detsim_hitreco__20240222T234956Z_reco2.root
Finished2025-11-17 22:09:49
Saved logsjustin-logs:247834.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

-----------------------
Input PFN = root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/74/82/nu_dune10kt_1x2x6_1431_266_20230828T033343Z_gen_g4_detsim_hitreco__20240222T234956Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 38th record. run: 1431 subRun: 1 event: 26638 at 17-Nov-2025 16:06:38 CST
Begin processing the 39th record. run: 1431 subRun: 1 event: 26639 at 17-Nov-2025 16:06:41 CST
Begin processing the 40th record. run: 1431 subRun: 1 event: 26640 at 17-Nov-2025 16:06:47 CST
Begin processing the 41st record. run: 1431 subRun: 1 event: 26641 at 17-Nov-2025 16:06:50 CST
Begin processing the 42nd record. run: 1431 subRun: 1 event: 26642 at 17-Nov-2025 16:06:52 CST
Begin processing the 43rd record. run: 1431 subRun: 1 event: 26643 at 17-Nov-2025 16:06:55 CST
Begin processing the 44th record. run: 1431 subRun: 1 event: 26644 at 17-Nov-2025 16:06:58 CST
Begin processing the 45th record. run: 1431 subRun: 1 event: 26645 at 17-Nov-2025 16:07:01 CST
Begin processing the 46th record. run: 1431 subRun: 1 event: 26646 at 17-Nov-2025 16:07:03 CST
Begin processing the 47th record. run: 1431 subRun: 1 event: 26647 at 17-Nov-2025 16:07:07 CST
Begin processing the 48th record. run: 1431 subRun: 1 event: 26648 at 17-Nov-2025 16:07:09 CST
Begin processing the 49th record. run: 1431 subRun: 1 event: 26649 at 17-Nov-2025 16:07:12 CST
Begin processing the 50th record. run: 1431 subRun: 1 event: 26650 at 17-Nov-2025 16:07:14 CST
Begin processing the 51st record. run: 1431 subRun: 1 event: 26651 at 17-Nov-2025 16:07:17 CST
Begin processing the 52nd record. run: 1431 subRun: 1 event: 26652 at 17-Nov-2025 16:07:19 CST
Begin processing the 53rd record. run: 1431 subRun: 1 event: 26653 at 17-Nov-2025 16:07:22 CST
Begin processing the 54th record. run: 1431 subRun: 1 event: 26654 at 17-Nov-2025 16:07:24 CST
Begin processing the 55th record. run: 1431 subRun: 1 event: 26655 at 17-Nov-2025 16:07:27 CST
Begin processing the 56th record. run: 1431 subRun: 1 event: 26656 at 17-Nov-2025 16:07:29 CST
Begin processing the 57th record. run: 1431 subRun: 1 event: 26657 at 17-Nov-2025 16:07:32 CST
Begin processing the 58th record. run: 1431 subRun: 1 event: 26658 at 17-Nov-2025 16:07:37 CST
Begin processing the 59th record. run: 1431 subRun: 1 event: 26659 at 17-Nov-2025 16:07:39 CST
Begin processing the 60th record. run: 1431 subRun: 1 event: 26660 at 17-Nov-2025 16:07:41 CST
Begin processing the 61st record. run: 1431 subRun: 1 event: 26661 at 17-Nov-2025 16:07:50 CST
Begin processing the 62nd record. run: 1431 subRun: 1 event: 26662 at 17-Nov-2025 16:07:53 CST
Begin processing the 63rd record. run: 1431 subRun: 1 event: 26663 at 17-Nov-2025 16:07:55 CST
Begin processing the 64th record. run: 1431 subRun: 1 event: 26664 at 17-Nov-2025 16:07:58 CST
Begin processing the 65th record. run: 1431 subRun: 1 event: 26665 at 17-Nov-2025 16:08:00 CST
Begin processing the 66th record. run: 1431 subRun: 1 event: 26666 at 17-Nov-2025 16:08:03 CST
Begin processing the 67th record. run: 1431 subRun: 1 event: 26667 at 17-Nov-2025 16:08:06 CST
Begin processing the 68th record. run: 1431 subRun: 1 event: 26668 at 17-Nov-2025 16:08:09 CST
Begin processing the 69th record. run: 1431 subRun: 1 event: 26669 at 17-Nov-2025 16:08:11 CST
Begin processing the 70th record. run: 1431 subRun: 1 event: 26670 at 17-Nov-2025 16:08:14 CST
Begin processing the 71st record. run: 1431 subRun: 1 event: 26671 at 17-Nov-2025 16:08:16 CST
Begin processing the 72nd record. run: 1431 subRun: 1 event: 26672 at 17-Nov-2025 16:08:19 CST
Begin processing the 73rd record. run: 1431 subRun: 1 event: 26673 at 17-Nov-2025 16:08:22 CST
Begin processing the 74th record. run: 1431 subRun: 1 event: 26674 at 17-Nov-2025 16:08:24 CST
Begin processing the 75th record. run: 1431 subRun: 1 event: 26675 at 17-Nov-2025 16:08:27 CST
Begin processing the 76th record. run: 1431 subRun: 1 event: 26676 at 17-Nov-2025 16:08:31 CST
Begin processing the 77th record. run: 1431 subRun: 1 event: 26677 at 17-Nov-2025 16:08:34 CST
Begin processing the 78th record. run: 1431 subRun: 1 event: 26678 at 17-Nov-2025 16:08:36 CST
Begin processing the 79th record. run: 1431 subRun: 1 event: 26679 at 17-Nov-2025 16:08:39 CST
Begin processing the 80th record. run: 1431 subRun: 1 event: 26680 at 17-Nov-2025 16:08:42 CST
Begin processing the 81st record. run: 1431 subRun: 1 event: 26681 at 17-Nov-2025 16:08:44 CST
Begin processing the 82nd record. run: 1431 subRun: 1 event: 26682 at 17-Nov-2025 16:08:47 CST
Begin processing the 83rd record. run: 1431 subRun: 1 event: 26683 at 17-Nov-2025 16:08:50 CST
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 192
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlSecondaryVertexingAlgorithm.cc line#: 186
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLSecondaryVertexing, STATUS_CODE_NOT_INITIALIZED
Begin processing the 84th record. run: 1431 subRun: 1 event: 26684 at 17-Nov-2025 16:08:51 CST
Begin processing the 85th record. run: 1431 subRun: 1 event: 26685 at 17-Nov-2025 16:08:54 CST
Begin processing the 86th record. run: 1431 subRun: 1 event: 26686 at 17-Nov-2025 16:08:57 CST
Begin processing the 87th record. run: 1431 subRun: 1 event: 26687 at 17-Nov-2025 16:09:00 CST
Begin processing the 88th record. run: 1431 subRun: 1 event: 26688 at 17-Nov-2025 16:09:04 CST
Begin processing the 89th record. run: 1431 subRun: 1 event: 26689 at 17-Nov-2025 16:09:07 CST
Begin processing the 90th record. run: 1431 subRun: 1 event: 26690 at 17-Nov-2025 16:09:09 CST
Begin processing the 91st record. run: 1431 subRun: 1 event: 26691 at 17-Nov-2025 16:09:12 CST
Begin processing the 92nd record. run: 1431 subRun: 1 event: 26692 at 17-Nov-2025 16:09:14 CST
Begin processing the 93rd record. run: 1431 subRun: 1 event: 26693 at 17-Nov-2025 16:09:17 CST
Begin processing the 94th record. run: 1431 subRun: 1 event: 26694 at 17-Nov-2025 16:09:20 CST
Begin processing the 95th record. run: 1431 subRun: 1 event: 26695 at 17-Nov-2025 16:09:22 CST
Begin processing the 96th record. run: 1431 subRun: 1 event: 26696 at 17-Nov-2025 16:09:24 CST
Begin processing the 97th record. run: 1431 subRun: 1 event: 26697 at 17-Nov-2025 16:09:27 CST
Begin processing the 98th record. run: 1431 subRun: 1 event: 26698 at 17-Nov-2025 16:09:29 CST
Begin processing the 99th record. run: 1431 subRun: 1 event: 26699 at 17-Nov-2025 16:09:32 CST
Begin processing the 100th record. run: 1431 subRun: 1 event: 26700 at 17-Nov-2025 16:09:34 CST
17-Nov-2025 16:09:37 CST  Closed output file "nu_dune10kt_1x2x6_1431_266_20230828T033343Z_gen_g4_detsim_hitreco__20240222T234956Z_reco2_reco2.root"
17-Nov-2025 16:09:37 CST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/74/82/nu_dune10kt_1x2x6_1431_266_20230828T033343Z_gen_g4_detsim_hitreco__20240222T234956Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.344693       2.23558       7.03354       2.05896      0.743254        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0309905     0.0621132     0.204142      0.0559894     0.023298        100    
reco:pandora:StandardPandora                   0.139569       2.16951       6.96487       2.00427      0.743973        100    
[art]:TriggerResults:TriggerResultInserter     9.879e-06    1.41882e-05   4.1148e-05    1.2859e-05    4.73983e-06      100    
end_path:out1:RootOutput                       2.255e-06    3.32933e-06   1.4858e-05     2.891e-06    1.47484e-06      100    
end_path:out1:RootOutput(write)               0.000714539   0.00378988     0.0147521    0.00313456    0.00282059       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2403.6 MB
  Peak resident set size usage (VmHWM): 1398.21 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
mv: cannot stat 'CheatingKalmanSplittingV.root': No such file or directory
mv: cannot stat 'CheatingKalmanSplittingW.root': No such file or directory
justIN time: 2026-02-04 07:43:27 UTC       justIN version: 01.06.00