justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 276334.83@dunegpschedd02.fnal.gov

Jobsub ID276334.83@dunegpschedd02.fnal.gov
Workflow ID12096
Stage ID1
User nameimawby@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-01-19 14:15:16
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_gk02_condce_mcore
Last heartbeat2026-01-19 14:59:55
From worker nodeHostnameiut2-c245.iu.edu
cpuinfoIntel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-19 14:33:32
Input filesfardet-hd:nue_dune10kt_1x2x6_1405_828_20230826T021213Z_gen_g4_detsim_hitreco__20240221T090842Z_reco2.root
JobscriptExit code0
Real time26m (1567s)
CPU time13m (788s = 50%)
Max RSS bytes1245933568 (1188 MiB)
Outputting started2026-01-19 14:59:40
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/metrics_nue_2/fnal/12096/1/001/Validation_nue_dune10kt_1x2x6_1405_828_20230826T021213Z_gen_g4_detsim_hitreco__20240221T090842Z_reco2.root
Finished2026-01-19 14:59:55
Saved logsjustin-logs:276334.83-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

272760a301f448969f3ed8/localProducts_larsoft_v10_13_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/32/3c/nue_dune10kt_1x2x6_1405_828_20230826T021213Z_gen_g4_detsim_hitreco__20240221T090842Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 41st record. run: 1405 subRun: 1 event: 82841 at 19-Jan-2026 09:52:22 EST
Begin processing the 42nd record. run: 1405 subRun: 1 event: 82842 at 19-Jan-2026 09:52:27 EST
Begin processing the 43rd record. run: 1405 subRun: 1 event: 82843 at 19-Jan-2026 09:52:34 EST
Begin processing the 44th record. run: 1405 subRun: 1 event: 82844 at 19-Jan-2026 09:52:39 EST
Begin processing the 45th record. run: 1405 subRun: 1 event: 82845 at 19-Jan-2026 09:52:46 EST
Begin processing the 46th record. run: 1405 subRun: 1 event: 82846 at 19-Jan-2026 09:52:51 EST
Begin processing the 47th record. run: 1405 subRun: 1 event: 82847 at 19-Jan-2026 09:52:56 EST
Begin processing the 48th record. run: 1405 subRun: 1 event: 82848 at 19-Jan-2026 09:53:14 EST
Begin processing the 49th record. run: 1405 subRun: 1 event: 82849 at 19-Jan-2026 09:53:38 EST
Begin processing the 50th record. run: 1405 subRun: 1 event: 82850 at 19-Jan-2026 09:53:42 EST
Begin processing the 51st record. run: 1405 subRun: 1 event: 82851 at 19-Jan-2026 09:53:46 EST
Begin processing the 52nd record. run: 1405 subRun: 1 event: 82852 at 19-Jan-2026 09:53:52 EST
Begin processing the 53rd record. run: 1405 subRun: 1 event: 82853 at 19-Jan-2026 09:54:00 EST
Begin processing the 54th record. run: 1405 subRun: 1 event: 82854 at 19-Jan-2026 09:54:11 EST
Begin processing the 55th record. run: 1405 subRun: 1 event: 82855 at 19-Jan-2026 09:54:28 EST
Begin processing the 56th record. run: 1405 subRun: 1 event: 82856 at 19-Jan-2026 09:54:33 EST
Begin processing the 57th record. run: 1405 subRun: 1 event: 82857 at 19-Jan-2026 09:54:39 EST
Begin processing the 58th record. run: 1405 subRun: 1 event: 82858 at 19-Jan-2026 09:54:45 EST
Begin processing the 59th record. run: 1405 subRun: 1 event: 82859 at 19-Jan-2026 09:54:49 EST
Begin processing the 60th record. run: 1405 subRun: 1 event: 82860 at 19-Jan-2026 09:54:57 EST
Begin processing the 61st record. run: 1405 subRun: 1 event: 82861 at 19-Jan-2026 09:55:07 EST
Begin processing the 62nd record. run: 1405 subRun: 1 event: 82862 at 19-Jan-2026 09:55:19 EST
Begin processing the 63rd record. run: 1405 subRun: 1 event: 82863 at 19-Jan-2026 09:55:45 EST
Begin processing the 64th record. run: 1405 subRun: 1 event: 82864 at 19-Jan-2026 09:55:51 EST
Begin processing the 65th record. run: 1405 subRun: 1 event: 82865 at 19-Jan-2026 09:55:59 EST
Begin processing the 66th record. run: 1405 subRun: 1 event: 82866 at 19-Jan-2026 09:56:04 EST
Begin processing the 67th record. run: 1405 subRun: 1 event: 82867 at 19-Jan-2026 09:56:14 EST
Begin processing the 68th record. run: 1405 subRun: 1 event: 82868 at 19-Jan-2026 09:56:18 EST
Begin processing the 69th record. run: 1405 subRun: 1 event: 82869 at 19-Jan-2026 09:56:24 EST
Begin processing the 70th record. run: 1405 subRun: 1 event: 82870 at 19-Jan-2026 09:56:29 EST
Begin processing the 71st record. run: 1405 subRun: 1 event: 82871 at 19-Jan-2026 09:56:34 EST
Begin processing the 72nd record. run: 1405 subRun: 1 event: 82872 at 19-Jan-2026 09:56:42 EST
Begin processing the 73rd record. run: 1405 subRun: 1 event: 82873 at 19-Jan-2026 09:56:51 EST
Begin processing the 74th record. run: 1405 subRun: 1 event: 82874 at 19-Jan-2026 09:56:57 EST
Begin processing the 75th record. run: 1405 subRun: 1 event: 82875 at 19-Jan-2026 09:57:01 EST
Begin processing the 76th record. run: 1405 subRun: 1 event: 82876 at 19-Jan-2026 09:57:13 EST
Begin processing the 77th record. run: 1405 subRun: 1 event: 82877 at 19-Jan-2026 09:57:18 EST
Begin processing the 78th record. run: 1405 subRun: 1 event: 82878 at 19-Jan-2026 09:57:24 EST
Begin processing the 79th record. run: 1405 subRun: 1 event: 82879 at 19-Jan-2026 09:57:29 EST
Begin processing the 80th record. run: 1405 subRun: 1 event: 82880 at 19-Jan-2026 09:57:34 EST
Begin processing the 81st record. run: 1405 subRun: 1 event: 82881 at 19-Jan-2026 09:57:39 EST
Begin processing the 82nd record. run: 1405 subRun: 1 event: 82882 at 19-Jan-2026 09:57:43 EST
Begin processing the 83rd record. run: 1405 subRun: 1 event: 82883 at 19-Jan-2026 09:57:55 EST
Begin processing the 84th record. run: 1405 subRun: 1 event: 82884 at 19-Jan-2026 09:58:02 EST
Begin processing the 85th record. run: 1405 subRun: 1 event: 82885 at 19-Jan-2026 09:58:05 EST
Begin processing the 86th record. run: 1405 subRun: 1 event: 82886 at 19-Jan-2026 09:58:13 EST
Begin processing the 87th record. run: 1405 subRun: 1 event: 82887 at 19-Jan-2026 09:58:17 EST
Begin processing the 88th record. run: 1405 subRun: 1 event: 82888 at 19-Jan-2026 09:58:27 EST
Begin processing the 89th record. run: 1405 subRun: 1 event: 82889 at 19-Jan-2026 09:58:33 EST
Begin processing the 90th record. run: 1405 subRun: 1 event: 82890 at 19-Jan-2026 09:58:38 EST
Begin processing the 91st record. run: 1405 subRun: 1 event: 82891 at 19-Jan-2026 09:58:43 EST
Begin processing the 92nd record. run: 1405 subRun: 1 event: 82892 at 19-Jan-2026 09:58:49 EST
Begin processing the 93rd record. run: 1405 subRun: 1 event: 82893 at 19-Jan-2026 09:58:59 EST
Begin processing the 94th record. run: 1405 subRun: 1 event: 82894 at 19-Jan-2026 09:59:05 EST
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 192
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlSecondaryVertexingAlgorithm.cc line#: 186
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLSecondaryVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, listName, pList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetList
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArTwoDReco/DLThreeDClusterSplittingAlgorithm.cc line#: 194
Begin processing the 95th record. run: 1405 subRun: 1 event: 82895 at 19-Jan-2026 09:59:05 EST
Begin processing the 96th record. run: 1405 subRun: 1 event: 82896 at 19-Jan-2026 09:59:10 EST
Begin processing the 97th record. run: 1405 subRun: 1 event: 82897 at 19-Jan-2026 09:59:15 EST
Begin processing the 98th record. run: 1405 subRun: 1 event: 82898 at 19-Jan-2026 09:59:21 EST
Begin processing the 99th record. run: 1405 subRun: 1 event: 82899 at 19-Jan-2026 09:59:25 EST
Begin processing the 100th record. run: 1405 subRun: 1 event: 82900 at 19-Jan-2026 09:59:32 EST
19-Jan-2026 09:59:39 EST  Closed output file "nue_dune10kt_1x2x6_1405_828_20230826T021213Z_gen_g4_detsim_hitreco__20240221T090842Z_reco2_reco2.root"
19-Jan-2026 09:59:39 EST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/32/3c/nue_dune10kt_1x2x6_1405_828_20230826T021213Z_gen_g4_detsim_hitreco__20240221T090842Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.105461       6.88342       44.8551       4.76791       6.57369        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0120004     0.0351862     0.155864      0.0328227     0.0162364       100    
reco:pandora:StandardPandora                   0.0510604      6.82848       44.7616       4.70868       6.55673        100    
[art]:TriggerResults:TriggerResultInserter    1.6712e-05    4.17496e-05   0.000108736   3.57075e-05   1.98175e-05      100    
end_path:out1:RootOutput                       2.988e-06    6.17688e-06   4.1025e-05    4.9825e-06    4.03802e-06      100    
end_path:out1:RootOutput(write)               0.00173677     0.0194244     0.103678      0.0129751     0.018375        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2229 MB
  Peak resident set size usage (VmHWM): 1245.93 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-05 17:07:58 UTC       justIN version: 01.06.00