justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 43097.60@dunegpschedd02.fnal.gov

Jobsub ID43097.60@dunegpschedd02.fnal.gov
Workflow ID2980
Stage ID1
User namelwhite86@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-09-19 08:37:29
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-09-19 10:10:28
From worker nodeHostnamee2550
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-09-19 10:02:42
Input filesfardet-hd:nu_dune10kt_1x2x6_1114_17_20230827T153311Z_gen_g4_detsim_hitreco__20240223T215055Z_reco2.root
JobscriptExit code0
Real time7m (455s)
CPU time4m (277s = 60%)
Max RSS bytes1382703104 (1318 MiB)
Outputting started2025-09-19 10:10:18
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/02980/1/001/trackShowerCountingValidation_nu_dune10kt_1x2x6_1114_17_20230827T153311Z_gen_g4_detsim_hitreco__20240223T215055Z_reco2.root
Finished2025-09-19 10:10:28
Saved logsjustin-logs:43097.60-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t 19-Sep-2025 05:07:15 CDT
Begin processing the 40th record. run: 1114 subRun: 1 event: 1740 at 19-Sep-2025 05:07:19 CDT
Begin processing the 41st record. run: 1114 subRun: 1 event: 1741 at 19-Sep-2025 05:07:22 CDT
Begin processing the 42nd record. run: 1114 subRun: 1 event: 1742 at 19-Sep-2025 05:07:24 CDT
Begin processing the 43rd record. run: 1114 subRun: 1 event: 1743 at 19-Sep-2025 05:07:28 CDT
Begin processing the 44th record. run: 1114 subRun: 1 event: 1744 at 19-Sep-2025 05:07:30 CDT
Begin processing the 45th record. run: 1114 subRun: 1 event: 1745 at 19-Sep-2025 05:07:35 CDT
Begin processing the 46th record. run: 1114 subRun: 1 event: 1746 at 19-Sep-2025 05:07:38 CDT
Begin processing the 47th record. run: 1114 subRun: 1 event: 1747 at 19-Sep-2025 05:07:40 CDT
Begin processing the 48th record. run: 1114 subRun: 1 event: 1748 at 19-Sep-2025 05:07:43 CDT
Begin processing the 49th record. run: 1114 subRun: 1 event: 1749 at 19-Sep-2025 05:07:46 CDT
Begin processing the 50th record. run: 1114 subRun: 1 event: 1750 at 19-Sep-2025 05:07:48 CDT
Begin processing the 51st record. run: 1114 subRun: 1 event: 1751 at 19-Sep-2025 05:07:51 CDT
Begin processing the 52nd record. run: 1114 subRun: 1 event: 1752 at 19-Sep-2025 05:07:54 CDT
Begin processing the 53rd record. run: 1114 subRun: 1 event: 1753 at 19-Sep-2025 05:07:56 CDT
Begin processing the 54th record. run: 1114 subRun: 1 event: 1754 at 19-Sep-2025 05:07:59 CDT
Begin processing the 55th record. run: 1114 subRun: 1 event: 1755 at 19-Sep-2025 05:08:02 CDT
Begin processing the 56th record. run: 1114 subRun: 1 event: 1756 at 19-Sep-2025 05:08:09 CDT
Begin processing the 57th record. run: 1114 subRun: 1 event: 1757 at 19-Sep-2025 05:08:12 CDT
Begin processing the 58th record. run: 1114 subRun: 1 event: 1758 at 19-Sep-2025 05:08:16 CDT
Begin processing the 59th record. run: 1114 subRun: 1 event: 1759 at 19-Sep-2025 05:08:19 CDT
Begin processing the 60th record. run: 1114 subRun: 1 event: 1760 at 19-Sep-2025 05:08:22 CDT
Begin processing the 61st record. run: 1114 subRun: 1 event: 1761 at 19-Sep-2025 05:08:24 CDT
Begin processing the 62nd record. run: 1114 subRun: 1 event: 1762 at 19-Sep-2025 05:08:27 CDT
Begin processing the 63rd record. run: 1114 subRun: 1 event: 1763 at 19-Sep-2025 05:08:30 CDT
Begin processing the 64th record. run: 1114 subRun: 1 event: 1764 at 19-Sep-2025 05:08:33 CDT
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArCNNTrackShowerCountingValidation, STATUS_CODE_FAILURE
Begin processing the 65th record. run: 1114 subRun: 1 event: 1765 at 19-Sep-2025 05:08:35 CDT
Begin processing the 66th record. run: 1114 subRun: 1 event: 1766 at 19-Sep-2025 05:08:38 CDT
Begin processing the 67th record. run: 1114 subRun: 1 event: 1767 at 19-Sep-2025 05:08:40 CDT
Begin processing the 68th record. run: 1114 subRun: 1 event: 1768 at 19-Sep-2025 05:08:43 CDT
Begin processing the 69th record. run: 1114 subRun: 1 event: 1769 at 19-Sep-2025 05:08:46 CDT
Begin processing the 70th record. run: 1114 subRun: 1 event: 1770 at 19-Sep-2025 05:08:48 CDT
Begin processing the 71st record. run: 1114 subRun: 1 event: 1771 at 19-Sep-2025 05:08:51 CDT
Begin processing the 72nd record. run: 1114 subRun: 1 event: 1772 at 19-Sep-2025 05:08:54 CDT
Begin processing the 73rd record. run: 1114 subRun: 1 event: 1773 at 19-Sep-2025 05:08:57 CDT
Begin processing the 74th record. run: 1114 subRun: 1 event: 1774 at 19-Sep-2025 05:09:01 CDT
Begin processing the 75th record. run: 1114 subRun: 1 event: 1775 at 19-Sep-2025 05:09:03 CDT
Begin processing the 76th record. run: 1114 subRun: 1 event: 1776 at 19-Sep-2025 05:09:07 CDT
Begin processing the 77th record. run: 1114 subRun: 1 event: 1777 at 19-Sep-2025 05:09:10 CDT
Begin processing the 78th record. run: 1114 subRun: 1 event: 1778 at 19-Sep-2025 05:09:13 CDT
Begin processing the 79th record. run: 1114 subRun: 1 event: 1779 at 19-Sep-2025 05:09:15 CDT
Begin processing the 80th record. run: 1114 subRun: 1 event: 1780 at 19-Sep-2025 05:09:18 CDT
Begin processing the 81st record. run: 1114 subRun: 1 event: 1781 at 19-Sep-2025 05:09:20 CDT
Begin processing the 82nd record. run: 1114 subRun: 1 event: 1782 at 19-Sep-2025 05:09:24 CDT
Begin processing the 83rd record. run: 1114 subRun: 1 event: 1783 at 19-Sep-2025 05:09:26 CDT
Begin processing the 84th record. run: 1114 subRun: 1 event: 1784 at 19-Sep-2025 05:09:29 CDT
Begin processing the 85th record. run: 1114 subRun: 1 event: 1785 at 19-Sep-2025 05:09:32 CDT
Begin processing the 86th record. run: 1114 subRun: 1 event: 1786 at 19-Sep-2025 05:09:35 CDT
Begin processing the 87th record. run: 1114 subRun: 1 event: 1787 at 19-Sep-2025 05:09:38 CDT
Begin processing the 88th record. run: 1114 subRun: 1 event: 1788 at 19-Sep-2025 05:09:41 CDT
Begin processing the 89th record. run: 1114 subRun: 1 event: 1789 at 19-Sep-2025 05:09:44 CDT
Begin processing the 90th record. run: 1114 subRun: 1 event: 1790 at 19-Sep-2025 05:09:47 CDT
Begin processing the 91st record. run: 1114 subRun: 1 event: 1791 at 19-Sep-2025 05:09:49 CDT
Begin processing the 92nd record. run: 1114 subRun: 1 event: 1792 at 19-Sep-2025 05:09:53 CDT
Begin processing the 93rd record. run: 1114 subRun: 1 event: 1793 at 19-Sep-2025 05:09:57 CDT
Begin processing the 94th record. run: 1114 subRun: 1 event: 1794 at 19-Sep-2025 05:10:00 CDT
Begin processing the 95th record. run: 1114 subRun: 1 event: 1795 at 19-Sep-2025 05:10:02 CDT
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, unknown exception
PandoraContentApi::GetList(*this, m_inputPfoListName, pPfoList) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraEventClassification/srcs/larpandoracontent/larpandoradlcontent/LArEventClassification/CNNTrackShowerCountingValidationAlgorithm.cc line#: 78
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArCNNTrackShowerCountingValidation, STATUS_CODE_NOT_INITIALIZED
Begin processing the 96th record. run: 1114 subRun: 1 event: 1796 at 19-Sep-2025 05:10:04 CDT
Begin processing the 97th record. run: 1114 subRun: 1 event: 1797 at 19-Sep-2025 05:10:06 CDT
Begin processing the 98th record. run: 1114 subRun: 1 event: 1798 at 19-Sep-2025 05:10:09 CDT
Begin processing the 99th record. run: 1114 subRun: 1 event: 1799 at 19-Sep-2025 05:10:11 CDT
Begin processing the 100th record. run: 1114 subRun: 1 event: 1800 at 19-Sep-2025 05:10:14 CDT
19-Sep-2025 05:10:18 CDT  Closed output file "nu_dune10kt_1x2x6_1114_17_20230827T153311Z_gen_g4_detsim_hitreco__20240223T215055Z_reco2_reco2.root"
19-Sep-2025 05:10:18 CDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e3/65/nu_dune10kt_1x2x6_1114_17_20230827T153311Z_gen_g4_detsim_hitreco__20240223T215055Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.951743       2.5987        10.3885       2.36326      0.995691        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0364564     0.0590313      0.27118      0.0539258     0.0381545       100    
reco:pandora2:StandardPandora                  0.896519       2.53065       10.2936       2.29511      0.992978        100    
[art]:TriggerResults:TriggerResultInserter     8.977e-06    1.56824e-05   5.7167e-05    1.2614e-05    8.50114e-06      100    
end_path:out1:RootOutput                       2.043e-06    3.2756e-06    1.4857e-05     2.585e-06    1.84328e-06      100    
end_path:out1:RootOutput(write)               0.000875968   0.00885797     0.0417375    0.00769807    0.00655977       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2367.16 MB
  Peak resident set size usage (VmHWM): 1382.7 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 1142860
-rw-r--r-- 1 slot1_8 slot1_8        208 Sep 19 05:02 all-input-dids.txt
-rw-r--r-- 1 slot1_8 slot1_8          0 Sep 19 05:02 debugprod.log
-rw-r--r-- 1 slot1_8 slot1_8      41520 Sep 19 05:10 jobscript.log
-rw-r--r-- 1 slot1_8 slot1_8        180 Sep 19 05:10 justin-processed-pfns.txt
drwxr-xr-x 4 slot1_8 slot1_8       4096 Sep 19 05:02 larpandoracontent
-rw-r--r-- 1 slot1_8 slot1_8 1170209677 Sep 19 05:10 nu_dune10kt_1x2x6_1114_17_20230827T153311Z_gen_g4_detsim_hitreco__20240223T215055Z_reco2_reco2.root
-rw-r--r-- 1 slot1_8 slot1_8        519 Sep 19 05:10 reco2_hist.root
-rw-r--r-- 1 slot1_8 slot1_8      12172 Sep 19 05:10 trackShowerCountingValidation_nu_dune10kt_1x2x6_1114_17_20230827T153311Z_gen_g4_detsim_hitreco__20240223T215055Z_reco2.root
justIN time: 2025-09-19 15:50:06 UTC       justIN version: 01.05.00