justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 233760.20@dunegpschedd02.fnal.gov

Jobsub ID233760.20@dunegpschedd02.fnal.gov
Workflow ID9327
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2025-10-28 14:36:17
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-10-28 15:32:48
From worker nodeHostnamee4021
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-28 14:48:07
Input filesfardet-hd:nu_dune10kt_1x2x6_1427_308_20230827T213340Z_gen_g4_detsim_hitreco__20240223T003555Z_reco2.root
JobscriptExit code0
Real time44m (2667s)
CPU time10m (632s = 23%)
Max RSS bytes1414328320 (1348 MiB)
Outputting started2025-10-28 15:32:35
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/kalmanAlg_nu_0/fnal/09327/1/002/ClusterValidation_WithAlg_nu_dune10kt_1x2x6_1427_308_20230827T213340Z_gen_g4_detsim_hitreco__20240223T003555Z_reco2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/kalmanAlg_nu_0/fnal/09327/1/002/ClusterValidation_WithoutAlg_nu_dune10kt_1x2x6_1427_308_20230827T213340Z_gen_g4_detsim_hitreco__20240223T003555Z_reco2.root
Finished2025-10-28 15:32:48
Saved logsjustin-logs:233760.20-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

f
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/9d/e5/nu_dune10kt_1x2x6_1427_308_20230827T213340Z_gen_g4_detsim_hitreco__20240223T003555Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Failure in algorithm Alg0004, LArDLSecondaryVertexing, STATUS_CODE_NOT_INITIALIZED
Begin processing the 24th record. run: 1427 subRun: 1 event: 30824 at 28-Oct-2025 10:17:54 CDT
Begin processing the 25th record. run: 1427 subRun: 1 event: 30825 at 28-Oct-2025 10:18:01 CDT
Begin processing the 26th record. run: 1427 subRun: 1 event: 30826 at 28-Oct-2025 10:18:08 CDT
Begin processing the 27th record. run: 1427 subRun: 1 event: 30827 at 28-Oct-2025 10:18:16 CDT
Begin processing the 28th record. run: 1427 subRun: 1 event: 30828 at 28-Oct-2025 10:18:23 CDT
Begin processing the 29th record. run: 1427 subRun: 1 event: 30829 at 28-Oct-2025 10:18:32 CDT
Begin processing the 30th record. run: 1427 subRun: 1 event: 30830 at 28-Oct-2025 10:18:38 CDT
Begin processing the 31st record. run: 1427 subRun: 1 event: 30831 at 28-Oct-2025 10:19:10 CDT
Begin processing the 32nd record. run: 1427 subRun: 1 event: 30832 at 28-Oct-2025 10:19:22 CDT
Begin processing the 33rd record. run: 1427 subRun: 1 event: 30833 at 28-Oct-2025 10:19:31 CDT
Begin processing the 34th record. run: 1427 subRun: 1 event: 30834 at 28-Oct-2025 10:20:02 CDT
Begin processing the 35th record. run: 1427 subRun: 1 event: 30835 at 28-Oct-2025 10:20:10 CDT
Begin processing the 36th record. run: 1427 subRun: 1 event: 30836 at 28-Oct-2025 10:20:23 CDT
Begin processing the 37th record. run: 1427 subRun: 1 event: 30837 at 28-Oct-2025 10:20:32 CDT
Begin processing the 38th record. run: 1427 subRun: 1 event: 30838 at 28-Oct-2025 10:20:42 CDT
Begin processing the 39th record. run: 1427 subRun: 1 event: 30839 at 28-Oct-2025 10:20:48 CDT
Begin processing the 40th record. run: 1427 subRun: 1 event: 30840 at 28-Oct-2025 10:20:56 CDT
Begin processing the 41st record. run: 1427 subRun: 1 event: 30841 at 28-Oct-2025 10:21:05 CDT
Begin processing the 42nd record. run: 1427 subRun: 1 event: 30842 at 28-Oct-2025 10:21:16 CDT
Begin processing the 43rd record. run: 1427 subRun: 1 event: 30843 at 28-Oct-2025 10:21:22 CDT
Begin processing the 44th record. run: 1427 subRun: 1 event: 30844 at 28-Oct-2025 10:21:34 CDT
Begin processing the 45th record. run: 1427 subRun: 1 event: 30845 at 28-Oct-2025 10:23:26 CDT
Begin processing the 46th record. run: 1427 subRun: 1 event: 30846 at 28-Oct-2025 10:23:36 CDT
Begin processing the 47th record. run: 1427 subRun: 1 event: 30847 at 28-Oct-2025 10:24:21 CDT
Begin processing the 48th record. run: 1427 subRun: 1 event: 30848 at 28-Oct-2025 10:25:00 CDT
Begin processing the 49th record. run: 1427 subRun: 1 event: 30849 at 28-Oct-2025 10:25:05 CDT
Begin processing the 50th record. run: 1427 subRun: 1 event: 30850 at 28-Oct-2025 10:25:11 CDT
Begin processing the 51st record. run: 1427 subRun: 1 event: 30851 at 28-Oct-2025 10:25:15 CDT
Begin processing the 52nd record. run: 1427 subRun: 1 event: 30852 at 28-Oct-2025 10:25:19 CDT
Begin processing the 53rd record. run: 1427 subRun: 1 event: 30853 at 28-Oct-2025 10:25:28 CDT
Begin processing the 54th record. run: 1427 subRun: 1 event: 30854 at 28-Oct-2025 10:25:32 CDT
Begin processing the 55th record. run: 1427 subRun: 1 event: 30855 at 28-Oct-2025 10:25:40 CDT
Begin processing the 56th record. run: 1427 subRun: 1 event: 30856 at 28-Oct-2025 10:25:45 CDT
Begin processing the 57th record. run: 1427 subRun: 1 event: 30857 at 28-Oct-2025 10:25:51 CDT
Begin processing the 58th record. run: 1427 subRun: 1 event: 30858 at 28-Oct-2025 10:26:05 CDT
Begin processing the 59th record. run: 1427 subRun: 1 event: 30859 at 28-Oct-2025 10:26:14 CDT
Begin processing the 60th record. run: 1427 subRun: 1 event: 30860 at 28-Oct-2025 10:26:18 CDT
Begin processing the 61st record. run: 1427 subRun: 1 event: 30861 at 28-Oct-2025 10:26:24 CDT
Begin processing the 62nd record. run: 1427 subRun: 1 event: 30862 at 28-Oct-2025 10:26:58 CDT
Begin processing the 63rd record. run: 1427 subRun: 1 event: 30863 at 28-Oct-2025 10:27:04 CDT
Begin processing the 64th record. run: 1427 subRun: 1 event: 30864 at 28-Oct-2025 10:27:10 CDT
Begin processing the 65th record. run: 1427 subRun: 1 event: 30865 at 28-Oct-2025 10:27:20 CDT
Begin processing the 66th record. run: 1427 subRun: 1 event: 30866 at 28-Oct-2025 10:27:29 CDT
Begin processing the 67th record. run: 1427 subRun: 1 event: 30867 at 28-Oct-2025 10:27:41 CDT
Begin processing the 68th record. run: 1427 subRun: 1 event: 30868 at 28-Oct-2025 10:27:44 CDT
Begin processing the 69th record. run: 1427 subRun: 1 event: 30869 at 28-Oct-2025 10:27:50 CDT
Begin processing the 70th record. run: 1427 subRun: 1 event: 30870 at 28-Oct-2025 10:28:03 CDT
Begin processing the 71st record. run: 1427 subRun: 1 event: 30871 at 28-Oct-2025 10:28:07 CDT
Begin processing the 72nd record. run: 1427 subRun: 1 event: 30872 at 28-Oct-2025 10:28:11 CDT
Begin processing the 73rd record. run: 1427 subRun: 1 event: 30873 at 28-Oct-2025 10:28:20 CDT
Begin processing the 74th record. run: 1427 subRun: 1 event: 30874 at 28-Oct-2025 10:28:25 CDT
Begin processing the 75th record. run: 1427 subRun: 1 event: 30875 at 28-Oct-2025 10:28:36 CDT
Begin processing the 76th record. run: 1427 subRun: 1 event: 30876 at 28-Oct-2025 10:29:10 CDT
Begin processing the 77th record. run: 1427 subRun: 1 event: 30877 at 28-Oct-2025 10:29:19 CDT
Begin processing the 78th record. run: 1427 subRun: 1 event: 30878 at 28-Oct-2025 10:29:22 CDT
Begin processing the 79th record. run: 1427 subRun: 1 event: 30879 at 28-Oct-2025 10:29:28 CDT
Begin processing the 80th record. run: 1427 subRun: 1 event: 30880 at 28-Oct-2025 10:29:35 CDT
Begin processing the 81st record. run: 1427 subRun: 1 event: 30881 at 28-Oct-2025 10:29:43 CDT
Begin processing the 82nd record. run: 1427 subRun: 1 event: 30882 at 28-Oct-2025 10:29:57 CDT
Begin processing the 83rd record. run: 1427 subRun: 1 event: 30883 at 28-Oct-2025 10:30:07 CDT
Begin processing the 84th record. run: 1427 subRun: 1 event: 30884 at 28-Oct-2025 10:30:13 CDT
Begin processing the 85th record. run: 1427 subRun: 1 event: 30885 at 28-Oct-2025 10:30:19 CDT
Begin processing the 86th record. run: 1427 subRun: 1 event: 30886 at 28-Oct-2025 10:30:25 CDT
Begin processing the 87th record. run: 1427 subRun: 1 event: 30887 at 28-Oct-2025 10:30:35 CDT
Begin processing the 88th record. run: 1427 subRun: 1 event: 30888 at 28-Oct-2025 10:30:43 CDT
Begin processing the 89th record. run: 1427 subRun: 1 event: 30889 at 28-Oct-2025 10:31:20 CDT
Begin processing the 90th record. run: 1427 subRun: 1 event: 30890 at 28-Oct-2025 10:31:27 CDT
Begin processing the 91st record. run: 1427 subRun: 1 event: 30891 at 28-Oct-2025 10:31:30 CDT
Begin processing the 92nd record. run: 1427 subRun: 1 event: 30892 at 28-Oct-2025 10:31:34 CDT
Begin processing the 93rd record. run: 1427 subRun: 1 event: 30893 at 28-Oct-2025 10:31:44 CDT
Begin processing the 94th record. run: 1427 subRun: 1 event: 30894 at 28-Oct-2025 10:31:56 CDT
Begin processing the 95th record. run: 1427 subRun: 1 event: 30895 at 28-Oct-2025 10:32:01 CDT
Begin processing the 96th record. run: 1427 subRun: 1 event: 30896 at 28-Oct-2025 10:32:11 CDT
Begin processing the 97th record. run: 1427 subRun: 1 event: 30897 at 28-Oct-2025 10:32:15 CDT
Begin processing the 98th record. run: 1427 subRun: 1 event: 30898 at 28-Oct-2025 10:32:23 CDT
Begin processing the 99th record. run: 1427 subRun: 1 event: 30899 at 28-Oct-2025 10:32:27 CDT
Begin processing the 100th record. run: 1427 subRun: 1 event: 30900 at 28-Oct-2025 10:32:30 CDT
28-Oct-2025 10:32:34 CDT  Closed output file "nu_dune10kt_1x2x6_1427_308_20230827T213340Z_gen_g4_detsim_hitreco__20240223T003555Z_reco2_reco2.root"
28-Oct-2025 10:32:34 CDT  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/9d/e5/nu_dune10kt_1x2x6_1427_308_20230827T213340Z_gen_g4_detsim_hitreco__20240223T003555Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.208158       6.04813       76.7501       3.98886       8.13263        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0320388     0.0683929     0.121449      0.0652637     0.0135776       100    
reco:pandora:StandardPandora                   0.143495       5.96171       76.6302       3.91078       8.12293        100    
[art]:TriggerResults:TriggerResultInserter    1.1241e-05    5.92374e-05   0.00216512    2.61595e-05   0.000215637      100    
end_path:out1:RootOutput                       3.026e-06    7.27229e-06   2.8984e-05     6.067e-06    4.30575e-06      100    
end_path:out1:RootOutput(write)               0.00111632     0.0176422     0.0655911     0.0118648     0.0149155       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2404.39 MB
  Peak resident set size usage (VmHWM): 1414.33 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2025-11-04 15:52:17 UTC       justIN version: 01.05.01