Jobsub ID 256258.143@dunegpschedd02.fnal.gov

Jobsub ID	256258.143@dunegpschedd02.fnal.gov
Workflow ID	10929
Stage ID	1
User name	imawby@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2025-12-02 09:52:31
Site	US_Michigan
Entry	HCC_US_Michigan_gate02
Last heartbeat	2025-12-02 10:13:37
From worker node	Hostname	c7-16-8.aglt2.org
	cpuinfo	AMD EPYC 7413 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	36000 (10 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-12-02 09:53:36
Input files	fardet-hd:nu_dune10kt_1x2x6_1100_880_20230826T040108Z_gen_g4_detsim_hitreco__20240223T002104Z_reco2.root
Outputting started	2025-12-02 10:08:15
Output files
Finished	2025-12-02 11:08:53
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/84/df/nu_dune10kt_1x2x6_1100_880_20230826T040108Z_gen_g4_detsim_hitreco__20240223T002104Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 23rd record. run: 1100 subRun: 1 event: 88023 at 02-Dec-2025 05:00:51 EST
Begin processing the 24th record. run: 1100 subRun: 1 event: 88024 at 02-Dec-2025 05:00:56 EST
Begin processing the 25th record. run: 1100 subRun: 1 event: 88025 at 02-Dec-2025 05:01:01 EST
Begin processing the 26th record. run: 1100 subRun: 1 event: 88026 at 02-Dec-2025 05:01:14 EST
Begin processing the 27th record. run: 1100 subRun: 1 event: 88027 at 02-Dec-2025 05:01:18 EST
Begin processing the 28th record. run: 1100 subRun: 1 event: 88028 at 02-Dec-2025 05:01:22 EST
Begin processing the 29th record. run: 1100 subRun: 1 event: 88029 at 02-Dec-2025 05:01:27 EST
Begin processing the 30th record. run: 1100 subRun: 1 event: 88030 at 02-Dec-2025 05:01:31 EST
Begin processing the 31st record. run: 1100 subRun: 1 event: 88031 at 02-Dec-2025 05:01:35 EST
Begin processing the 32nd record. run: 1100 subRun: 1 event: 88032 at 02-Dec-2025 05:01:39 EST
Begin processing the 33rd record. run: 1100 subRun: 1 event: 88033 at 02-Dec-2025 05:01:44 EST
Begin processing the 34th record. run: 1100 subRun: 1 event: 88034 at 02-Dec-2025 05:01:48 EST
Begin processing the 35th record. run: 1100 subRun: 1 event: 88035 at 02-Dec-2025 05:01:53 EST
Begin processing the 36th record. run: 1100 subRun: 1 event: 88036 at 02-Dec-2025 05:01:56 EST
Begin processing the 37th record. run: 1100 subRun: 1 event: 88037 at 02-Dec-2025 05:02:00 EST
Begin processing the 38th record. run: 1100 subRun: 1 event: 88038 at 02-Dec-2025 05:02:06 EST
Begin processing the 39th record. run: 1100 subRun: 1 event: 88039 at 02-Dec-2025 05:02:10 EST
Begin processing the 40th record. run: 1100 subRun: 1 event: 88040 at 02-Dec-2025 05:02:14 EST
Begin processing the 41st record. run: 1100 subRun: 1 event: 88041 at 02-Dec-2025 05:02:18 EST
Begin processing the 42nd record. run: 1100 subRun: 1 event: 88042 at 02-Dec-2025 05:02:26 EST
Begin processing the 43rd record. run: 1100 subRun: 1 event: 88043 at 02-Dec-2025 05:02:31 EST
Begin processing the 44th record. run: 1100 subRun: 1 event: 88044 at 02-Dec-2025 05:02:37 EST
Begin processing the 45th record. run: 1100 subRun: 1 event: 88045 at 02-Dec-2025 05:02:41 EST
Begin processing the 46th record. run: 1100 subRun: 1 event: 88046 at 02-Dec-2025 05:02:47 EST
Begin processing the 47th record. run: 1100 subRun: 1 event: 88047 at 02-Dec-2025 05:02:53 EST
Begin processing the 48th record. run: 1100 subRun: 1 event: 88048 at 02-Dec-2025 05:02:58 EST
Begin processing the 49th record. run: 1100 subRun: 1 event: 88049 at 02-Dec-2025 05:03:05 EST
Begin processing the 50th record. run: 1100 subRun: 1 event: 88050 at 02-Dec-2025 05:03:10 EST
Begin processing the 51st record. run: 1100 subRun: 1 event: 88051 at 02-Dec-2025 05:03:14 EST
Begin processing the 52nd record. run: 1100 subRun: 1 event: 88052 at 02-Dec-2025 05:03:23 EST
Begin processing the 53rd record. run: 1100 subRun: 1 event: 88053 at 02-Dec-2025 05:03:30 EST
Begin processing the 54th record. run: 1100 subRun: 1 event: 88054 at 02-Dec-2025 05:03:35 EST
Begin processing the 55th record. run: 1100 subRun: 1 event: 88055 at 02-Dec-2025 05:03:40 EST
Begin processing the 56th record. run: 1100 subRun: 1 event: 88056 at 02-Dec-2025 05:03:44 EST
Begin processing the 57th record. run: 1100 subRun: 1 event: 88057 at 02-Dec-2025 05:03:47 EST
Begin processing the 58th record. run: 1100 subRun: 1 event: 88058 at 02-Dec-2025 05:03:50 EST
Begin processing the 59th record. run: 1100 subRun: 1 event: 88059 at 02-Dec-2025 05:03:58 EST
Begin processing the 60th record. run: 1100 subRun: 1 event: 88060 at 02-Dec-2025 05:04:05 EST
Begin processing the 61st record. run: 1100 subRun: 1 event: 88061 at 02-Dec-2025 05:04:13 EST
Begin processing the 62nd record. run: 1100 subRun: 1 event: 88062 at 02-Dec-2025 05:04:19 EST
Begin processing the 63rd record. run: 1100 subRun: 1 event: 88063 at 02-Dec-2025 05:04:23 EST
Begin processing the 64th record. run: 1100 subRun: 1 event: 88064 at 02-Dec-2025 05:04:27 EST
Begin processing the 65th record. run: 1100 subRun: 1 event: 88065 at 02-Dec-2025 05:04:31 EST
Begin processing the 66th record. run: 1100 subRun: 1 event: 88066 at 02-Dec-2025 05:04:35 EST
Begin processing the 67th record. run: 1100 subRun: 1 event: 88067 at 02-Dec-2025 05:04:40 EST
Begin processing the 68th record. run: 1100 subRun: 1 event: 88068 at 02-Dec-2025 05:04:45 EST
Begin processing the 69th record. run: 1100 subRun: 1 event: 88069 at 02-Dec-2025 05:04:50 EST
Begin processing the 70th record. run: 1100 subRun: 1 event: 88070 at 02-Dec-2025 05:04:55 EST
Begin processing the 71st record. run: 1100 subRun: 1 event: 88071 at 02-Dec-2025 05:05:00 EST
Begin processing the 72nd record. run: 1100 subRun: 1 event: 88072 at 02-Dec-2025 05:05:05 EST
Begin processing the 73rd record. run: 1100 subRun: 1 event: 88073 at 02-Dec-2025 05:05:13 EST
Begin processing the 74th record. run: 1100 subRun: 1 event: 88074 at 02-Dec-2025 05:05:17 EST
Begin processing the 75th record. run: 1100 subRun: 1 event: 88075 at 02-Dec-2025 05:05:49 EST
Begin processing the 76th record. run: 1100 subRun: 1 event: 88076 at 02-Dec-2025 05:05:53 EST
Begin processing the 77th record. run: 1100 subRun: 1 event: 88077 at 02-Dec-2025 05:05:59 EST
Begin processing the 78th record. run: 1100 subRun: 1 event: 88078 at 02-Dec-2025 05:06:02 EST
Begin processing the 79th record. run: 1100 subRun: 1 event: 88079 at 02-Dec-2025 05:06:06 EST
Begin processing the 80th record. run: 1100 subRun: 1 event: 88080 at 02-Dec-2025 05:06:11 EST
Begin processing the 81st record. run: 1100 subRun: 1 event: 88081 at 02-Dec-2025 05:06:15 EST
Begin processing the 82nd record. run: 1100 subRun: 1 event: 88082 at 02-Dec-2025 05:06:20 EST
Begin processing the 83rd record. run: 1100 subRun: 1 event: 88083 at 02-Dec-2025 05:06:26 EST
Begin processing the 84th record. run: 1100 subRun: 1 event: 88084 at 02-Dec-2025 05:06:31 EST
Begin processing the 85th record. run: 1100 subRun: 1 event: 88085 at 02-Dec-2025 05:06:35 EST
Begin processing the 86th record. run: 1100 subRun: 1 event: 88086 at 02-Dec-2025 05:06:38 EST
Begin processing the 87th record. run: 1100 subRun: 1 event: 88087 at 02-Dec-2025 05:06:43 EST
Begin processing the 88th record. run: 1100 subRun: 1 event: 88088 at 02-Dec-2025 05:06:47 EST
Begin processing the 89th record. run: 1100 subRun: 1 event: 88089 at 02-Dec-2025 05:06:50 EST
Begin processing the 90th record. run: 1100 subRun: 1 event: 88090 at 02-Dec-2025 05:06:55 EST
Begin processing the 91st record. run: 1100 subRun: 1 event: 88091 at 02-Dec-2025 05:07:01 EST
Begin processing the 92nd record. run: 1100 subRun: 1 event: 88092 at 02-Dec-2025 05:07:24 EST
Begin processing the 93rd record. run: 1100 subRun: 1 event: 88093 at 02-Dec-2025 05:07:29 EST
Begin processing the 94th record. run: 1100 subRun: 1 event: 88094 at 02-Dec-2025 05:07:33 EST
Begin processing the 95th record. run: 1100 subRun: 1 event: 88095 at 02-Dec-2025 05:07:44 EST
Begin processing the 96th record. run: 1100 subRun: 1 event: 88096 at 02-Dec-2025 05:07:49 EST
Begin processing the 97th record. run: 1100 subRun: 1 event: 88097 at 02-Dec-2025 05:07:52 EST
Begin processing the 98th record. run: 1100 subRun: 1 event: 88098 at 02-Dec-2025 05:07:55 EST
Begin processing the 99th record. run: 1100 subRun: 1 event: 88099 at 02-Dec-2025 05:07:59 EST
Begin processing the 100th record. run: 1100 subRun: 1 event: 88100 at 02-Dec-2025 05:08:04 EST
02-Dec-2025 05:08:12 EST  Closed output file "nu_dune10kt_1x2x6_1100_880_20230826T040108Z_gen_g4_detsim_hitreco__20240223T002104Z_reco2_reco2.root"
02-Dec-2025 05:08:12 EST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/84/df/nu_dune10kt_1x2x6_1100_880_20230826T040108Z_gen_g4_detsim_hitreco__20240223T002104Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      2.50804       6.59342       49.569        4.5673        7.31618        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0159564     0.0391096      0.24026      0.035274      0.0227789       100    
reco:pandora:StandardPandora                    2.47257       6.53404       49.5393       4.51142       7.3089         100    
[art]:TriggerResults:TriggerResultInserter    1.4558e-05    2.88718e-05   0.000115096   2.4847e-05    1.63757e-05      100    
end_path:out1:RootOutput                       2.645e-06    6.19038e-06   2.0218e-05     6.172e-06    2.1197e-06       100    
end_path:out1:RootOutput(write)                0.0013576     0.0199547     0.0962559     0.0151665     0.0187532       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2290.18 MB
  Peak resident set size usage (VmHWM): 1293.97 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===