Jobsub ID 40211.100@dunegpschedd01.fnal.gov

Jobsub ID	40211.100@dunegpschedd01.fnal.gov
Workflow ID	1978
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-04 13:50:17
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2025-09-04 13:57:02
From worker node	Hostname	n161.dur.scotgrid.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-04 13:51:06
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_801_20231204T100512Z_gen_g4_detsim_hitreco__20240509T201343Z_reco2.root
Jobscript	Exit code	0
	Real time	5m (332s)
	CPU time	1m (62s = 18%)
	Max RSS bytes	1440473088 (1373 MiB)
Outputting started	2025-09-04 13:56:40
Output files
Finished	2025-09-04 13:57:02
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

nt: 80130 at 04-Sep-2025 14:53:20 BST
Begin processing the 31st record. run: 74517726 subRun: 1 event: 80131 at 04-Sep-2025 14:53:21 BST
Begin processing the 32nd record. run: 74517726 subRun: 1 event: 80132 at 04-Sep-2025 14:53:23 BST
Begin processing the 33rd record. run: 74517726 subRun: 1 event: 80133 at 04-Sep-2025 14:53:29 BST
Begin processing the 34th record. run: 74517726 subRun: 1 event: 80134 at 04-Sep-2025 14:53:31 BST
Begin processing the 35th record. run: 74517726 subRun: 1 event: 80135 at 04-Sep-2025 14:53:38 BST
Begin processing the 36th record. run: 74517726 subRun: 1 event: 80136 at 04-Sep-2025 14:53:40 BST
Begin processing the 37th record. run: 74517726 subRun: 1 event: 80137 at 04-Sep-2025 14:53:42 BST
Begin processing the 38th record. run: 74517726 subRun: 1 event: 80138 at 04-Sep-2025 14:53:43 BST
Begin processing the 39th record. run: 74517726 subRun: 1 event: 80139 at 04-Sep-2025 14:53:45 BST
Begin processing the 40th record. run: 74517726 subRun: 1 event: 80140 at 04-Sep-2025 14:53:47 BST
Begin processing the 41st record. run: 74517726 subRun: 1 event: 80141 at 04-Sep-2025 14:53:49 BST
Begin processing the 42nd record. run: 74517726 subRun: 1 event: 80142 at 04-Sep-2025 14:53:51 BST
Begin processing the 43rd record. run: 74517726 subRun: 1 event: 80143 at 04-Sep-2025 14:53:53 BST
Begin processing the 44th record. run: 74517726 subRun: 1 event: 80144 at 04-Sep-2025 14:53:57 BST
Begin processing the 45th record. run: 74517726 subRun: 1 event: 80145 at 04-Sep-2025 14:53:58 BST
Begin processing the 46th record. run: 74517726 subRun: 1 event: 80146 at 04-Sep-2025 14:54:00 BST
Begin processing the 47th record. run: 74517726 subRun: 1 event: 80147 at 04-Sep-2025 14:54:01 BST
Begin processing the 48th record. run: 74517726 subRun: 1 event: 80148 at 04-Sep-2025 14:54:04 BST
Begin processing the 49th record. run: 74517726 subRun: 1 event: 80149 at 04-Sep-2025 14:54:05 BST
Begin processing the 50th record. run: 74517726 subRun: 1 event: 80150 at 04-Sep-2025 14:54:07 BST
Begin processing the 51st record. run: 74517726 subRun: 1 event: 80151 at 04-Sep-2025 14:54:09 BST
Begin processing the 52nd record. run: 74517726 subRun: 1 event: 80152 at 04-Sep-2025 14:54:11 BST
Begin processing the 53rd record. run: 74517726 subRun: 1 event: 80153 at 04-Sep-2025 14:54:12 BST
Begin processing the 54th record. run: 74517726 subRun: 1 event: 80154 at 04-Sep-2025 14:54:14 BST
Begin processing the 55th record. run: 74517726 subRun: 1 event: 80155 at 04-Sep-2025 14:54:29 BST
Begin processing the 56th record. run: 74517726 subRun: 1 event: 80156 at 04-Sep-2025 14:54:31 BST
Begin processing the 57th record. run: 74517726 subRun: 1 event: 80157 at 04-Sep-2025 14:54:32 BST
Begin processing the 58th record. run: 74517726 subRun: 1 event: 80158 at 04-Sep-2025 14:54:34 BST
Begin processing the 59th record. run: 74517726 subRun: 1 event: 80159 at 04-Sep-2025 14:54:39 BST
Begin processing the 60th record. run: 74517726 subRun: 1 event: 80160 at 04-Sep-2025 14:54:41 BST
Begin processing the 61st record. run: 74517726 subRun: 1 event: 80161 at 04-Sep-2025 14:54:43 BST
Begin processing the 62nd record. run: 74517726 subRun: 1 event: 80162 at 04-Sep-2025 14:54:44 BST
Begin processing the 63rd record. run: 74517726 subRun: 1 event: 80163 at 04-Sep-2025 14:54:45 BST
Begin processing the 64th record. run: 74517726 subRun: 1 event: 80164 at 04-Sep-2025 14:54:46 BST
Begin processing the 65th record. run: 74517726 subRun: 1 event: 80165 at 04-Sep-2025 14:54:48 BST
Begin processing the 66th record. run: 74517726 subRun: 1 event: 80166 at 04-Sep-2025 14:54:50 BST
Begin processing the 67th record. run: 74517726 subRun: 1 event: 80167 at 04-Sep-2025 14:54:52 BST
Begin processing the 68th record. run: 74517726 subRun: 1 event: 80168 at 04-Sep-2025 14:54:54 BST
Begin processing the 69th record. run: 74517726 subRun: 1 event: 80169 at 04-Sep-2025 14:54:55 BST
Begin processing the 70th record. run: 74517726 subRun: 1 event: 80170 at 04-Sep-2025 14:54:57 BST
Begin processing the 71st record. run: 74517726 subRun: 1 event: 80171 at 04-Sep-2025 14:54:59 BST
Begin processing the 72nd record. run: 74517726 subRun: 1 event: 80172 at 04-Sep-2025 14:55:01 BST
Begin processing the 73rd record. run: 74517726 subRun: 1 event: 80173 at 04-Sep-2025 14:55:03 BST
Begin processing the 74th record. run: 74517726 subRun: 1 event: 80174 at 04-Sep-2025 14:55:04 BST
Begin processing the 75th record. run: 74517726 subRun: 1 event: 80175 at 04-Sep-2025 14:55:05 BST
Begin processing the 76th record. run: 74517726 subRun: 1 event: 80176 at 04-Sep-2025 14:55:07 BST
Begin processing the 77th record. run: 74517726 subRun: 1 event: 80177 at 04-Sep-2025 14:55:09 BST
Begin processing the 78th record. run: 74517726 subRun: 1 event: 80178 at 04-Sep-2025 14:55:11 BST
Begin processing the 79th record. run: 74517726 subRun: 1 event: 80179 at 04-Sep-2025 14:55:13 BST
Begin processing the 80th record. run: 74517726 subRun: 1 event: 80180 at 04-Sep-2025 14:55:15 BST
Begin processing the 81st record. run: 74517726 subRun: 1 event: 80181 at 04-Sep-2025 14:55:16 BST
Begin processing the 82nd record. run: 74517726 subRun: 1 event: 80182 at 04-Sep-2025 14:55:28 BST
Begin processing the 83rd record. run: 74517726 subRun: 1 event: 80183 at 04-Sep-2025 14:55:28 BST
Begin processing the 84th record. run: 74517726 subRun: 1 event: 80184 at 04-Sep-2025 14:55:30 BST
Begin processing the 85th record. run: 74517726 subRun: 1 event: 80185 at 04-Sep-2025 14:55:32 BST
Begin processing the 86th record. run: 74517726 subRun: 1 event: 80186 at 04-Sep-2025 14:55:34 BST
Begin processing the 87th record. run: 74517726 subRun: 1 event: 80187 at 04-Sep-2025 14:55:40 BST
Begin processing the 88th record. run: 74517726 subRun: 1 event: 80188 at 04-Sep-2025 14:55:42 BST
Begin processing the 89th record. run: 74517726 subRun: 1 event: 80189 at 04-Sep-2025 14:56:05 BST
Begin processing the 90th record. run: 74517726 subRun: 1 event: 80190 at 04-Sep-2025 14:56:06 BST
Begin processing the 91st record. run: 74517726 subRun: 1 event: 80191 at 04-Sep-2025 14:56:07 BST
Begin processing the 92nd record. run: 74517726 subRun: 1 event: 80192 at 04-Sep-2025 14:56:09 BST
Begin processing the 93rd record. run: 74517726 subRun: 1 event: 80193 at 04-Sep-2025 14:56:28 BST
Begin processing the 94th record. run: 74517726 subRun: 1 event: 80194 at 04-Sep-2025 14:56:30 BST
Begin processing the 95th record. run: 74517726 subRun: 1 event: 80195 at 04-Sep-2025 14:56:32 BST
Begin processing the 96th record. run: 74517726 subRun: 1 event: 80196 at 04-Sep-2025 14:56:34 BST
Begin processing the 97th record. run: 74517726 subRun: 1 event: 80197 at 04-Sep-2025 14:56:34 BST
Begin processing the 98th record. run: 74517726 subRun: 1 event: 80198 at 04-Sep-2025 14:56:35 BST
Begin processing the 99th record. run: 74517726 subRun: 1 event: 80199 at 04-Sep-2025 14:56:36 BST
Begin processing the 100th record. run: 74517726 subRun: 1 event: 80200 at 04-Sep-2025 14:56:38 BST
04-Sep-2025 14:56:38 BST  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/47/61/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_801_20231204T100512Z_gen_g4_detsim_hitreco__20240509T201343Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.371787       1.9338        11.0377       1.41307       1.89087        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0219895     0.0327255     0.0441004     0.0228427     0.0106109       100    
prod:pandorapid:Chi2ParticleID                 0.126637      0.198328      0.488944      0.191802      0.0585322       100    
prod:energyreconue:EnergyReco                   0.10414      0.391095      0.946011      0.388228      0.117937        100    
prod:energyreconumu:EnergyReco                0.00334571     0.223149       1.55993       0.19812       0.25828        100    
prod:energyreconc:EnergyReco                  0.00327161     0.0911163      1.0222       0.0631903     0.139809        100    
[art]:TriggerResults:TriggerResultInserter    1.2392e-05    1.52609e-05   6.8899e-05    1.4341e-05    5.68167e-06      100    
end_path:ana:PIDASelectionAna                  0.0882438     0.996771       8.55938      0.614442       1.46121        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12290.7 MB
  Peak resident set size usage (VmHWM): 1440.47 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 dune001 dune 20010 Sep  4 14:56 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_801_20231204T100512Z_gen_g4_detsim_hitreco__20240509T201343Z_reco2_pida_2025-09-04T_135114Z.root
-rw-r--r--. 1 dune001 dune 14308 Sep  4 14:56 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_801_20231204T100512Z_gen_g4_detsim_hitreco__20240509T201343Z_reco2_reco_2025-09-04T_135114Z.log
-rw-r--r--. 1 dune001 dune 12033 Sep  4 14:56 jobscript.log
-rw-r--r--. 1 dune001 dune   217 Sep  4 14:56 justin-processed-pfns.txt
-rw-r--r--. 1 dune001 dune   138 Sep  4 14:51 all-input-dids.txt
-rw-r--r--. 1 dune001 dune     0 Sep  4 14:51 debugprod.log