Jobsub ID 40211.105@dunegpschedd01.fnal.gov

Jobsub ID	40211.105@dunegpschedd01.fnal.gov
Workflow ID	1978
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-04 13:50:17
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2025-09-04 14:02:35
From worker node	Hostname	n158.dur.scotgrid.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-04 13:51:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_70_20231118T084727Z_gen_g4_detsim_hitreco__20240503T052323Z_reco2.root
Jobscript	Exit code	0
	Real time	10m (656s)
	CPU time	1m (75s = 11%)
	Max RSS bytes	1437143040 (1370 MiB)
Outputting started	2025-09-04 14:02:14
Output files
Finished	2025-09-04 14:02:35
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

: 1 event: 14130 at 04-Sep-2025 14:58:44 BST
Begin processing the 131st record. run: 6171714 subRun: 1 event: 14131 at 04-Sep-2025 14:58:45 BST
Begin processing the 132nd record. run: 6171714 subRun: 1 event: 14132 at 04-Sep-2025 14:58:53 BST
Begin processing the 133rd record. run: 6171714 subRun: 1 event: 14133 at 04-Sep-2025 14:58:55 BST
Begin processing the 134th record. run: 6171714 subRun: 1 event: 14134 at 04-Sep-2025 14:58:57 BST
Begin processing the 135th record. run: 6171714 subRun: 1 event: 14135 at 04-Sep-2025 14:59:04 BST
Begin processing the 136th record. run: 6171714 subRun: 1 event: 14136 at 04-Sep-2025 14:59:06 BST
Begin processing the 137th record. run: 6171714 subRun: 1 event: 14137 at 04-Sep-2025 14:59:08 BST
Begin processing the 138th record. run: 6171714 subRun: 1 event: 14138 at 04-Sep-2025 14:59:10 BST
Begin processing the 139th record. run: 6171714 subRun: 1 event: 14139 at 04-Sep-2025 14:59:14 BST
Begin processing the 140th record. run: 6171714 subRun: 1 event: 14140 at 04-Sep-2025 14:59:16 BST
Begin processing the 141st record. run: 6171714 subRun: 1 event: 14141 at 04-Sep-2025 14:59:18 BST
Begin processing the 142nd record. run: 6171714 subRun: 1 event: 14142 at 04-Sep-2025 14:59:21 BST
Begin processing the 143rd record. run: 6171714 subRun: 1 event: 14143 at 04-Sep-2025 14:59:25 BST
Begin processing the 144th record. run: 6171714 subRun: 1 event: 14144 at 04-Sep-2025 14:59:26 BST
Begin processing the 145th record. run: 6171714 subRun: 1 event: 14145 at 04-Sep-2025 14:59:30 BST
Begin processing the 146th record. run: 6171714 subRun: 1 event: 14146 at 04-Sep-2025 14:59:32 BST
Begin processing the 147th record. run: 6171714 subRun: 1 event: 14147 at 04-Sep-2025 14:59:33 BST
Begin processing the 148th record. run: 6171714 subRun: 1 event: 14148 at 04-Sep-2025 14:59:35 BST
Begin processing the 149th record. run: 6171714 subRun: 1 event: 14149 at 04-Sep-2025 15:00:01 BST
Begin processing the 150th record. run: 6171714 subRun: 1 event: 14150 at 04-Sep-2025 15:00:03 BST
Begin processing the 151st record. run: 6171714 subRun: 1 event: 14151 at 04-Sep-2025 15:00:05 BST
Begin processing the 152nd record. run: 6171714 subRun: 1 event: 14152 at 04-Sep-2025 15:00:18 BST
Begin processing the 153rd record. run: 6171714 subRun: 1 event: 14153 at 04-Sep-2025 15:00:20 BST
Begin processing the 154th record. run: 6171714 subRun: 1 event: 14154 at 04-Sep-2025 15:00:22 BST
Begin processing the 155th record. run: 6171714 subRun: 1 event: 14155 at 04-Sep-2025 15:00:24 BST
Begin processing the 156th record. run: 6171714 subRun: 1 event: 14156 at 04-Sep-2025 15:00:26 BST
Begin processing the 157th record. run: 6171714 subRun: 1 event: 14157 at 04-Sep-2025 15:00:28 BST
Begin processing the 158th record. run: 6171714 subRun: 1 event: 14158 at 04-Sep-2025 15:00:30 BST
Begin processing the 159th record. run: 6171714 subRun: 1 event: 14159 at 04-Sep-2025 15:00:37 BST
Begin processing the 160th record. run: 6171714 subRun: 1 event: 14160 at 04-Sep-2025 15:00:39 BST
Begin processing the 161st record. run: 6171714 subRun: 1 event: 14161 at 04-Sep-2025 15:00:41 BST
Begin processing the 162nd record. run: 6171714 subRun: 1 event: 14162 at 04-Sep-2025 15:00:43 BST
Begin processing the 163rd record. run: 6171714 subRun: 1 event: 14163 at 04-Sep-2025 15:00:44 BST
Begin processing the 164th record. run: 6171714 subRun: 1 event: 14164 at 04-Sep-2025 15:00:52 BST
Begin processing the 165th record. run: 6171714 subRun: 1 event: 14165 at 04-Sep-2025 15:00:53 BST
Begin processing the 166th record. run: 6171714 subRun: 1 event: 14166 at 04-Sep-2025 15:00:57 BST
Begin processing the 167th record. run: 6171714 subRun: 1 event: 14167 at 04-Sep-2025 15:00:59 BST
Begin processing the 168th record. run: 6171714 subRun: 1 event: 14168 at 04-Sep-2025 15:01:00 BST
Begin processing the 169th record. run: 6171714 subRun: 1 event: 14169 at 04-Sep-2025 15:01:02 BST
Begin processing the 170th record. run: 6171714 subRun: 1 event: 14170 at 04-Sep-2025 15:01:04 BST
Begin processing the 171st record. run: 6171714 subRun: 1 event: 14171 at 04-Sep-2025 15:01:06 BST
Begin processing the 172nd record. run: 6171714 subRun: 1 event: 14172 at 04-Sep-2025 15:01:07 BST
Begin processing the 173rd record. run: 6171714 subRun: 1 event: 14173 at 04-Sep-2025 15:01:11 BST
Begin processing the 174th record. run: 6171714 subRun: 1 event: 14174 at 04-Sep-2025 15:01:12 BST
Begin processing the 175th record. run: 6171714 subRun: 1 event: 14175 at 04-Sep-2025 15:01:14 BST
Begin processing the 176th record. run: 6171714 subRun: 1 event: 14176 at 04-Sep-2025 15:01:17 BST
Begin processing the 177th record. run: 6171714 subRun: 1 event: 14177 at 04-Sep-2025 15:01:19 BST
Begin processing the 178th record. run: 6171714 subRun: 1 event: 14178 at 04-Sep-2025 15:01:22 BST
Begin processing the 179th record. run: 6171714 subRun: 1 event: 14179 at 04-Sep-2025 15:01:24 BST
Begin processing the 180th record. run: 6171714 subRun: 1 event: 14180 at 04-Sep-2025 15:01:26 BST
Begin processing the 181st record. run: 6171714 subRun: 1 event: 14181 at 04-Sep-2025 15:01:30 BST
Begin processing the 182nd record. run: 6171714 subRun: 1 event: 14182 at 04-Sep-2025 15:01:32 BST
Begin processing the 183rd record. run: 6171714 subRun: 1 event: 14183 at 04-Sep-2025 15:01:34 BST
Begin processing the 184th record. run: 6171714 subRun: 1 event: 14184 at 04-Sep-2025 15:01:36 BST
Begin processing the 185th record. run: 6171714 subRun: 1 event: 14185 at 04-Sep-2025 15:01:38 BST
Begin processing the 186th record. run: 6171714 subRun: 1 event: 14186 at 04-Sep-2025 15:01:39 BST
Begin processing the 187th record. run: 6171714 subRun: 1 event: 14187 at 04-Sep-2025 15:01:39 BST
Begin processing the 188th record. run: 6171714 subRun: 1 event: 14188 at 04-Sep-2025 15:01:42 BST
Begin processing the 189th record. run: 6171714 subRun: 1 event: 14189 at 04-Sep-2025 15:01:44 BST
Begin processing the 190th record. run: 6171714 subRun: 1 event: 14190 at 04-Sep-2025 15:01:46 BST
Begin processing the 191st record. run: 6171714 subRun: 1 event: 14191 at 04-Sep-2025 15:01:48 BST
Begin processing the 192nd record. run: 6171714 subRun: 1 event: 14192 at 04-Sep-2025 15:01:52 BST
Begin processing the 193rd record. run: 6171714 subRun: 1 event: 14193 at 04-Sep-2025 15:01:54 BST
Begin processing the 194th record. run: 6171714 subRun: 1 event: 14194 at 04-Sep-2025 15:01:55 BST
Begin processing the 195th record. run: 6171714 subRun: 1 event: 14195 at 04-Sep-2025 15:01:58 BST
Begin processing the 196th record. run: 6171714 subRun: 1 event: 14196 at 04-Sep-2025 15:02:02 BST
Begin processing the 197th record. run: 6171714 subRun: 1 event: 14197 at 04-Sep-2025 15:02:05 BST
Begin processing the 198th record. run: 6171714 subRun: 1 event: 14198 at 04-Sep-2025 15:02:06 BST
Begin processing the 199th record. run: 6171714 subRun: 1 event: 14199 at 04-Sep-2025 15:02:09 BST
Begin processing the 200th record. run: 6171714 subRun: 1 event: 14200 at 04-Sep-2025 15:02:11 BST
04-Sep-2025 15:02:13 BST  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/90/67/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_70_20231118T084727Z_gen_g4_detsim_hitreco__20240503T052323Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.336053       1.94619       12.015        1.42855       1.76167        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0212577     0.0320685     0.0437556     0.0225556     0.0104845       200    
prod:pandorapid:Chi2ParticleID                 0.125241      0.196581      0.495447      0.189078      0.0555748       200    
prod:energyreconue:EnergyReco                  0.0940602     0.391108       1.10863      0.380099      0.108928        200    
prod:energyreconumu:EnergyReco                0.00323812     0.228426       2.93773      0.193254      0.322941        200    
prod:energyreconc:EnergyReco                   0.003136      0.0907914     0.949098      0.072615      0.131868        200    
[art]:TriggerResults:TriggerResultInserter    1.1132e-05    1.4816e-05    9.1571e-05    1.3843e-05    6.08581e-06      200    
end_path:ana:PIDASelectionAna                  0.065994       1.00677       9.64127       0.62241       1.3604         200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12288.1 MB
  Peak resident set size usage (VmHWM): 1437.14 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 80
-rw-r--r--. 1 dune001 dune 28733 Sep  4 15:02 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_70_20231118T084727Z_gen_g4_detsim_hitreco__20240503T052323Z_reco2_pida_2025-09-04T_135130Z.root
-rw-r--r--. 1 dune001 dune 24102 Sep  4 15:02 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_70_20231118T084727Z_gen_g4_detsim_hitreco__20240503T052323Z_reco2_reco_2025-09-04T_135130Z.log
-rw-r--r--. 1 dune001 dune 12028 Sep  4 15:02 jobscript.log
-rw-r--r--. 1 dune001 dune   215 Sep  4 15:02 justin-processed-pfns.txt
-rw-r--r--. 1 dune001 dune   136 Sep  4 14:51 all-input-dids.txt
-rw-r--r--. 1 dune001 dune     0 Sep  4 14:51 debugprod.log