Jobsub ID 22761.72@dunegpschedd02.fnal.gov

Jobsub ID	22761.72@dunegpschedd02.fnal.gov
Workflow ID	1657
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-20 00:04:32
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2025-08-20 00:10:33
From worker node	Hostname	wh09.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-20 00:05:58
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_72_20231207T151536Z_gen_g4_detsim_hitreco__20240510T044926Z_reco2.root
Jobscript	Exit code	0
	Real time	4m (253s)
	CPU time	0m (48s = 18%)
	Max RSS bytes	1331703808 (1270 MiB)
Outputting started	2025-08-20 00:10:12
Output files
Finished	2025-08-20 00:10:33
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

20-Aug-2025 01:07:26 BST
Begin processing the 30th record. run: 66049673 subRun: 1 event: 7230 at 20-Aug-2025 01:07:28 BST
Begin processing the 31st record. run: 66049673 subRun: 1 event: 7231 at 20-Aug-2025 01:07:30 BST
Begin processing the 32nd record. run: 66049673 subRun: 1 event: 7232 at 20-Aug-2025 01:07:31 BST
Begin processing the 33rd record. run: 66049673 subRun: 1 event: 7233 at 20-Aug-2025 01:07:32 BST
Begin processing the 34th record. run: 66049673 subRun: 1 event: 7234 at 20-Aug-2025 01:07:38 BST
Begin processing the 35th record. run: 66049673 subRun: 1 event: 7235 at 20-Aug-2025 01:07:55 BST
Begin processing the 36th record. run: 66049673 subRun: 1 event: 7236 at 20-Aug-2025 01:07:56 BST
Begin processing the 37th record. run: 66049673 subRun: 1 event: 7237 at 20-Aug-2025 01:07:58 BST
Begin processing the 38th record. run: 66049673 subRun: 1 event: 7238 at 20-Aug-2025 01:07:59 BST
Begin processing the 39th record. run: 66049673 subRun: 1 event: 7239 at 20-Aug-2025 01:08:01 BST
Begin processing the 40th record. run: 66049673 subRun: 1 event: 7240 at 20-Aug-2025 01:08:03 BST
Begin processing the 41st record. run: 66049673 subRun: 1 event: 7241 at 20-Aug-2025 01:08:06 BST
Begin processing the 42nd record. run: 66049673 subRun: 1 event: 7242 at 20-Aug-2025 01:08:08 BST
Begin processing the 43rd record. run: 66049673 subRun: 1 event: 7243 at 20-Aug-2025 01:08:09 BST
Begin processing the 44th record. run: 66049673 subRun: 1 event: 7244 at 20-Aug-2025 01:08:10 BST
Begin processing the 45th record. run: 66049673 subRun: 1 event: 7245 at 20-Aug-2025 01:08:12 BST
Begin processing the 46th record. run: 66049673 subRun: 1 event: 7246 at 20-Aug-2025 01:08:14 BST
Begin processing the 47th record. run: 66049673 subRun: 1 event: 7247 at 20-Aug-2025 01:08:15 BST
Begin processing the 48th record. run: 66049673 subRun: 1 event: 7248 at 20-Aug-2025 01:08:17 BST
Begin processing the 49th record. run: 66049673 subRun: 1 event: 7249 at 20-Aug-2025 01:08:18 BST
Begin processing the 50th record. run: 66049673 subRun: 1 event: 7250 at 20-Aug-2025 01:08:28 BST
Begin processing the 51st record. run: 66049673 subRun: 1 event: 7251 at 20-Aug-2025 01:08:30 BST
Begin processing the 52nd record. run: 66049673 subRun: 1 event: 7252 at 20-Aug-2025 01:08:32 BST
Begin processing the 53rd record. run: 66049673 subRun: 1 event: 7253 at 20-Aug-2025 01:08:34 BST
Begin processing the 54th record. run: 66049673 subRun: 1 event: 7254 at 20-Aug-2025 01:08:36 BST
Begin processing the 55th record. run: 66049673 subRun: 1 event: 7255 at 20-Aug-2025 01:08:38 BST
Begin processing the 56th record. run: 66049673 subRun: 1 event: 7256 at 20-Aug-2025 01:08:40 BST
Begin processing the 57th record. run: 66049673 subRun: 1 event: 7257 at 20-Aug-2025 01:08:41 BST
Begin processing the 58th record. run: 66049673 subRun: 1 event: 7258 at 20-Aug-2025 01:08:43 BST
Begin processing the 59th record. run: 66049673 subRun: 1 event: 7259 at 20-Aug-2025 01:08:45 BST
Begin processing the 60th record. run: 66049673 subRun: 1 event: 7260 at 20-Aug-2025 01:08:47 BST
Begin processing the 61st record. run: 66049673 subRun: 1 event: 7261 at 20-Aug-2025 01:08:50 BST
Begin processing the 62nd record. run: 66049673 subRun: 1 event: 7262 at 20-Aug-2025 01:08:52 BST
Begin processing the 63rd record. run: 66049673 subRun: 1 event: 7263 at 20-Aug-2025 01:08:57 BST
Begin processing the 64th record. run: 66049673 subRun: 1 event: 7264 at 20-Aug-2025 01:08:58 BST
Begin processing the 65th record. run: 66049673 subRun: 1 event: 7265 at 20-Aug-2025 01:09:00 BST
Begin processing the 66th record. run: 66049673 subRun: 1 event: 7266 at 20-Aug-2025 01:09:01 BST
Begin processing the 67th record. run: 66049673 subRun: 1 event: 7267 at 20-Aug-2025 01:09:03 BST
Begin processing the 68th record. run: 66049673 subRun: 1 event: 7268 at 20-Aug-2025 01:09:03 BST
Begin processing the 69th record. run: 66049673 subRun: 1 event: 7269 at 20-Aug-2025 01:09:05 BST
Begin processing the 70th record. run: 66049673 subRun: 1 event: 7270 at 20-Aug-2025 01:09:07 BST
Begin processing the 71st record. run: 66049673 subRun: 1 event: 7271 at 20-Aug-2025 01:09:08 BST
Begin processing the 72nd record. run: 66049673 subRun: 1 event: 7272 at 20-Aug-2025 01:09:11 BST
Begin processing the 73rd record. run: 66049673 subRun: 1 event: 7273 at 20-Aug-2025 01:09:13 BST
Begin processing the 74th record. run: 66049673 subRun: 1 event: 7274 at 20-Aug-2025 01:09:14 BST
Begin processing the 75th record. run: 66049673 subRun: 1 event: 7275 at 20-Aug-2025 01:09:16 BST
Begin processing the 76th record. run: 66049673 subRun: 1 event: 7276 at 20-Aug-2025 01:09:21 BST
Begin processing the 77th record. run: 66049673 subRun: 1 event: 7277 at 20-Aug-2025 01:09:23 BST
Begin processing the 78th record. run: 66049673 subRun: 1 event: 7278 at 20-Aug-2025 01:09:25 BST
Begin processing the 79th record. run: 66049673 subRun: 1 event: 7279 at 20-Aug-2025 01:09:26 BST
Begin processing the 80th record. run: 66049673 subRun: 1 event: 7280 at 20-Aug-2025 01:09:27 BST
Begin processing the 81st record. run: 66049673 subRun: 1 event: 7281 at 20-Aug-2025 01:09:28 BST
Begin processing the 82nd record. run: 66049673 subRun: 1 event: 7282 at 20-Aug-2025 01:09:29 BST
Begin processing the 83rd record. run: 66049673 subRun: 1 event: 7283 at 20-Aug-2025 01:09:31 BST
Begin processing the 84th record. run: 66049673 subRun: 1 event: 7284 at 20-Aug-2025 01:09:32 BST
Begin processing the 85th record. run: 66049673 subRun: 1 event: 7285 at 20-Aug-2025 01:09:33 BST
Begin processing the 86th record. run: 66049673 subRun: 1 event: 7286 at 20-Aug-2025 01:09:37 BST
Begin processing the 87th record. run: 66049673 subRun: 1 event: 7287 at 20-Aug-2025 01:09:38 BST
Begin processing the 88th record. run: 66049673 subRun: 1 event: 7288 at 20-Aug-2025 01:09:40 BST
Begin processing the 89th record. run: 66049673 subRun: 1 event: 7289 at 20-Aug-2025 01:09:43 BST
Begin processing the 90th record. run: 66049673 subRun: 1 event: 7290 at 20-Aug-2025 01:09:45 BST
Begin processing the 91st record. run: 66049673 subRun: 1 event: 7291 at 20-Aug-2025 01:09:46 BST
Begin processing the 92nd record. run: 66049673 subRun: 1 event: 7292 at 20-Aug-2025 01:09:47 BST
Begin processing the 93rd record. run: 66049673 subRun: 1 event: 7293 at 20-Aug-2025 01:09:49 BST
Begin processing the 94th record. run: 66049673 subRun: 1 event: 7294 at 20-Aug-2025 01:09:50 BST
Begin processing the 95th record. run: 66049673 subRun: 1 event: 7295 at 20-Aug-2025 01:09:51 BST
Begin processing the 96th record. run: 66049673 subRun: 1 event: 7296 at 20-Aug-2025 01:09:53 BST
Begin processing the 97th record. run: 66049673 subRun: 1 event: 7297 at 20-Aug-2025 01:09:55 BST
Begin processing the 98th record. run: 66049673 subRun: 1 event: 7298 at 20-Aug-2025 01:09:57 BST
Begin processing the 99th record. run: 66049673 subRun: 1 event: 7299 at 20-Aug-2025 01:09:58 BST
Begin processing the 100th record. run: 66049673 subRun: 1 event: 7300 at 20-Aug-2025 01:10:01 BST
20-Aug-2025 01:10:10 BST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d3/ff/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_72_20231207T151536Z_gen_g4_detsim_hitreco__20240510T044926Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.392485        1.602        8.9824        1.38181       1.12875        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0211481     0.0422284     0.0675385     0.041728      0.012066        100    
prod:pandorapid:Chi2ParticleID                 0.122886      0.208807      0.422196       0.20667      0.0612651       100    
prod:energyreconue:EnergyReco                  0.216197      0.469108       1.22013      0.437711      0.145116        100    
prod:energyreconumu:EnergyReco                 0.003655      0.123108       1.45247      0.0896654     0.189902        100    
prod:energyreconc:EnergyReco                   0.0034528     0.0620898     0.797367      0.0471129     0.101971        100    
[art]:TriggerResults:TriggerResultInserter    1.4268e-05    1.82289e-05   7.8022e-05    1.63775e-05   6.92943e-06      100    
end_path:ana:PIDASelectionAna                  0.0034679     0.696215       5.52514      0.551768      0.725823        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6812.67 MB
  Peak resident set size usage (VmHWM): 1331.7 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 job0008 job0008 19565 Aug 20 01:10 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_72_20231207T151536Z_gen_g4_detsim_hitreco__20240510T044926Z_reco2_pida_2025-08-20T_000604Z.root
-rw-r--r--. 1 job0008 job0008 14114 Aug 20 01:10 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_72_20231207T151536Z_gen_g4_detsim_hitreco__20240510T044926Z_reco2_reco_2025-08-20T_000604Z.log
-rw-r--r--. 1 job0008 job0008 11893 Aug 20 01:10 jobscript.log
-rw-r--r--. 1 job0008 job0008   186 Aug 20 01:10 justin-processed-pfns.txt
-rw-r--r--. 1 job0008 job0008   137 Aug 20 01:06 all-input-dids.txt
-rw-r--r--. 1 job0008 job0008     0 Aug 20 01:06 debugprod.log