Jobsub ID 40583.121@dunegpschedd01.fnal.gov

Jobsub ID	40583.121@dunegpschedd01.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-05 01:44:55
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2025-09-05 01:49:39
From worker node	Hostname	n200.dur.scotgrid.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-05 01:46:01
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_713_20231207T064546Z_gen_g4_detsim_hitreco__20240510T055505Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (199s)
	CPU time	0m (37s = 18%)
	Max RSS bytes	1104609280 (1053 MiB)
Outputting started	2025-09-05 01:49:21
Output files
Finished	2025-09-05 01:49:39
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

: 71330 at 05-Sep-2025 02:47:12 BST
Begin processing the 31st record. run: 74540126 subRun: 1 event: 71331 at 05-Sep-2025 02:47:14 BST
Begin processing the 32nd record. run: 74540126 subRun: 1 event: 71332 at 05-Sep-2025 02:47:15 BST
Begin processing the 33rd record. run: 74540126 subRun: 1 event: 71333 at 05-Sep-2025 02:47:17 BST
Begin processing the 34th record. run: 74540126 subRun: 1 event: 71334 at 05-Sep-2025 02:47:18 BST
Begin processing the 35th record. run: 74540126 subRun: 1 event: 71335 at 05-Sep-2025 02:47:20 BST
Begin processing the 36th record. run: 74540126 subRun: 1 event: 71336 at 05-Sep-2025 02:47:22 BST
Begin processing the 37th record. run: 74540126 subRun: 1 event: 71337 at 05-Sep-2025 02:47:23 BST
Begin processing the 38th record. run: 74540126 subRun: 1 event: 71338 at 05-Sep-2025 02:47:25 BST
Begin processing the 39th record. run: 74540126 subRun: 1 event: 71339 at 05-Sep-2025 02:47:26 BST
Begin processing the 40th record. run: 74540126 subRun: 1 event: 71340 at 05-Sep-2025 02:47:28 BST
Begin processing the 41st record. run: 74540126 subRun: 1 event: 71341 at 05-Sep-2025 02:47:30 BST
Begin processing the 42nd record. run: 74540126 subRun: 1 event: 71342 at 05-Sep-2025 02:47:31 BST
Begin processing the 43rd record. run: 74540126 subRun: 1 event: 71343 at 05-Sep-2025 02:47:33 BST
Begin processing the 44th record. run: 74540126 subRun: 1 event: 71344 at 05-Sep-2025 02:47:34 BST
Begin processing the 45th record. run: 74540126 subRun: 1 event: 71345 at 05-Sep-2025 02:47:36 BST
Begin processing the 46th record. run: 74540126 subRun: 1 event: 71346 at 05-Sep-2025 02:47:37 BST
Begin processing the 47th record. run: 74540126 subRun: 1 event: 71347 at 05-Sep-2025 02:47:39 BST
Begin processing the 48th record. run: 74540126 subRun: 1 event: 71348 at 05-Sep-2025 02:47:41 BST
Begin processing the 49th record. run: 74540126 subRun: 1 event: 71349 at 05-Sep-2025 02:47:42 BST
Begin processing the 50th record. run: 74540126 subRun: 1 event: 71350 at 05-Sep-2025 02:47:44 BST
Begin processing the 51st record. run: 74540126 subRun: 1 event: 71351 at 05-Sep-2025 02:47:45 BST
Begin processing the 52nd record. run: 74540126 subRun: 1 event: 71352 at 05-Sep-2025 02:47:47 BST
Begin processing the 53rd record. run: 74540126 subRun: 1 event: 71353 at 05-Sep-2025 02:47:48 BST
Begin processing the 54th record. run: 74540126 subRun: 1 event: 71354 at 05-Sep-2025 02:47:55 BST
Begin processing the 55th record. run: 74540126 subRun: 1 event: 71355 at 05-Sep-2025 02:48:04 BST
Begin processing the 56th record. run: 74540126 subRun: 1 event: 71356 at 05-Sep-2025 02:48:05 BST
Begin processing the 57th record. run: 74540126 subRun: 1 event: 71357 at 05-Sep-2025 02:48:07 BST
Begin processing the 58th record. run: 74540126 subRun: 1 event: 71358 at 05-Sep-2025 02:48:08 BST
Begin processing the 59th record. run: 74540126 subRun: 1 event: 71359 at 05-Sep-2025 02:48:10 BST
Begin processing the 60th record. run: 74540126 subRun: 1 event: 71360 at 05-Sep-2025 02:48:12 BST
Begin processing the 61st record. run: 74540126 subRun: 1 event: 71361 at 05-Sep-2025 02:48:13 BST
Begin processing the 62nd record. run: 74540126 subRun: 1 event: 71362 at 05-Sep-2025 02:48:15 BST
Begin processing the 63rd record. run: 74540126 subRun: 1 event: 71363 at 05-Sep-2025 02:48:17 BST
Begin processing the 64th record. run: 74540126 subRun: 1 event: 71364 at 05-Sep-2025 02:48:19 BST
Begin processing the 65th record. run: 74540126 subRun: 1 event: 71365 at 05-Sep-2025 02:48:20 BST
Begin processing the 66th record. run: 74540126 subRun: 1 event: 71366 at 05-Sep-2025 02:48:22 BST
Begin processing the 67th record. run: 74540126 subRun: 1 event: 71367 at 05-Sep-2025 02:48:24 BST
Begin processing the 68th record. run: 74540126 subRun: 1 event: 71368 at 05-Sep-2025 02:48:25 BST
Begin processing the 69th record. run: 74540126 subRun: 1 event: 71369 at 05-Sep-2025 02:48:26 BST
Begin processing the 70th record. run: 74540126 subRun: 1 event: 71370 at 05-Sep-2025 02:48:29 BST
Begin processing the 71st record. run: 74540126 subRun: 1 event: 71371 at 05-Sep-2025 02:48:31 BST
Begin processing the 72nd record. run: 74540126 subRun: 1 event: 71372 at 05-Sep-2025 02:48:33 BST
Begin processing the 73rd record. run: 74540126 subRun: 1 event: 71373 at 05-Sep-2025 02:48:35 BST
Begin processing the 74th record. run: 74540126 subRun: 1 event: 71374 at 05-Sep-2025 02:48:36 BST
Begin processing the 75th record. run: 74540126 subRun: 1 event: 71375 at 05-Sep-2025 02:48:38 BST
Begin processing the 76th record. run: 74540126 subRun: 1 event: 71376 at 05-Sep-2025 02:48:40 BST
Begin processing the 77th record. run: 74540126 subRun: 1 event: 71377 at 05-Sep-2025 02:48:41 BST
Begin processing the 78th record. run: 74540126 subRun: 1 event: 71378 at 05-Sep-2025 02:48:42 BST
Begin processing the 79th record. run: 74540126 subRun: 1 event: 71379 at 05-Sep-2025 02:48:44 BST
Begin processing the 80th record. run: 74540126 subRun: 1 event: 71380 at 05-Sep-2025 02:48:45 BST
Begin processing the 81st record. run: 74540126 subRun: 1 event: 71381 at 05-Sep-2025 02:48:47 BST
Begin processing the 82nd record. run: 74540126 subRun: 1 event: 71382 at 05-Sep-2025 02:48:48 BST
Begin processing the 83rd record. run: 74540126 subRun: 1 event: 71383 at 05-Sep-2025 02:48:50 BST
Begin processing the 84th record. run: 74540126 subRun: 1 event: 71384 at 05-Sep-2025 02:48:52 BST
Begin processing the 85th record. run: 74540126 subRun: 1 event: 71385 at 05-Sep-2025 02:48:53 BST
Begin processing the 86th record. run: 74540126 subRun: 1 event: 71386 at 05-Sep-2025 02:48:54 BST
Begin processing the 87th record. run: 74540126 subRun: 1 event: 71387 at 05-Sep-2025 02:48:56 BST
Begin processing the 88th record. run: 74540126 subRun: 1 event: 71388 at 05-Sep-2025 02:48:58 BST
Begin processing the 89th record. run: 74540126 subRun: 1 event: 71389 at 05-Sep-2025 02:49:00 BST
Begin processing the 90th record. run: 74540126 subRun: 1 event: 71390 at 05-Sep-2025 02:49:02 BST
Begin processing the 91st record. run: 74540126 subRun: 1 event: 71391 at 05-Sep-2025 02:49:04 BST
Begin processing the 92nd record. run: 74540126 subRun: 1 event: 71392 at 05-Sep-2025 02:49:06 BST
Begin processing the 93rd record. run: 74540126 subRun: 1 event: 71393 at 05-Sep-2025 02:49:07 BST
Begin processing the 94th record. run: 74540126 subRun: 1 event: 71394 at 05-Sep-2025 02:49:09 BST
Begin processing the 95th record. run: 74540126 subRun: 1 event: 71395 at 05-Sep-2025 02:49:10 BST
Begin processing the 96th record. run: 74540126 subRun: 1 event: 71396 at 05-Sep-2025 02:49:12 BST
Begin processing the 97th record. run: 74540126 subRun: 1 event: 71397 at 05-Sep-2025 02:49:14 BST
Begin processing the 98th record. run: 74540126 subRun: 1 event: 71398 at 05-Sep-2025 02:49:15 BST
Begin processing the 99th record. run: 74540126 subRun: 1 event: 71399 at 05-Sep-2025 02:49:17 BST
Begin processing the 100th record. run: 74540126 subRun: 1 event: 71400 at 05-Sep-2025 02:49:18 BST
05-Sep-2025 02:49:20 BST  Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/ea/da/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_713_20231207T064546Z_gen_g4_detsim_hitreco__20240510T055505Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.399271       1.38885       5.50395       1.30858      0.583683        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0210772     0.0316107     0.0425813     0.0219818     0.0104083       100    
prod:pandorapid:Chi2ParticleID                 0.124585      0.181305      0.276829      0.187784      0.0247324       100    
prod:energyreconue:EnergyReco                  0.221009      0.469988      0.899551      0.441249      0.0960535       100    
prod:energyreconumu:EnergyReco                0.00155098     0.111839       1.63144      0.0702937     0.207015        100    
prod:energyreconc:EnergyReco                  0.00317145     0.0417061     0.435379      0.036797      0.0517496       100    
[art]:TriggerResults:TriggerResultInserter    1.1317e-05    1.35792e-05   7.0493e-05    1.25335e-05   6.04958e-06      100    
end_path:ana:PIDASelectionAna                  0.0243988     0.552041       2.23875      0.521146      0.262126        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11954.8 MB
  Peak resident set size usage (VmHWM): 1104.61 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 dune001 dune 20374 Sep  5 02:49 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_713_20231207T064546Z_gen_g4_detsim_hitreco__20240510T055505Z_reco2_pida_2025-09-05T_014606Z.root
-rw-r--r--. 1 dune001 dune 14314 Sep  5 02:49 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_713_20231207T064546Z_gen_g4_detsim_hitreco__20240510T055505Z_reco2_reco_2025-09-05T_014606Z.log
-rw-r--r--. 1 dune001 dune 12037 Sep  5 02:49 jobscript.log
-rw-r--r--. 1 dune001 dune   219 Sep  5 02:49 justin-processed-pfns.txt
-rw-r--r--. 1 dune001 dune   138 Sep  5 02:46 all-input-dids.txt
-rw-r--r--. 1 dune001 dune     0 Sep  5 02:46 debugprod.log