Jobsub ID 33887.61@dunegpschedd02.fnal.gov

Jobsub ID	33887.61@dunegpschedd02.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-05 04:13:03
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2025-09-05 04:21:12
From worker node	Hostname	n200.dur.scotgrid.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-05 04:14:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_764_20231207T090214Z_gen_g4_detsim_hitreco__20240510T024624Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (399s)
	CPU time	0m (42s = 10%)
	Max RSS bytes	1188663296 (1133 MiB)
Outputting started	2025-09-05 04:20:54
Output files
Finished	2025-09-05 04:21:12
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

1 event: 76429 at 05-Sep-2025 05:16:40 BST
Begin processing the 30th record. run: 6452759 subRun: 1 event: 76430 at 05-Sep-2025 05:16:43 BST
Begin processing the 31st record. run: 6452759 subRun: 1 event: 76431 at 05-Sep-2025 05:16:45 BST
Begin processing the 32nd record. run: 6452759 subRun: 1 event: 76432 at 05-Sep-2025 05:16:46 BST
Begin processing the 33rd record. run: 6452759 subRun: 1 event: 76433 at 05-Sep-2025 05:16:50 BST
Begin processing the 34th record. run: 6452759 subRun: 1 event: 76434 at 05-Sep-2025 05:16:52 BST
Begin processing the 35th record. run: 6452759 subRun: 1 event: 76435 at 05-Sep-2025 05:16:59 BST
Begin processing the 36th record. run: 6452759 subRun: 1 event: 76436 at 05-Sep-2025 05:17:02 BST
Begin processing the 37th record. run: 6452759 subRun: 1 event: 76437 at 05-Sep-2025 05:17:04 BST
Begin processing the 38th record. run: 6452759 subRun: 1 event: 76438 at 05-Sep-2025 05:17:06 BST
Begin processing the 39th record. run: 6452759 subRun: 1 event: 76439 at 05-Sep-2025 05:17:07 BST
Begin processing the 40th record. run: 6452759 subRun: 1 event: 76440 at 05-Sep-2025 05:17:10 BST
Begin processing the 41st record. run: 6452759 subRun: 1 event: 76441 at 05-Sep-2025 05:17:13 BST
Begin processing the 42nd record. run: 6452759 subRun: 1 event: 76442 at 05-Sep-2025 05:17:15 BST
Begin processing the 43rd record. run: 6452759 subRun: 1 event: 76443 at 05-Sep-2025 05:17:19 BST
Begin processing the 44th record. run: 6452759 subRun: 1 event: 76444 at 05-Sep-2025 05:17:23 BST
Begin processing the 45th record. run: 6452759 subRun: 1 event: 76445 at 05-Sep-2025 05:17:24 BST
Begin processing the 46th record. run: 6452759 subRun: 1 event: 76446 at 05-Sep-2025 05:17:27 BST
Begin processing the 47th record. run: 6452759 subRun: 1 event: 76447 at 05-Sep-2025 05:17:30 BST
Begin processing the 48th record. run: 6452759 subRun: 1 event: 76448 at 05-Sep-2025 05:17:33 BST
Begin processing the 49th record. run: 6452759 subRun: 1 event: 76449 at 05-Sep-2025 05:17:42 BST
Begin processing the 50th record. run: 6452759 subRun: 1 event: 76450 at 05-Sep-2025 05:17:45 BST
Begin processing the 51st record. run: 6452759 subRun: 1 event: 76451 at 05-Sep-2025 05:17:48 BST
Begin processing the 52nd record. run: 6452759 subRun: 1 event: 76452 at 05-Sep-2025 05:17:52 BST
Begin processing the 53rd record. run: 6452759 subRun: 1 event: 76453 at 05-Sep-2025 05:17:55 BST
Begin processing the 54th record. run: 6452759 subRun: 1 event: 76454 at 05-Sep-2025 05:18:16 BST
Begin processing the 55th record. run: 6452759 subRun: 1 event: 76455 at 05-Sep-2025 05:18:18 BST
Begin processing the 56th record. run: 6452759 subRun: 1 event: 76456 at 05-Sep-2025 05:18:22 BST
Begin processing the 57th record. run: 6452759 subRun: 1 event: 76457 at 05-Sep-2025 05:18:26 BST
Begin processing the 58th record. run: 6452759 subRun: 1 event: 76458 at 05-Sep-2025 05:18:29 BST
Begin processing the 59th record. run: 6452759 subRun: 1 event: 76459 at 05-Sep-2025 05:18:33 BST
Begin processing the 60th record. run: 6452759 subRun: 1 event: 76460 at 05-Sep-2025 05:18:36 BST
Begin processing the 61st record. run: 6452759 subRun: 1 event: 76461 at 05-Sep-2025 05:18:37 BST
Begin processing the 62nd record. run: 6452759 subRun: 1 event: 76462 at 05-Sep-2025 05:18:40 BST
Begin processing the 63rd record. run: 6452759 subRun: 1 event: 76463 at 05-Sep-2025 05:18:44 BST
Begin processing the 64th record. run: 6452759 subRun: 1 event: 76464 at 05-Sep-2025 05:18:47 BST
Begin processing the 65th record. run: 6452759 subRun: 1 event: 76465 at 05-Sep-2025 05:18:48 BST
Begin processing the 66th record. run: 6452759 subRun: 1 event: 76466 at 05-Sep-2025 05:18:53 BST
Begin processing the 67th record. run: 6452759 subRun: 1 event: 76467 at 05-Sep-2025 05:18:55 BST
Begin processing the 68th record. run: 6452759 subRun: 1 event: 76468 at 05-Sep-2025 05:18:58 BST
Begin processing the 69th record. run: 6452759 subRun: 1 event: 76469 at 05-Sep-2025 05:19:01 BST
Begin processing the 70th record. run: 6452759 subRun: 1 event: 76470 at 05-Sep-2025 05:19:04 BST
Begin processing the 71st record. run: 6452759 subRun: 1 event: 76471 at 05-Sep-2025 05:19:07 BST
Begin processing the 72nd record. run: 6452759 subRun: 1 event: 76472 at 05-Sep-2025 05:19:14 BST
Begin processing the 73rd record. run: 6452759 subRun: 1 event: 76473 at 05-Sep-2025 05:19:17 BST
Begin processing the 74th record. run: 6452759 subRun: 1 event: 76474 at 05-Sep-2025 05:19:20 BST
Begin processing the 75th record. run: 6452759 subRun: 1 event: 76475 at 05-Sep-2025 05:19:23 BST
Begin processing the 76th record. run: 6452759 subRun: 1 event: 76476 at 05-Sep-2025 05:19:25 BST
Begin processing the 77th record. run: 6452759 subRun: 1 event: 76477 at 05-Sep-2025 05:19:28 BST
Begin processing the 78th record. run: 6452759 subRun: 1 event: 76478 at 05-Sep-2025 05:19:34 BST
Begin processing the 79th record. run: 6452759 subRun: 1 event: 76479 at 05-Sep-2025 05:19:37 BST
Begin processing the 80th record. run: 6452759 subRun: 1 event: 76480 at 05-Sep-2025 05:19:39 BST
Begin processing the 81st record. run: 6452759 subRun: 1 event: 76481 at 05-Sep-2025 05:19:42 BST
Begin processing the 82nd record. run: 6452759 subRun: 1 event: 76482 at 05-Sep-2025 05:19:45 BST
Begin processing the 83rd record. run: 6452759 subRun: 1 event: 76483 at 05-Sep-2025 05:19:49 BST
Begin processing the 84th record. run: 6452759 subRun: 1 event: 76484 at 05-Sep-2025 05:19:51 BST
Begin processing the 85th record. run: 6452759 subRun: 1 event: 76485 at 05-Sep-2025 05:19:55 BST
Begin processing the 86th record. run: 6452759 subRun: 1 event: 76486 at 05-Sep-2025 05:19:57 BST
Begin processing the 87th record. run: 6452759 subRun: 1 event: 76487 at 05-Sep-2025 05:20:00 BST
Begin processing the 88th record. run: 6452759 subRun: 1 event: 76488 at 05-Sep-2025 05:20:03 BST
Begin processing the 89th record. run: 6452759 subRun: 1 event: 76489 at 05-Sep-2025 05:20:06 BST
Begin processing the 90th record. run: 6452759 subRun: 1 event: 76490 at 05-Sep-2025 05:20:11 BST
Begin processing the 91st record. run: 6452759 subRun: 1 event: 76491 at 05-Sep-2025 05:20:17 BST
Begin processing the 92nd record. run: 6452759 subRun: 1 event: 76492 at 05-Sep-2025 05:20:25 BST
Begin processing the 93rd record. run: 6452759 subRun: 1 event: 76493 at 05-Sep-2025 05:20:27 BST
Begin processing the 94th record. run: 6452759 subRun: 1 event: 76494 at 05-Sep-2025 05:20:31 BST
Begin processing the 95th record. run: 6452759 subRun: 1 event: 76495 at 05-Sep-2025 05:20:33 BST
Begin processing the 96th record. run: 6452759 subRun: 1 event: 76496 at 05-Sep-2025 05:20:35 BST
Begin processing the 97th record. run: 6452759 subRun: 1 event: 76497 at 05-Sep-2025 05:20:38 BST
Begin processing the 98th record. run: 6452759 subRun: 1 event: 76498 at 05-Sep-2025 05:20:41 BST
Begin processing the 99th record. run: 6452759 subRun: 1 event: 76499 at 05-Sep-2025 05:20:47 BST
Begin processing the 100th record. run: 6452759 subRun: 1 event: 76500 at 05-Sep-2025 05:20:51 BST
05-Sep-2025 05:20:54 BST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a9/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_764_20231207T090214Z_gen_g4_detsim_hitreco__20240510T024624Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.615531       2.58565       10.8483       2.28159       1.32764        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0343118     0.0643154     0.119493      0.073521      0.0213312       100    
prod:pandorapid:Chi2ParticleID                 0.203621      0.320552      0.506172      0.322064      0.0460202       100    
prod:energyreconue:EnergyReco                   0.34985      0.814791        1.497       0.820658      0.186696        100    
prod:energyreconumu:EnergyReco                0.00150165     0.147093       1.25115      0.116699      0.164249        100    
prod:energyreconc:EnergyReco                  0.00339458     0.0982234     0.731626      0.081368      0.113732        100    
[art]:TriggerResults:TriggerResultInserter    1.1589e-05    1.4283e-05     6.175e-05    1.27505e-05   5.33438e-06      100    
end_path:ana:PIDASelectionAna                 0.00308373      1.14027       7.02478      0.898448      0.949917        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12033.2 MB
  Peak resident set size usage (VmHWM): 1188.66 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 dune001 dune 20196 Sep  5 05:20 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_764_20231207T090214Z_gen_g4_detsim_hitreco__20240510T024624Z_reco2_pida_2025-09-05T_041419Z.root
-rw-r--r--. 1 dune001 dune 14115 Sep  5 05:20 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_764_20231207T090214Z_gen_g4_detsim_hitreco__20240510T024624Z_reco2_reco_2025-09-05T_041419Z.log
-rw-r--r--. 1 dune001 dune 11894 Sep  5 05:20 jobscript.log
-rw-r--r--. 1 dune001 dune   186 Sep  5 05:20 justin-processed-pfns.txt
-rw-r--r--. 1 dune001 dune   137 Sep  5 05:14 all-input-dids.txt
-rw-r--r--. 1 dune001 dune     0 Sep  5 05:14 debugprod.log