Jobsub ID 40211.109@dunegpschedd01.fnal.gov

Jobsub ID	40211.109@dunegpschedd01.fnal.gov
Workflow ID	1978
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-04 13:50:17
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2025-09-04 13:58:59
From worker node	Hostname	n170.dur.scotgrid.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-04 13:51:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_886_20231202T205526Z_gen_g4_detsim_hitreco__20240508T070521Z_reco2.root
Jobscript	Exit code	0
	Real time	7m (449s)
	CPU time	0m (54s = 12%)
	Max RSS bytes	1582125056 (1508 MiB)
Outputting started	2025-09-04 13:58:41
Output files
Finished	2025-09-04 13:58:59
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

d. run: 66006610 subRun: 1 event: 88630 at 04-Sep-2025 14:53:49 BST
Begin processing the 31st record. run: 66006610 subRun: 1 event: 88631 at 04-Sep-2025 14:53:51 BST
Begin processing the 32nd record. run: 66006610 subRun: 1 event: 88632 at 04-Sep-2025 14:53:55 BST
Begin processing the 33rd record. run: 66006610 subRun: 1 event: 88633 at 04-Sep-2025 14:53:57 BST
Begin processing the 34th record. run: 66006610 subRun: 1 event: 88634 at 04-Sep-2025 14:54:00 BST
Begin processing the 35th record. run: 66006610 subRun: 1 event: 88635 at 04-Sep-2025 14:54:02 BST
Begin processing the 36th record. run: 66006610 subRun: 1 event: 88636 at 04-Sep-2025 14:54:05 BST
Begin processing the 37th record. run: 66006610 subRun: 1 event: 88637 at 04-Sep-2025 14:54:07 BST
Begin processing the 38th record. run: 66006610 subRun: 1 event: 88638 at 04-Sep-2025 14:54:10 BST
Begin processing the 39th record. run: 66006610 subRun: 1 event: 88639 at 04-Sep-2025 14:54:12 BST
Begin processing the 40th record. run: 66006610 subRun: 1 event: 88640 at 04-Sep-2025 14:54:18 BST
Begin processing the 41st record. run: 66006610 subRun: 1 event: 88641 at 04-Sep-2025 14:54:21 BST
Begin processing the 42nd record. run: 66006610 subRun: 1 event: 88642 at 04-Sep-2025 14:54:24 BST
Begin processing the 43rd record. run: 66006610 subRun: 1 event: 88643 at 04-Sep-2025 14:54:24 BST
Begin processing the 44th record. run: 66006610 subRun: 1 event: 88644 at 04-Sep-2025 14:54:28 BST
Begin processing the 45th record. run: 66006610 subRun: 1 event: 88645 at 04-Sep-2025 14:54:30 BST
Begin processing the 46th record. run: 66006610 subRun: 1 event: 88646 at 04-Sep-2025 14:54:33 BST
Begin processing the 47th record. run: 66006610 subRun: 1 event: 88647 at 04-Sep-2025 14:54:35 BST
Begin processing the 48th record. run: 66006610 subRun: 1 event: 88648 at 04-Sep-2025 14:54:38 BST
Begin processing the 49th record. run: 66006610 subRun: 1 event: 88649 at 04-Sep-2025 14:54:40 BST
Begin processing the 50th record. run: 66006610 subRun: 1 event: 88650 at 04-Sep-2025 14:54:43 BST
Begin processing the 51st record. run: 66006610 subRun: 1 event: 88651 at 04-Sep-2025 14:54:59 BST
Begin processing the 52nd record. run: 66006610 subRun: 1 event: 88652 at 04-Sep-2025 14:55:02 BST
Begin processing the 53rd record. run: 66006610 subRun: 1 event: 88653 at 04-Sep-2025 14:55:04 BST
Begin processing the 54th record. run: 66006610 subRun: 1 event: 88654 at 04-Sep-2025 14:55:24 BST
Begin processing the 55th record. run: 66006610 subRun: 1 event: 88655 at 04-Sep-2025 14:55:28 BST
Begin processing the 56th record. run: 66006610 subRun: 1 event: 88656 at 04-Sep-2025 14:55:32 BST
Begin processing the 57th record. run: 66006610 subRun: 1 event: 88657 at 04-Sep-2025 14:55:39 BST
Begin processing the 58th record. run: 66006610 subRun: 1 event: 88658 at 04-Sep-2025 14:55:45 BST
Begin processing the 59th record. run: 66006610 subRun: 1 event: 88659 at 04-Sep-2025 14:55:48 BST
Begin processing the 60th record. run: 66006610 subRun: 1 event: 88660 at 04-Sep-2025 14:55:51 BST
Begin processing the 61st record. run: 66006610 subRun: 1 event: 88661 at 04-Sep-2025 14:55:54 BST
Begin processing the 62nd record. run: 66006610 subRun: 1 event: 88662 at 04-Sep-2025 14:55:57 BST
Begin processing the 63rd record. run: 66006610 subRun: 1 event: 88663 at 04-Sep-2025 14:56:00 BST
Begin processing the 64th record. run: 66006610 subRun: 1 event: 88664 at 04-Sep-2025 14:56:03 BST
Begin processing the 65th record. run: 66006610 subRun: 1 event: 88665 at 04-Sep-2025 14:56:05 BST
Begin processing the 66th record. run: 66006610 subRun: 1 event: 88666 at 04-Sep-2025 14:56:06 BST
Begin processing the 67th record. run: 66006610 subRun: 1 event: 88667 at 04-Sep-2025 14:56:09 BST
Begin processing the 68th record. run: 66006610 subRun: 1 event: 88668 at 04-Sep-2025 14:56:12 BST
Begin processing the 69th record. run: 66006610 subRun: 1 event: 88669 at 04-Sep-2025 14:56:15 BST
Begin processing the 70th record. run: 66006610 subRun: 1 event: 88670 at 04-Sep-2025 14:56:17 BST
Begin processing the 71st record. run: 66006610 subRun: 1 event: 88671 at 04-Sep-2025 14:56:20 BST
Begin processing the 72nd record. run: 66006610 subRun: 1 event: 88672 at 04-Sep-2025 14:56:29 BST
Begin processing the 73rd record. run: 66006610 subRun: 1 event: 88673 at 04-Sep-2025 14:56:32 BST
Begin processing the 74th record. run: 66006610 subRun: 1 event: 88674 at 04-Sep-2025 14:56:32 BST
Begin processing the 75th record. run: 66006610 subRun: 1 event: 88675 at 04-Sep-2025 14:56:35 BST
Begin processing the 76th record. run: 66006610 subRun: 1 event: 88676 at 04-Sep-2025 14:56:39 BST
Begin processing the 77th record. run: 66006610 subRun: 1 event: 88677 at 04-Sep-2025 14:56:44 BST
Begin processing the 78th record. run: 66006610 subRun: 1 event: 88678 at 04-Sep-2025 14:56:47 BST
Begin processing the 79th record. run: 66006610 subRun: 1 event: 88679 at 04-Sep-2025 14:56:51 BST
Begin processing the 80th record. run: 66006610 subRun: 1 event: 88680 at 04-Sep-2025 14:56:53 BST
Begin processing the 81st record. run: 66006610 subRun: 1 event: 88681 at 04-Sep-2025 14:56:56 BST
Begin processing the 82nd record. run: 66006610 subRun: 1 event: 88682 at 04-Sep-2025 14:57:06 BST
Begin processing the 83rd record. run: 66006610 subRun: 1 event: 88683 at 04-Sep-2025 14:57:10 BST
Begin processing the 84th record. run: 66006610 subRun: 1 event: 88684 at 04-Sep-2025 14:57:19 BST
Begin processing the 85th record. run: 66006610 subRun: 1 event: 88685 at 04-Sep-2025 14:57:22 BST
Begin processing the 86th record. run: 66006610 subRun: 1 event: 88686 at 04-Sep-2025 14:57:25 BST
Begin processing the 87th record. run: 66006610 subRun: 1 event: 88687 at 04-Sep-2025 14:57:29 BST
Begin processing the 88th record. run: 66006610 subRun: 1 event: 88688 at 04-Sep-2025 14:57:32 BST
Begin processing the 89th record. run: 66006610 subRun: 1 event: 88689 at 04-Sep-2025 14:57:36 BST
Begin processing the 90th record. run: 66006610 subRun: 1 event: 88690 at 04-Sep-2025 14:57:43 BST
Begin processing the 91st record. run: 66006610 subRun: 1 event: 88691 at 04-Sep-2025 14:57:46 BST
Begin processing the 92nd record. run: 66006610 subRun: 1 event: 88692 at 04-Sep-2025 14:58:09 BST
Begin processing the 93rd record. run: 66006610 subRun: 1 event: 88693 at 04-Sep-2025 14:58:12 BST
Begin processing the 94th record. run: 66006610 subRun: 1 event: 88694 at 04-Sep-2025 14:58:16 BST
Begin processing the 95th record. run: 66006610 subRun: 1 event: 88695 at 04-Sep-2025 14:58:19 BST
Begin processing the 96th record. run: 66006610 subRun: 1 event: 88696 at 04-Sep-2025 14:58:22 BST
Begin processing the 97th record. run: 66006610 subRun: 1 event: 88697 at 04-Sep-2025 14:58:25 BST
Begin processing the 98th record. run: 66006610 subRun: 1 event: 88698 at 04-Sep-2025 14:58:29 BST
Begin processing the 99th record. run: 66006610 subRun: 1 event: 88699 at 04-Sep-2025 14:58:34 BST
Begin processing the 100th record. run: 66006610 subRun: 1 event: 88700 at 04-Sep-2025 14:58:37 BST
04-Sep-2025 14:58:40 BST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/1b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_886_20231202T205526Z_gen_g4_detsim_hitreco__20240508T070521Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.567697       2.88809       17.4838       2.27566       2.21397        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0345239     0.0653431     0.102823      0.0716336     0.0197649       100    
prod:pandorapid:Chi2ParticleID                  0.20423      0.325329      0.686486      0.315937      0.0672037       100    
prod:energyreconue:EnergyReco                  0.147795      0.635697       1.09862      0.624108      0.116838        100    
prod:energyreconumu:EnergyReco                0.00326327      0.35713       3.55182      0.320685       0.37105        100    
prod:energyreconc:EnergyReco                  0.00319742     0.0999338     0.848295      0.0633301     0.139977        100    
[art]:TriggerResults:TriggerResultInserter    1.2115e-05    1.38423e-05   6.0386e-05    1.32085e-05   4.77881e-06      100    
end_path:ana:PIDASelectionAna                  0.140112       1.40429       11.2536      0.938967       1.64866        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12430.4 MB
  Peak resident set size usage (VmHWM): 1582.13 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 60
-rw-r--r--. 1 dune001 dune 20556 Sep  4 14:58 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_886_20231202T205526Z_gen_g4_detsim_hitreco__20240508T070521Z_reco2_pida_2025-09-04T_135115Z.root
-rw-r--r--. 1 dune001 dune 14218 Sep  4 14:58 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_886_20231202T205526Z_gen_g4_detsim_hitreco__20240508T070521Z_reco2_reco_2025-09-04T_135115Z.log
-rw-r--r--. 1 dune001 dune 11973 Sep  4 14:58 jobscript.log
-rw-r--r--. 1 dune001 dune   187 Sep  4 14:58 justin-processed-pfns.txt
-rw-r--r--. 1 dune001 dune   138 Sep  4 14:51 all-input-dids.txt
-rw-r--r--. 1 dune001 dune     0 Sep  4 14:51 debugprod.log