Jobsub ID 40583.35@dunegpschedd01.fnal.gov

Jobsub ID	40583.35@dunegpschedd01.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-05 01:44:55
Site	UK_Bristol
Entry	CMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat	2025-09-05 01:50:45
From worker node	Hostname	hd81.dice.priv
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	259200 (72 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-05 01:46:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_1138_20231204T011133Z_gen_g4_detsim_hitreco__20240509T202935Z_reco2.root
Jobscript	Exit code	0
	Real time	4m (245s)
	CPU time	1m (62s = 25%)
	Max RSS bytes	1338998784 (1276 MiB)
Outputting started	2025-09-05 01:50:22
Output files
Finished	2025-09-05 01:50:45
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

run: 6419587 subRun: 1 event: 113830 at 05-Sep-2025 01:47:49 UTC
Begin processing the 31st record. run: 6419587 subRun: 1 event: 113831 at 05-Sep-2025 01:47:59 UTC
Begin processing the 32nd record. run: 6419587 subRun: 1 event: 113832 at 05-Sep-2025 01:48:01 UTC
Begin processing the 33rd record. run: 6419587 subRun: 1 event: 113833 at 05-Sep-2025 01:48:02 UTC
Begin processing the 34th record. run: 6419587 subRun: 1 event: 113834 at 05-Sep-2025 01:48:04 UTC
Begin processing the 35th record. run: 6419587 subRun: 1 event: 113835 at 05-Sep-2025 01:48:07 UTC
Begin processing the 36th record. run: 6419587 subRun: 1 event: 113836 at 05-Sep-2025 01:48:09 UTC
Begin processing the 37th record. run: 6419587 subRun: 1 event: 113837 at 05-Sep-2025 01:48:11 UTC
Begin processing the 38th record. run: 6419587 subRun: 1 event: 113838 at 05-Sep-2025 01:48:13 UTC
Begin processing the 39th record. run: 6419587 subRun: 1 event: 113839 at 05-Sep-2025 01:48:15 UTC
Begin processing the 40th record. run: 6419587 subRun: 1 event: 113840 at 05-Sep-2025 01:48:17 UTC
Begin processing the 41st record. run: 6419587 subRun: 1 event: 113841 at 05-Sep-2025 01:48:19 UTC
Begin processing the 42nd record. run: 6419587 subRun: 1 event: 113842 at 05-Sep-2025 01:48:26 UTC
Begin processing the 43rd record. run: 6419587 subRun: 1 event: 113843 at 05-Sep-2025 01:48:26 UTC
Begin processing the 44th record. run: 6419587 subRun: 1 event: 113844 at 05-Sep-2025 01:48:27 UTC
Begin processing the 45th record. run: 6419587 subRun: 1 event: 113845 at 05-Sep-2025 01:48:28 UTC
Begin processing the 46th record. run: 6419587 subRun: 1 event: 113846 at 05-Sep-2025 01:48:29 UTC
Begin processing the 47th record. run: 6419587 subRun: 1 event: 113847 at 05-Sep-2025 01:48:32 UTC
Begin processing the 48th record. run: 6419587 subRun: 1 event: 113848 at 05-Sep-2025 01:48:34 UTC
Begin processing the 49th record. run: 6419587 subRun: 1 event: 113849 at 05-Sep-2025 01:48:36 UTC
Begin processing the 50th record. run: 6419587 subRun: 1 event: 113850 at 05-Sep-2025 01:48:38 UTC
Begin processing the 51st record. run: 6419587 subRun: 1 event: 113851 at 05-Sep-2025 01:48:40 UTC
Begin processing the 52nd record. run: 6419587 subRun: 1 event: 113852 at 05-Sep-2025 01:48:42 UTC
Begin processing the 53rd record. run: 6419587 subRun: 1 event: 113853 at 05-Sep-2025 01:48:44 UTC
Begin processing the 54th record. run: 6419587 subRun: 1 event: 113854 at 05-Sep-2025 01:48:44 UTC
Begin processing the 55th record. run: 6419587 subRun: 1 event: 113855 at 05-Sep-2025 01:48:47 UTC
Begin processing the 56th record. run: 6419587 subRun: 1 event: 113856 at 05-Sep-2025 01:48:49 UTC
Begin processing the 57th record. run: 6419587 subRun: 1 event: 113857 at 05-Sep-2025 01:48:51 UTC
Begin processing the 58th record. run: 6419587 subRun: 1 event: 113858 at 05-Sep-2025 01:48:53 UTC
Begin processing the 59th record. run: 6419587 subRun: 1 event: 113859 at 05-Sep-2025 01:48:55 UTC
Begin processing the 60th record. run: 6419587 subRun: 1 event: 113860 at 05-Sep-2025 01:48:57 UTC
Begin processing the 61st record. run: 6419587 subRun: 1 event: 113861 at 05-Sep-2025 01:48:59 UTC
Begin processing the 62nd record. run: 6419587 subRun: 1 event: 113862 at 05-Sep-2025 01:49:02 UTC
Begin processing the 63rd record. run: 6419587 subRun: 1 event: 113863 at 05-Sep-2025 01:49:04 UTC
Begin processing the 64th record. run: 6419587 subRun: 1 event: 113864 at 05-Sep-2025 01:49:06 UTC
Begin processing the 65th record. run: 6419587 subRun: 1 event: 113865 at 05-Sep-2025 01:49:07 UTC
Begin processing the 66th record. run: 6419587 subRun: 1 event: 113866 at 05-Sep-2025 01:49:10 UTC
Begin processing the 67th record. run: 6419587 subRun: 1 event: 113867 at 05-Sep-2025 01:49:12 UTC
Begin processing the 68th record. run: 6419587 subRun: 1 event: 113868 at 05-Sep-2025 01:49:14 UTC
Begin processing the 69th record. run: 6419587 subRun: 1 event: 113869 at 05-Sep-2025 01:49:16 UTC
Begin processing the 70th record. run: 6419587 subRun: 1 event: 113870 at 05-Sep-2025 01:49:18 UTC
Begin processing the 71st record. run: 6419587 subRun: 1 event: 113871 at 05-Sep-2025 01:49:20 UTC
Begin processing the 72nd record. run: 6419587 subRun: 1 event: 113872 at 05-Sep-2025 01:49:22 UTC
Begin processing the 73rd record. run: 6419587 subRun: 1 event: 113873 at 05-Sep-2025 01:49:24 UTC
Begin processing the 74th record. run: 6419587 subRun: 1 event: 113874 at 05-Sep-2025 01:49:26 UTC
Begin processing the 75th record. run: 6419587 subRun: 1 event: 113875 at 05-Sep-2025 01:49:28 UTC
Begin processing the 76th record. run: 6419587 subRun: 1 event: 113876 at 05-Sep-2025 01:49:30 UTC
Begin processing the 77th record. run: 6419587 subRun: 1 event: 113877 at 05-Sep-2025 01:49:35 UTC
Begin processing the 78th record. run: 6419587 subRun: 1 event: 113878 at 05-Sep-2025 01:49:37 UTC
Begin processing the 79th record. run: 6419587 subRun: 1 event: 113879 at 05-Sep-2025 01:49:39 UTC
Begin processing the 80th record. run: 6419587 subRun: 1 event: 113880 at 05-Sep-2025 01:49:40 UTC
Begin processing the 81st record. run: 6419587 subRun: 1 event: 113881 at 05-Sep-2025 01:49:44 UTC
Begin processing the 82nd record. run: 6419587 subRun: 1 event: 113882 at 05-Sep-2025 01:49:45 UTC
Begin processing the 83rd record. run: 6419587 subRun: 1 event: 113883 at 05-Sep-2025 01:49:47 UTC
Begin processing the 84th record. run: 6419587 subRun: 1 event: 113884 at 05-Sep-2025 01:49:49 UTC
Begin processing the 85th record. run: 6419587 subRun: 1 event: 113885 at 05-Sep-2025 01:49:50 UTC
Begin processing the 86th record. run: 6419587 subRun: 1 event: 113886 at 05-Sep-2025 01:49:51 UTC
Begin processing the 87th record. run: 6419587 subRun: 1 event: 113887 at 05-Sep-2025 01:49:53 UTC
Begin processing the 88th record. run: 6419587 subRun: 1 event: 113888 at 05-Sep-2025 01:49:55 UTC
Begin processing the 89th record. run: 6419587 subRun: 1 event: 113889 at 05-Sep-2025 01:49:59 UTC
Begin processing the 90th record. run: 6419587 subRun: 1 event: 113890 at 05-Sep-2025 01:50:01 UTC
Begin processing the 91st record. run: 6419587 subRun: 1 event: 113891 at 05-Sep-2025 01:50:03 UTC
Begin processing the 92nd record. run: 6419587 subRun: 1 event: 113892 at 05-Sep-2025 01:50:05 UTC
Begin processing the 93rd record. run: 6419587 subRun: 1 event: 113893 at 05-Sep-2025 01:50:07 UTC
Begin processing the 94th record. run: 6419587 subRun: 1 event: 113894 at 05-Sep-2025 01:50:07 UTC
Begin processing the 95th record. run: 6419587 subRun: 1 event: 113895 at 05-Sep-2025 01:50:09 UTC
Begin processing the 96th record. run: 6419587 subRun: 1 event: 113896 at 05-Sep-2025 01:50:12 UTC
Begin processing the 97th record. run: 6419587 subRun: 1 event: 113897 at 05-Sep-2025 01:50:14 UTC
Begin processing the 98th record. run: 6419587 subRun: 1 event: 113898 at 05-Sep-2025 01:50:16 UTC
Begin processing the 99th record. run: 6419587 subRun: 1 event: 113899 at 05-Sep-2025 01:50:17 UTC
Begin processing the 100th record. run: 6419587 subRun: 1 event: 113900 at 05-Sep-2025 01:50:19 UTC
05-Sep-2025 01:50:21 UTC  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e4/b9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_1138_20231204T011133Z_gen_g4_detsim_hitreco__20240509T202935Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.449645       1.64874       6.56271       1.5709       0.681167        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0244846     0.0477338     0.300906      0.0487488     0.0292985       100    
prod:pandorapid:Chi2ParticleID                  0.1434       0.216849      0.331648      0.226753      0.0335894       100    
prod:energyreconue:EnergyReco                  0.252686      0.568666       1.28792       0.53874      0.142636        100    
prod:energyreconumu:EnergyReco                0.00195958     0.114056       1.7023       0.0841523      0.18878        100    
prod:energyreconc:EnergyReco                  0.00454444     0.055007      0.719801      0.0461687     0.0796926       100    
[art]:TriggerResults:TriggerResultInserter    1.3966e-05    1.84279e-05   0.000118233   1.6246e-05    1.10436e-05      100    
end_path:ana:PIDASelectionAna                 0.00387211     0.645873       2.47982      0.620567      0.314233        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11107.7 MB
  Peak resident set size usage (VmHWM): 1339 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r-- 1 dnpil001 users 20205 Sep  5 01:50 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_1138_20231204T011133Z_gen_g4_detsim_hitreco__20240509T202935Z_reco2_pida_2025-09-05T_014624Z.root
-rw-r--r-- 1 dnpil001 users 14215 Sep  5 01:50 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_1138_20231204T011133Z_gen_g4_detsim_hitreco__20240509T202935Z_reco2_reco_2025-09-05T_014624Z.log
-rw-r--r-- 1 dnpil001 users 11970 Sep  5 01:50 jobscript.log
-rw-r--r-- 1 dnpil001 users   187 Sep  5 01:50 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users   138 Sep  5 01:46 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users     0 Sep  5 01:46 debugprod.log