Jobsub ID 40211.153@dunegpschedd01.fnal.gov

Jobsub ID	40211.153@dunegpschedd01.fnal.gov
Workflow ID	1978
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-04 13:50:17
Site	UK_Bristol
Entry	CMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat	2025-09-04 13:58:24
From worker node	Hostname	hd81.dice.priv
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	259200 (72 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-04 13:51:01
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50256571_173_20231121T150713Z_gen_g4_detsim_hitreco__20240503T053027Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (419s)
	CPU time	1m (75s = 17%)
	Max RSS bytes	1647419392 (1571 MiB)
Outputting started	2025-09-04 13:58:01
Output files
Finished	2025-09-04 13:58:24
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

rocessing the 131st record. run: 50256571 subRun: 1 event: 34731 at 04-Sep-2025 13:55:49 UTC
Begin processing the 132nd record. run: 50256571 subRun: 1 event: 34732 at 04-Sep-2025 13:55:51 UTC
Begin processing the 133rd record. run: 50256571 subRun: 1 event: 34733 at 04-Sep-2025 13:55:52 UTC
Begin processing the 134th record. run: 50256571 subRun: 1 event: 34734 at 04-Sep-2025 13:55:54 UTC
Begin processing the 135th record. run: 50256571 subRun: 1 event: 34735 at 04-Sep-2025 13:55:57 UTC
Begin processing the 136th record. run: 50256571 subRun: 1 event: 34736 at 04-Sep-2025 13:55:58 UTC
Begin processing the 137th record. run: 50256571 subRun: 1 event: 34737 at 04-Sep-2025 13:56:01 UTC
Begin processing the 138th record. run: 50256571 subRun: 1 event: 34738 at 04-Sep-2025 13:56:03 UTC
Begin processing the 139th record. run: 50256571 subRun: 1 event: 34739 at 04-Sep-2025 13:56:05 UTC
Begin processing the 140th record. run: 50256571 subRun: 1 event: 34740 at 04-Sep-2025 13:56:07 UTC
Begin processing the 141st record. run: 50256571 subRun: 1 event: 34741 at 04-Sep-2025 13:56:09 UTC
Begin processing the 142nd record. run: 50256571 subRun: 1 event: 34742 at 04-Sep-2025 13:56:11 UTC
Begin processing the 143rd record. run: 50256571 subRun: 1 event: 34743 at 04-Sep-2025 13:56:12 UTC
Begin processing the 144th record. run: 50256571 subRun: 1 event: 34744 at 04-Sep-2025 13:56:14 UTC
Begin processing the 145th record. run: 50256571 subRun: 1 event: 34745 at 04-Sep-2025 13:56:16 UTC
Begin processing the 146th record. run: 50256571 subRun: 1 event: 34746 at 04-Sep-2025 13:56:18 UTC
Begin processing the 147th record. run: 50256571 subRun: 1 event: 34747 at 04-Sep-2025 13:56:20 UTC
Begin processing the 148th record. run: 50256571 subRun: 1 event: 34748 at 04-Sep-2025 13:56:22 UTC
Begin processing the 149th record. run: 50256571 subRun: 1 event: 34749 at 04-Sep-2025 13:56:23 UTC
Begin processing the 150th record. run: 50256571 subRun: 1 event: 34750 at 04-Sep-2025 13:56:25 UTC
Begin processing the 151st record. run: 50256571 subRun: 1 event: 34751 at 04-Sep-2025 13:56:26 UTC
Begin processing the 152nd record. run: 50256571 subRun: 1 event: 34752 at 04-Sep-2025 13:56:28 UTC
Begin processing the 153rd record. run: 50256571 subRun: 1 event: 34753 at 04-Sep-2025 13:56:30 UTC
Begin processing the 154th record. run: 50256571 subRun: 1 event: 34754 at 04-Sep-2025 13:56:32 UTC
Begin processing the 155th record. run: 50256571 subRun: 1 event: 34755 at 04-Sep-2025 13:56:34 UTC
Begin processing the 156th record. run: 50256571 subRun: 1 event: 34756 at 04-Sep-2025 13:56:36 UTC
Begin processing the 157th record. run: 50256571 subRun: 1 event: 34757 at 04-Sep-2025 13:56:38 UTC
Begin processing the 158th record. run: 50256571 subRun: 1 event: 34758 at 04-Sep-2025 13:56:40 UTC
Begin processing the 159th record. run: 50256571 subRun: 1 event: 34759 at 04-Sep-2025 13:56:42 UTC
Begin processing the 160th record. run: 50256571 subRun: 1 event: 34760 at 04-Sep-2025 13:56:44 UTC
Begin processing the 161st record. run: 50256571 subRun: 1 event: 34761 at 04-Sep-2025 13:56:46 UTC
Begin processing the 162nd record. run: 50256571 subRun: 1 event: 34762 at 04-Sep-2025 13:56:47 UTC
Begin processing the 163rd record. run: 50256571 subRun: 1 event: 34763 at 04-Sep-2025 13:56:49 UTC
Begin processing the 164th record. run: 50256571 subRun: 1 event: 34764 at 04-Sep-2025 13:56:51 UTC
Begin processing the 165th record. run: 50256571 subRun: 1 event: 34765 at 04-Sep-2025 13:56:53 UTC
Begin processing the 166th record. run: 50256571 subRun: 1 event: 34766 at 04-Sep-2025 13:56:55 UTC
Begin processing the 167th record. run: 50256571 subRun: 1 event: 34767 at 04-Sep-2025 13:56:57 UTC
Begin processing the 168th record. run: 50256571 subRun: 1 event: 34768 at 04-Sep-2025 13:56:58 UTC
Begin processing the 169th record. run: 50256571 subRun: 1 event: 34769 at 04-Sep-2025 13:57:00 UTC
Begin processing the 170th record. run: 50256571 subRun: 1 event: 34770 at 04-Sep-2025 13:57:02 UTC
Begin processing the 171st record. run: 50256571 subRun: 1 event: 34771 at 04-Sep-2025 13:57:04 UTC
Begin processing the 172nd record. run: 50256571 subRun: 1 event: 34772 at 04-Sep-2025 13:57:06 UTC
Begin processing the 173rd record. run: 50256571 subRun: 1 event: 34773 at 04-Sep-2025 13:57:09 UTC
Begin processing the 174th record. run: 50256571 subRun: 1 event: 34774 at 04-Sep-2025 13:57:12 UTC
Begin processing the 175th record. run: 50256571 subRun: 1 event: 34775 at 04-Sep-2025 13:57:14 UTC
Begin processing the 176th record. run: 50256571 subRun: 1 event: 34776 at 04-Sep-2025 13:57:16 UTC
Begin processing the 177th record. run: 50256571 subRun: 1 event: 34777 at 04-Sep-2025 13:57:17 UTC
Begin processing the 178th record. run: 50256571 subRun: 1 event: 34778 at 04-Sep-2025 13:57:19 UTC
Begin processing the 179th record. run: 50256571 subRun: 1 event: 34779 at 04-Sep-2025 13:57:21 UTC
Begin processing the 180th record. run: 50256571 subRun: 1 event: 34780 at 04-Sep-2025 13:57:23 UTC
Begin processing the 181st record. run: 50256571 subRun: 1 event: 34781 at 04-Sep-2025 13:57:25 UTC
Begin processing the 182nd record. run: 50256571 subRun: 1 event: 34782 at 04-Sep-2025 13:57:27 UTC
Begin processing the 183rd record. run: 50256571 subRun: 1 event: 34783 at 04-Sep-2025 13:57:28 UTC
Begin processing the 184th record. run: 50256571 subRun: 1 event: 34784 at 04-Sep-2025 13:57:30 UTC
Begin processing the 185th record. run: 50256571 subRun: 1 event: 34785 at 04-Sep-2025 13:57:32 UTC
Begin processing the 186th record. run: 50256571 subRun: 1 event: 34786 at 04-Sep-2025 13:57:35 UTC
Begin processing the 187th record. run: 50256571 subRun: 1 event: 34787 at 04-Sep-2025 13:57:38 UTC
Begin processing the 188th record. run: 50256571 subRun: 1 event: 34788 at 04-Sep-2025 13:57:39 UTC
Begin processing the 189th record. run: 50256571 subRun: 1 event: 34789 at 04-Sep-2025 13:57:41 UTC
Begin processing the 190th record. run: 50256571 subRun: 1 event: 34790 at 04-Sep-2025 13:57:43 UTC
Begin processing the 191st record. run: 50256571 subRun: 1 event: 34791 at 04-Sep-2025 13:57:45 UTC
Begin processing the 192nd record. run: 50256571 subRun: 1 event: 34792 at 04-Sep-2025 13:57:46 UTC
Begin processing the 193rd record. run: 50256571 subRun: 1 event: 34793 at 04-Sep-2025 13:57:48 UTC
Begin processing the 194th record. run: 50256571 subRun: 1 event: 34794 at 04-Sep-2025 13:57:49 UTC
Begin processing the 195th record. run: 50256571 subRun: 1 event: 34795 at 04-Sep-2025 13:57:50 UTC
Begin processing the 196th record. run: 50256571 subRun: 1 event: 34796 at 04-Sep-2025 13:57:52 UTC
Begin processing the 197th record. run: 50256571 subRun: 1 event: 34797 at 04-Sep-2025 13:57:53 UTC
Begin processing the 198th record. run: 50256571 subRun: 1 event: 34798 at 04-Sep-2025 13:57:53 UTC
Begin processing the 199th record. run: 50256571 subRun: 1 event: 34799 at 04-Sep-2025 13:57:56 UTC
Begin processing the 200th record. run: 50256571 subRun: 1 event: 34800 at 04-Sep-2025 13:57:57 UTC
04-Sep-2025 13:58:00 UTC  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/71/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50256571_173_20231121T150713Z_gen_g4_detsim_hitreco__20240503T053027Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.392802       1.56673       7.53619       1.54911      0.607184        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0242105     0.0471148      0.1046       0.048324      0.0173099       200    
prod:pandorapid:Chi2ParticleID                  0.1428       0.215268      0.413957      0.216745      0.0348526       200    
prod:energyreconue:EnergyReco                  0.108157      0.423083      0.867466      0.438128       0.10053        200    
prod:energyreconumu:EnergyReco                 0.0041881     0.218821       1.72244      0.221654      0.198486        200    
prod:energyreconc:EnergyReco                  0.00390073     0.0461466     0.577507      0.0457836     0.0517719       200    
[art]:TriggerResults:TriggerResultInserter    1.1922e-05    1.70851e-05   9.9117e-05    1.5975e-05    6.97098e-06      200    
end_path:ana:PIDASelectionAna                  0.0990395     0.615742       3.8774       0.581793      0.299099        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11448.3 MB
  Peak resident set size usage (VmHWM): 1647.42 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 72
-rw-r--r-- 1 dnpil001 users 28322 Sep  4 13:58 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50256571_173_20231121T150713Z_gen_g4_detsim_hitreco__20240503T053027Z_reco2_pida_2025-09-04T_135107Z.root
-rw-r--r-- 1 dnpil001 users 24218 Sep  4 13:58 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50256571_173_20231121T150713Z_gen_g4_detsim_hitreco__20240503T053027Z_reco2_reco_2025-09-04T_135107Z.log
-rw-r--r-- 1 dnpil001 users 12049 Sep  4 13:58 jobscript.log
-rw-r--r-- 1 dnpil001 users   187 Sep  4 13:58 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users   138 Sep  4 13:51 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users     0 Sep  4 13:51 debugprod.log