Jobsub ID 40683.177@dunegpschedd01.fnal.gov

Jobsub ID	40683.177@dunegpschedd01.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-05 05:03:05
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2025-09-05 05:11:46
From worker node	Hostname	wn-pep-010.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-05 05:06:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_468_20231123T021951Z_gen_g4_detsim_hitreco__20240507T175311Z_reco2.root
Jobscript	Exit code	0
	Real time	5m (318s)
	CPU time	1m (60s = 18%)
	Max RSS bytes	1460580352 (1392 MiB)
Outputting started	2025-09-05 05:11:31
Output files
Finished	2025-09-05 05:11:46
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

Begin processing the 131st record. run: 6209163 subRun: 1 event: 93731 at 05-Sep-2025 07:09:43 CEST
Begin processing the 132nd record. run: 6209163 subRun: 1 event: 93732 at 05-Sep-2025 07:09:44 CEST
Begin processing the 133rd record. run: 6209163 subRun: 1 event: 93733 at 05-Sep-2025 07:09:45 CEST
Begin processing the 134th record. run: 6209163 subRun: 1 event: 93734 at 05-Sep-2025 07:09:47 CEST
Begin processing the 135th record. run: 6209163 subRun: 1 event: 93735 at 05-Sep-2025 07:09:48 CEST
Begin processing the 136th record. run: 6209163 subRun: 1 event: 93736 at 05-Sep-2025 07:09:50 CEST
Begin processing the 137th record. run: 6209163 subRun: 1 event: 93737 at 05-Sep-2025 07:09:51 CEST
Begin processing the 138th record. run: 6209163 subRun: 1 event: 93738 at 05-Sep-2025 07:09:53 CEST
Begin processing the 139th record. run: 6209163 subRun: 1 event: 93739 at 05-Sep-2025 07:09:54 CEST
Begin processing the 140th record. run: 6209163 subRun: 1 event: 93740 at 05-Sep-2025 07:09:56 CEST
Begin processing the 141st record. run: 6209163 subRun: 1 event: 93741 at 05-Sep-2025 07:09:58 CEST
Begin processing the 142nd record. run: 6209163 subRun: 1 event: 93742 at 05-Sep-2025 07:09:59 CEST
Begin processing the 143rd record. run: 6209163 subRun: 1 event: 93743 at 05-Sep-2025 07:10:00 CEST
Begin processing the 144th record. run: 6209163 subRun: 1 event: 93744 at 05-Sep-2025 07:10:01 CEST
Begin processing the 145th record. run: 6209163 subRun: 1 event: 93745 at 05-Sep-2025 07:10:03 CEST
Begin processing the 146th record. run: 6209163 subRun: 1 event: 93746 at 05-Sep-2025 07:10:04 CEST
Begin processing the 147th record. run: 6209163 subRun: 1 event: 93747 at 05-Sep-2025 07:10:06 CEST
Begin processing the 148th record. run: 6209163 subRun: 1 event: 93748 at 05-Sep-2025 07:10:07 CEST
Begin processing the 149th record. run: 6209163 subRun: 1 event: 93749 at 05-Sep-2025 07:10:08 CEST
Begin processing the 150th record. run: 6209163 subRun: 1 event: 93750 at 05-Sep-2025 07:10:09 CEST
Begin processing the 151st record. run: 6209163 subRun: 1 event: 93751 at 05-Sep-2025 07:10:11 CEST
Begin processing the 152nd record. run: 6209163 subRun: 1 event: 93752 at 05-Sep-2025 07:10:12 CEST
Begin processing the 153rd record. run: 6209163 subRun: 1 event: 93753 at 05-Sep-2025 07:10:13 CEST
Begin processing the 154th record. run: 6209163 subRun: 1 event: 93754 at 05-Sep-2025 07:10:14 CEST
Begin processing the 155th record. run: 6209163 subRun: 1 event: 93755 at 05-Sep-2025 07:10:16 CEST
Begin processing the 156th record. run: 6209163 subRun: 1 event: 93756 at 05-Sep-2025 07:10:17 CEST
Begin processing the 157th record. run: 6209163 subRun: 1 event: 93757 at 05-Sep-2025 07:10:18 CEST
Begin processing the 158th record. run: 6209163 subRun: 1 event: 93758 at 05-Sep-2025 07:10:19 CEST
Begin processing the 159th record. run: 6209163 subRun: 1 event: 93759 at 05-Sep-2025 07:10:20 CEST
Begin processing the 160th record. run: 6209163 subRun: 1 event: 93760 at 05-Sep-2025 07:10:21 CEST
Begin processing the 161st record. run: 6209163 subRun: 1 event: 93761 at 05-Sep-2025 07:10:22 CEST
Begin processing the 162nd record. run: 6209163 subRun: 1 event: 93762 at 05-Sep-2025 07:10:24 CEST
Begin processing the 163rd record. run: 6209163 subRun: 1 event: 93763 at 05-Sep-2025 07:10:25 CEST
Begin processing the 164th record. run: 6209163 subRun: 1 event: 93764 at 05-Sep-2025 07:10:26 CEST
Begin processing the 165th record. run: 6209163 subRun: 1 event: 93765 at 05-Sep-2025 07:10:28 CEST
Begin processing the 166th record. run: 6209163 subRun: 1 event: 93766 at 05-Sep-2025 07:10:29 CEST
Begin processing the 167th record. run: 6209163 subRun: 1 event: 93767 at 05-Sep-2025 07:10:30 CEST
Begin processing the 168th record. run: 6209163 subRun: 1 event: 93768 at 05-Sep-2025 07:10:32 CEST
Begin processing the 169th record. run: 6209163 subRun: 1 event: 93769 at 05-Sep-2025 07:10:35 CEST
Begin processing the 170th record. run: 6209163 subRun: 1 event: 93770 at 05-Sep-2025 07:10:45 CEST
Begin processing the 171st record. run: 6209163 subRun: 1 event: 93771 at 05-Sep-2025 07:10:46 CEST
Begin processing the 172nd record. run: 6209163 subRun: 1 event: 93772 at 05-Sep-2025 07:10:47 CEST
Begin processing the 173rd record. run: 6209163 subRun: 1 event: 93773 at 05-Sep-2025 07:10:48 CEST
Begin processing the 174th record. run: 6209163 subRun: 1 event: 93774 at 05-Sep-2025 07:10:50 CEST
Begin processing the 175th record. run: 6209163 subRun: 1 event: 93775 at 05-Sep-2025 07:10:51 CEST
Begin processing the 176th record. run: 6209163 subRun: 1 event: 93776 at 05-Sep-2025 07:10:52 CEST
Begin processing the 177th record. run: 6209163 subRun: 1 event: 93777 at 05-Sep-2025 07:10:57 CEST
Begin processing the 178th record. run: 6209163 subRun: 1 event: 93778 at 05-Sep-2025 07:10:59 CEST
Begin processing the 179th record. run: 6209163 subRun: 1 event: 93779 at 05-Sep-2025 07:11:00 CEST
Begin processing the 180th record. run: 6209163 subRun: 1 event: 93780 at 05-Sep-2025 07:11:01 CEST
Begin processing the 181st record. run: 6209163 subRun: 1 event: 93781 at 05-Sep-2025 07:11:02 CEST
Begin processing the 182nd record. run: 6209163 subRun: 1 event: 93782 at 05-Sep-2025 07:11:03 CEST
Begin processing the 183rd record. run: 6209163 subRun: 1 event: 93783 at 05-Sep-2025 07:11:05 CEST
Begin processing the 184th record. run: 6209163 subRun: 1 event: 93784 at 05-Sep-2025 07:11:07 CEST
Begin processing the 185th record. run: 6209163 subRun: 1 event: 93785 at 05-Sep-2025 07:11:08 CEST
Begin processing the 186th record. run: 6209163 subRun: 1 event: 93786 at 05-Sep-2025 07:11:09 CEST
Begin processing the 187th record. run: 6209163 subRun: 1 event: 93787 at 05-Sep-2025 07:11:11 CEST
Begin processing the 188th record. run: 6209163 subRun: 1 event: 93788 at 05-Sep-2025 07:11:12 CEST
Begin processing the 189th record. run: 6209163 subRun: 1 event: 93789 at 05-Sep-2025 07:11:14 CEST
Begin processing the 190th record. run: 6209163 subRun: 1 event: 93790 at 05-Sep-2025 07:11:16 CEST
Begin processing the 191st record. run: 6209163 subRun: 1 event: 93791 at 05-Sep-2025 07:11:17 CEST
Begin processing the 192nd record. run: 6209163 subRun: 1 event: 93792 at 05-Sep-2025 07:11:18 CEST
Begin processing the 193rd record. run: 6209163 subRun: 1 event: 93793 at 05-Sep-2025 07:11:20 CEST
Begin processing the 194th record. run: 6209163 subRun: 1 event: 93794 at 05-Sep-2025 07:11:21 CEST
Begin processing the 195th record. run: 6209163 subRun: 1 event: 93795 at 05-Sep-2025 07:11:22 CEST
Begin processing the 196th record. run: 6209163 subRun: 1 event: 93796 at 05-Sep-2025 07:11:24 CEST
Begin processing the 197th record. run: 6209163 subRun: 1 event: 93797 at 05-Sep-2025 07:11:25 CEST
Begin processing the 198th record. run: 6209163 subRun: 1 event: 93798 at 05-Sep-2025 07:11:26 CEST
Begin processing the 199th record. run: 6209163 subRun: 1 event: 93799 at 05-Sep-2025 07:11:27 CEST
Begin processing the 200th record. run: 6209163 subRun: 1 event: 93800 at 05-Sep-2025 07:11:28 CEST
05-Sep-2025 07:11:30 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b2/54/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_468_20231123T021951Z_gen_g4_detsim_hitreco__20240507T175311Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.281663       1.16276       7.37038       1.03878      0.645573        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0147415     0.0406005      0.13728      0.0394781     0.0232717       200    
prod:pandorapid:Chi2ParticleID                 0.0857438     0.148905      0.334475      0.145158      0.0327945       200    
prod:energyreconue:EnergyReco                  0.154782       0.37411      0.969685      0.354656      0.0944872       200    
prod:energyreconumu:EnergyReco                0.00148682     0.0979114      2.6422       0.0643111     0.236672        200    
prod:energyreconc:EnergyReco                   0.0031113     0.0472464     0.639031      0.0369608     0.063489        200    
[art]:TriggerResults:TriggerResultInserter    1.1639e-05    1.56712e-05   0.000278179   1.3395e-05    1.9475e-05       200    
end_path:ana:PIDASelectionAna                  0.0176026     0.453536       2.74277      0.389894      0.286268        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8005.79 MB
  Peak resident set size usage (VmHWM): 1460.58 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 80
-rw-r--r--. 1 dune012 dune 29395 Sep  5 07:11 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_468_20231123T021951Z_gen_g4_detsim_hitreco__20240507T175311Z_reco2_pida_2025-09-05T_050616Z.root
-rw-r--r--. 1 dune012 dune 24218 Sep  5 07:11 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_468_20231123T021951Z_gen_g4_detsim_hitreco__20240507T175311Z_reco2_reco_2025-09-05T_050616Z.log
-rw-r--r--. 1 dune012 dune 12048 Sep  5 07:11 jobscript.log
-rw-r--r--. 1 dune012 dune   186 Sep  5 07:11 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune   137 Sep  5 07:06 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     0 Sep  5 07:06 debugprod.log