Jobsub ID 40708.37@dunegpschedd01.fnal.gov

Jobsub ID	40708.37@dunegpschedd01.fnal.gov
Workflow ID	1980
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-05 06:25:09
Site	CH_UNIBE-LHEP
Entry	OSG_CH_UNIBE_LHEP_ce02
Last heartbeat	2025-09-05 06:32:37
From worker node	Hostname	wn-1-10.local
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-05 06:25:52
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_468_20231123T021951Z_gen_g4_detsim_hitreco__20240507T175311Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (382s)
	CPU time	1m (63s = 16%)
	Max RSS bytes	1255972864 (1197 MiB)
Outputting started	2025-09-05 06:32:15
Output files
Finished	2025-09-05 06:32:37
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

2 CEST
Begin processing the 131st record. run: 6209163 subRun: 1 event: 93731 at 05-Sep-2025 08:30:03 CEST
Begin processing the 132nd record. run: 6209163 subRun: 1 event: 93732 at 05-Sep-2025 08:30:04 CEST
Begin processing the 133rd record. run: 6209163 subRun: 1 event: 93733 at 05-Sep-2025 08:30:06 CEST
Begin processing the 134th record. run: 6209163 subRun: 1 event: 93734 at 05-Sep-2025 08:30:08 CEST
Begin processing the 135th record. run: 6209163 subRun: 1 event: 93735 at 05-Sep-2025 08:30:10 CEST
Begin processing the 136th record. run: 6209163 subRun: 1 event: 93736 at 05-Sep-2025 08:30:11 CEST
Begin processing the 137th record. run: 6209163 subRun: 1 event: 93737 at 05-Sep-2025 08:30:12 CEST
Begin processing the 138th record. run: 6209163 subRun: 1 event: 93738 at 05-Sep-2025 08:30:15 CEST
Begin processing the 139th record. run: 6209163 subRun: 1 event: 93739 at 05-Sep-2025 08:30:16 CEST
Begin processing the 140th record. run: 6209163 subRun: 1 event: 93740 at 05-Sep-2025 08:30:20 CEST
Begin processing the 141st record. run: 6209163 subRun: 1 event: 93741 at 05-Sep-2025 08:30:21 CEST
Begin processing the 142nd record. run: 6209163 subRun: 1 event: 93742 at 05-Sep-2025 08:30:22 CEST
Begin processing the 143rd record. run: 6209163 subRun: 1 event: 93743 at 05-Sep-2025 08:30:24 CEST
Begin processing the 144th record. run: 6209163 subRun: 1 event: 93744 at 05-Sep-2025 08:30:25 CEST
Begin processing the 145th record. run: 6209163 subRun: 1 event: 93745 at 05-Sep-2025 08:30:27 CEST
Begin processing the 146th record. run: 6209163 subRun: 1 event: 93746 at 05-Sep-2025 08:30:29 CEST
Begin processing the 147th record. run: 6209163 subRun: 1 event: 93747 at 05-Sep-2025 08:30:31 CEST
Begin processing the 148th record. run: 6209163 subRun: 1 event: 93748 at 05-Sep-2025 08:30:32 CEST
Begin processing the 149th record. run: 6209163 subRun: 1 event: 93749 at 05-Sep-2025 08:30:34 CEST
Begin processing the 150th record. run: 6209163 subRun: 1 event: 93750 at 05-Sep-2025 08:30:35 CEST
Begin processing the 151st record. run: 6209163 subRun: 1 event: 93751 at 05-Sep-2025 08:30:36 CEST
Begin processing the 152nd record. run: 6209163 subRun: 1 event: 93752 at 05-Sep-2025 08:30:38 CEST
Begin processing the 153rd record. run: 6209163 subRun: 1 event: 93753 at 05-Sep-2025 08:30:40 CEST
Begin processing the 154th record. run: 6209163 subRun: 1 event: 93754 at 05-Sep-2025 08:30:41 CEST
Begin processing the 155th record. run: 6209163 subRun: 1 event: 93755 at 05-Sep-2025 08:30:43 CEST
Begin processing the 156th record. run: 6209163 subRun: 1 event: 93756 at 05-Sep-2025 08:30:44 CEST
Begin processing the 157th record. run: 6209163 subRun: 1 event: 93757 at 05-Sep-2025 08:30:45 CEST
Begin processing the 158th record. run: 6209163 subRun: 1 event: 93758 at 05-Sep-2025 08:30:47 CEST
Begin processing the 159th record. run: 6209163 subRun: 1 event: 93759 at 05-Sep-2025 08:30:48 CEST
Begin processing the 160th record. run: 6209163 subRun: 1 event: 93760 at 05-Sep-2025 08:30:49 CEST
Begin processing the 161st record. run: 6209163 subRun: 1 event: 93761 at 05-Sep-2025 08:30:51 CEST
Begin processing the 162nd record. run: 6209163 subRun: 1 event: 93762 at 05-Sep-2025 08:30:52 CEST
Begin processing the 163rd record. run: 6209163 subRun: 1 event: 93763 at 05-Sep-2025 08:30:53 CEST
Begin processing the 164th record. run: 6209163 subRun: 1 event: 93764 at 05-Sep-2025 08:30:55 CEST
Begin processing the 165th record. run: 6209163 subRun: 1 event: 93765 at 05-Sep-2025 08:30:57 CEST
Begin processing the 166th record. run: 6209163 subRun: 1 event: 93766 at 05-Sep-2025 08:30:59 CEST
Begin processing the 167th record. run: 6209163 subRun: 1 event: 93767 at 05-Sep-2025 08:31:00 CEST
Begin processing the 168th record. run: 6209163 subRun: 1 event: 93768 at 05-Sep-2025 08:31:02 CEST
Begin processing the 169th record. run: 6209163 subRun: 1 event: 93769 at 05-Sep-2025 08:31:07 CEST
Begin processing the 170th record. run: 6209163 subRun: 1 event: 93770 at 05-Sep-2025 08:31:20 CEST
Begin processing the 171st record. run: 6209163 subRun: 1 event: 93771 at 05-Sep-2025 08:31:21 CEST
Begin processing the 172nd record. run: 6209163 subRun: 1 event: 93772 at 05-Sep-2025 08:31:22 CEST
Begin processing the 173rd record. run: 6209163 subRun: 1 event: 93773 at 05-Sep-2025 08:31:23 CEST
Begin processing the 174th record. run: 6209163 subRun: 1 event: 93774 at 05-Sep-2025 08:31:25 CEST
Begin processing the 175th record. run: 6209163 subRun: 1 event: 93775 at 05-Sep-2025 08:31:27 CEST
Begin processing the 176th record. run: 6209163 subRun: 1 event: 93776 at 05-Sep-2025 08:31:28 CEST
Begin processing the 177th record. run: 6209163 subRun: 1 event: 93777 at 05-Sep-2025 08:31:35 CEST
Begin processing the 178th record. run: 6209163 subRun: 1 event: 93778 at 05-Sep-2025 08:31:36 CEST
Begin processing the 179th record. run: 6209163 subRun: 1 event: 93779 at 05-Sep-2025 08:31:38 CEST
Begin processing the 180th record. run: 6209163 subRun: 1 event: 93780 at 05-Sep-2025 08:31:39 CEST
Begin processing the 181st record. run: 6209163 subRun: 1 event: 93781 at 05-Sep-2025 08:31:40 CEST
Begin processing the 182nd record. run: 6209163 subRun: 1 event: 93782 at 05-Sep-2025 08:31:42 CEST
Begin processing the 183rd record. run: 6209163 subRun: 1 event: 93783 at 05-Sep-2025 08:31:44 CEST
Begin processing the 184th record. run: 6209163 subRun: 1 event: 93784 at 05-Sep-2025 08:31:46 CEST
Begin processing the 185th record. run: 6209163 subRun: 1 event: 93785 at 05-Sep-2025 08:31:48 CEST
Begin processing the 186th record. run: 6209163 subRun: 1 event: 93786 at 05-Sep-2025 08:31:49 CEST
Begin processing the 187th record. run: 6209163 subRun: 1 event: 93787 at 05-Sep-2025 08:31:51 CEST
Begin processing the 188th record. run: 6209163 subRun: 1 event: 93788 at 05-Sep-2025 08:31:53 CEST
Begin processing the 189th record. run: 6209163 subRun: 1 event: 93789 at 05-Sep-2025 08:31:54 CEST
Begin processing the 190th record. run: 6209163 subRun: 1 event: 93790 at 05-Sep-2025 08:31:57 CEST
Begin processing the 191st record. run: 6209163 subRun: 1 event: 93791 at 05-Sep-2025 08:31:58 CEST
Begin processing the 192nd record. run: 6209163 subRun: 1 event: 93792 at 05-Sep-2025 08:32:00 CEST
Begin processing the 193rd record. run: 6209163 subRun: 1 event: 93793 at 05-Sep-2025 08:32:01 CEST
Begin processing the 194th record. run: 6209163 subRun: 1 event: 93794 at 05-Sep-2025 08:32:03 CEST
Begin processing the 195th record. run: 6209163 subRun: 1 event: 93795 at 05-Sep-2025 08:32:04 CEST
Begin processing the 196th record. run: 6209163 subRun: 1 event: 93796 at 05-Sep-2025 08:32:06 CEST
Begin processing the 197th record. run: 6209163 subRun: 1 event: 93797 at 05-Sep-2025 08:32:07 CEST
Begin processing the 198th record. run: 6209163 subRun: 1 event: 93798 at 05-Sep-2025 08:32:09 CEST
Begin processing the 199th record. run: 6209163 subRun: 1 event: 93799 at 05-Sep-2025 08:32:10 CEST
Begin processing the 200th record. run: 6209163 subRun: 1 event: 93800 at 05-Sep-2025 08:32:11 CEST
05-Sep-2025 08:32:14 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b2/54/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_468_20231123T021951Z_gen_g4_detsim_hitreco__20240507T175311Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.339965       1.3513        8.24236       1.16541      0.824832        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0179933     0.0367963     0.112024      0.0363572     0.0136934       200    
prod:pandorapid:Chi2ParticleID                  0.10555      0.166827      0.322839      0.168875      0.0260016       200    
prod:energyreconue:EnergyReco                  0.187514      0.424689       1.05964      0.392378      0.0937591       200    
prod:energyreconumu:EnergyReco                0.00152394     0.101003       2.04311      0.0718933     0.194807        200    
prod:energyreconc:EnergyReco                  0.00291467     0.0468446     0.648939      0.0319674     0.0728837       200    
[art]:TriggerResults:TriggerResultInserter     9.917e-06    1.50477e-05   4.5136e-05    1.3541e-05    5.16973e-06      200    
end_path:ana:PIDASelectionAna                  0.020841      0.574793       4.35966      0.462107      0.503086        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8006.73 MB
  Peak resident set size usage (VmHWM): 1255.97 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 69
-rw-r--r-- 1 dune002 dune 29395 Sep  5 08:32 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_468_20231123T021951Z_gen_g4_detsim_hitreco__20240507T175311Z_reco2_pida_2025-09-05T_062556Z.root
-rw-r--r-- 1 dune002 dune 24218 Sep  5 08:32 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_468_20231123T021951Z_gen_g4_detsim_hitreco__20240507T175311Z_reco2_reco_2025-09-05T_062556Z.log
-rw-r--r-- 1 dune002 dune 12048 Sep  5 08:32 jobscript.log
-rw-r--r-- 1 dune002 dune   186 Sep  5 08:32 justin-processed-pfns.txt
-rw-r--r-- 1 dune002 dune   137 Sep  5 08:25 all-input-dids.txt
-rw-r--r-- 1 dune002 dune     0 Sep  5 08:26 debugprod.log