Jobsub ID 33407.80@dunegpschedd02.fnal.gov

Jobsub ID	33407.80@dunegpschedd02.fnal.gov
Workflow ID	1979
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-04 11:52:11
Site	NL_SURFsara
Entry	DUNE_SurfSARA_arc03
Last heartbeat	2025-09-04 11:58:45
From worker node	Hostname	wn-la-13.gina.surf.nl
	cpuinfo	AMD EPYC 9754 128-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-04 11:56:29
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_77_20231201T042314Z_gen_g4_detsim_hitreco__20240507T203519Z_reco2.root
Jobscript	Exit code	0
	Real time	1m (77s)
	CPU time	1m (65s = 84%)
	Max RSS bytes	1358766080 (1295 MiB)
Outputting started	2025-09-04 11:57:47
Output files
Finished	2025-09-04 11:58:45
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

3:57:26 CEST
Begin processing the 30th record. run: 6374820 subRun: 1 event: 7730 at 04-Sep-2025 13:57:26 CEST
Begin processing the 31st record. run: 6374820 subRun: 1 event: 7731 at 04-Sep-2025 13:57:26 CEST
Begin processing the 32nd record. run: 6374820 subRun: 1 event: 7732 at 04-Sep-2025 13:57:26 CEST
Begin processing the 33rd record. run: 6374820 subRun: 1 event: 7733 at 04-Sep-2025 13:57:27 CEST
Begin processing the 34th record. run: 6374820 subRun: 1 event: 7734 at 04-Sep-2025 13:57:27 CEST
Begin processing the 35th record. run: 6374820 subRun: 1 event: 7735 at 04-Sep-2025 13:57:27 CEST
Begin processing the 36th record. run: 6374820 subRun: 1 event: 7736 at 04-Sep-2025 13:57:27 CEST
Begin processing the 37th record. run: 6374820 subRun: 1 event: 7737 at 04-Sep-2025 13:57:28 CEST
Begin processing the 38th record. run: 6374820 subRun: 1 event: 7738 at 04-Sep-2025 13:57:28 CEST
Begin processing the 39th record. run: 6374820 subRun: 1 event: 7739 at 04-Sep-2025 13:57:28 CEST
Begin processing the 40th record. run: 6374820 subRun: 1 event: 7740 at 04-Sep-2025 13:57:28 CEST
Begin processing the 41st record. run: 6374820 subRun: 1 event: 7741 at 04-Sep-2025 13:57:29 CEST
Begin processing the 42nd record. run: 6374820 subRun: 1 event: 7742 at 04-Sep-2025 13:57:29 CEST
Begin processing the 43rd record. run: 6374820 subRun: 1 event: 7743 at 04-Sep-2025 13:57:29 CEST
Begin processing the 44th record. run: 6374820 subRun: 1 event: 7744 at 04-Sep-2025 13:57:30 CEST
Begin processing the 45th record. run: 6374820 subRun: 1 event: 7745 at 04-Sep-2025 13:57:31 CEST
Begin processing the 46th record. run: 6374820 subRun: 1 event: 7746 at 04-Sep-2025 13:57:31 CEST
Begin processing the 47th record. run: 6374820 subRun: 1 event: 7747 at 04-Sep-2025 13:57:31 CEST
Begin processing the 48th record. run: 6374820 subRun: 1 event: 7748 at 04-Sep-2025 13:57:31 CEST
Begin processing the 49th record. run: 6374820 subRun: 1 event: 7749 at 04-Sep-2025 13:57:32 CEST
Begin processing the 50th record. run: 6374820 subRun: 1 event: 7750 at 04-Sep-2025 13:57:32 CEST
Begin processing the 51st record. run: 6374820 subRun: 1 event: 7751 at 04-Sep-2025 13:57:32 CEST
Begin processing the 52nd record. run: 6374820 subRun: 1 event: 7752 at 04-Sep-2025 13:57:32 CEST
Begin processing the 53rd record. run: 6374820 subRun: 1 event: 7753 at 04-Sep-2025 13:57:33 CEST
Begin processing the 54th record. run: 6374820 subRun: 1 event: 7754 at 04-Sep-2025 13:57:34 CEST
Begin processing the 55th record. run: 6374820 subRun: 1 event: 7755 at 04-Sep-2025 13:57:34 CEST
Begin processing the 56th record. run: 6374820 subRun: 1 event: 7756 at 04-Sep-2025 13:57:34 CEST
Begin processing the 57th record. run: 6374820 subRun: 1 event: 7757 at 04-Sep-2025 13:57:35 CEST
Begin processing the 58th record. run: 6374820 subRun: 1 event: 7758 at 04-Sep-2025 13:57:35 CEST
Begin processing the 59th record. run: 6374820 subRun: 1 event: 7759 at 04-Sep-2025 13:57:35 CEST
Begin processing the 60th record. run: 6374820 subRun: 1 event: 7760 at 04-Sep-2025 13:57:37 CEST
Begin processing the 61st record. run: 6374820 subRun: 1 event: 7761 at 04-Sep-2025 13:57:37 CEST
Begin processing the 62nd record. run: 6374820 subRun: 1 event: 7762 at 04-Sep-2025 13:57:37 CEST
Begin processing the 63rd record. run: 6374820 subRun: 1 event: 7763 at 04-Sep-2025 13:57:37 CEST
Begin processing the 64th record. run: 6374820 subRun: 1 event: 7764 at 04-Sep-2025 13:57:37 CEST
Begin processing the 65th record. run: 6374820 subRun: 1 event: 7765 at 04-Sep-2025 13:57:37 CEST
Begin processing the 66th record. run: 6374820 subRun: 1 event: 7766 at 04-Sep-2025 13:57:38 CEST
Begin processing the 67th record. run: 6374820 subRun: 1 event: 7767 at 04-Sep-2025 13:57:38 CEST
Begin processing the 68th record. run: 6374820 subRun: 1 event: 7768 at 04-Sep-2025 13:57:38 CEST
Begin processing the 69th record. run: 6374820 subRun: 1 event: 7769 at 04-Sep-2025 13:57:38 CEST
Begin processing the 70th record. run: 6374820 subRun: 1 event: 7770 at 04-Sep-2025 13:57:39 CEST
Begin processing the 71st record. run: 6374820 subRun: 1 event: 7771 at 04-Sep-2025 13:57:39 CEST
Begin processing the 72nd record. run: 6374820 subRun: 1 event: 7772 at 04-Sep-2025 13:57:39 CEST
Begin processing the 73rd record. run: 6374820 subRun: 1 event: 7773 at 04-Sep-2025 13:57:39 CEST
Begin processing the 74th record. run: 6374820 subRun: 1 event: 7774 at 04-Sep-2025 13:57:39 CEST
Begin processing the 75th record. run: 6374820 subRun: 1 event: 7775 at 04-Sep-2025 13:57:40 CEST
Begin processing the 76th record. run: 6374820 subRun: 1 event: 7776 at 04-Sep-2025 13:57:40 CEST
Begin processing the 77th record. run: 6374820 subRun: 1 event: 7777 at 04-Sep-2025 13:57:40 CEST
Begin processing the 78th record. run: 6374820 subRun: 1 event: 7778 at 04-Sep-2025 13:57:40 CEST
Begin processing the 79th record. run: 6374820 subRun: 1 event: 7779 at 04-Sep-2025 13:57:40 CEST
Begin processing the 80th record. run: 6374820 subRun: 1 event: 7780 at 04-Sep-2025 13:57:41 CEST
Begin processing the 81st record. run: 6374820 subRun: 1 event: 7781 at 04-Sep-2025 13:57:41 CEST
Begin processing the 82nd record. run: 6374820 subRun: 1 event: 7782 at 04-Sep-2025 13:57:41 CEST
Begin processing the 83rd record. run: 6374820 subRun: 1 event: 7783 at 04-Sep-2025 13:57:41 CEST
Begin processing the 84th record. run: 6374820 subRun: 1 event: 7784 at 04-Sep-2025 13:57:42 CEST
Begin processing the 85th record. run: 6374820 subRun: 1 event: 7785 at 04-Sep-2025 13:57:42 CEST
Begin processing the 86th record. run: 6374820 subRun: 1 event: 7786 at 04-Sep-2025 13:57:42 CEST
Begin processing the 87th record. run: 6374820 subRun: 1 event: 7787 at 04-Sep-2025 13:57:42 CEST
Begin processing the 88th record. run: 6374820 subRun: 1 event: 7788 at 04-Sep-2025 13:57:42 CEST
Begin processing the 89th record. run: 6374820 subRun: 1 event: 7789 at 04-Sep-2025 13:57:42 CEST
Begin processing the 90th record. run: 6374820 subRun: 1 event: 7790 at 04-Sep-2025 13:57:43 CEST
Begin processing the 91st record. run: 6374820 subRun: 1 event: 7791 at 04-Sep-2025 13:57:43 CEST
Begin processing the 92nd record. run: 6374820 subRun: 1 event: 7792 at 04-Sep-2025 13:57:43 CEST
Begin processing the 93rd record. run: 6374820 subRun: 1 event: 7793 at 04-Sep-2025 13:57:44 CEST
Begin processing the 94th record. run: 6374820 subRun: 1 event: 7794 at 04-Sep-2025 13:57:44 CEST
Begin processing the 95th record. run: 6374820 subRun: 1 event: 7795 at 04-Sep-2025 13:57:44 CEST
Begin processing the 96th record. run: 6374820 subRun: 1 event: 7796 at 04-Sep-2025 13:57:44 CEST
Begin processing the 97th record. run: 6374820 subRun: 1 event: 7797 at 04-Sep-2025 13:57:45 CEST
Begin processing the 98th record. run: 6374820 subRun: 1 event: 7798 at 04-Sep-2025 13:57:45 CEST
Begin processing the 99th record. run: 6374820 subRun: 1 event: 7799 at 04-Sep-2025 13:57:46 CEST
Begin processing the 100th record. run: 6374820 subRun: 1 event: 7800 at 04-Sep-2025 13:57:46 CEST
04-Sep-2025 13:57:46 CEST  Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/d4/75/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_77_20231201T042314Z_gen_g4_detsim_hitreco__20240507T203519Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0488027     0.233213       2.4685       0.156157      0.297347        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00229805    0.00401215     0.0126785    0.00406697    0.00140065       100    
prod:pandorapid:Chi2ParticleID                 0.010434       0.01839      0.0389397     0.0172499    0.00512215       100    
prod:energyreconue:EnergyReco                  0.0174671     0.0413677     0.115109      0.0373541     0.0141193       100    
prod:energyreconumu:EnergyReco                0.00345155     0.0488577     0.730703      0.0202242     0.111684        100    
prod:energyreconc:EnergyReco                  0.00319124     0.0150097     0.087158      0.0135265     0.0133854       100    
[art]:TriggerResults:TriggerResultInserter    1.4262e-05    1.94999e-05   8.7903e-05    1.7867e-05    7.96271e-06      100    
end_path:ana:PIDASelectionAna                  0.0104405     0.104979       1.53465      0.0647217     0.170478        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18642.4 MB
  Peak resident set size usage (VmHWM): 1358.77 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 dune003 dune 20364 Sep  4 13:57 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_77_20231201T042314Z_gen_g4_detsim_hitreco__20240507T203519Z_reco2_pida_2025-09-04T_115642Z.root
-rw-r--r--. 1 dune003 dune 14211 Sep  4 13:57 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_77_20231201T042314Z_gen_g4_detsim_hitreco__20240507T203519Z_reco2_reco_2025-09-04T_115642Z.log
-rw-r--r--. 1 dune003 dune 11957 Sep  4 13:57 jobscript.log
-rw-r--r--. 1 dune003 dune   217 Sep  4 13:57 justin-processed-pfns.txt
-rw-r--r--. 1 dune003 dune   136 Sep  4 13:56 all-input-dids.txt
-rw-r--r--. 1 dune003 dune     0 Sep  4 13:56 debugprod.log