Jobsub ID 33414.132@dunegpschedd02.fnal.gov

Jobsub ID	33414.132@dunegpschedd02.fnal.gov
Workflow ID	1978
Stage ID	1
User name	hsouza@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-04 12:18:12
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2025-09-04 12:21:00
From worker node	Hostname	wn-choc-021.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	3145728000 (3000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-04 12:19:49
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_707_20231203T110318Z_gen_g4_detsim_hitreco__20240509T200045Z_reco2.root
Jobscript	Exit code	0
	Real time	0m (53s)
	CPU time	0m (47s = 88%)
	Max RSS bytes	1309499392 (1248 MiB)
Outputting started	2025-09-04 12:20:43
Output files
Finished	2025-09-04 12:21:00
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

. run: 50574782 subRun: 1 event: 70731 at 04-Sep-2025 14:20:23 CEST
Begin processing the 32nd record. run: 50574782 subRun: 1 event: 70732 at 04-Sep-2025 14:20:23 CEST
Begin processing the 33rd record. run: 50574782 subRun: 1 event: 70733 at 04-Sep-2025 14:20:23 CEST
Begin processing the 34th record. run: 50574782 subRun: 1 event: 70734 at 04-Sep-2025 14:20:23 CEST
Begin processing the 35th record. run: 50574782 subRun: 1 event: 70735 at 04-Sep-2025 14:20:23 CEST
Begin processing the 36th record. run: 50574782 subRun: 1 event: 70736 at 04-Sep-2025 14:20:25 CEST
Begin processing the 37th record. run: 50574782 subRun: 1 event: 70737 at 04-Sep-2025 14:20:25 CEST
Begin processing the 38th record. run: 50574782 subRun: 1 event: 70738 at 04-Sep-2025 14:20:25 CEST
Begin processing the 39th record. run: 50574782 subRun: 1 event: 70739 at 04-Sep-2025 14:20:25 CEST
Begin processing the 40th record. run: 50574782 subRun: 1 event: 70740 at 04-Sep-2025 14:20:26 CEST
Begin processing the 41st record. run: 50574782 subRun: 1 event: 70741 at 04-Sep-2025 14:20:26 CEST
Begin processing the 42nd record. run: 50574782 subRun: 1 event: 70742 at 04-Sep-2025 14:20:26 CEST
Begin processing the 43rd record. run: 50574782 subRun: 1 event: 70743 at 04-Sep-2025 14:20:26 CEST
Begin processing the 44th record. run: 50574782 subRun: 1 event: 70744 at 04-Sep-2025 14:20:26 CEST
Begin processing the 45th record. run: 50574782 subRun: 1 event: 70745 at 04-Sep-2025 14:20:26 CEST
Begin processing the 46th record. run: 50574782 subRun: 1 event: 70746 at 04-Sep-2025 14:20:26 CEST
Begin processing the 47th record. run: 50574782 subRun: 1 event: 70747 at 04-Sep-2025 14:20:26 CEST
Begin processing the 48th record. run: 50574782 subRun: 1 event: 70748 at 04-Sep-2025 14:20:26 CEST
Begin processing the 49th record. run: 50574782 subRun: 1 event: 70749 at 04-Sep-2025 14:20:26 CEST
Begin processing the 50th record. run: 50574782 subRun: 1 event: 70750 at 04-Sep-2025 14:20:26 CEST
Begin processing the 51st record. run: 50574782 subRun: 1 event: 70751 at 04-Sep-2025 14:20:27 CEST
Begin processing the 52nd record. run: 50574782 subRun: 1 event: 70752 at 04-Sep-2025 14:20:27 CEST
Begin processing the 53rd record. run: 50574782 subRun: 1 event: 70753 at 04-Sep-2025 14:20:27 CEST
Begin processing the 54th record. run: 50574782 subRun: 1 event: 70754 at 04-Sep-2025 14:20:27 CEST
Begin processing the 55th record. run: 50574782 subRun: 1 event: 70755 at 04-Sep-2025 14:20:27 CEST
Begin processing the 56th record. run: 50574782 subRun: 1 event: 70756 at 04-Sep-2025 14:20:27 CEST
Begin processing the 57th record. run: 50574782 subRun: 1 event: 70757 at 04-Sep-2025 14:20:27 CEST
Begin processing the 58th record. run: 50574782 subRun: 1 event: 70758 at 04-Sep-2025 14:20:27 CEST
Begin processing the 59th record. run: 50574782 subRun: 1 event: 70759 at 04-Sep-2025 14:20:29 CEST
Begin processing the 60th record. run: 50574782 subRun: 1 event: 70760 at 04-Sep-2025 14:20:29 CEST
Begin processing the 61st record. run: 50574782 subRun: 1 event: 70761 at 04-Sep-2025 14:20:29 CEST
Begin processing the 62nd record. run: 50574782 subRun: 1 event: 70762 at 04-Sep-2025 14:20:29 CEST
Begin processing the 63rd record. run: 50574782 subRun: 1 event: 70763 at 04-Sep-2025 14:20:30 CEST
Begin processing the 64th record. run: 50574782 subRun: 1 event: 70764 at 04-Sep-2025 14:20:30 CEST
Begin processing the 65th record. run: 50574782 subRun: 1 event: 70765 at 04-Sep-2025 14:20:30 CEST
Begin processing the 66th record. run: 50574782 subRun: 1 event: 70766 at 04-Sep-2025 14:20:30 CEST
Begin processing the 67th record. run: 50574782 subRun: 1 event: 70767 at 04-Sep-2025 14:20:32 CEST
Begin processing the 68th record. run: 50574782 subRun: 1 event: 70768 at 04-Sep-2025 14:20:32 CEST
Begin processing the 69th record. run: 50574782 subRun: 1 event: 70769 at 04-Sep-2025 14:20:32 CEST
Begin processing the 70th record. run: 50574782 subRun: 1 event: 70770 at 04-Sep-2025 14:20:32 CEST
Begin processing the 71st record. run: 50574782 subRun: 1 event: 70771 at 04-Sep-2025 14:20:32 CEST
Begin processing the 72nd record. run: 50574782 subRun: 1 event: 70772 at 04-Sep-2025 14:20:32 CEST
Begin processing the 73rd record. run: 50574782 subRun: 1 event: 70773 at 04-Sep-2025 14:20:32 CEST
Begin processing the 74th record. run: 50574782 subRun: 1 event: 70774 at 04-Sep-2025 14:20:32 CEST
Begin processing the 75th record. run: 50574782 subRun: 1 event: 70775 at 04-Sep-2025 14:20:32 CEST
Begin processing the 76th record. run: 50574782 subRun: 1 event: 70776 at 04-Sep-2025 14:20:32 CEST
Begin processing the 77th record. run: 50574782 subRun: 1 event: 70777 at 04-Sep-2025 14:20:33 CEST
Begin processing the 78th record. run: 50574782 subRun: 1 event: 70778 at 04-Sep-2025 14:20:33 CEST
Begin processing the 79th record. run: 50574782 subRun: 1 event: 70779 at 04-Sep-2025 14:20:33 CEST
Begin processing the 80th record. run: 50574782 subRun: 1 event: 70780 at 04-Sep-2025 14:20:33 CEST
Begin processing the 81st record. run: 50574782 subRun: 1 event: 70781 at 04-Sep-2025 14:20:33 CEST
Begin processing the 82nd record. run: 50574782 subRun: 1 event: 70782 at 04-Sep-2025 14:20:34 CEST
Begin processing the 83rd record. run: 50574782 subRun: 1 event: 70783 at 04-Sep-2025 14:20:34 CEST
Begin processing the 84th record. run: 50574782 subRun: 1 event: 70784 at 04-Sep-2025 14:20:34 CEST
Begin processing the 85th record. run: 50574782 subRun: 1 event: 70785 at 04-Sep-2025 14:20:34 CEST
Begin processing the 86th record. run: 50574782 subRun: 1 event: 70786 at 04-Sep-2025 14:20:34 CEST
Begin processing the 87th record. run: 50574782 subRun: 1 event: 70787 at 04-Sep-2025 14:20:34 CEST
Begin processing the 88th record. run: 50574782 subRun: 1 event: 70788 at 04-Sep-2025 14:20:34 CEST
Begin processing the 89th record. run: 50574782 subRun: 1 event: 70789 at 04-Sep-2025 14:20:34 CEST
Begin processing the 90th record. run: 50574782 subRun: 1 event: 70790 at 04-Sep-2025 14:20:34 CEST
Begin processing the 91st record. run: 50574782 subRun: 1 event: 70791 at 04-Sep-2025 14:20:34 CEST
Begin processing the 92nd record. run: 50574782 subRun: 1 event: 70792 at 04-Sep-2025 14:20:35 CEST
Begin processing the 93rd record. run: 50574782 subRun: 1 event: 70793 at 04-Sep-2025 14:20:35 CEST
Begin processing the 94th record. run: 50574782 subRun: 1 event: 70794 at 04-Sep-2025 14:20:35 CEST
Begin processing the 95th record. run: 50574782 subRun: 1 event: 70795 at 04-Sep-2025 14:20:41 CEST
Begin processing the 96th record. run: 50574782 subRun: 1 event: 70796 at 04-Sep-2025 14:20:42 CEST
Begin processing the 97th record. run: 50574782 subRun: 1 event: 70797 at 04-Sep-2025 14:20:42 CEST
Begin processing the 98th record. run: 50574782 subRun: 1 event: 70798 at 04-Sep-2025 14:20:42 CEST
Begin processing the 99th record. run: 50574782 subRun: 1 event: 70799 at 04-Sep-2025 14:20:42 CEST
Begin processing the 100th record. run: 50574782 subRun: 1 event: 70800 at 04-Sep-2025 14:20:42 CEST
04-Sep-2025 14:20:42 CEST  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/47/60/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_707_20231203T110318Z_gen_g4_detsim_hitreco__20240509T200045Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0243203     0.206507       5.06107      0.0471901     0.592008        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000494433   0.000772993   0.00168488    0.000738442   0.00023052       100    
prod:pandorapid:Chi2ParticleID                0.000979239   0.00365739     0.0291559     0.002168     0.00461308       100    
prod:energyreconue:EnergyReco                 0.00438674     0.0148577      0.26775     0.00707293     0.0301278       100    
prod:energyreconumu:EnergyReco                0.00191322     0.0968936      3.05656      0.0105592     0.370002        100    
prod:energyreconc:EnergyReco                  0.00331025     0.017171      0.253214      0.0116474     0.0282907       100    
[art]:TriggerResults:TriggerResultInserter    1.3174e-05    1.52232e-05   6.6229e-05    1.43145e-05   5.33242e-06      100    
end_path:ana:PIDASelectionAna                 0.00450463     0.072838       1.45327      0.0189782     0.200119        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5699.95 MB
  Peak resident set size usage (VmHWM): 1309.5 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 56
-rw-r--r--. 1 dune012 dune 20106 Sep  4 14:20 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_707_20231203T110318Z_gen_g4_detsim_hitreco__20240509T200045Z_reco2_pida_2025-09-04T_121953Z.root
-rw-r--r--. 1 dune012 dune 14410 Sep  4 14:20 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_707_20231203T110318Z_gen_g4_detsim_hitreco__20240509T200045Z_reco2_reco_2025-09-04T_121953Z.log
-rw-r--r--. 1 dune012 dune 12110 Sep  4 14:20 jobscript.log
-rw-r--r--. 1 dune012 dune   217 Sep  4 14:20 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune   138 Sep  4 14:19 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     0 Sep  4 14:20 debugprod.log