Jobsub ID 40095.183@dunegpschedd02.fnal.gov

Jobsub ID	40095.183@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 15:29:34
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce04
Last heartbeat	2025-09-15 17:03:36
From worker node	Hostname	wn124
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 15:30:32
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_474_20231206T230645Z_gen_g4_detsim_hitreco__20240510T063118Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_760_20231207T024722Z_gen_g4_detsim_hitreco__20240510T032001Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_538_20231205T070556Z_gen_g4_detsim_hitreco__20240510T020258Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_16_20231202T110437Z_gen_g4_detsim_hitreco__20240507T214630Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_487_20231208T055950Z_gen_g4_detsim_hitreco__20240510T050349Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_34_20231120T123816Z_gen_g4_detsim_hitreco__20240503T062057Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_567_20231204T221646Z_gen_g4_detsim_hitreco__20240509T221543Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_958_20231203T162112Z_gen_g4_detsim_hitreco__20240508T060746Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_466_20231208T055947Z_gen_g4_detsim_hitreco__20240510T050420Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 17:03:36
Saved logs	justin-logs:40095.183-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nge:EnergyReco                     0.0004179     0.0774561      9.3864      0.00416624     0.569981        300    
reco:energyrecnumumcs:EnergyReco                      0.00108663     0.0774173      9.02112     0.00529122     0.550773        300    
reco:energyrecnumumcsllhd:EnergyReco                  0.000930803    0.0762332      8.82788     0.00508989     0.539264        300    
reco:opdec:Deconvolution                               0.627932       1.12546       3.77874       1.06877      0.330109        300    
reco:ophitspe:OpHitFinderDeco                           1.78596       1.85923       2.94139       1.8502       0.0833359       300    
reco:opflash:OpFlashFinder                            0.000295523   0.00193224     0.0135808    0.00174843    0.00128385       300    
reco:opslicer:OpSlicer                                0.000235851    0.397971       1.64073      0.335383      0.311152        300    
reco:rns:RandomNumberSaver                            2.5671e-05    3.81514e-05   0.000370728   3.6412e-05    2.0638e-05       300    
reco:anglereconue:NuAngularReco                       0.000209289   0.00205774     0.0581669    0.000478547   0.00594896       300    
reco:anglereconumu:NuAngularReco                      0.000112135   0.00193095     0.0585287    0.000347121   0.00608808       300    
reco:anglereconuepfps:NuAngularReco                   0.000157137    0.0020444     0.0583168    0.000451676   0.00618672       300    
reco:anglereconumupfps:NuAngularReco                  0.000168787   0.00204765     0.0592991    0.00043725    0.00629064       300    
reco:anglerecohits:NuAngularReco                      0.000274252   0.00287984     0.0663984    0.00097198    0.00715719       300    
[art]:TriggerResults:TriggerResultInserter            1.0721e-05    1.81396e-05   6.0432e-05    1.76305e-05   5.68473e-06      300    
end_path:cafmaker:CAFMaker                            0.00137894     0.0268072     0.873566      0.0052102     0.0819384       300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3803.02 MB
  Peak resident set size usage (VmHWM): 2081.39 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 13:59:13 -03 ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0e/13/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_16_20231202T110437Z_gen_g4_detsim_hitreco__20240507T214630Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3505!
Begin processing the 295th record. run: 6430442 subRun: 1 event: 53895 at 15-Sep-2025 13:57:47 -03
Boundary wire vector sizes: 1644, 1016, 998
minwire 0: 2044
minwire 1: 393
minwire 2: 1940
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00485742, 0.85755, 0.137592, 
Output 1: 0.814675, 0.161728, 0.0190309, 0.00456601, 
Output 2: 0.457222, 0.502296, 0.0369403, 0.00354138, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2151!
Begin processing the 296th record. run: 6430442 subRun: 1 event: 53896 at 15-Sep-2025 13:58:04 -03
Boundary wire vector sizes: 69, 83, 50
minwire 0: 420
minwire 1: 2496
minwire 2: 147
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00105901, 0.98629, 0.0126509, 
Output 1: 0.987752, 0.0119309, 0.000199764, 0.000117599, 
Output 2: 0.998206, 0.00173622, 4.31152e-05, 1.43963e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 779!
Begin processing the 297th record. run: 6430442 subRun: 1 event: 53897 at 15-Sep-2025 13:58:15 -03
Boundary wire vector sizes: 359, 360, 237
minwire 0: 2223
minwire 1: 1377
minwire 2: 1900
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996956, 0.000577425, 0.00246663, 
Output 1: 0.472709, 0.428802, 0.0682105, 0.0302786, 
Output 2: 0.914253, 0.0792266, 0.00588931, 0.00063089, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2165!
Begin processing the 298th record. run: 6430442 subRun: 1 event: 53898 at 15-Sep-2025 13:58:26 -03
Boundary wire vector sizes: 36, 48, 42
minwire 0: 892
minwire 1: 2149
minwire 2: 673
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.991325, 0.000596005, 0.00807938, 
Output 1: 0.00537733, 0.989963, 0.00454668, 0.0001125, 
Output 2: 0.992119, 0.00771271, 0.00016525, 3.50693e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1137!
Begin processing the 299th record. run: 6430442 subRun: 1 event: 53899 at 15-Sep-2025 13:58:37 -03
Boundary wire vector sizes: 324, 388, 195
minwire 0: 1896
minwire 1: 1537
minwire 2: 1534
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00113581, 0.994619, 0.00424495, 
Output 1: 0.0535008, 0.285166, 0.499105, 0.162228, 
Output 2: 0.988731, 0.0106242, 0.000599682, 4.49812e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2633!
Begin processing the 300th record. run: 6430442 subRun: 1 event: 53900 at 15-Sep-2025 13:58:49 -03
Boundary wire vector sizes: 3044, 2555, 2297
minwire 0: 1286
minwire 1: 1211
minwire 2: 1001
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.958119, 0.0043804, 0.0375003, 
Output 1: 0.306423, 0.613642, 0.0610345, 0.0189003, 
Output 2: 0.102003, 0.366469, 0.350665, 0.180863, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3246!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e4/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_474_20231206T230645Z_gen_g4_detsim_hitreco__20240510T063118Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cb/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_760_20231207T024722Z_gen_g4_detsim_hitreco__20240510T032001Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/36/fd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_538_20231205T070556Z_gen_g4_detsim_hitreco__20240510T020258Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0e/13/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_16_20231202T110437Z_gen_g4_detsim_hitreco__20240507T214630Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7e/39/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_487_20231208T055950Z_gen_g4_detsim_hitreco__20240510T050349Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/dc/03/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_34_20231120T123816Z_gen_g4_detsim_hitreco__20240503T062057Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9e/54/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_567_20231204T221646Z_gen_g4_detsim_hitreco__20240509T221543Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f8/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_958_20231203T162112Z_gen_g4_detsim_hitreco__20240508T060746Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_466_20231208T055947Z_gen_g4_detsim_hitreco__20240510T050420Z_reco2.root
.:
total 13260
-rw-r--r-- 1 nobody nobody 5031935 Sep 15 13:59 flatcaf.root
-rw-r--r-- 1 nobody nobody 4091589 Sep 15 13:59 caf.root
-rw-r--r-- 1 nobody nobody 4091589 Sep 15 13:59 caf_fd_hd_atmo_2501_20250915T153048Z.root
-rw-r--r-- 1 nobody nobody  191808 Sep 15 13:59 caf_20250915T153048Z.log
-rw-r--r-- 1 nobody nobody   80808 Sep 15 13:59 jobscript.log
-rw-r--r-- 1 nobody nobody    1675 Sep 15 13:59 caf_20250915T153048Z.file
-rw-r--r-- 1 nobody nobody    1675 Sep 15 13:59 caf_20250915T153048Z.pfns
-rw-r--r-- 1 nobody nobody    1675 Sep 15 12:30 file.list
-rw-r--r-- 1 nobody nobody    1675 Sep 15 13:59 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1234 Sep 15 12:30 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1234 Sep 15 13:59 caf_20250915T153048Z.did
-rw-r--r-- 1 nobody nobody    1234 Sep 15 12:30 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 15 12:31 debugprod.log