Jobsub ID 101241.2@dunegpschedd02.fnal.gov

Jobsub ID	101241.2@dunegpschedd02.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 10:03:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-23 10:36:36
From worker node	Hostname	wn77
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:04:57
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_233_20231201T071501Z_gen_g4_detsim_hitreco__20240507T192255Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_558_20231208T093226Z_gen_g4_detsim_hitreco__20240510T064313Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_43_20231204T042604Z_gen_g4_detsim_hitreco__20240509T205516Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_991_20231205T081707Z_gen_g4_detsim_hitreco__20240509T222156Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_1193_20231203T233207Z_gen_g4_detsim_hitreco__20240509T193548Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_359_20231204T064926Z_gen_g4_detsim_hitreco__20240509T205307Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_811_20231202T151556Z_gen_g4_detsim_hitreco__20240508T031226Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_684_20231207T093335Z_gen_g4_detsim_hitreco__20240510T040430Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_263_20231208T040402Z_gen_g4_detsim_hitreco__20240510T034745Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_78_20231208T053455Z_gen_g4_detsim_hitreco__20240510T060550Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 10:36:36
Saved logs	justin-logs:101241.2-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000730722    0.0491843     0.767944     0.00485325     0.123867        100    
reco:opdec:Deconvolution                               0.629943       1.03561       1.94486       1.00577      0.167268        100    
reco:ophitspe:OpHitFinderDeco                           1.77016       1.78126       1.90344       1.77874      0.0157457       100    
reco:opflash:OpFlashFinder                            0.000397887    0.0018663     0.0057153    0.00173325    0.000819754      100    
reco:opslicer:OpSlicer                                0.00330397     0.427664       1.42868      0.341002      0.315664        100    
reco:rns:RandomNumberSaver                            2.5491e-05    4.21071e-05   0.000514833   3.59115e-05   4.83217e-05      100    
reco:anglereconue:NuAngularReco                       0.000249621    0.0018784     0.0212267    0.000564865   0.00330182       100    
reco:anglereconumu:NuAngularReco                      0.000130505   0.00171843     0.0209919    0.000378412   0.00328548       100    
reco:anglereconuepfps:NuAngularReco                   0.000186488   0.00189497     0.0211059    0.000513997   0.00341269       100    
reco:anglereconumupfps:NuAngularReco                  0.000184358   0.00182771     0.0217748    0.000491727   0.00335192       100    
reco:anglerecohits:NuAngularReco                      0.000260842   0.00272911     0.0256179    0.00109132    0.00411021       100    
[art]:TriggerResults:TriggerResultInserter            1.1081e-05    1.88032e-05   7.3193e-05    1.7581e-05    7.63652e-06      100    
end_path:cafmaker:CAFMaker                             0.0016127     0.0241203     0.234121     0.00602452     0.0439315       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3668.27 MB
  Peak resident set size usage (VmHWM): 1984.79 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 07:32:23 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/12/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_558_20231208T093226Z_gen_g4_detsim_hitreco__20240510T064313Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1133!
Begin processing the 95th record. run: 65974923 subRun: 1 event: 23395 at 23-Sep-2025 07:27:19 -03
Boundary wire vector sizes: 262, 297, 201
minwire 0: 1297
minwire 1: 2177
minwire 2: 866
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.994926, 0.000568453, 0.00450555, 
Output 1: 0.00197381, 0.994447, 0.00351111, 6.79476e-05, 
Output 2: 0.995389, 0.0045219, 8.78712e-05, 1.67319e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2080!
Begin processing the 96th record. run: 65974923 subRun: 1 event: 23396 at 23-Sep-2025 07:27:30 -03
Boundary wire vector sizes: 236, 359, 256
minwire 0: 1571
minwire 1: 1089
minwire 2: 1225
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.956878, 0.00152427, 0.0415976, 
Output 1: 0.0111431, 0.176712, 0.749835, 0.0623094, 
Output 2: 0.874053, 0.119544, 0.00626193, 0.00014106, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2343!
Begin processing the 97th record. run: 65974923 subRun: 1 event: 23397 at 23-Sep-2025 07:27:42 -03
Boundary wire vector sizes: 206, 149, 171
minwire 0: 1371
minwire 1: 1755
minwire 2: 1092
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0070792, 0.162098, 0.830823, 
Output 1: 0.103412, 0.677375, 0.2128, 0.00641351, 
Output 2: 0.987812, 0.011786, 0.000388058, 1.44187e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2540!
Begin processing the 98th record. run: 65974923 subRun: 1 event: 23398 at 23-Sep-2025 07:27:53 -03
Boundary wire vector sizes: 146, 201, 152
minwire 0: 849
minwire 1: 1512
minwire 2: 559
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0906491, 0.0183544, 0.890997, 
Output 1: 0.230763, 0.724253, 0.0437219, 0.00126171, 
Output 2: 0.0379164, 0.332723, 0.563065, 0.0662957, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1882!
Begin processing the 99th record. run: 65974923 subRun: 1 event: 23399 at 23-Sep-2025 07:28:05 -03
Boundary wire vector sizes: 267, 228, 173
minwire 0: 2086
minwire 1: 268
minwire 2: 2233
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.123535, 0.257746, 0.618719, 
Output 1: 0.136867, 0.76155, 0.0970722, 0.00451066, 
Output 2: 0.66438, 0.293104, 0.0403585, 0.0021572, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1095!
Begin processing the 100th record. run: 65974923 subRun: 1 event: 23400 at 23-Sep-2025 07:28:15 -03
Boundary wire vector sizes: 482, 171, 349
minwire 0: 1374
minwire 1: 957
minwire 2: 1429
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.250426, 0.00277493, 0.746799, 
Output 1: 0.0367807, 0.557356, 0.38538, 0.0204832, 
Output 2: 0.300151, 0.677108, 0.0225386, 0.000202093, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2066!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/26/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_233_20231201T071501Z_gen_g4_detsim_hitreco__20240507T192255Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/12/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_558_20231208T093226Z_gen_g4_detsim_hitreco__20240510T064313Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/a3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_43_20231204T042604Z_gen_g4_detsim_hitreco__20240509T205516Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ea/f3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_991_20231205T081707Z_gen_g4_detsim_hitreco__20240509T222156Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/89/cd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_1193_20231203T233207Z_gen_g4_detsim_hitreco__20240509T193548Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d9/de/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_359_20231204T064926Z_gen_g4_detsim_hitreco__20240509T205307Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_811_20231202T151556Z_gen_g4_detsim_hitreco__20240508T031226Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d6/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_684_20231207T093335Z_gen_g4_detsim_hitreco__20240510T040430Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4b/2e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_263_20231208T040402Z_gen_g4_detsim_hitreco__20240510T034745Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ef/7d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_78_20231208T053455Z_gen_g4_detsim_hitreco__20240510T060550Z_reco2.root
.:
total 5028
-rw-r--r-- 1 nobody nobody 1966821 Sep 23 07:32 flatcaf.root
-rw-r--r-- 1 nobody nobody 1511986 Sep 23 07:32 caf.root
-rw-r--r-- 1 nobody nobody 1511986 Sep 23 07:32 caf_fd_hd_atmo_3301_20250923T100510Z.root
-rw-r--r-- 1 nobody nobody   75749 Sep 23 07:32 caf_20250923T100510Z.log
-rw-r--r-- 1 nobody nobody   40254 Sep 23 07:32 jobscript.log
-rw-r--r-- 1 nobody nobody    1897 Sep 23 07:32 caf_20250923T100510Z.file
-rw-r--r-- 1 nobody nobody    1897 Sep 23 07:32 caf_20250923T100510Z.pfns
-rw-r--r-- 1 nobody nobody    1897 Sep 23 07:05 file.list
-rw-r--r-- 1 nobody nobody    1897 Sep 23 07:32 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 23 07:05 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 23 07:32 caf_20250923T100510Z.did
-rw-r--r-- 1 nobody nobody    1375 Sep 23 07:05 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:05 debugprod.log