Jobsub ID 106434.26@dunegpschedd01.fnal.gov

Jobsub ID	106434.26@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:57:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-23 11:11:47
From worker node	Hostname	wn38
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:03:45
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_273_20231122T115805Z_gen_g4_detsim_hitreco__20240503T060236Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_483_20231202T100138Z_gen_g4_detsim_hitreco__20240507T214403Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_518_20231203T223231Z_gen_g4_detsim_hitreco__20240509T215444Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_672_20231202T203857Z_gen_g4_detsim_hitreco__20240508T070048Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_1103_20231204T031509Z_gen_g4_detsim_hitreco__20240509T212520Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_443_20231201T150203Z_gen_g4_detsim_hitreco__20240507T203511Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_94_20231202T150347Z_gen_g4_detsim_hitreco__20240508T033205Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_286_20231207T050043Z_gen_g4_detsim_hitreco__20240510T051351Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_609_20231201T170338Z_gen_g4_detsim_hitreco__20240507T201405Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_1044_20231204T010257Z_gen_g4_detsim_hitreco__20240509T214314Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 11:11:47
Saved logs	justin-logs:106434.26-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

09        100    
reco:ophitspe:OpHitFinderDeco                           1.79366       1.82454       1.98151       1.81929      0.0221269       100    
reco:opflash:OpFlashFinder                            0.00181247    0.00836682     0.107742     0.00520006     0.011981        100    
reco:opslicer:OpSlicer                                0.000979238    0.425041       1.59097       0.35037      0.370867        100    
reco:rns:RandomNumberSaver                            4.1134e-05    0.000233757   0.00937682    0.000106198   0.000931035      100    
reco:anglereconue:NuAngularReco                       0.00143034    0.00982283     0.164791     0.00364198     0.0216036       100    
reco:anglereconumu:NuAngularReco                      0.00016988    0.00525799     0.162771     0.000650306    0.0198834       100    
reco:anglereconuepfps:NuAngularReco                   0.000853828   0.00841484     0.169144     0.00254769     0.0223317       100    
reco:anglereconumupfps:NuAngularReco                  0.000239615   0.00643303     0.165554     0.000844329    0.0207975       100    
reco:anglerecohits:NuAngularReco                      0.000405784    0.0113893     0.250101     0.00343067     0.0318297       100    
[art]:TriggerResults:TriggerResultInserter            1.8133e-05    3.64402e-05   0.000234222   2.35525e-05   3.9363e-05       100    
end_path:cafmaker:CAFMaker                            0.00793412     0.175212       6.5914       0.0249923     0.746124        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3818.38 MB
  Peak resident set size usage (VmHWM): 2143.06 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 08:04:29 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/4c/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_483_20231202T100138Z_gen_g4_detsim_hitreco__20240507T214403Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Begin processing the 92nd record. run: 50261805 subRun: 1 event: 27392 at 23-Sep-2025 07:58:02 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 306!
Begin processing the 93rd record. run: 50261805 subRun: 1 event: 27393 at 23-Sep-2025 07:58:14 -03
Boundary wire vector sizes: 618, 506, 344
minwire 0: 1565
minwire 1: 1716
minwire 2: 1221
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00148767, 0.994251, 0.00426085, 
Output 1: 0.0123366, 0.175724, 0.739007, 0.0729327, 
Output 2: 0.950186, 0.0470364, 0.0026981, 7.94698e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2754!
Begin processing the 94th record. run: 50261805 subRun: 1 event: 27394 at 23-Sep-2025 07:58:30 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 158!
Begin processing the 95th record. run: 50261805 subRun: 1 event: 27395 at 23-Sep-2025 07:58:41 -03
Boundary wire vector sizes: 629, 850, 573
minwire 0: 2007
minwire 1: 57
minwire 2: 2174
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0495955, 0.120278, 0.830127, 
Output 1: 0.0733204, 0.890101, 0.0323091, 0.00426921, 
Output 2: 0.0375052, 0.95266, 0.00978803, 4.68e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1679!
Begin processing the 96th record. run: 50261805 subRun: 1 event: 27396 at 23-Sep-2025 07:58:57 -03
Boundary wire vector sizes: 717, 650, 416
minwire 0: 122
minwire 1: 2113
minwire 2: 132
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996497, 0.000884707, 0.00261854, 
Output 1: 0.310566, 0.534823, 0.106675, 0.0479364, 
Output 2: 0.86244, 0.122441, 0.0134597, 0.00165965, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1249!
Begin processing the 97th record. run: 50261805 subRun: 1 event: 27397 at 23-Sep-2025 07:59:13 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 188!
Begin processing the 98th record. run: 50261805 subRun: 1 event: 27398 at 23-Sep-2025 07:59:24 -03
Boundary wire vector sizes: 71, 72, 60
minwire 0: 518
minwire 1: 2175
minwire 2: 4
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.817959, 0.00154615, 0.180495, 
Output 1: 0.0170758, 0.267572, 0.66487, 0.0504822, 
Output 2: 0.78866, 0.201737, 0.009402, 0.000201647, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1892!
Begin processing the 99th record. run: 50261805 subRun: 1 event: 27399 at 23-Sep-2025 07:59:37 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 382!
Begin processing the 100th record. run: 50261805 subRun: 1 event: 27400 at 23-Sep-2025 07:59:48 -03
Boundary wire vector sizes: 62, 69, 62
minwire 0: 1703
minwire 1: 347
minwire 2: 1793
Used alternate method to get min and max wires due to vertex determination failure: 505, 804
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 606, 905
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00237733, 0.00177148, 0.995851, 
Output 1: 0.920946, 0.0785338, 0.000433393, 8.64948e-05, 
Output 2: 0.999101, 0.000875447, 2.08049e-05, 2.8716e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1020!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0f/29/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_273_20231122T115805Z_gen_g4_detsim_hitreco__20240503T060236Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/4c/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_483_20231202T100138Z_gen_g4_detsim_hitreco__20240507T214403Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/76/59/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_518_20231203T223231Z_gen_g4_detsim_hitreco__20240509T215444Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bb/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_672_20231202T203857Z_gen_g4_detsim_hitreco__20240508T070048Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/da/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_1103_20231204T031509Z_gen_g4_detsim_hitreco__20240509T212520Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/49/60/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_443_20231201T150203Z_gen_g4_detsim_hitreco__20240507T203511Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9d/43/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_94_20231202T150347Z_gen_g4_detsim_hitreco__20240508T033205Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/52/71/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_286_20231207T050043Z_gen_g4_detsim_hitreco__20240510T051351Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d2/87/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_609_20231201T170338Z_gen_g4_detsim_hitreco__20240507T201405Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/9c/7a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_1044_20231204T010257Z_gen_g4_detsim_hitreco__20240509T214314Z_reco2.root
.:
total 5508
-rw-r--r-- 1 nobody nobody 2153619 Sep 23 08:04 flatcaf.root
-rw-r--r-- 1 nobody nobody 1658929 Sep 23 08:04 caf.root
-rw-r--r-- 1 nobody nobody 1658929 Sep 23 08:04 caf_fd_hd_atmo_3302_20250923T100500Z.root
-rw-r--r-- 1 nobody nobody   70501 Sep 23 08:04 caf_20250923T100500Z.log
-rw-r--r-- 1 nobody nobody   43770 Sep 23 08:04 jobscript.log
-rw-r--r-- 1 nobody nobody    1930 Sep 23 08:04 caf_20250923T100500Z.file
-rw-r--r-- 1 nobody nobody    1930 Sep 23 08:04 caf_20250923T100500Z.pfns
-rw-r--r-- 1 nobody nobody    1930 Sep 23 07:05 file.list
-rw-r--r-- 1 nobody nobody    1930 Sep 23 08:04 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:05 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 08:04 caf_20250923T100500Z.did
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:05 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:05 debugprod.log