Jobsub ID 94908.162@dunegpschedd02.fnal.gov

Jobsub ID	94908.162@dunegpschedd02.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 08:38:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-23 10:59:23
From worker node	Hostname	wn60
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 08:42:08
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_162_20231207T035512Z_gen_g4_detsim_hitreco__20240510T043424Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_428_20231122T052559Z_gen_g4_detsim_hitreco__20240507T171029Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1095_20231204T080256Z_gen_g4_detsim_hitreco__20240509T214119Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_445_20231207T201930Z_gen_g4_detsim_hitreco__20240510T045409Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_415_20231202T120215Z_gen_g4_detsim_hitreco__20240507T212007Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_805_20231203T044825Z_gen_g4_detsim_hitreco__20240508T064929Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_588_20231208T071711Z_gen_g4_detsim_hitreco__20240510T055726Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_114_20231207T003350Z_gen_g4_detsim_hitreco__20240510T031915Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_819_20231205T032847Z_gen_g4_detsim_hitreco__20240510T020854Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_411_20231201T115929Z_gen_g4_detsim_hitreco__20240507T194331Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 10:59:23
Saved logs	justin-logs:94908.162-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.0529397      2.86936     0.00471474     0.215629        500    
reco:opdec:Deconvolution                               0.594811       1.05509       3.17169       1.02056       0.25853        500    
reco:ophitspe:OpHitFinderDeco                           1.76578       1.79674       1.96638       1.78141      0.032412        500    
reco:opflash:OpFlashFinder                            0.000250391   0.00199085    0.00796097    0.00190628    0.000924559      500    
reco:opslicer:OpSlicer                                9.5266e-05      0.36904       1.42625      0.292163      0.303413        500    
reco:rns:RandomNumberSaver                            2.7579e-05    7.78467e-05   0.00075074    6.1696e-05    9.33245e-05      500    
reco:anglereconue:NuAngularReco                       0.000231124   0.00277693     0.113615     0.000681996    0.0078769       500    
reco:anglereconumu:NuAngularReco                      0.000119524   0.00233345     0.110794     0.000418207   0.00760989       500    
reco:anglereconuepfps:NuAngularReco                   0.000209302   0.00252514     0.111685     0.000560562   0.00771247       500    
reco:anglereconumupfps:NuAngularReco                  0.000207343   0.00246205     0.111111     0.000521891   0.00769034       500    
reco:anglerecohits:NuAngularReco                      0.000174185   0.00347158     0.121045      0.001213     0.00856102       500    
[art]:TriggerResults:TriggerResultInserter            1.2357e-05    2.59906e-05   0.000623003   2.1947e-05    3.77914e-05      500    
end_path:cafmaker:CAFMaker                             0.0017512     0.0339998     0.836612     0.00830003     0.086176        500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3650 MB
  Peak resident set size usage (VmHWM): 1975.62 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 07:55:09 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/c5/32/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_415_20231202T120215Z_gen_g4_detsim_hitreco__20240507T212007Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 0: 0.359428, 0.00686342, 0.633709, 
Output 1: 0.0460855, 0.924421, 0.028411, 0.0010828, 
Output 2: 0.0871567, 0.850331, 0.0616495, 0.000862677, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2185!
Begin processing the 493rd record. run: 66050399 subRun: 1 event: 44593 at 23-Sep-2025 07:50:55 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 189!
Begin processing the 494th record. run: 66050399 subRun: 1 event: 44594 at 23-Sep-2025 07:51:05 -03
Boundary wire vector sizes: 244, 133, 167
minwire 0: 1519
minwire 1: 571
minwire 2: 1720
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0739383, 0.00894034, 0.917121, 
Output 1: 0.00418838, 0.0249175, 0.0325398, 0.938354, 
Output 2: 0.652857, 0.321184, 0.0230692, 0.0028895, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1782!
Begin processing the 495th record. run: 66050399 subRun: 1 event: 44595 at 23-Sep-2025 07:51:18 -03
Boundary wire vector sizes: 200, 562, 269
minwire 0: 349
minwire 1: 1970
minwire 2: 268
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996322, 0.00048361, 0.00319419, 
Output 1: 0.0046385, 0.991515, 0.0037721, 7.41278e-05, 
Output 2: 0.996396, 0.00352759, 7.44991e-05, 1.44959e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1308!
Begin processing the 496th record. run: 66050399 subRun: 1 event: 44596 at 23-Sep-2025 07:51:30 -03
Boundary wire vector sizes: 77, 68, 86
minwire 0: 1700
minwire 1: 1896
minwire 2: 1294
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00338362, 0.00268763, 0.993929, 
Output 1: 0.00614547, 0.98778, 0.0059028, 0.000171675, 
Output 2: 0.997588, 0.00235575, 5.4515e-05, 1.49671e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1838!
Begin processing the 497th record. run: 66050399 subRun: 1 event: 44597 at 23-Sep-2025 07:51:43 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 239!
Begin processing the 498th record. run: 66050399 subRun: 1 event: 44598 at 23-Sep-2025 07:51:52 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 703!
Begin processing the 499th record. run: 66050399 subRun: 1 event: 44599 at 23-Sep-2025 07:52:03 -03
Boundary wire vector sizes: 2457, 1528, 1379
minwire 0: 1752
minwire 1: 484
minwire 2: 1915
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000530929, 0.996569, 0.0028997, 
Output 1: 0.992608, 0.00715318, 0.000146219, 9.25662e-05, 
Output 2: 0.997777, 0.00215197, 5.41668e-05, 1.69796e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2523!
Begin processing the 500th record. run: 66050399 subRun: 1 event: 44600 at 23-Sep-2025 07:52:25 -03
Boundary wire vector sizes: 126, 172, 88
minwire 0: 1909
minwire 1: 1382
minwire 2: 1743
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00147703, 0.980382, 0.0181415, 
Output 1: 0.0540916, 0.892446, 0.0517836, 0.00167916, 
Output 2: 0.993634, 0.00619817, 0.000162372, 5.10792e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1817!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ca/d4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_162_20231207T035512Z_gen_g4_detsim_hitreco__20240510T043424Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5a/75/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_428_20231122T052559Z_gen_g4_detsim_hitreco__20240507T171029Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7e/ae/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1095_20231204T080256Z_gen_g4_detsim_hitreco__20240509T214119Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/76/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_445_20231207T201930Z_gen_g4_detsim_hitreco__20240510T045409Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/c5/32/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_415_20231202T120215Z_gen_g4_detsim_hitreco__20240507T212007Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ec/23/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_805_20231203T044825Z_gen_g4_detsim_hitreco__20240508T064929Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/59/12/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_588_20231208T071711Z_gen_g4_detsim_hitreco__20240510T055726Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/03/f4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_114_20231207T003350Z_gen_g4_detsim_hitreco__20240510T031915Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9e/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_819_20231205T032847Z_gen_g4_detsim_hitreco__20240510T020854Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ba/51/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_411_20231201T115929Z_gen_g4_detsim_hitreco__20240507T194331Z_reco2.root
.:
total 20888
-rw-r--r-- 1 nobody nobody 7903867 Sep 23 07:55 flatcaf.root
-rw-r--r-- 1 nobody nobody 6504941 Sep 23 07:55 caf.root
-rw-r--r-- 1 nobody nobody 6504941 Sep 23 07:55 caf_fd_hd_atmo_3302_20250923T084220Z.root
-rw-r--r-- 1 nobody nobody  302377 Sep 23 07:55 caf_20250923T084220Z.log
-rw-r--r-- 1 nobody nobody  122192 Sep 23 07:55 jobscript.log
-rw-r--r-- 1 nobody nobody    1928 Sep 23 07:55 caf_20250923T084220Z.file
-rw-r--r-- 1 nobody nobody    1928 Sep 23 07:55 caf_20250923T084220Z.pfns
-rw-r--r-- 1 nobody nobody    1928 Sep 23 05:42 file.list
-rw-r--r-- 1 nobody nobody    1928 Sep 23 07:55 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 05:42 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:55 caf_20250923T084220Z.did
-rw-r--r-- 1 nobody nobody    1376 Sep 23 05:42 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 05:42 debugprod.log