Jobsub ID 107917.48@dunegpschedd01.fnal.gov

Jobsub ID	107917.48@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 10:17:39
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce04
Last heartbeat	2025-09-23 10:52:33
From worker node	Hostname	wn123
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:20:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_516_20231208T024659Z_gen_g4_detsim_hitreco__20240510T040626Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_141_20231202T150353Z_gen_g4_detsim_hitreco__20240507T223826Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_555_20231207T045503Z_gen_g4_detsim_hitreco__20240510T053848Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_416_20231121T150256Z_gen_g4_detsim_hitreco__20240507T181353Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_329_20231123T011848Z_gen_g4_detsim_hitreco__20240507T175035Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_436_20231203T040553Z_gen_g4_detsim_hitreco__20240508T072706Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_261_20231208T045044Z_gen_g4_detsim_hitreco__20240510T065351Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_843_20231205T071219Z_gen_g4_detsim_hitreco__20240509T222109Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_379_20231202T201219Z_gen_g4_detsim_hitreco__20240508T065736Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_514_20231204T071317Z_gen_g4_detsim_hitreco__20240509T205838Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 10:52:33
Saved logs	justin-logs:107917.48-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

1.01345      0.177755        100    
reco:ophitspe:OpHitFinderDeco                           1.7642        1.77336       1.78284       1.77348     0.00375852       100    
reco:opflash:OpFlashFinder                            0.000256753   0.00181469    0.00563577    0.00176552    0.000832428      100    
reco:opslicer:OpSlicer                                 0.0004184     0.424327       1.47617       0.38124       0.30571        100    
reco:rns:RandomNumberSaver                            2.4871e-05    4.02898e-05   0.000442482   3.53815e-05   4.11362e-05      100    
reco:anglereconue:NuAngularReco                       0.00021149    0.00167356     0.0200979    0.000454003   0.00337718       100    
reco:anglereconumu:NuAngularReco                      8.2024e-05    0.00151519     0.0192008    0.000338776   0.00328851       100    
reco:anglereconuepfps:NuAngularReco                   0.000117656   0.00166576     0.020073     0.000438162   0.00337353       100    
reco:anglereconumupfps:NuAngularReco                  0.000108215   0.00162258     0.0202263    0.000426691   0.00339025       100    
reco:anglerecohits:NuAngularReco                      0.000362008   0.00249075     0.0277203    0.000932415   0.00423596       100    
[art]:TriggerResults:TriggerResultInserter             1.026e-05    1.71134e-05   5.8433e-05    1.68655e-05   4.53407e-06      100    
end_path:cafmaker:CAFMaker                            0.00124946     0.022122      0.269818     0.00507708     0.0453533       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3651.01 MB
  Peak resident set size usage (VmHWM): 1968.66 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 07:48:06 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/bd/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_141_20231202T150353Z_gen_g4_detsim_hitreco__20240507T223826Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1505!
Begin processing the 95th record. run: 74574126 subRun: 1 event: 51695 at 23-Sep-2025 07:43:53 -03
Boundary wire vector sizes: 369, 146, 250
minwire 0: 1171
minwire 1: 988
minwire 2: 1403
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.62537, 0.00650284, 0.368127, 
Output 1: 0.0477825, 0.874252, 0.0757868, 0.00217819, 
Output 2: 0.0710715, 0.798792, 0.127659, 0.00247771, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1566!
Begin processing the 96th record. run: 74574126 subRun: 1 event: 51696 at 23-Sep-2025 07:44:04 -03
Boundary wire vector sizes: 740, 777, 425
minwire 0: 1060
minwire 1: 1084
minwire 2: 1078
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.760025, 0.228315, 0.0116601, 
Output 1: 0.583105, 0.410608, 0.00594437, 0.000342317, 
Output 2: 0.383118, 0.591887, 0.0241936, 0.000801178, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2267!
Begin processing the 97th record. run: 74574126 subRun: 1 event: 51697 at 23-Sep-2025 07:44:16 -03
Boundary wire vector sizes: 64, 155, 123
minwire 0: 1769
minwire 1: 1439
minwire 2: 1371
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.925029, 0.0351734, 0.0397976, 
Output 1: 0.997246, 0.00238676, 0.000191708, 0.000175082, 
Output 2: 0.984935, 0.014667, 0.000309539, 8.80989e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1576!
Begin processing the 98th record. run: 74574126 subRun: 1 event: 51698 at 23-Sep-2025 07:44:27 -03
Boundary wire vector sizes: 182, 125, 99
minwire 0: 1720
minwire 1: 724
minwire 2: 1982
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00186858, 0.919619, 0.0785122, 
Output 1: 0.975339, 0.0237517, 0.000772574, 0.000137009, 
Output 2: 0.996104, 0.00376772, 0.000104893, 2.33528e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1184!
Begin processing the 99th record. run: 74574126 subRun: 1 event: 51699 at 23-Sep-2025 07:44:37 -03
Boundary wire vector sizes: 650, 932, 617
minwire 0: 646
minwire 1: 1837
minwire 2: 956
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000263038, 0.998006, 0.00173108, 
Output 1: 0.834185, 0.164958, 0.000804358, 5.26319e-05, 
Output 2: 0.998794, 0.00118074, 2.34245e-05, 1.50161e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2794!
Begin processing the 100th record. run: 74574126 subRun: 1 event: 51700 at 23-Sep-2025 07:44:50 -03
Boundary wire vector sizes: 54, 62, 47
minwire 0: 1822
minwire 1: 1827
minwire 2: 1425
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.846993, 0.00313209, 0.149875, 
Output 1: 0.00478714, 0.990879, 0.00415232, 0.000182048, 
Output 2: 0.848, 0.148425, 0.00352337, 5.15275e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1880!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e7/1d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_516_20231208T024659Z_gen_g4_detsim_hitreco__20240510T040626Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/bd/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_141_20231202T150353Z_gen_g4_detsim_hitreco__20240507T223826Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b4/2b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_555_20231207T045503Z_gen_g4_detsim_hitreco__20240510T053848Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/49/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_416_20231121T150256Z_gen_g4_detsim_hitreco__20240507T181353Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/89/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_329_20231123T011848Z_gen_g4_detsim_hitreco__20240507T175035Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a5/1f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_436_20231203T040553Z_gen_g4_detsim_hitreco__20240508T072706Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/a3/6e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_261_20231208T045044Z_gen_g4_detsim_hitreco__20240510T065351Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6d/aa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_843_20231205T071219Z_gen_g4_detsim_hitreco__20240509T222109Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b6/4a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_379_20231202T201219Z_gen_g4_detsim_hitreco__20240508T065736Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/37/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_514_20231204T071317Z_gen_g4_detsim_hitreco__20240509T205838Z_reco2.root
.:
total 4348
-rw-r--r-- 1 nobody nobody 1710122 Sep 23 07:48 flatcaf.root
-rw-r--r-- 1 nobody nobody 1290628 Sep 23 07:48 caf.root
-rw-r--r-- 1 nobody nobody 1290628 Sep 23 07:48 caf_fd_hd_atmo_3302_20250923T102030Z.root
-rw-r--r-- 1 nobody nobody   74552 Sep 23 07:48 caf_20250923T102030Z.log
-rw-r--r-- 1 nobody nobody   37233 Sep 23 07:48 jobscript.log
-rw-r--r-- 1 nobody nobody    1962 Sep 23 07:48 caf_20250923T102030Z.file
-rw-r--r-- 1 nobody nobody    1962 Sep 23 07:48 caf_20250923T102030Z.pfns
-rw-r--r-- 1 nobody nobody    1962 Sep 23 07:20 file.list
-rw-r--r-- 1 nobody nobody    1962 Sep 23 07:48 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:20 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:48 caf_20250923T102030Z.did
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:20 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:20 debugprod.log