Jobsub ID 94625.126@dunegpschedd02.fnal.gov

Jobsub ID	94625.126@dunegpschedd02.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 08:35:36
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-23 10:58:53
From worker node	Hostname	wn42
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 08:38:29
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_525_20231202T062635Z_gen_g4_detsim_hitreco__20240508T034154Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_198_20231201T121655Z_gen_g4_detsim_hitreco__20240507T191858Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_177_20231202T075337Z_gen_g4_detsim_hitreco__20240507T205010Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_736_20231207T121018Z_gen_g4_detsim_hitreco__20240510T060156Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_610_20231207T225957Z_gen_g4_detsim_hitreco__20240510T050602Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_627_20231202T160957Z_gen_g4_detsim_hitreco__20240508T043519Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_16_20231208T032512Z_gen_g4_detsim_hitreco__20240510T042718Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_121_20231202T073143Z_gen_g4_detsim_hitreco__20240507T204555Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_258_20231120T135901Z_gen_g4_detsim_hitreco__20240507T181456Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_274_20231203T110000Z_gen_g4_detsim_hitreco__20240509T194317Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 10:58:53
Saved logs	justin-logs:94625.126-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

45     0.00460023     0.205082        500    
reco:opdec:Deconvolution                               0.595072       1.09064       2.8045        1.02772       0.30176        500    
reco:ophitspe:OpHitFinderDeco                           1.76361       1.79174       1.87055       1.77759      0.0309613       500    
reco:opflash:OpFlashFinder                            0.000195037   0.00213779     0.0159049    0.00201421    0.00116085       500    
reco:opslicer:OpSlicer                                6.5401e-05      0.37542       1.63124      0.282552      0.315843        500    
reco:rns:RandomNumberSaver                            2.5627e-05    7.11814e-05   0.00176629    5.43005e-05   0.000118577      500    
reco:anglereconue:NuAngularReco                       0.000227464   0.00329301     0.213199     0.000666561    0.0131314       500    
reco:anglereconumu:NuAngularReco                      0.000133204    0.0028689     0.212115     0.000404886    0.0126718       500    
reco:anglereconuepfps:NuAngularReco                   0.000181231   0.00309107     0.210818     0.000591041    0.0126864       500    
reco:anglereconumupfps:NuAngularReco                  0.000172546   0.00299723     0.214602     0.000537865    0.0127867       500    
reco:anglerecohits:NuAngularReco                      0.000168836   0.00403155     0.223461     0.00120067     0.0137062       500    
[art]:TriggerResults:TriggerResultInserter            1.2309e-05    2.46922e-05   0.000637578   2.13965e-05   3.85765e-05      500    
end_path:cafmaker:CAFMaker                            0.00175155     0.0420868      2.39397      0.0100087     0.154288        500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3784.09 MB
  Peak resident set size usage (VmHWM): 2117.87 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 07:54:38 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/ae/7a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_627_20231202T160957Z_gen_g4_detsim_hitreco__20240508T043519Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 2: 0.997148, 0.00276625, 7.64724e-05, 9.03572e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 736!
Begin processing the 493rd record. run: 66050399 subRun: 1 event: 61093 at 23-Sep-2025 07:50:13 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 272!
Begin processing the 494th record. run: 66050399 subRun: 1 event: 61094 at 23-Sep-2025 07:50:22 -03
Boundary wire vector sizes: 47, 30, 31
minwire 0: 1181
minwire 1: 287
minwire 2: 1169
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 502, 801
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 1216, 1515
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0119896, 0.00200997, 0.986, 
Output 1: 0.166338, 0.613362, 0.213139, 0.00716089, 
Output 2: 0.991796, 0.00780068, 0.000390659, 1.30013e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1392!
Begin processing the 495th record. run: 66050399 subRun: 1 event: 61095 at 23-Sep-2025 07:50:34 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 424!
Begin processing the 496th record. run: 66050399 subRun: 1 event: 61096 at 23-Sep-2025 07:50:43 -03
Boundary wire vector sizes: 79, 54, 57
minwire 0: 424
minwire 1: 2799
minwire 2: 8
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0126658, 0.977518, 0.00981658, 
Output 1: 0.00934876, 0.0923172, 0.82976, 0.0685739, 
Output 2: 0.975821, 0.0230508, 0.00109577, 3.20085e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 989!
Begin processing the 497th record. run: 66050399 subRun: 1 event: 61097 at 23-Sep-2025 07:50:55 -03
Boundary wire vector sizes: 1746, 1677, 1487
minwire 0: 994
minwire 1: 1233
minwire 2: 1041
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00439508, 0.932197, 0.0634078, 
Output 1: 0.0244084, 0.108013, 0.201115, 0.666463, 
Output 2: 0.191458, 0.353883, 0.275116, 0.179544, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2703!
Begin processing the 498th record. run: 66050399 subRun: 1 event: 61098 at 23-Sep-2025 07:51:25 -03
Boundary wire vector sizes: 52, 39, 42
minwire 0: 2161
minwire 1: 736
minwire 2: 2280
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.213572, 0.00829336, 0.778134, 
Output 1: 0.0264236, 0.366847, 0.536312, 0.070417, 
Output 2: 0.238678, 0.740382, 0.020764, 0.000176115, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1250!
Begin processing the 499th record. run: 66050399 subRun: 1 event: 61099 at 23-Sep-2025 07:51:37 -03
Boundary wire vector sizes: 488, 348, 383
minwire 0: 1304
minwire 1: 1098
minwire 2: 1507
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.026458, 0.891482, 0.0820597, 
Output 1: 0.0915541, 0.381642, 0.450046, 0.0767575, 
Output 2: 0.639866, 0.339047, 0.020293, 0.000794456, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2090!
Begin processing the 500th record. run: 66050399 subRun: 1 event: 61100 at 23-Sep-2025 07:51:51 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 721!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b9/14/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_525_20231202T062635Z_gen_g4_detsim_hitreco__20240508T034154Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/69/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_198_20231201T121655Z_gen_g4_detsim_hitreco__20240507T191858Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/64/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_177_20231202T075337Z_gen_g4_detsim_hitreco__20240507T205010Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bf/78/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_736_20231207T121018Z_gen_g4_detsim_hitreco__20240510T060156Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b9/3a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_610_20231207T225957Z_gen_g4_detsim_hitreco__20240510T050602Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/ae/7a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_627_20231202T160957Z_gen_g4_detsim_hitreco__20240508T043519Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/78/38/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_16_20231208T032512Z_gen_g4_detsim_hitreco__20240510T042718Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ea/7f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_121_20231202T073143Z_gen_g4_detsim_hitreco__20240507T204555Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9e/3e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_258_20231120T135901Z_gen_g4_detsim_hitreco__20240507T181456Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/84/68/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_274_20231203T110000Z_gen_g4_detsim_hitreco__20240509T194317Z_reco2.root
.:
total 22520
-rw-r--r-- 1 nobody nobody 8518613 Sep 23 07:54 flatcaf.root
-rw-r--r-- 1 nobody nobody 7030930 Sep 23 07:54 caf.root
-rw-r--r-- 1 nobody nobody 7030930 Sep 23 07:54 caf_fd_hd_atmo_3301_20250923T083841Z.root
-rw-r--r-- 1 nobody nobody  309428 Sep 23 07:54 caf_20250923T083841Z.log
-rw-r--r-- 1 nobody nobody  118243 Sep 23 07:54 jobscript.log
-rw-r--r-- 1 nobody nobody    1897 Sep 23 07:54 caf_20250923T083841Z.file
-rw-r--r-- 1 nobody nobody    1897 Sep 23 07:54 caf_20250923T083841Z.pfns
-rw-r--r-- 1 nobody nobody    1897 Sep 23 05:38 file.list
-rw-r--r-- 1 nobody nobody    1897 Sep 23 07:54 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 23 05:38 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 23 07:54 caf_20250923T083841Z.did
-rw-r--r-- 1 nobody nobody    1375 Sep 23 05:38 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 05:39 debugprod.log