Jobsub ID 46601.141@dunegpschedd01.fnal.gov

Jobsub ID	46601.141@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:17:41
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-15 13:43:27
From worker node	Hostname	wn39
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 12:19:50
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_256_20231207T110854Z_gen_g4_detsim_hitreco__20240510T052703Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_415_20231208T063720Z_gen_g4_detsim_hitreco__20240510T050100Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_203_20231208T033958Z_gen_g4_detsim_hitreco__20240510T045317Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_319_20231208T002315Z_gen_g4_detsim_hitreco__20240510T064217Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_224_20231201T071021Z_gen_g4_detsim_hitreco__20240507T184449Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_368_20231203T123416Z_gen_g4_detsim_hitreco__20240508T050551Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_27_20231121T145452Z_gen_g4_detsim_hitreco__20240503T070750Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_819_20231203T082610Z_gen_g4_detsim_hitreco__20240508T060800Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_142_20231201T050604Z_gen_g4_detsim_hitreco__20240507T203002Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_760_20231208T083036Z_gen_g4_detsim_hitreco__20240510T063418Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 13:43:27
Saved logs	justin-logs:46601.141-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

o:energyrecnumumcsllhd:EnergyReco                  0.000741493    0.0750054      2.39998     0.00776039     0.254264        200    
reco:opdec:Deconvolution                               0.637141       1.09976       4.13286       1.05743      0.326635        200    
reco:ophitspe:OpHitFinderDeco                           1.83195       1.84503       1.94341       1.8425       0.0126205       200    
reco:opflash:OpFlashFinder                            0.000295519    0.0021604     0.0219139    0.00180653    0.00207852       200    
reco:opslicer:OpSlicer                                0.000241228    0.403165       1.18196      0.308733      0.325196        200    
reco:rns:RandomNumberSaver                            2.8194e-05    4.19018e-05   0.000510789    3.698e-05    3.43158e-05      200    
reco:anglereconue:NuAngularReco                       0.000229725   0.00434377      0.18147     0.000618988    0.0191915       200    
reco:anglereconumu:NuAngularReco                      0.000144688   0.00421871     0.182902     0.000472757    0.0194067       200    
reco:anglereconuepfps:NuAngularReco                   0.000203843    0.0043635     0.186665     0.000579945    0.0195976       200    
reco:anglereconumupfps:NuAngularReco                  0.000194448    0.0043189     0.183306     0.000558791    0.0193436       200    
reco:anglerecohits:NuAngularReco                      0.00017311    0.00536194     0.192711     0.00120368     0.0206201       200    
[art]:TriggerResults:TriggerResultInserter            1.5769e-05    2.16245e-05   6.5424e-05    2.0773e-05    4.88314e-06      200    
end_path:cafmaker:CAFMaker                            0.00172482     0.0508352      2.22123     0.00688604     0.215138        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3653.31 MB
  Peak resident set size usage (VmHWM): 1977.29 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 10:39:01 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/06/67/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_203_20231208T033958Z_gen_g4_detsim_hitreco__20240510T045317Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 309, 169, 216
minwire 0: 788
minwire 1: 1322
minwire 2: 847
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00154963, 0.964268, 0.0341822, 
Output 1: 0.865902, 0.110667, 0.0194046, 0.0040261, 
Output 2: 0.956948, 0.0416975, 0.00122005, 0.000134434, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1341!
Begin processing the 195th record. run: 6472619 subRun: 1 event: 41595 at 15-Sep-2025 10:30:11 -03
Boundary wire vector sizes: 564, 354, 420
minwire 0: 2220
minwire 1: 0
minwire 2: 2515
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000394117, 0.997329, 0.00227642, 
Output 1: 0.0263415, 0.965644, 0.00771359, 0.000300967, 
Output 2: 0.991115, 0.00871155, 0.000169112, 4.46548e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1389!
Begin processing the 196th record. run: 6472619 subRun: 1 event: 41596 at 15-Sep-2025 10:30:23 -03
Boundary wire vector sizes: 546, 636, 365
minwire 0: 1392
minwire 1: 1841
minwire 2: 1006
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0219432, 0.770366, 0.207691, 
Output 1: 0.823123, 0.146922, 0.0225477, 0.00740658, 
Output 2: 0.238935, 0.714421, 0.0431004, 0.00354346, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2697!
Begin processing the 197th record. run: 6472619 subRun: 1 event: 41597 at 15-Sep-2025 10:30:37 -03
Boundary wire vector sizes: 265, 233, 183
minwire 0: 1249
minwire 1: 1688
minwire 2: 1077
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00108563, 0.963767, 0.035147, 
Output 1: 0.447864, 0.548313, 0.00371331, 0.000110177, 
Output 2: 0.997174, 0.00275714, 6.51191e-05, 3.53945e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1998!
Begin processing the 198th record. run: 6472619 subRun: 1 event: 41598 at 15-Sep-2025 10:30:49 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 710!
Begin processing the 199th record. run: 6472619 subRun: 1 event: 41599 at 15-Sep-2025 10:31:00 -03
Boundary wire vector sizes: 642, 493, 576
minwire 0: 1408
minwire 1: 849
minwire 2: 1481
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00639894, 0.52974, 0.463861, 
Output 1: 0.38518, 0.586679, 0.023659, 0.00448159, 
Output 2: 0.60826, 0.374425, 0.016146, 0.00116933, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2330!
Begin processing the 200th record. run: 6472619 subRun: 1 event: 41600 at 15-Sep-2025 10:31:14 -03
Boundary wire vector sizes: 1361, 654, 736
minwire 0: 414
minwire 1: 1829
minwire 2: 296
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00320929, 0.988989, 0.00780213, 
Output 1: 0.861829, 0.134979, 0.00297351, 0.000218059, 
Output 2: 0.132595, 0.859199, 0.00814174, 6.38508e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2187!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e5/3c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_256_20231207T110854Z_gen_g4_detsim_hitreco__20240510T052703Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ac/99/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_415_20231208T063720Z_gen_g4_detsim_hitreco__20240510T050100Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/06/67/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_203_20231208T033958Z_gen_g4_detsim_hitreco__20240510T045317Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9f/48/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_319_20231208T002315Z_gen_g4_detsim_hitreco__20240510T064217Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/56/d1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_224_20231201T071021Z_gen_g4_detsim_hitreco__20240507T184449Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_368_20231203T123416Z_gen_g4_detsim_hitreco__20240508T050551Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ba/d1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_27_20231121T145452Z_gen_g4_detsim_hitreco__20240503T070750Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c5/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_819_20231203T082610Z_gen_g4_detsim_hitreco__20240508T060800Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/41/86/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_142_20231201T050604Z_gen_g4_detsim_hitreco__20240507T203002Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6c/1d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_760_20231208T083036Z_gen_g4_detsim_hitreco__20240510T063418Z_reco2.root
.:
total 10972
-rw-r--r-- 1 nobody nobody 4194039 Sep 15 10:39 flatcaf.root
-rw-r--r-- 1 nobody nobody 3393111 Sep 15 10:39 caf.root
-rw-r--r-- 1 nobody nobody 3393111 Sep 15 10:39 caf_fd_hd_atmo_2501_20250915T122010Z.root
-rw-r--r-- 1 nobody nobody  131143 Sep 15 10:39 caf_20250915T122010Z.log
-rw-r--r-- 1 nobody nobody   70756 Sep 15 10:39 jobscript.log
-rw-r--r-- 1 nobody nobody    1867 Sep 15 10:39 caf_20250915T122010Z.file
-rw-r--r-- 1 nobody nobody    1867 Sep 15 10:39 caf_20250915T122010Z.pfns
-rw-r--r-- 1 nobody nobody    1867 Sep 15 09:20 file.list
-rw-r--r-- 1 nobody nobody    1867 Sep 15 10:39 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 15 09:20 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 15 10:39 caf_20250915T122010Z.did
-rw-r--r-- 1 nobody nobody    1375 Sep 15 09:20 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 15 09:21 debugprod.log