Jobsub ID 155570.186@dunegpschedd01.fnal.gov

Jobsub ID	155570.186@dunegpschedd01.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 15:41:55
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2025-09-23 16:25:41
From worker node	Hostname	wn1205240.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 15:44:09
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_382_20231205T025610Z_gen_g4_detsim_hitreco__20240510T020501Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_609_20231203T025406Z_gen_g4_detsim_hitreco__20240508T062507Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_737_20231207T075048Z_gen_g4_detsim_hitreco__20240510T024354Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_465_20231123T021949Z_gen_g4_detsim_hitreco__20240507T174139Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_274_20231207T041739Z_gen_g4_detsim_hitreco__20240510T052845Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_207_20231122T091444Z_gen_g4_detsim_hitreco__20240503T063821Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_235_20231203T043820Z_gen_g4_detsim_hitreco__20240508T073836Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_790_20231207T065537Z_gen_g4_detsim_hitreco__20240510T055500Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_465_20231201T090558Z_gen_g4_detsim_hitreco__20240507T203553Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_557_20231203T063141Z_gen_g4_detsim_hitreco__20240508T075931Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 16:25:41
Saved logs	justin-logs:155570.186-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.0394418      5.50353     0.00210392     0.280387        500    
reco:opdec:Deconvolution                               0.0805778     0.225617      0.483696       0.22427      0.0615463       500    
reco:ophitspe:OpHitFinderDeco                          0.233321      0.238144      0.269393      0.238111     0.00205841       500    
reco:opflash:OpFlashFinder                            0.000140044   0.000883483   0.00240045    0.000829615   0.00039482       500    
reco:opslicer:OpSlicer                                2.8554e-05     0.176417      0.710999      0.133456      0.148637        500    
reco:rns:RandomNumberSaver                            1.3585e-05    2.01539e-05   0.000188115   2.00275e-05   8.36937e-06      500    
reco:anglereconue:NuAngularReco                        7.975e-05    0.00151632     0.0736092    0.00025684    0.00599826       500    
reco:anglereconumu:NuAngularReco                      5.7619e-05    0.00141114     0.0712137    0.000152087    0.0059631       500    
reco:anglereconuepfps:NuAngularReco                   8.7866e-05     0.0014752     0.0725077    0.000221593   0.00597108       500    
reco:anglereconumupfps:NuAngularReco                  8.0993e-05    0.00147043     0.0729142    0.000197041    0.0059949       500    
reco:anglerecohits:NuAngularReco                      6.9791e-05    0.00191732     0.0757557    0.000474816   0.00646426       500    
[art]:TriggerResults:TriggerResultInserter             7.494e-06    8.7167e-06    3.0578e-05    8.4205e-06    2.03416e-06      500    
end_path:cafmaker:CAFMaker                            0.00101109     0.0202018      1.19906     0.00309505     0.0880164       500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3847.78 MB
  Peak resident set size usage (VmHWM): 2176.38 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 17:25:23 BST ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/e9/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_274_20231207T041739Z_gen_g4_detsim_hitreco__20240510T052845Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 0: 0.981147, 0.00206942, 0.0167838, 
Output 1: 0.430766, 0.563212, 0.00582935, 0.000192611, 
Output 2: 0.983401, 0.0163107, 0.000280515, 7.29918e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 809!
Begin processing the 493rd record. run: 6209163 subRun: 1 event: 93193 at 23-Sep-2025 17:16:32 BST
Boundary wire vector sizes: 51, 73, 42
minwire 0: 300
minwire 1: 1853
minwire 2: 19
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.913659, 0.000833743, 0.0855074, 
Output 1: 0.995772, 0.00389174, 0.000204602, 0.000131275, 
Output 2: 0.840726, 0.155816, 0.00307439, 0.000383983, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1205!
Begin processing the 494th record. run: 6209163 subRun: 1 event: 93194 at 23-Sep-2025 17:16:34 BST
Boundary wire vector sizes: 1872, 1322, 1519
minwire 0: 635
minwire 1: 1285
minwire 2: 677
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00237286, 0.947887, 0.0497398, 
Output 1: 0.572308, 0.419415, 0.00784589, 0.000430688, 
Output 2: 0.934656, 0.0634831, 0.0017788, 8.20907e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2543!
Begin processing the 495th record. run: 6209163 subRun: 1 event: 93195 at 23-Sep-2025 17:16:39 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 429!
Begin processing the 496th record. run: 6209163 subRun: 1 event: 93196 at 23-Sep-2025 17:16:41 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 986!
Begin processing the 497th record. run: 6209163 subRun: 1 event: 93197 at 23-Sep-2025 17:16:43 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 827!
Begin processing the 498th record. run: 6209163 subRun: 1 event: 93198 at 23-Sep-2025 17:16:45 BST
Boundary wire vector sizes: 213, 243, 165
minwire 0: 2382
minwire 1: 122
minwire 2: 2729
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00057696, 0.996786, 0.00263697, 
Output 1: 0.0116919, 0.984328, 0.00384648, 0.000133711, 
Output 2: 0.996299, 0.00362037, 7.83315e-05, 2.14109e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 845!
Begin processing the 499th record. run: 6209163 subRun: 1 event: 93199 at 23-Sep-2025 17:16:47 BST
Boundary wire vector sizes: 247, 330, 281
minwire 0: 2016
minwire 1: 360
minwire 2: 2263
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00120162, 0.990978, 0.00782073, 
Output 1: 0.98801, 0.0115538, 0.00040689, 2.88561e-05, 
Output 2: 0.103679, 0.883544, 0.0126777, 9.93084e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1219!
Begin processing the 500th record. run: 6209163 subRun: 1 event: 93200 at 23-Sep-2025 17:16:50 BST
Boundary wire vector sizes: 10419, 14555, 9855
minwire 0: 344
minwire 1: 1291
minwire 2: 7
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.372523, 0.508192, 0.119285, 
Output 1: 0.279885, 0.526786, 0.121593, 0.0717351, 
Output 2: 0.012405, 0.0450836, 0.18617, 0.756342, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3314!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/70/7b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_382_20231205T025610Z_gen_g4_detsim_hitreco__20240510T020501Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/73/c1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_609_20231203T025406Z_gen_g4_detsim_hitreco__20240508T062507Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bd/48/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_737_20231207T075048Z_gen_g4_detsim_hitreco__20240510T024354Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ff/8f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_465_20231123T021949Z_gen_g4_detsim_hitreco__20240507T174139Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/e9/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_274_20231207T041739Z_gen_g4_detsim_hitreco__20240510T052845Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2e/68/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_207_20231122T091444Z_gen_g4_detsim_hitreco__20240503T063821Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/03/8b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_235_20231203T043820Z_gen_g4_detsim_hitreco__20240508T073836Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/83/ee/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_790_20231207T065537Z_gen_g4_detsim_hitreco__20240510T055500Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/f3/c0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_465_20231201T090558Z_gen_g4_detsim_hitreco__20240507T203553Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f0/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_557_20231203T063141Z_gen_g4_detsim_hitreco__20240508T075931Z_reco2.root
.:
total 27196
-rw-r--r-- 1 dune01 dune01 10262399 Sep 23 17:25 flatcaf.root
-rw-r--r-- 1 dune01 dune01  8555590 Sep 23 17:25 caf.root
-rw-r--r-- 1 dune01 dune01  8555590 Sep 23 17:25 caf_fd_hd_atmo_3301_20250923T154418Z.root
-rw-r--r-- 1 dune01 dune01   301112 Sep 23 17:25 caf_20250923T154418Z.log
-rw-r--r-- 1 dune01 dune01   119354 Sep 23 17:25 jobscript.log
-rw-r--r-- 1 dune01 dune01     1929 Sep 23 17:25 caf_20250923T154418Z.file
-rw-r--r-- 1 dune01 dune01     1929 Sep 23 17:25 caf_20250923T154418Z.pfns
-rw-r--r-- 1 dune01 dune01     1929 Sep 23 16:44 file.list
-rw-r--r-- 1 dune01 dune01     1929 Sep 23 17:25 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01     1375 Sep 23 16:44 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01     1375 Sep 23 17:25 caf_20250923T154418Z.did
-rw-r--r-- 1 dune01 dune01     1375 Sep 23 16:44 did.list
-rw-r--r-- 1 dune01 dune01        0 Sep 23 16:44 debugprod.log