Jobsub ID 155570.190@dunegpschedd01.fnal.gov

Jobsub ID	155570.190@dunegpschedd01.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 15:41:55
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce01
Last heartbeat	2025-09-23 15:59:59
From worker node	Hostname	wn2205240.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7513 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 15:43:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370890_301_20231201T082202Z_gen_g4_detsim_hitreco__20240507T202311Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_831_20231202T103919Z_gen_g4_detsim_hitreco__20240507T214526Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_786_20231207T200706Z_gen_g4_detsim_hitreco__20240510T033109Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_782_20231207T121924Z_gen_g4_detsim_hitreco__20240510T042001Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_325_20231207T083907Z_gen_g4_detsim_hitreco__20240510T035519Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_249_20231208T040339Z_gen_g4_detsim_hitreco__20240510T035146Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_226_20231207T132048Z_gen_g4_detsim_hitreco__20240510T062405Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_466_20231204T140347Z_gen_g4_detsim_hitreco__20240509T194458Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_583_20231203T081524Z_gen_g4_detsim_hitreco__20240509T210104Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_659_20231202T160927Z_gen_g4_detsim_hitreco__20240508T043527Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 15:59:59
Saved logs	justin-logs:155570.190-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

661024    0.0212985     0.486109     0.00318872     0.0682537       100    
reco:opdec:Deconvolution                               0.0867701     0.246753      0.527176      0.253881      0.0808082       100    
reco:ophitspe:OpHitFinderDeco                          0.236443      0.245647       0.25446      0.246093     0.00390452       100    
reco:opflash:OpFlashFinder                            0.000165922   0.00118329    0.00589885    0.00119918    0.000718324      100    
reco:opslicer:OpSlicer                                4.4644e-05     0.211533      0.777427      0.186861      0.186331        100    
reco:rns:RandomNumberSaver                            2.2332e-05    4.90909e-05   0.000617913   4.2655e-05    5.86112e-05      100    
reco:anglereconue:NuAngularReco                       0.000176392   0.00100983     0.0191291    0.000367632   0.00244966       100    
reco:anglereconumu:NuAngularReco                      7.9239e-05    0.00078899     0.0183355    0.000172775   0.00238207       100    
reco:anglereconuepfps:NuAngularReco                   0.000131247   0.000941639    0.0185726    0.000290547    0.0024297       100    
reco:anglereconumupfps:NuAngularReco                  0.000114646   0.000856161    0.0193798    0.000232277   0.00244445       100    
reco:anglerecohits:NuAngularReco                      9.3055e-05    0.00153107     0.0218943    0.000739281   0.00283708       100    
[art]:TriggerResults:TriggerResultInserter            1.1552e-05    1.53662e-05   7.6544e-05    1.43565e-05   6.66331e-06      100    
end_path:cafmaker:CAFMaker                            0.00123114     0.0132712     0.208674     0.00432144     0.0310522       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3648.43 MB
  Peak resident set size usage (VmHWM): 1974.21 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 16:59:36 BST ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/c6/ba/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_831_20231202T103919Z_gen_g4_detsim_hitreco__20240507T214526Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 0: 0.997487, 0.00048851, 0.0020243, 
Output 1: 0.087315, 0.876365, 0.0351093, 0.00121056, 
Output 2: 0.997668, 0.0022679, 6.22285e-05, 1.4557e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1743!
Begin processing the 93rd record. run: 6370890 subRun: 1 event: 30193 at 23-Sep-2025 16:51:38 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 500!
Begin processing the 94th record. run: 6370890 subRun: 1 event: 30194 at 23-Sep-2025 16:51:40 BST
Boundary wire vector sizes: 1118, 915, 1009
minwire 0: 2166
minwire 1: 125
minwire 2: 1820
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0807643, 0.108288, 0.810948, 
Output 1: 0.152234, 0.58971, 0.236448, 0.0216086, 
Output 2: 0.0702343, 0.439614, 0.415559, 0.0745924, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2401!
Begin processing the 95th record. run: 6370890 subRun: 1 event: 30195 at 23-Sep-2025 16:51:45 BST
Boundary wire vector sizes: 122, 147, 93
minwire 0: 1363
minwire 1: 828
minwire 2: 1624
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.746243, 0.00182533, 0.251931, 
Output 1: 0.994099, 0.00554657, 0.000247715, 0.000106398, 
Output 2: 0.720289, 0.273754, 0.00551001, 0.000447311, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1302!
Begin processing the 96th record. run: 6370890 subRun: 1 event: 30196 at 23-Sep-2025 16:51:48 BST
Boundary wire vector sizes: 152, 108, 115
minwire 0: 2147
minwire 1: 1240
minwire 2: 1856
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000266963, 0.998572, 0.00116094, 
Output 1: 0.00503326, 0.990691, 0.00414142, 0.000134465, 
Output 2: 0.998276, 0.00168005, 4.27578e-05, 1.27771e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1821!
Begin processing the 97th record. run: 6370890 subRun: 1 event: 30197 at 23-Sep-2025 16:51:51 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1287!
Begin processing the 98th record. run: 6370890 subRun: 1 event: 30198 at 23-Sep-2025 16:51:54 BST
Boundary wire vector sizes: 66, 64, 61
minwire 0: 996
minwire 1: 1405
minwire 2: 921
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0388542, 0.00453659, 0.956609, 
Output 1: 0.0242194, 0.0796323, 0.229322, 0.666826, 
Output 2: 0.943644, 0.0532692, 0.00277549, 0.000311813, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1479!
Begin processing the 99th record. run: 6370890 subRun: 1 event: 30199 at 23-Sep-2025 16:51:57 BST
Boundary wire vector sizes: 657, 719, 763
minwire 0: 209
minwire 1: 1750
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.93102, 0.00500238, 0.0639774, 
Output 1: 0.14374, 0.549137, 0.271666, 0.0354564, 
Output 2: 0.120108, 0.49653, 0.334352, 0.0490096, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1729!
Begin processing the 100th record. run: 6370890 subRun: 1 event: 30200 at 23-Sep-2025 16:52:01 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 904!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b4/5d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370890_301_20231201T082202Z_gen_g4_detsim_hitreco__20240507T202311Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/c6/ba/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_831_20231202T103919Z_gen_g4_detsim_hitreco__20240507T214526Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7f/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_786_20231207T200706Z_gen_g4_detsim_hitreco__20240510T033109Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/43/3d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_782_20231207T121924Z_gen_g4_detsim_hitreco__20240510T042001Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/37/72/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_325_20231207T083907Z_gen_g4_detsim_hitreco__20240510T035519Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/36/13/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_249_20231208T040339Z_gen_g4_detsim_hitreco__20240510T035146Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/dd/fb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_226_20231207T132048Z_gen_g4_detsim_hitreco__20240510T062405Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/17/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_466_20231204T140347Z_gen_g4_detsim_hitreco__20240509T194458Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/61/19/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_583_20231203T081524Z_gen_g4_detsim_hitreco__20240509T210104Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/28/76/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_659_20231202T160927Z_gen_g4_detsim_hitreco__20240508T043527Z_reco2.root
.:
total 3724
-rw-r--r-- 1 dune01 dune01 1482316 Sep 23 16:59 flatcaf.root
-rw-r--r-- 1 dune01 dune01 1088440 Sep 23 16:59 caf.root
-rw-r--r-- 1 dune01 dune01 1088440 Sep 23 16:59 caf_fd_hd_atmo_3301_20250923T154354Z.root
-rw-r--r-- 1 dune01 dune01   67989 Sep 23 16:59 caf_20250923T154354Z.log
-rw-r--r-- 1 dune01 dune01   37983 Sep 23 16:59 jobscript.log
-rw-r--r-- 1 dune01 dune01    1959 Sep 23 16:59 caf_20250923T154354Z.file
-rw-r--r-- 1 dune01 dune01    1959 Sep 23 16:59 caf_20250923T154354Z.pfns
-rw-r--r-- 1 dune01 dune01    1959 Sep 23 16:43 file.list
-rw-r--r-- 1 dune01 dune01    1959 Sep 23 16:59 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01    1375 Sep 23 16:43 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01    1375 Sep 23 16:59 caf_20250923T154354Z.did
-rw-r--r-- 1 dune01 dune01    1375 Sep 23 16:43 did.list
-rw-r--r-- 1 dune01 dune01       0 Sep 23 16:44 debugprod.log