Jobsub ID 40069.63@dunegpschedd02.fnal.gov

Jobsub ID	40069.63@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 14:56:48
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-09-15 16:16:13
From worker node	Hostname	comp20-14
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 14:57:46
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_403_20231208T045520Z_gen_g4_detsim_hitreco__20240510T065208Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_276_20231202T141606Z_gen_g4_detsim_hitreco__20240508T055456Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_465_20231207T123938Z_gen_g4_detsim_hitreco__20240510T061608Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_302_20231204T063458Z_gen_g4_detsim_hitreco__20240509T205439Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_767_20231202T115710Z_gen_g4_detsim_hitreco__20240507T215936Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_742_20231203T050204Z_gen_g4_detsim_hitreco__20240508T072748Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_695_20231204T100454Z_gen_g4_detsim_hitreco__20240509T194704Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_466_20231202T062552Z_gen_g4_detsim_hitreco__20240508T034314Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_275_20231203T040108Z_gen_g4_detsim_hitreco__20240508T044324Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_282_20231201T122225Z_gen_g4_detsim_hitreco__20240507T191812Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 16:16:13
Saved logs	justin-logs:40069.63-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.0371932     0.258253      0.915484      0.245126      0.109366        600    
reco:ophitspe:OpHitFinderDeco                          0.0896722     0.147672      0.529502      0.129949      0.0636234       600    
reco:opflash:OpFlashFinder                            0.000189888   0.00115905    0.00275679    0.00112222    0.000494118      600    
reco:opslicer:OpSlicer                                4.4367e-05     0.214432      0.804462      0.172434      0.179618        600    
reco:rns:RandomNumberSaver                            1.6428e-05    2.6835e-05    0.000293354   2.45605e-05   1.2789e-05       600    
reco:anglereconue:NuAngularReco                       0.000130176   0.00191035     0.0813153    0.000315239   0.00632239       600    
reco:anglereconumu:NuAngularReco                      6.4508e-05      0.00174      0.0808748    0.00017166    0.00620029       600    
reco:anglereconuepfps:NuAngularReco                   0.00010005    0.00181802     0.0817191    0.000249238   0.00622052       600    
reco:anglereconumupfps:NuAngularReco                  8.7614e-05     0.0018019     0.0819383    0.000225258   0.00623473       600    
reco:anglerecohits:NuAngularReco                      7.9019e-05    0.00229406     0.0859814    0.000535709   0.00673988       600    
[art]:TriggerResults:TriggerResultInserter             9.23e-06     1.17564e-05   3.9721e-05    1.12215e-05   2.02864e-06      600    
end_path:cafmaker:CAFMaker                            0.00134384     0.0238946      1.00214     0.00406483     0.0784626       600    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3817.75 MB
  Peak resident set size usage (VmHWM): 2134.99 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 16:54:06 BST ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/85/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_695_20231204T100454Z_gen_g4_detsim_hitreco__20240509T194704Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1702!
Begin processing the 595th record. run: 66006611 subRun: 1 event: 74295 at 15-Sep-2025 16:46:04 BST
Boundary wire vector sizes: 332, 122, 219
minwire 0: 2426
minwire 1: 914
minwire 2: 2285
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995561, 0.000371403, 0.0040679, 
Output 1: 0.988858, 0.0106246, 0.000335563, 0.000181628, 
Output 2: 0.995976, 0.00392443, 8.18339e-05, 1.74442e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1439!
Begin processing the 596th record. run: 66006611 subRun: 1 event: 74296 at 15-Sep-2025 16:46:07 BST
Boundary wire vector sizes: 54, 80, 68
minwire 0: 1898
minwire 1: 648
minwire 2: 1989
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00312717, 0.00175204, 0.995121, 
Output 1: 0.152184, 0.83898, 0.00832507, 0.000510878, 
Output 2: 0.99659, 0.0032505, 0.00015016, 9.72468e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1540!
Begin processing the 597th record. run: 66006611 subRun: 1 event: 74297 at 15-Sep-2025 16:46:09 BST
Boundary wire vector sizes: 146, 80, 112
minwire 0: 983
minwire 1: 2199
minwire 2: 630
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.977709, 0.000737575, 0.0215534, 
Output 1: 0.0239349, 0.634193, 0.332548, 0.00932376, 
Output 2: 0.968468, 0.0304966, 0.00101441, 2.06196e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1618!
Begin processing the 598th record. run: 66006611 subRun: 1 event: 74298 at 15-Sep-2025 16:46:12 BST
Boundary wire vector sizes: 213, 235, 184
minwire 0: 715
minwire 1: 2699
minwire 2: 33
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995605, 0.000634513, 0.00376022, 
Output 1: 0.0360545, 0.956938, 0.0068268, 0.000180341, 
Output 2: 0.996317, 0.00357327, 0.000107133, 2.97156e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1442!
Begin processing the 599th record. run: 66006611 subRun: 1 event: 74299 at 15-Sep-2025 16:46:14 BST
Boundary wire vector sizes: 468, 622, 556
minwire 0: 753
minwire 1: 1715
minwire 2: 399
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0194959, 0.595968, 0.384536, 
Output 1: 0.152762, 0.824185, 0.021889, 0.00116383, 
Output 2: 0.378408, 0.548542, 0.0688043, 0.00424616, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2231!
Begin processing the 600th record. run: 66006611 subRun: 1 event: 74300 at 15-Sep-2025 16:46:17 BST
Boundary wire vector sizes: 4408, 4365, 3810
minwire 0: 305
minwire 1: 1152
minwire 2: 3
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.867513, 0.0184533, 0.114034, 
Output 1: 0.298437, 0.428567, 0.156544, 0.116452, 
Output 2: 0.0139345, 0.0541365, 0.205096, 0.726833, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3146!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/81/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_403_20231208T045520Z_gen_g4_detsim_hitreco__20240510T065208Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d0/f4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_276_20231202T141606Z_gen_g4_detsim_hitreco__20240508T055456Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/9b/f3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_465_20231207T123938Z_gen_g4_detsim_hitreco__20240510T061608Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/97/ac/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_302_20231204T063458Z_gen_g4_detsim_hitreco__20240509T205439Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/89/45/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_767_20231202T115710Z_gen_g4_detsim_hitreco__20240507T215936Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/94/03/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_742_20231203T050204Z_gen_g4_detsim_hitreco__20240508T072748Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/85/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_695_20231204T100454Z_gen_g4_detsim_hitreco__20240509T194704Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/3b/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_466_20231202T062552Z_gen_g4_detsim_hitreco__20240508T034314Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/52/54/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_275_20231203T040108Z_gen_g4_detsim_hitreco__20240508T044324Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/b6/a1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_282_20231201T122225Z_gen_g4_detsim_hitreco__20240507T191812Z_reco2.root
.:
total 30048
-rw-r--r-- 1 pltdune002 pltdune 11299790 Sep 15 16:54 flatcaf.root
-rw-r--r-- 1 pltdune002 pltdune  9447666 Sep 15 16:54 caf.root
-rw-r--r-- 1 pltdune002 pltdune  9447666 Sep 15 16:54 caf_fd_hd_atmo_2501_20250915T145753Z.root
-rw-r--r-- 1 pltdune002 pltdune   343885 Sep 15 16:54 caf_20250915T145753Z.log
-rw-r--r-- 1 pltdune002 pltdune   142930 Sep 15 16:54 jobscript.log
-rw-r--r-- 1 pltdune002 pltdune     1966 Sep 15 16:54 caf_20250915T145753Z.file
-rw-r--r-- 1 pltdune002 pltdune     1966 Sep 15 16:54 caf_20250915T145753Z.pfns
-rw-r--r-- 1 pltdune002 pltdune     1966 Sep 15 15:57 file.list
-rw-r--r-- 1 pltdune002 pltdune     1966 Sep 15 16:54 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune002 pltdune     1378 Sep 15 15:57 all-input-dids.txt
-rw-r--r-- 1 pltdune002 pltdune     1378 Sep 15 16:54 caf_20250915T145753Z.did
-rw-r--r-- 1 pltdune002 pltdune     1378 Sep 15 15:57 did.list
-rw-r--r-- 1 pltdune002 pltdune        0 Sep 15 15:58 debugprod.log