Jobsub ID 40098.16@dunegpschedd02.fnal.gov

Jobsub ID	40098.16@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 15:37:49
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-09-15 16:33:21
From worker node	Hostname	comp22-01
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 15:39:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_212_20231201T220601Z_gen_g4_detsim_hitreco__20240507T203857Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_339_20231201T183853Z_gen_g4_detsim_hitreco__20240507T212509Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_898_20231202T091810Z_gen_g4_detsim_hitreco__20240507T210640Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_532_20231207T120328Z_gen_g4_detsim_hitreco__20240510T055108Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_648_20231207T162530Z_gen_g4_detsim_hitreco__20240510T031421Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_563_20231207T031938Z_gen_g4_detsim_hitreco__20240510T043930Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_540_20231208T010417Z_gen_g4_detsim_hitreco__20240510T031309Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_551_20231202T060545Z_gen_g4_detsim_hitreco__20240507T193836Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_120_20231202T132144Z_gen_g4_detsim_hitreco__20240507T222538Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_877_20231205T071516Z_gen_g4_detsim_hitreco__20240509T222208Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 16:33:21
Saved logs	justin-logs:40098.16-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.215618      0.0800986       300    
reco:ophitspe:OpHitFinderDeco                          0.0935388     0.167492       7.61479       0.11022      0.434017        300    
reco:opflash:OpFlashFinder                            0.000257986   0.00167882     0.0044326    0.00167279    0.000527245      300    
reco:opslicer:OpSlicer                                9.5199e-05     0.196249      0.890143      0.142451      0.161758        300    
reco:rns:RandomNumberSaver                            1.7103e-05    2.42226e-05   0.000278837   2.20025e-05   1.53903e-05      300    
reco:anglereconue:NuAngularReco                       0.000127437   0.00166638     0.0613793    0.000303245   0.00591097       300    
reco:anglereconumu:NuAngularReco                      6.2107e-05    0.00152198     0.0608493    0.000171308   0.00583596       300    
reco:anglereconuepfps:NuAngularReco                   9.5208e-05    0.00159868     0.0617048    0.000228433    0.0058677       300    
reco:anglereconumupfps:NuAngularReco                  8.3102e-05    0.00158206     0.0619445    0.000210168   0.00588621       300    
reco:anglerecohits:NuAngularReco                      6.8701e-05    0.00204282     0.0669467    0.000506294   0.00638972       300    
[art]:TriggerResults:TriggerResultInserter             8.241e-06    1.04159e-05   3.4353e-05     9.759e-06    2.8685e-06       300    
end_path:cafmaker:CAFMaker                            0.00131317     0.0208948     0.727116     0.00335454     0.0737724       300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3885.27 MB
  Peak resident set size usage (VmHWM): 2209.8 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 17:11:13 BST ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/5d/19/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_532_20231207T120328Z_gen_g4_detsim_hitreco__20240510T055108Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00105296, 0.996351, 0.00259619, 
Output 1: 0.028154, 0.245971, 0.614132, 0.111743, 
Output 2: 0.56238, 0.409178, 0.0278542, 0.000587741, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2475!
Begin processing the 294th record. run: 6400858 subRun: 1 event: 89894 at 15-Sep-2025 17:03:17 BST
Boundary wire vector sizes: 437, 518, 483
minwire 0: 1231
minwire 1: 1716
minwire 2: 932
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000437807, 0.998051, 0.00151132, 
Output 1: 0.00821139, 0.83854, 0.14802, 0.0052288, 
Output 2: 0.995188, 0.00463305, 0.000174762, 4.24388e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2305!
Begin processing the 295th record. run: 6400858 subRun: 1 event: 89895 at 15-Sep-2025 17:03:21 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1213!
Begin processing the 296th record. run: 6400858 subRun: 1 event: 89896 at 15-Sep-2025 17:03:24 BST
Boundary wire vector sizes: 86, 95, 101
minwire 0: 917
minwire 1: 1506
minwire 2: 739
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00249279, 0.00142999, 0.996077, 
Output 1: 0.0525532, 0.941888, 0.00537244, 0.000185899, 
Output 2: 0.996771, 0.00308966, 0.000133595, 6.09598e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2024!
Begin processing the 297th record. run: 6400858 subRun: 1 event: 89897 at 15-Sep-2025 17:03:27 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 759!
Begin processing the 298th record. run: 6400858 subRun: 1 event: 89898 at 15-Sep-2025 17:03:30 BST
Boundary wire vector sizes: 100, 97, 103
minwire 0: 156
minwire 1: 2056
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.715071, 0.142961, 0.141968, 
Output 1: 0.0602803, 0.181798, 0.291952, 0.46597, 
Output 2: 0.522338, 0.440901, 0.0343411, 0.00241984, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1078!
Begin processing the 299th record. run: 6400858 subRun: 1 event: 89899 at 15-Sep-2025 17:03:33 BST
Boundary wire vector sizes: 50, 34, 30
minwire 0: 2604
minwire 1: 401
minwire 2: 2693
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0034277, 0.00172759, 0.994845, 
Output 1: 0.144652, 0.848932, 0.00629661, 0.000119969, 
Output 2: 0.998242, 0.0016807, 7.4751e-05, 3.00257e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 951!
Begin processing the 300th record. run: 6400858 subRun: 1 event: 89900 at 15-Sep-2025 17:03:36 BST
Boundary wire vector sizes: 2990, 2651, 1757
minwire 0: 124
minwire 1: 2042
minwire 2: 43
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.285389, 0.195227, 0.519384, 
Output 1: 0.759728, 0.233985, 0.00479127, 0.00149607, 
Output 2: 0.353674, 0.525651, 0.0994983, 0.0211768, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2213!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/a5/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_212_20231201T220601Z_gen_g4_detsim_hitreco__20240507T203857Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ce/74/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_339_20231201T183853Z_gen_g4_detsim_hitreco__20240507T212509Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/99/79/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_898_20231202T091810Z_gen_g4_detsim_hitreco__20240507T210640Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/5d/19/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_532_20231207T120328Z_gen_g4_detsim_hitreco__20240510T055108Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c1/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_648_20231207T162530Z_gen_g4_detsim_hitreco__20240510T031421Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/00/9a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_563_20231207T031938Z_gen_g4_detsim_hitreco__20240510T043930Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/3e/03/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_540_20231208T010417Z_gen_g4_detsim_hitreco__20240510T031309Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/e5/37/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_551_20231202T060545Z_gen_g4_detsim_hitreco__20240507T193836Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/25/a0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_120_20231202T132144Z_gen_g4_detsim_hitreco__20240507T222538Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/3d/6a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_877_20231205T071516Z_gen_g4_detsim_hitreco__20240509T222208Z_reco2.root
.:
total 13264
-rw-r--r-- 1 pltdune002 pltdune 5021888 Sep 15 17:11 flatcaf.root
-rw-r--r-- 1 pltdune002 pltdune 4099515 Sep 15 17:11 caf.root
-rw-r--r-- 1 pltdune002 pltdune 4099515 Sep 15 17:11 caf_fd_hd_atmo_2501_20250915T153945Z.root
-rw-r--r-- 1 pltdune002 pltdune  188920 Sep 15 17:11 caf_20250915T153945Z.log
-rw-r--r-- 1 pltdune002 pltdune   78639 Sep 15 17:11 jobscript.log
-rw-r--r-- 1 pltdune002 pltdune    1965 Sep 15 17:11 caf_20250915T153945Z.file
-rw-r--r-- 1 pltdune002 pltdune    1965 Sep 15 17:11 caf_20250915T153945Z.pfns
-rw-r--r-- 1 pltdune002 pltdune    1965 Sep 15 16:39 file.list
-rw-r--r-- 1 pltdune002 pltdune    1965 Sep 15 17:11 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune002 pltdune    1375 Sep 15 16:39 all-input-dids.txt
-rw-r--r-- 1 pltdune002 pltdune    1375 Sep 15 17:11 caf_20250915T153945Z.did
-rw-r--r-- 1 pltdune002 pltdune    1375 Sep 15 16:39 did.list
-rw-r--r-- 1 pltdune002 pltdune       0 Sep 15 16:39 debugprod.log