Jobsub ID 39997.136@dunegpschedd02.fnal.gov

Jobsub ID	39997.136@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 13:05:43
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-09-15 13:52:30
From worker node	Hostname	comp20-14
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 13:06:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_661_20231207T120652Z_gen_g4_detsim_hitreco__20240510T060301Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_257_20231204T012345Z_gen_g4_detsim_hitreco__20240509T212744Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_175_20231203T104628Z_gen_g4_detsim_hitreco__20240508T042351Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_179_20231202T132508Z_gen_g4_detsim_hitreco__20240507T222639Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_381_20231202T150428Z_gen_g4_detsim_hitreco__20240508T023902Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_6_20231201T042236Z_gen_g4_detsim_hitreco__20240507T203556Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_116_20231202T162245Z_gen_g4_detsim_hitreco__20240508T031645Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_608_20231207T044028Z_gen_g4_detsim_hitreco__20240510T055805Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_46_20231201T094756Z_gen_g4_detsim_hitreco__20240507T185330Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_481_20231201T120842Z_gen_g4_detsim_hitreco__20240507T194242Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 13:52:30
Saved logs	justin-logs:39997.136-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00461697       100    
reco:opflash:OpFlashFinder                            0.000188823   0.00108347    0.00372823    0.000990498   0.000576483      100    
reco:opslicer:OpSlicer                                5.2408e-05     0.196311      0.772369      0.130691      0.183534        100    
reco:rns:RandomNumberSaver                            1.8409e-05    2.80458e-05   0.000350581   2.3352e-05    3.27926e-05      100    
reco:anglereconue:NuAngularReco                       0.000138919   0.00208103     0.0563777    0.000266593   0.00714743       100    
reco:anglereconumu:NuAngularReco                      6.8269e-05    0.00194258     0.055483     0.000128401   0.00707149       100    
reco:anglereconuepfps:NuAngularReco                   0.000103338   0.00203868     0.0554059    0.000200513   0.00707034       100    
reco:anglereconumupfps:NuAngularReco                  8.9556e-05     0.0019909     0.054648     0.00017747    0.00701632       100    
reco:anglerecohits:NuAngularReco                      7.5977e-05    0.00250711     0.0588873    0.000514976   0.00761305       100    
[art]:TriggerResults:TriggerResultInserter             8.946e-06    1.16528e-05    5.722e-05    1.0628e-05    4.91402e-06      100    
end_path:cafmaker:CAFMaker                            0.00134117     0.0265276      0.77588     0.00342841     0.0909877       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3655.64 MB
  Peak resident set size usage (VmHWM): 1975.64 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 14:24:20 BST ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/b0/9f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_257_20231204T012345Z_gen_g4_detsim_hitreco__20240509T212744Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00350168, 0.0021629, 0.994335, 
Output 1: 0.0411399, 0.953758, 0.0048927, 0.000209216, 
Output 2: 0.99809, 0.00183292, 7.33835e-05, 3.99095e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1188!
Begin processing the 95th record. run: 74552530 subRun: 1 event: 66195 at 15-Sep-2025 14:16:28 BST
Boundary wire vector sizes: 352, 423, 340
minwire 0: 1089
minwire 1: 220
minwire 2: 893
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.335656, 0.0233995, 0.640944, 
Output 1: 0.327505, 0.660803, 0.0111239, 0.000568835, 
Output 2: 0.279603, 0.659707, 0.0591586, 0.00153139, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2504!
Begin processing the 96th record. run: 74552530 subRun: 1 event: 66196 at 15-Sep-2025 14:16:32 BST
Boundary wire vector sizes: 115, 105, 68
minwire 0: 2036
minwire 1: 1274
minwire 2: 1831
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000628829, 0.993975, 0.0053958, 
Output 1: 0.966481, 0.0331878, 0.000256892, 7.44344e-05, 
Output 2: 0.999081, 0.000890877, 2.29974e-05, 5.12874e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1594!
Begin processing the 97th record. run: 74552530 subRun: 1 event: 66197 at 15-Sep-2025 14:16:35 BST
Boundary wire vector sizes: 60, 58, 54
minwire 0: 1124
minwire 1: 795
minwire 2: 809
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 1044, 1343
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 31, 330
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00290121, 0.00148987, 0.995609, 
Output 1: 0.194395, 0.768037, 0.0365832, 0.00098497, 
Output 2: 0.998653, 0.00125063, 9.17247e-05, 4.21647e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1873!
Begin processing the 98th record. run: 74552530 subRun: 1 event: 66198 at 15-Sep-2025 14:16:39 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 775!
Begin processing the 99th record. run: 74552530 subRun: 1 event: 66199 at 15-Sep-2025 14:16:41 BST
Boundary wire vector sizes: 209, 189, 90
minwire 0: 1754
minwire 1: 640
minwire 2: 2107
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00614249, 0.990218, 0.00363956, 
Output 1: 0.00752504, 0.777464, 0.208159, 0.00685169, 
Output 2: 0.996407, 0.0034117, 0.000175355, 5.56947e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1277!
Begin processing the 100th record. run: 74552530 subRun: 1 event: 66200 at 15-Sep-2025 14:16:45 BST
Boundary wire vector sizes: 173, 429, 296
minwire 0: 1441
minwire 1: 1611
minwire 2: 852
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997011, 0.000741144, 0.00224753, 
Output 1: 0.00438811, 0.992162, 0.00337656, 7.32194e-05, 
Output 2: 0.997194, 0.00273602, 6.86177e-05, 1.45686e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2326!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/98/85/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_661_20231207T120652Z_gen_g4_detsim_hitreco__20240510T060301Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/b0/9f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_257_20231204T012345Z_gen_g4_detsim_hitreco__20240509T212744Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/0b/78/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_175_20231203T104628Z_gen_g4_detsim_hitreco__20240508T042351Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/e9/b9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_179_20231202T132508Z_gen_g4_detsim_hitreco__20240507T222639Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/20/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_381_20231202T150428Z_gen_g4_detsim_hitreco__20240508T023902Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/1c/ba/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_6_20231201T042236Z_gen_g4_detsim_hitreco__20240507T203556Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/f6/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_116_20231202T162245Z_gen_g4_detsim_hitreco__20240508T031645Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/8a/24/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_608_20231207T044028Z_gen_g4_detsim_hitreco__20240510T055805Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/f8/b0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_46_20231201T094756Z_gen_g4_detsim_hitreco__20240507T185330Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/a6/99/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_481_20231201T120842Z_gen_g4_detsim_hitreco__20240507T194242Z_reco2.root
.:
total 5720
-rw-r--r-- 1 pltdune002 pltdune 2230889 Sep 15 14:24 flatcaf.root
-rw-r--r-- 1 pltdune002 pltdune 1737678 Sep 15 14:24 caf.root
-rw-r--r-- 1 pltdune002 pltdune 1737678 Sep 15 14:24 caf_fd_hd_atmo_2501_20250915T130637Z.root
-rw-r--r-- 1 pltdune002 pltdune   72361 Sep 15 14:24 caf_20250915T130637Z.log
-rw-r--r-- 1 pltdune002 pltdune   40729 Sep 15 14:24 jobscript.log
-rw-r--r-- 1 pltdune002 pltdune    1880 Sep 15 14:24 caf_20250915T130637Z.file
-rw-r--r-- 1 pltdune002 pltdune    1880 Sep 15 14:24 caf_20250915T130637Z.pfns
-rw-r--r-- 1 pltdune002 pltdune    1880 Sep 15 14:06 file.list
-rw-r--r-- 1 pltdune002 pltdune    1880 Sep 15 14:24 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune002 pltdune    1372 Sep 15 14:06 all-input-dids.txt
-rw-r--r-- 1 pltdune002 pltdune    1372 Sep 15 14:24 caf_20250915T130637Z.did
-rw-r--r-- 1 pltdune002 pltdune    1372 Sep 15 14:06 did.list
-rw-r--r-- 1 pltdune002 pltdune       0 Sep 15 14:06 debugprod.log