Jobsub ID 40343.52@dunegpschedd02.fnal.gov

Jobsub ID	40343.52@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-16 06:34:40
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2025-09-16 07:27:08
From worker node	Hostname	heplnc085.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-16 06:35:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_886_20231204T024352Z_gen_g4_detsim_hitreco__20240509T213537Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_650_20231208T062008Z_gen_g4_detsim_hitreco__20240510T051505Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_257_20231208T034031Z_gen_g4_detsim_hitreco__20240510T045626Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_730_20231204T140408Z_gen_g4_detsim_hitreco__20240509T203412Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_429_20231207T075639Z_gen_g4_detsim_hitreco__20240510T061148Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_808_20231204T045540Z_gen_g4_detsim_hitreco__20240509T210842Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_308_20231203T042024Z_gen_g4_detsim_hitreco__20240508T064056Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_481_20231202T171118Z_gen_g4_detsim_hitreco__20240507T204952Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_501_20231202T142607Z_gen_g4_detsim_hitreco__20240508T055743Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_53_20231207T111735Z_gen_g4_detsim_hitreco__20240510T053659Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-16 07:27:08
Saved logs	justin-logs:40343.52-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000863396    0.0628703      2.62689     0.00475472     0.272791        200    
reco:opdec:Deconvolution                               0.0843562     0.307075      0.697382      0.307359      0.104438        200    
reco:ophitspe:OpHitFinderDeco                          0.199404      0.209883       0.24386      0.209528      0.005568        200    
reco:opflash:OpFlashFinder                            0.000327844   0.00197345     0.0194965    0.00190859    0.00150096       200    
reco:opslicer:OpSlicer                                0.000775487    0.349186       1.08435      0.300827      0.273295        200    
reco:rns:RandomNumberSaver                            2.9933e-05    5.62781e-05   0.000563459   5.1542e-05    3.88005e-05      200    
reco:anglereconue:NuAngularReco                       0.00023869    0.00199082     0.039504     0.000600409   0.00452899       200    
reco:anglereconumu:NuAngularReco                      0.000136438   0.00173585     0.0395349    0.00040303     0.0044476       200    
reco:anglereconuepfps:NuAngularReco                   0.000210154   0.00188728     0.038932     0.000540284   0.00440791       200    
reco:anglereconumupfps:NuAngularReco                  0.000179894   0.00185862     0.041744     0.000531284   0.00454643       200    
reco:anglerecohits:NuAngularReco                      0.000186439   0.00285475     0.0432829    0.00123085    0.00521203       200    
[art]:TriggerResults:TriggerResultInserter            1.4886e-05    2.49053e-05    7.989e-05    2.3352e-05    6.46078e-06      200    
end_path:cafmaker:CAFMaker                            0.00181302     0.0277252     0.472326     0.00888967     0.0582396       200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3660.24 MB
  Peak resident set size usage (VmHWM): 1993.15 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 08:05:17 BST ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/1e/2c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_257_20231208T034031Z_gen_g4_detsim_hitreco__20240510T045626Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 0: 0.0034692, 0.013455, 0.983076, 
Output 1: 0.696356, 0.295571, 0.00765904, 0.000414094, 
Output 2: 0.997328, 0.00260404, 6.37048e-05, 3.94561e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1601!
Begin processing the 193rd record. run: 6492237 subRun: 1 event: 65093 at 16-Sep-2025 08:01:46 BST
Boundary wire vector sizes: 2048, 1555, 1309
minwire 0: 84
minwire 1: 1976
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00196544, 0.986384, 0.0116504, 
Output 1: 0.409491, 0.333324, 0.108604, 0.148581, 
Output 2: 0.147079, 0.310378, 0.306393, 0.23615, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2156!
Begin processing the 194th record. run: 6492237 subRun: 1 event: 65094 at 16-Sep-2025 08:01:55 BST
Boundary wire vector sizes: 794, 746, 619
minwire 0: 1547
minwire 1: 542
minwire 2: 1044
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00217086, 0.953788, 0.0440414, 
Output 1: 0.557007, 0.432647, 0.00978961, 0.00055645, 
Output 2: 0.409142, 0.570153, 0.0201571, 0.00054849, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2220!
Begin processing the 195th record. run: 6492237 subRun: 1 event: 65095 at 16-Sep-2025 08:02:03 BST
Boundary wire vector sizes: 220, 259, 124
minwire 0: 2248
minwire 1: 803
minwire 2: 1968
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0113096, 0.524965, 0.463725, 
Output 1: 0.364328, 0.598177, 0.0357341, 0.00176117, 
Output 2: 0.824405, 0.17296, 0.00257484, 5.98602e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1920!
Begin processing the 196th record. run: 6492237 subRun: 1 event: 65096 at 16-Sep-2025 08:02:09 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 598!
Begin processing the 197th record. run: 6492237 subRun: 1 event: 65097 at 16-Sep-2025 08:02:13 BST
Boundary wire vector sizes: 6036, 5881, 4276
minwire 0: 1796
minwire 1: 108
minwire 2: 1726
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0118335, 0.974529, 0.0136372, 
Output 1: 0.420032, 0.399333, 0.0967561, 0.0838788, 
Output 2: 0.051639, 0.201415, 0.349505, 0.397441, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3150!
Begin processing the 198th record. run: 6492237 subRun: 1 event: 65098 at 16-Sep-2025 08:03:02 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 708!
Begin processing the 199th record. run: 6492237 subRun: 1 event: 65099 at 16-Sep-2025 08:03:06 BST
Boundary wire vector sizes: 297, 350, 320
minwire 0: 1976
minwire 1: 185
minwire 2: 2156
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0637994, 0.194818, 0.741382, 
Output 1: 0.200904, 0.44813, 0.277906, 0.0730598, 
Output 2: 0.153416, 0.569904, 0.245532, 0.0311489, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1386!
Begin processing the 200th record. run: 6492237 subRun: 1 event: 65100 at 16-Sep-2025 08:03:12 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 525!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a9/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_886_20231204T024352Z_gen_g4_detsim_hitreco__20240509T213537Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/45/dd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_650_20231208T062008Z_gen_g4_detsim_hitreco__20240510T051505Z_reco2.root
root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/1e/2c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_257_20231208T034031Z_gen_g4_detsim_hitreco__20240510T045626Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c8/84/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_730_20231204T140408Z_gen_g4_detsim_hitreco__20240509T203412Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/25/d5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_429_20231207T075639Z_gen_g4_detsim_hitreco__20240510T061148Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fd/44/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_808_20231204T045540Z_gen_g4_detsim_hitreco__20240509T210842Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/54/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_308_20231203T042024Z_gen_g4_detsim_hitreco__20240508T064056Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fa/02/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_481_20231202T171118Z_gen_g4_detsim_hitreco__20240507T204952Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/dd/d9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_501_20231202T142607Z_gen_g4_detsim_hitreco__20240508T055743Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/39/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_53_20231207T111735Z_gen_g4_detsim_hitreco__20240510T053659Z_reco2.root
.:
total 8776
-rw-r--r-- 1 pltdune01 pltdune 3380490 Sep 16 08:05 flatcaf.root
-rw-r--r-- 1 pltdune01 pltdune 2689715 Sep 16 08:05 caf.root
-rw-r--r-- 1 pltdune01 pltdune 2689715 Sep 16 08:05 caf_fd_hd_atmo_2501_20250916T063544Z.root
-rw-r--r-- 1 pltdune01 pltdune  131243 Sep 16 08:05 caf_20250916T063544Z.log
-rw-r--r-- 1 pltdune01 pltdune   56716 Sep 16 08:05 jobscript.log
-rw-r--r-- 1 pltdune01 pltdune    1925 Sep 16 08:05 caf_20250916T063544Z.file
-rw-r--r-- 1 pltdune01 pltdune    1925 Sep 16 08:05 caf_20250916T063544Z.pfns
-rw-r--r-- 1 pltdune01 pltdune    1925 Sep 16 07:35 file.list
-rw-r--r-- 1 pltdune01 pltdune    1925 Sep 16 08:05 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune01 pltdune    1374 Sep 16 07:35 all-input-dids.txt
-rw-r--r-- 1 pltdune01 pltdune    1374 Sep 16 08:05 caf_20250916T063544Z.did
-rw-r--r-- 1 pltdune01 pltdune    1374 Sep 16 07:35 did.list
-rw-r--r-- 1 pltdune01 pltdune       0 Sep 16 07:36 debugprod.log