Jobsub ID 46706.8@dunegpschedd01.fnal.gov

Jobsub ID	46706.8@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 15:43:50
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce05
Last heartbeat	2025-09-15 17:24:51
From worker node	Hostname	dune001-7770472.0-lcg2541.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 15:45:26
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_519_20231208T042647Z_gen_g4_detsim_hitreco__20240510T050515Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_166_20231201T220552Z_gen_g4_detsim_hitreco__20240507T203937Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_474_20231202T125358Z_gen_g4_detsim_hitreco__20240507T222008Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_463_20231208T095952Z_gen_g4_detsim_hitreco__20240510T064941Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_687_20231207T162615Z_gen_g4_detsim_hitreco__20240510T031834Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_1058_20231203T030530Z_gen_g4_detsim_hitreco__20240508T070634Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_622_20231203T151628Z_gen_g4_detsim_hitreco__20240508T054530Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_313_20231203T094542Z_gen_g4_detsim_hitreco__20240508T075503Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_56_20231201T113309Z_gen_g4_detsim_hitreco__20240507T185653Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 17:24:51
Saved logs	justin-logs:46706.8-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

reco:energyrecnumumcs:EnergyReco                      0.000912308    0.0442194      2.34543     0.00515848      0.16206        700    
reco:energyrecnumumcsllhd:EnergyReco                  0.000724794    0.0439544      2.35539     0.00527427     0.161594        700    
reco:opdec:Deconvolution                               0.0838739     0.339042      0.683715      0.338006      0.105231        700    
reco:ophitspe:OpHitFinderDeco                          0.241536      0.265018       0.95343      0.261291      0.0322451       700    
reco:opflash:OpFlashFinder                            0.000210894   0.00203093    0.00687338    0.00182161    0.00114996       700    
reco:opslicer:OpSlicer                                5.7101e-05     0.383372       1.44734      0.319547      0.298598        700    
reco:rns:RandomNumberSaver                             2.564e-05    5.80326e-05   0.000612112   5.34355e-05   3.32029e-05      700    
reco:anglereconue:NuAngularReco                       0.000187434   0.00188212     0.0868083    0.000552156   0.00615737       700    
reco:anglereconumu:NuAngularReco                      9.1762e-05    0.00166648     0.0862179    0.000330597   0.00614026       700    
reco:anglereconuepfps:NuAngularReco                   0.000145003   0.00183037     0.0871934    0.000476995   0.00624671       700    
reco:anglereconumupfps:NuAngularReco                  0.000126042   0.00175829     0.0865116    0.000431139   0.00616888       700    
reco:anglerecohits:NuAngularReco                      0.000136093   0.00267756     0.0953823    0.00109051    0.00701136       700    
[art]:TriggerResults:TriggerResultInserter             1.228e-05    1.90669e-05   0.000103962    1.783e-05    5.89758e-06      700    
end_path:cafmaker:CAFMaker                            0.00158216     0.0259032     0.995888      0.0084251      0.07582        700    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3709.66 MB
  Peak resident set size usage (VmHWM): 1965.78 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 17:03:05 UTC ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f1/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_313_20231203T094542Z_gen_g4_detsim_hitreco__20240508T075503Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1895!
Begin processing the 693rd record. run: 6419113 subRun: 1 event: 62293 at 15-Sep-2025 17:02:35 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 199!
Begin processing the 694th record. run: 6419113 subRun: 1 event: 62294 at 15-Sep-2025 17:02:38 UTC
Boundary wire vector sizes: 57, 68, 59
minwire 0: 498
minwire 1: 2146
minwire 2: 368
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.991424, 0.000811992, 0.00776367, 
Output 1: 0.333629, 0.663529, 0.00275879, 8.32607e-05, 
Output 2: 0.992059, 0.0078009, 0.000135954, 3.64439e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 682!
Begin processing the 695th record. run: 6419113 subRun: 1 event: 62295 at 15-Sep-2025 17:02:42 UTC
Boundary wire vector sizes: 57, 60, 52
minwire 0: 1197
minwire 1: 1466
minwire 2: 984
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 358, 657
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00426211, 0.00279209, 0.992946, 
Output 1: 0.963201, 0.036037, 0.000622461, 0.000139086, 
Output 2: 0.998015, 0.00194347, 3.68993e-05, 4.71565e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1801!
Begin processing the 696th record. run: 6419113 subRun: 1 event: 62296 at 15-Sep-2025 17:02:46 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 588!
Begin processing the 697th record. run: 6419113 subRun: 1 event: 62297 at 15-Sep-2025 17:02:49 UTC
Boundary wire vector sizes: 173, 185, 148
minwire 0: 2145
minwire 1: 858
minwire 2: 2128
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00559815, 0.987206, 0.00719604, 
Output 1: 0.107487, 0.453506, 0.375926, 0.0630811, 
Output 2: 0.214562, 0.745072, 0.0395973, 0.000769152, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1686!
Begin processing the 698th record. run: 6419113 subRun: 1 event: 62298 at 15-Sep-2025 17:02:54 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 337!
Begin processing the 699th record. run: 6419113 subRun: 1 event: 62299 at 15-Sep-2025 17:02:56 UTC
Boundary wire vector sizes: 131, 112, 101
minwire 0: 2933
minwire 1: 686
minwire 2: 2719
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000874324, 0.996927, 0.00219881, 
Output 1: 0.0269195, 0.391378, 0.517296, 0.0644066, 
Output 2: 0.997544, 0.00231351, 0.000136787, 6.19601e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1112!
Begin processing the 700th record. run: 6419113 subRun: 1 event: 62300 at 15-Sep-2025 17:03:00 UTC
Boundary wire vector sizes: 43, 48, 35
minwire 0: 1651
minwire 1: 800
minwire 2: 1784
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.415324, 0.0419574, 0.542719, 
Output 1: 0.908934, 0.0880765, 0.0028609, 0.000129034, 
Output 2: 0.408588, 0.582614, 0.00867711, 0.000120658, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 726!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/97/e9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_519_20231208T042647Z_gen_g4_detsim_hitreco__20240510T050515Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/04/02/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_166_20231201T220552Z_gen_g4_detsim_hitreco__20240507T203937Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/1a/b9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_474_20231202T125358Z_gen_g4_detsim_hitreco__20240507T222008Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/02/46/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_463_20231208T095952Z_gen_g4_detsim_hitreco__20240510T064941Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/28/07/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_687_20231207T162615Z_gen_g4_detsim_hitreco__20240510T031834Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/da/39/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_1058_20231203T030530Z_gen_g4_detsim_hitreco__20240508T070634Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bd/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_622_20231203T151628Z_gen_g4_detsim_hitreco__20240508T054530Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f1/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_313_20231203T094542Z_gen_g4_detsim_hitreco__20240508T075503Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b9/91/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_56_20231201T113309Z_gen_g4_detsim_hitreco__20240507T185653Z_reco2.root
.:
total 29728
-rw-r--r-- 1 dune001 dune 11185019 Sep 15 17:03 flatcaf.root
-rw-r--r-- 1 dune001 dune  9297008 Sep 15 17:03 caf.root
-rw-r--r-- 1 dune001 dune  9297008 Sep 15 17:03 caf_fd_hd_atmo_2501_20250915T154541Z.root
-rw-r--r-- 1 dune001 dune   418054 Sep 15 17:03 caf_20250915T154541Z.log
-rw-r--r-- 1 dune001 dune   152944 Sep 15 17:03 jobscript.log
-rw-r--r-- 1 dune001 dune     1710 Sep 15 17:03 caf_20250915T154541Z.file
-rw-r--r-- 1 dune001 dune     1710 Sep 15 17:03 caf_20250915T154541Z.pfns
-rw-r--r-- 1 dune001 dune     1710 Sep 15 15:45 file.list
-rw-r--r-- 1 dune001 dune     1710 Sep 15 17:03 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune     1239 Sep 15 15:45 all-input-dids.txt
-rw-r--r-- 1 dune001 dune     1239 Sep 15 17:03 caf_20250915T154541Z.did
-rw-r--r-- 1 dune001 dune     1239 Sep 15 15:45 did.list
-rw-r--r-- 1 dune001 dune        0 Sep 15 15:46 debugprod.log