Jobsub ID 41104.11@dunegpschedd02.fnal.gov

Jobsub ID	41104.11@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 17:17:32
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce01
Last heartbeat	2025-09-17 19:02:08
From worker node	Hostname	wn1208251.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-17 17:18:28
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_446_20231207T031638Z_gen_g4_detsim_hitreco__20240510T043614Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_624_20231208T032205Z_gen_g4_detsim_hitreco__20240510T064730Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_256_20231207T124229Z_gen_g4_detsim_hitreco__20240510T042435Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_742_20231207T045153Z_gen_g4_detsim_hitreco__20240510T061558Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_812_20231202T182107Z_gen_g4_detsim_hitreco__20240508T065049Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_306_20231207T113856Z_gen_g4_detsim_hitreco__20240510T053334Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_964_20231202T140205Z_gen_g4_detsim_hitreco__20240507T223604Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_749_20231204T120445Z_gen_g4_detsim_hitreco__20240509T202718Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_769_20231204T052544Z_gen_g4_detsim_hitreco__20240509T213517Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_165_20231203T164416Z_gen_g4_detsim_hitreco__20240509T193750Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-17 19:02:08
Saved logs	justin-logs:41104.11-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000865353    0.055079       2.78078     0.00556053     0.217472        800    
reco:energyrecnumumcsllhd:EnergyReco                  0.000656749    0.0546346      2.81459     0.00569091     0.216805        800    
reco:opdec:Deconvolution                               0.0586642      0.36794       1.07353       0.36141      0.127188        800    
reco:ophitspe:OpHitFinderDeco                          0.123007      0.223835       0.30839      0.250384      0.0512854       800    
reco:opflash:OpFlashFinder                            0.000288434   0.00244014     0.0151998    0.00210668     0.0017371       800    
reco:opslicer:OpSlicer                                5.9011e-05     0.432621       1.91313      0.343357      0.350744        800    
reco:rns:RandomNumberSaver                             2.625e-05    4.39633e-05   0.000579794   4.0411e-05    2.81134e-05      800    
reco:anglereconue:NuAngularReco                       0.00022778    0.00221294     0.0989852    0.00058826    0.00661852       800    
reco:anglereconumu:NuAngularReco                      0.000101301   0.00199511     0.0993183    0.000350586   0.00653565       800    
reco:anglereconuepfps:NuAngularReco                   0.000160703   0.00215887     0.100106     0.000498741   0.00670563       800    
reco:anglereconumupfps:NuAngularReco                  0.000135736   0.00211213     0.101648     0.000458846   0.00665812       800    
reco:anglerecohits:NuAngularReco                      0.000141016    0.0030894     0.116085     0.00114041     0.0076527       800    
[art]:TriggerResults:TriggerResultInserter            1.2944e-05    2.01449e-05   0.000173527   1.82045e-05   1.02206e-05      800    
end_path:cafmaker:CAFMaker                            0.00224613     0.0300007      1.45538     0.00750533     0.0917855       800    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3699.27 MB
  Peak resident set size usage (VmHWM): 2001.92 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 20:01:11 BST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/91/6e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_769_20231204T052544Z_gen_g4_detsim_hitreco__20240509T213517Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 1: 0.384432, 0.594438, 0.0162876, 0.00484242, 
Output 2: 0.0300478, 0.153152, 0.337842, 0.478959, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3095!
Begin processing the 793rd record. run: 6422149 subRun: 1 event: 74993 at 17-Sep-2025 20:00:20 BST
Boundary wire vector sizes: 1470, 1303, 1249
minwire 0: 580
minwire 1: 875
minwire 2: 595
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0637973, 0.253206, 0.682997, 
Output 1: 0.265447, 0.390261, 0.195103, 0.149189, 
Output 2: 0.122111, 0.358815, 0.331898, 0.187176, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2465!
Begin processing the 794th record. run: 6422149 subRun: 1 event: 74994 at 17-Sep-2025 20:00:29 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 762!
Begin processing the 795th record. run: 6422149 subRun: 1 event: 74995 at 17-Sep-2025 20:00:33 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 451!
Begin processing the 796th record. run: 6422149 subRun: 1 event: 74996 at 17-Sep-2025 20:00:37 BST
Boundary wire vector sizes: 354, 428, 414
minwire 0: 1747
minwire 1: 1648
minwire 2: 1373
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.485255, 0.167831, 0.346914, 
Output 1: 0.0485497, 0.69094, 0.239101, 0.0214092, 
Output 2: 0.040716, 0.264781, 0.531496, 0.163007, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2819!
Begin processing the 797th record. run: 6422149 subRun: 1 event: 74997 at 17-Sep-2025 20:00:43 BST
Boundary wire vector sizes: 424, 1331, 849
minwire 0: 985
minwire 1: 898
minwire 2: 711
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997587, 0.000304337, 0.00210874, 
Output 1: 0.0981946, 0.608562, 0.259132, 0.0341113, 
Output 2: 0.990044, 0.00957548, 0.000366036, 1.46322e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3002!
Begin processing the 798th record. run: 6422149 subRun: 1 event: 74998 at 17-Sep-2025 20:00:53 BST
Boundary wire vector sizes: 720, 1193, 651
minwire 0: 1616
minwire 1: 404
minwire 2: 2003
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0127204, 0.895242, 0.092038, 
Output 1: 0.745021, 0.246297, 0.00786519, 0.000817092, 
Output 2: 0.391731, 0.592665, 0.0151326, 0.00047219, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2160!
Begin processing the 799th record. run: 6422149 subRun: 1 event: 74999 at 17-Sep-2025 20:00:59 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 563!
Begin processing the 800th record. run: 6422149 subRun: 1 event: 75000 at 17-Sep-2025 20:01:03 BST
Boundary wire vector sizes: 96, 97, 74
minwire 0: 1883
minwire 1: 629
minwire 2: 1752
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 702, 1001
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0116605, 0.00417334, 0.984166, 
Output 1: 0.122656, 0.580336, 0.248096, 0.0489114, 
Output 2: 0.973658, 0.0250898, 0.00119183, 6.00338e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1476!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8d/cb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_446_20231207T031638Z_gen_g4_detsim_hitreco__20240510T043614Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d6/0d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_624_20231208T032205Z_gen_g4_detsim_hitreco__20240510T064730Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/13/60/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_256_20231207T124229Z_gen_g4_detsim_hitreco__20240510T042435Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/40/77/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_742_20231207T045153Z_gen_g4_detsim_hitreco__20240510T061558Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2e/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_812_20231202T182107Z_gen_g4_detsim_hitreco__20240508T065049Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/4d/ce/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_306_20231207T113856Z_gen_g4_detsim_hitreco__20240510T053334Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/80/75/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_964_20231202T140205Z_gen_g4_detsim_hitreco__20240507T223604Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/f8/9f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_749_20231204T120445Z_gen_g4_detsim_hitreco__20240509T202718Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/91/6e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_769_20231204T052544Z_gen_g4_detsim_hitreco__20240509T213517Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1a/6d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_165_20231203T164416Z_gen_g4_detsim_hitreco__20240509T193750Z_reco2.root
.:
total 34796
-rw-r--r-- 1 dune01 dune01 13120734 Sep 17 20:01 flatcaf.root
-rw-r--r-- 1 dune01 dune01 10902026 Sep 17 20:01 caf.root
-rw-r--r-- 1 dune01 dune01 10902026 Sep 17 20:01 caf_fd_hd_atmo_2501_20250917T171842Z.root
-rw-r--r-- 1 dune01 dune01   475536 Sep 17 20:01 caf_20250917T171842Z.log
-rw-r--r-- 1 dune01 dune01   173400 Sep 17 20:01 jobscript.log
-rw-r--r-- 1 dune01 dune01     1928 Sep 17 20:01 caf_20250917T171842Z.file
-rw-r--r-- 1 dune01 dune01     1928 Sep 17 20:01 caf_20250917T171842Z.pfns
-rw-r--r-- 1 dune01 dune01     1928 Sep 17 18:18 file.list
-rw-r--r-- 1 dune01 dune01     1928 Sep 17 20:01 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01     1376 Sep 17 18:18 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01     1376 Sep 17 20:01 caf_20250917T171842Z.did
-rw-r--r-- 1 dune01 dune01     1376 Sep 17 18:18 did.list
-rw-r--r-- 1 dune01 dune01        0 Sep 17 18:19 debugprod.log