Jobsub ID 40626.5@dunegpschedd02.fnal.gov

Jobsub ID	40626.5@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-16 21:50:24
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2025-09-16 22:03:58
From worker node	Hostname	wn1907251.tier2.hep.manchester.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-16 21:51:14
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_344_20231204T212511Z_gen_g4_detsim_hitreco__20240510T025447Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_1152_20231203T053108Z_gen_g4_detsim_hitreco__20240508T063615Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_342_20231204T215526Z_gen_g4_detsim_hitreco__20240509T221203Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_728_20231203T233241Z_gen_g4_detsim_hitreco__20240509T220016Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_157_20231203T165220Z_gen_g4_detsim_hitreco__20240509T212647Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-16 22:03:58
Saved logs	justin-logs:40626.5-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3.9653e-05    0.000366352    0.0144897    0.000196745   0.00142786       100    
reco:cvnmap:CVNMapper                                 3.2232e-05     0.0211413     0.131949      0.020451      0.0217036       100    
reco:cvneva:CVNEvaluator                              1.8683e-05     0.455701       4.89849      0.540943      0.517916        100    
reco:energyrecnumu:EnergyReco                         0.00035913     0.0753175      2.19656      0.0255875     0.256537        100    
reco:energyrecnue:EnergyReco                          0.000170379   0.00357016     0.0585449    0.00106323    0.00938592       100    
reco:energyrecnc:EnergyReco                           0.00245851     0.062733       0.2282       0.0698621     0.0437195       100    
reco:energyrecnumurange:EnergyReco                    0.000409271    0.0453955      1.74988     0.00371267     0.209864        100    
reco:energyrecnumumcs:EnergyReco                      0.00092235     0.0470932      1.76239     0.00438531     0.213023        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000811476    0.0433204      1.45793      0.0044593     0.184921        100    
reco:opdec:Deconvolution                               0.0653039     0.300408      0.598305      0.295799      0.108015        100    
reco:ophitspe:OpHitFinderDeco                          0.0820291     0.0935358     0.115793      0.093084     0.00548096       100    
reco:opflash:OpFlashFinder                            0.000461576   0.00380435     0.186857     0.00181376     0.0184199       100    
reco:opslicer:OpSlicer                                0.00111252     0.371152       1.2084       0.294084      0.296604        100    
reco:rns:RandomNumberSaver                            2.6673e-05    4.69452e-05   0.000614422   3.7524e-05    5.79081e-05      100    
reco:anglereconue:NuAngularReco                       0.000202547   0.00209933     0.0407725    0.000577778   0.00561869       100    
reco:anglereconumu:NuAngularReco                      7.9796e-05     0.0018785     0.0390233    0.000384172   0.00554419       100    
reco:anglereconuepfps:NuAngularReco                   0.000127069   0.00200952     0.0333841    0.000486077   0.00528663       100    
reco:anglereconumupfps:NuAngularReco                  0.000129941    0.0019727     0.0352276    0.000478252   0.00537807       100    
reco:anglerecohits:NuAngularReco                      0.000409029    0.0028527     0.0406275    0.00110734    0.00619573       100    
[art]:TriggerResults:TriggerResultInserter            1.1609e-05    1.93104e-05   9.3354e-05    1.7297e-05    8.7648e-06       100    
end_path:cafmaker:CAFMaker                            0.00164146     0.0339131     0.758726     0.00967667     0.0963524       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3661.98 MB
  Peak resident set size usage (VmHWM): 1963.68 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 23:03:32 BST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/5d/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_1152_20231203T053108Z_gen_g4_detsim_hitreco__20240508T063615Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 0: 993
minwire 1: 1899
minwire 2: 718
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996165, 0.000541577, 0.00329349, 
Output 1: 0.00509086, 0.110824, 0.86639, 0.017695, 
Output 2: 0.95834, 0.0402119, 0.00143255, 1.59092e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2043!
Begin processing the 92nd record. run: 74518431 subRun: 1 event: 34492 at 16-Sep-2025 23:03:03 BST
Boundary wire vector sizes: 184, 164, 162
minwire 0: 456
minwire 1: 1336
minwire 2: 380
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0340664, 0.116852, 0.849082, 
Output 1: 0.0303366, 0.152819, 0.639641, 0.177204, 
Output 2: 0.972471, 0.0265444, 0.000946284, 3.84821e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1510!
Begin processing the 93rd record. run: 74518431 subRun: 1 event: 34493 at 16-Sep-2025 23:03:06 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 539!
Begin processing the 94th record. run: 74518431 subRun: 1 event: 34494 at 16-Sep-2025 23:03:09 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1510!
Begin processing the 95th record. run: 74518431 subRun: 1 event: 34495 at 16-Sep-2025 23:03:12 BST
Boundary wire vector sizes: 261, 194, 217
minwire 0: 314
minwire 1: 2202
minwire 2: 173
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00041332, 0.998032, 0.00155426, 
Output 1: 0.128853, 0.772533, 0.0915774, 0.00703647, 
Output 2: 0.997335, 0.00255181, 0.000108049, 4.93715e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1402!
Begin processing the 96th record. run: 74518431 subRun: 1 event: 34496 at 16-Sep-2025 23:03:16 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1227!
Begin processing the 97th record. run: 74518431 subRun: 1 event: 34497 at 16-Sep-2025 23:03:19 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 499!
Begin processing the 98th record. run: 74518431 subRun: 1 event: 34498 at 16-Sep-2025 23:03:21 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 702!
Begin processing the 99th record. run: 74518431 subRun: 1 event: 34499 at 16-Sep-2025 23:03:24 BST
Boundary wire vector sizes: 44, 51, 26
minwire 0: 1452
minwire 1: 750
minwire 2: 1400
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 305, 604
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.233462, 0.0314259, 0.735112, 
Output 1: 0.523161, 0.408748, 0.0581024, 0.00998865, 
Output 2: 0.754859, 0.24139, 0.00365651, 9.45115e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 805!
Begin processing the 100th record. run: 74518431 subRun: 1 event: 34500 at 16-Sep-2025 23:03:27 BST
Boundary wire vector sizes: 246, 201, 222
minwire 0: 2134
minwire 1: 915
minwire 2: 1912
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00646455, 0.929033, 0.0645024, 
Output 1: 0.200993, 0.737123, 0.0552674, 0.00661652, 
Output 2: 0.360471, 0.58768, 0.0495607, 0.00228787, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1946!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/21/82/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_344_20231204T212511Z_gen_g4_detsim_hitreco__20240510T025447Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/5d/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_1152_20231203T053108Z_gen_g4_detsim_hitreco__20240508T063615Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/57/c5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_342_20231204T215526Z_gen_g4_detsim_hitreco__20240509T221203Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/06/85/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_728_20231203T233241Z_gen_g4_detsim_hitreco__20240509T220016Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/61/48/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_157_20231203T165220Z_gen_g4_detsim_hitreco__20240509T212647Z_reco2.root
.:
total 4844
-rw-r--r-- 1 dune01 dune01 1909161 Sep 16 23:03 flatcaf.root
-rw-r--r-- 1 dune01 dune01 1445611 Sep 16 23:03 caf.root
-rw-r--r-- 1 dune01 dune01 1445611 Sep 16 23:03 caf_fd_hd_atmo_2501_20250916T215124Z.root
-rw-r--r-- 1 dune01 dune01   71980 Sep 16 23:03 caf_20250916T215124Z.log
-rw-r--r-- 1 dune01 dune01   37351 Sep 16 23:03 jobscript.log
-rw-r--r-- 1 dune01 dune01     979 Sep 16 23:03 caf_20250916T215124Z.file
-rw-r--r-- 1 dune01 dune01     979 Sep 16 23:03 caf_20250916T215124Z.pfns
-rw-r--r-- 1 dune01 dune01     979 Sep 16 22:51 file.list
-rw-r--r-- 1 dune01 dune01     979 Sep 16 23:03 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01     688 Sep 16 22:51 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01     688 Sep 16 23:03 caf_20250916T215124Z.did
-rw-r--r-- 1 dune01 dune01     688 Sep 16 22:51 did.list
-rw-r--r-- 1 dune01 dune01       0 Sep 16 22:51 debugprod.log