Jobsub ID 149785.188@dunegpschedd02.fnal.gov

Jobsub ID	149785.188@dunegpschedd02.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 15:41:58
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce01
Last heartbeat	2025-09-23 18:11:00
From worker node	Hostname	wn1907240.tier2.hep.manchester.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-23 15:43:59
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_639_20231207T044220Z_gen_g4_detsim_hitreco__20240510T060233Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_472_20231208T004704Z_gen_g4_detsim_hitreco__20240510T030843Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_411_20231201T115929Z_gen_g4_detsim_hitreco__20240507T194331Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_989_20231204T230330Z_gen_g4_detsim_hitreco__20240509T222235Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_561_20231203T104431Z_gen_g4_detsim_hitreco__20240508T075625Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_651_20231207T073759Z_gen_g4_detsim_hitreco__20240510T055001Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_49_20231204T055129Z_gen_g4_detsim_hitreco__20240509T215150Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_195_20231204T061836Z_gen_g4_detsim_hitreco__20240509T204406Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_162_20231207T034815Z_gen_g4_detsim_hitreco__20240510T053204Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_741_20231203T121104Z_gen_g4_detsim_hitreco__20240509T200544Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (8796s)
	CPU time	2h (7351s = 83%)
	Max RSS bytes	2241269760 (2137 MiB)
Outputting started	2025-09-23 18:10:36
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/03302/1/001/caf_fd_hd_atmo_3302_20250923T154410Z.root
Finished	2025-09-23 18:11:00
Saved logs	justin-logs:149785.188-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

9479     0.00104713     0.0198257      1000    
reco:energyrecnc:EnergyReco                           0.00212002     0.142599      0.744619       0.17461      0.106638       1000    
reco:energyrecnumurange:EnergyReco                     0.0002835     0.0876193      9.06513     0.00423783     0.444699       1000    
reco:energyrecnumumcs:EnergyReco                      0.000769192    0.0882223      8.93461     0.00494361     0.438887       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.00066698     0.0874888      8.91609     0.00494298     0.444034       1000    
reco:opdec:Deconvolution                               0.057636       0.39362       1.34554       0.38139      0.149732       1000    
reco:ophitspe:OpHitFinderDeco                          0.120374      0.229293      0.414992      0.249076      0.0491984      1000    
reco:opflash:OpFlashFinder                            0.000198891   0.00224513    0.00634419    0.00211148    0.00107559      1000    
reco:opslicer:OpSlicer                                5.0958e-05     0.433562       1.83626      0.324981      0.373827       1000    
reco:rns:RandomNumberSaver                            2.2378e-05    4.79444e-05   0.000798533   3.69855e-05   4.87319e-05     1000    
reco:anglereconue:NuAngularReco                       0.000177064   0.00389815     0.302443     0.000623207    0.0158414      1000    
reco:anglereconumu:NuAngularReco                      9.0855e-05     0.0036923      0.36336     0.000409414    0.0168594      1000    
reco:anglereconuepfps:NuAngularReco                   0.000134855    0.0038158      0.35989     0.000550199    0.0167926      1000    
reco:anglereconumupfps:NuAngularReco                  0.000124194   0.00381165     0.367719     0.000518904    0.0170923      1000    
reco:anglerecohits:NuAngularReco                      0.000124184   0.00484937     0.392038     0.00114656     0.0183387      1000    
[art]:TriggerResults:TriggerResultInserter            1.0928e-05    1.93382e-05   0.000525757   1.7595e-05    1.71001e-05     1000    
end_path:cafmaker:CAFMaker                            0.00157832     0.0469799      2.99974     0.00894423     0.174461       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3904.34 MB
  Peak resident set size usage (VmHWM): 2241.27 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1194!
Begin processing the 995th record. run: 6411871 subRun: 1 event: 74195 at 23-Sep-2025 19:09:54 BST
Boundary wire vector sizes: 4511, 2563, 2097
minwire 0: 792
minwire 1: 1147
minwire 2: 643
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00770162, 0.973254, 0.0190442, 
Output 1: 0.325879, 0.335006, 0.152557, 0.186558, 
Output 2: 0.0712859, 0.263835, 0.346502, 0.318377, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3116!
Begin processing the 996th record. run: 6411871 subRun: 1 event: 74196 at 23-Sep-2025 19:10:11 BST
Boundary wire vector sizes: 317, 146, 235
minwire 0: 1194
minwire 1: 1143
minwire 2: 1416
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995018, 0.000391335, 0.00459033, 
Output 1: 0.991488, 0.00830745, 0.000129541, 7.52537e-05, 
Output 2: 0.996849, 0.00307119, 6.35279e-05, 1.64748e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1308!
Begin processing the 997th record. run: 6411871 subRun: 1 event: 74197 at 23-Sep-2025 19:10:16 BST
Boundary wire vector sizes: 108, 84, 82
minwire 0: 558
minwire 1: 2892
minwire 2: 7
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0189342, 0.00390867, 0.977157, 
Output 1: 0.0195526, 0.298559, 0.616269, 0.0656192, 
Output 2: 0.994259, 0.00546331, 0.000267224, 1.01323e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 937!
Begin processing the 998th record. run: 6411871 subRun: 1 event: 74198 at 23-Sep-2025 19:10:20 BST
Boundary wire vector sizes: 102, 64, 55
minwire 0: 987
minwire 1: 2200
minwire 2: 722
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000281399, 0.998723, 0.000995615, 
Output 1: 0.0194668, 0.974066, 0.00618685, 0.000280582, 
Output 2: 0.998934, 0.00103319, 3.17817e-05, 1.09072e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1383!
Begin processing the 999th record. run: 6411871 subRun: 1 event: 74199 at 23-Sep-2025 19:10:24 BST
Boundary wire vector sizes: 237, 174, 151
minwire 0: 2165
minwire 1: 474
minwire 2: 1977
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000516831, 0.996465, 0.00301778, 
Output 1: 0.614747, 0.375325, 0.00915674, 0.000771693, 
Output 2: 0.995307, 0.00457786, 0.000108977, 6.12927e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1580!
Begin processing the 1000th record. run: 6411871 subRun: 1 event: 74200 at 23-Sep-2025 19:10:28 BST
Boundary wire vector sizes: 331, 236, 275
minwire 0: 2285
minwire 1: 1112
minwire 2: 2062
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.986981, 0.0098511, 0.00316779, 
Output 1: 0.0159726, 0.924333, 0.0586776, 0.00101678, 
Output 2: 0.952183, 0.0470535, 0.000752053, 1.11717e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2036!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/95/65/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_639_20231207T044220Z_gen_g4_detsim_hitreco__20240510T060233Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b3/51/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_472_20231208T004704Z_gen_g4_detsim_hitreco__20240510T030843Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ba/51/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_411_20231201T115929Z_gen_g4_detsim_hitreco__20240507T194331Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/61/bf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_989_20231204T230330Z_gen_g4_detsim_hitreco__20240509T222235Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/57/2e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_561_20231203T104431Z_gen_g4_detsim_hitreco__20240508T075625Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/0c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_651_20231207T073759Z_gen_g4_detsim_hitreco__20240510T055001Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7d/0a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_49_20231204T055129Z_gen_g4_detsim_hitreco__20240509T215150Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ab/21/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_195_20231204T061836Z_gen_g4_detsim_hitreco__20240509T204406Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/51/f6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_162_20231207T034815Z_gen_g4_detsim_hitreco__20240510T053204Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_741_20231203T121104Z_gen_g4_detsim_hitreco__20240509T200544Z_reco2.root
.:
total 49248
-rw-r--r-- 1 dune01 dune01 18485673 Sep 23 19:10 flatcaf.root
-rw-r--r-- 1 dune01 dune01 15529387 Sep 23 19:10 caf.root
-rw-r--r-- 1 dune01 dune01 15529387 Sep 23 19:10 caf_fd_hd_atmo_3302_20250923T154410Z.root
-rw-r--r-- 1 dune01 dune01   589206 Sep 23 19:10 caf_20250923T154410Z.log
-rw-r--r-- 1 dune01 dune01   243703 Sep 23 19:10 jobscript.log
-rw-r--r-- 1 dune01 dune01     1926 Sep 23 19:10 caf_20250923T154410Z.file
-rw-r--r-- 1 dune01 dune01     1926 Sep 23 19:10 caf_20250923T154410Z.pfns
-rw-r--r-- 1 dune01 dune01     1926 Sep 23 16:44 file.list
-rw-r--r-- 1 dune01 dune01     1926 Sep 23 19:10 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01     1376 Sep 23 16:44 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01     1376 Sep 23 19:10 caf_20250923T154410Z.did
-rw-r--r-- 1 dune01 dune01     1376 Sep 23 16:44 did.list
-rw-r--r-- 1 dune01 dune01        0 Sep 23 16:44 debugprod.log