Jobsub ID 46603.10@dunegpschedd01.fnal.gov

Jobsub ID	46603.10@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:31:41
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-09-15 14:22:09
From worker node	Hostname	comp15-03
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 12:33:38
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_302_20231203T072841Z_gen_g4_detsim_hitreco__20240508T054215Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_29_20231204T055008Z_gen_g4_detsim_hitreco__20240509T214921Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_581_20231207T043911Z_gen_g4_detsim_hitreco__20240510T055630Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_347_20231202T200635Z_gen_g4_detsim_hitreco__20240508T065932Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_482_20231203T080427Z_gen_g4_detsim_hitreco__20240509T205420Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_1004_20231204T010123Z_gen_g4_detsim_hitreco__20240509T202655Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_65_20231208T084856Z_gen_g4_detsim_hitreco__20240510T062640Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_743_20231207T160221Z_gen_g4_detsim_hitreco__20240510T045527Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_110_20231204T120340Z_gen_g4_detsim_hitreco__20240509T214536Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_855_20231202T130923Z_gen_g4_detsim_hitreco__20240507T222124Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5186s)
	CPU time	1h (3939s = 75%)
	Max RSS bytes	2340335616 (2231 MiB)
Outputting started	2025-09-15 14:00:05
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T123345Z.root
Finished	2025-09-15 14:22:09
Saved logs	justin-logs:46603.10-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00641578      1000    
reco:energyrecnc:EnergyReco                           0.00149048     0.133715       1.19397      0.165313      0.105463       1000    
reco:energyrecnumurange:EnergyReco                    0.000171942    0.0261469      1.8166      0.00166475      0.11744       1000    
reco:energyrecnumumcs:EnergyReco                      0.000399415    0.0266601      1.81308     0.00192997     0.117527       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000302854    0.0266091      1.80943     0.00187478     0.117497       1000    
reco:opdec:Deconvolution                               0.0532616     0.242823       2.39065      0.238475      0.0982397      1000    
reco:ophitspe:OpHitFinderDeco                           0.13914      0.246076      0.314509      0.263711      0.0371865      1000    
reco:opflash:OpFlashFinder                            0.000142791   0.00102721    0.00304096    0.000993672   0.000460109     1000    
reco:opslicer:OpSlicer                                 2.821e-05     0.204571      0.908843      0.165875      0.168672       1000    
reco:rns:RandomNumberSaver                             1.726e-05    2.70895e-05   0.000298303    2.547e-05    1.05433e-05     1000    
reco:anglereconue:NuAngularReco                       0.000134652   0.00117094     0.0689889    0.000300568   0.00386803      1000    
reco:anglereconumu:NuAngularReco                       5.652e-05    0.00101881     0.0693616    0.000152932   0.00386049      1000    
reco:anglereconuepfps:NuAngularReco                   9.0521e-05    0.00109658     0.0693605    0.000228008   0.00387326      1000    
reco:anglereconumupfps:NuAngularReco                  7.3701e-05    0.00106434     0.0703174    0.000192677   0.00389115      1000    
reco:anglerecohits:NuAngularReco                      5.9211e-05    0.00152049     0.0760644    0.000482905   0.00439244      1000    
[art]:TriggerResults:TriggerResultInserter             8.38e-06     1.0551e-05     4.259e-05     1.022e-05    1.8464e-06      1000    
end_path:cafmaker:CAFMaker                            0.000976422    0.0149632      1.08303     0.00306886     0.054761       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4013.7 MB
  Peak resident set size usage (VmHWM): 2340.34 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 510, 400, 292
minwire 0: 1227
minwire 1: 2088
minwire 2: 908
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00023176, 0.998324, 0.00144441, 
Output 1: 0.679316, 0.318411, 0.00211122, 0.000161626, 
Output 2: 0.998419, 0.00154809, 3.13595e-05, 1.84747e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2533!
Begin processing the 992nd record. run: 6406500 subRun: 1 event: 85592 at 15-Sep-2025 14:59:39 BST
Boundary wire vector sizes: 498, 379, 251
minwire 0: 1613
minwire 1: 687
minwire 2: 1913
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99375, 0.000889726, 0.00536008, 
Output 1: 0.0167809, 0.0836251, 0.43736, 0.462235, 
Output 2: 0.979997, 0.0191141, 0.000823265, 6.5675e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1310!
Begin processing the 993rd record. run: 6406500 subRun: 1 event: 85593 at 15-Sep-2025 14:59:42 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 375!
Begin processing the 994th record. run: 6406500 subRun: 1 event: 85594 at 15-Sep-2025 14:59:45 BST
Boundary wire vector sizes: 156, 149, 134
minwire 0: 979
minwire 1: 2512
minwire 2: 352
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0268206, 0.00333277, 0.969847, 
Output 1: 0.0154028, 0.944115, 0.0389267, 0.00155585, 
Output 2: 0.0276133, 0.890994, 0.0803725, 0.00102034, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1861!
Begin processing the 995th record. run: 6406500 subRun: 1 event: 85595 at 15-Sep-2025 14:59:47 BST
Boundary wire vector sizes: 94, 173, 125
minwire 0: 883
minwire 1: 1650
minwire 2: 808
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00031877, 0.998742, 0.000939671, 
Output 1: 0.00364241, 0.99227, 0.00396802, 0.000119207, 
Output 2: 0.99849, 0.00146741, 4.12837e-05, 1.25085e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1290!
Begin processing the 996th record. run: 6406500 subRun: 1 event: 85596 at 15-Sep-2025 14:59:50 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 837!
Begin processing the 997th record. run: 6406500 subRun: 1 event: 85597 at 15-Sep-2025 14:59:53 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 588!
Begin processing the 998th record. run: 6406500 subRun: 1 event: 85598 at 15-Sep-2025 14:59:55 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1153!
Begin processing the 999th record. run: 6406500 subRun: 1 event: 85599 at 15-Sep-2025 14:59:57 BST
Boundary wire vector sizes: 230, 246, 201
minwire 0: 1045
minwire 1: 1243
minwire 2: 853
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0344441, 0.00652802, 0.959028, 
Output 1: 0.138473, 0.701324, 0.154587, 0.00561546, 
Output 2: 0.766054, 0.215624, 0.0177922, 0.000529817, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2093!
Begin processing the 1000th record. run: 6406500 subRun: 1 event: 85600 at 15-Sep-2025 15:00:00 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 934!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c9/99/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_302_20231203T072841Z_gen_g4_detsim_hitreco__20240508T054215Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/cb/83/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_29_20231204T055008Z_gen_g4_detsim_hitreco__20240509T214921Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/cf/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_581_20231207T043911Z_gen_g4_detsim_hitreco__20240510T055630Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4c/d9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_347_20231202T200635Z_gen_g4_detsim_hitreco__20240508T065932Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6f/da/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_482_20231203T080427Z_gen_g4_detsim_hitreco__20240509T205420Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/64/fb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_1004_20231204T010123Z_gen_g4_detsim_hitreco__20240509T202655Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/66/0a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_65_20231208T084856Z_gen_g4_detsim_hitreco__20240510T062640Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/8a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_743_20231207T160221Z_gen_g4_detsim_hitreco__20240510T045527Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2b/6d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_110_20231204T120340Z_gen_g4_detsim_hitreco__20240509T214536Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/cd/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_855_20231202T130923Z_gen_g4_detsim_hitreco__20240507T222124Z_reco2.root
.:
total 46428
-rw-r--r-- 1 pltdune002 pltdune 17441835 Sep 15 15:00 flatcaf.root
-rw-r--r-- 1 pltdune002 pltdune 14570220 Sep 15 15:00 caf.root
-rw-r--r-- 1 pltdune002 pltdune 14570220 Sep 15 15:00 caf_fd_hd_atmo_2501_20250915T123345Z.root
-rw-r--r-- 1 pltdune002 pltdune   588428 Sep 15 15:00 caf_20250915T123345Z.log
-rw-r--r-- 1 pltdune002 pltdune   214763 Sep 15 15:00 jobscript.log
-rw-r--r-- 1 pltdune002 pltdune     1933 Sep 15 15:00 caf_20250915T123345Z.file
-rw-r--r-- 1 pltdune002 pltdune     1933 Sep 15 15:00 caf_20250915T123345Z.pfns
-rw-r--r-- 1 pltdune002 pltdune     1933 Sep 15 13:33 file.list
-rw-r--r-- 1 pltdune002 pltdune     1933 Sep 15 15:00 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune002 pltdune     1373 Sep 15 13:33 all-input-dids.txt
-rw-r--r-- 1 pltdune002 pltdune     1373 Sep 15 15:00 caf_20250915T123345Z.did
-rw-r--r-- 1 pltdune002 pltdune     1373 Sep 15 13:33 did.list
-rw-r--r-- 1 pltdune002 pltdune        0 Sep 15 13:33 debugprod.log