Jobsub ID 40075.107@dunegpschedd02.fnal.gov

Jobsub ID	40075.107@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 15:06:48
Site	UK_Oxford
Entry	DUNE_UK_SGrid_Oxford_arc01
Last heartbeat	2025-09-15 17:30:34
From worker node	Hostname	t2wn069.physics.ox.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 15:08:44
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_119_20231201T181910Z_gen_g4_detsim_hitreco__20240507T210808Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_260_20231207T060419Z_gen_g4_detsim_hitreco__20240510T062821Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_907_20231205T040030Z_gen_g4_detsim_hitreco__20240510T021300Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_297_20231120T140137Z_gen_g4_detsim_hitreco__20240507T181616Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_470_20231208T092948Z_gen_g4_detsim_hitreco__20240510T074028Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_238_20231201T104611Z_gen_g4_detsim_hitreco__20240507T183748Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_560_20231207T112543Z_gen_g4_detsim_hitreco__20240510T053432Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_536_20231207T131243Z_gen_g4_detsim_hitreco__20240510T044747Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1185_20231205T112818Z_gen_g4_detsim_hitreco__20240509T222619Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_130_20231207T065732Z_gen_g4_detsim_hitreco__20240510T044715Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (7153s)
	CPU time	1h (4819s = 67%)
	Max RSS bytes	2272395264 (2167 MiB)
Outputting started	2025-09-15 17:07:58
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T150857Z.root
Finished	2025-09-15 17:30:34
Saved logs	justin-logs:40075.107-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00175131     0.145465       2.21693      0.149572      0.147449       1100    
reco:energyrecnumurange:EnergyReco                    0.00027752     0.0553475      13.2158     0.00351905     0.443181       1100    
reco:energyrecnumumcs:EnergyReco                      0.000682199    0.0571852      15.0486     0.00415956     0.490939       1100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000486532    0.0549064      12.6647     0.00411207     0.423942       1100    
reco:opdec:Deconvolution                               0.0509801     0.322274       1.12045      0.303696      0.136513       1100    
reco:ophitspe:OpHitFinderDeco                          0.091724      0.211597      0.480386      0.221108      0.0469529      1100    
reco:opflash:OpFlashFinder                             0.000219     0.00190298    0.00567721    0.00182713    0.000810194     1100    
reco:opslicer:OpSlicer                                6.4632e-05      0.31594       1.35546      0.246553      0.257937       1100    
reco:rns:RandomNumberSaver                            2.5468e-05    6.7939e-05    0.000546925    6.42e-05     3.04552e-05     1100    
reco:anglereconue:NuAngularReco                       0.000174377   0.00215577     0.0959929    0.000626575   0.00613254      1100    
reco:anglereconumu:NuAngularReco                      7.9559e-05    0.00179474     0.0937369    0.000348493   0.00601052      1100    
reco:anglereconuepfps:NuAngularReco                   0.000117671   0.00194835     0.0938804    0.000500057   0.00602165      1100    
reco:anglereconumupfps:NuAngularReco                  0.000103053   0.00187912     0.0937776    0.000421701   0.00602197      1100    
reco:anglerecohits:NuAngularReco                      0.000111449   0.00276582     0.101495     0.00101387    0.00681737      1100    
[art]:TriggerResults:TriggerResultInserter             9.939e-06    1.96577e-05   8.4869e-05    1.8349e-05     7.085e-06      1100    
end_path:cafmaker:CAFMaker                            0.00142838      0.02869       1.18425     0.00895066     0.076714       1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3745.2 MB
  Peak resident set size usage (VmHWM): 2272.4 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Classifier summary: 
Output 0: 0.0070667, 0.00190179, 0.991032, 
Output 1: 0.0112144, 0.0876012, 0.881116, 0.0200682, 
Output 2: 0.984695, 0.0144032, 0.000885833, 1.61462e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1542!
Begin processing the 1094th record. run: 6467376 subRun: 1 event: 13094 at 15-Sep-2025 18:06:27 BST
Boundary wire vector sizes: 55, 57, 58
minwire 0: 1692
minwire 1: 1381
minwire 2: 1120
Used alternate method to get min and max wires due to vertex determination failure: 11, 310
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 321, 620
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0252933, 0.0058834, 0.968823, 
Output 1: 0.810145, 0.176783, 0.0120859, 0.000986327, 
Output 2: 0.9872, 0.0123631, 0.000406179, 3.0329e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1793!
Begin processing the 1095th record. run: 6467376 subRun: 1 event: 13095 at 15-Sep-2025 18:06:32 BST
Boundary wire vector sizes: 7215, 9225, 5967
minwire 0: 240
minwire 1: 86
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.472237, 0.510902, 0.0168611, 
Output 1: 0.206373, 0.332738, 0.256206, 0.204683, 
Output 2: 0.0107653, 0.0366458, 0.210169, 0.742419, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3401!
Begin processing the 1096th record. run: 6467376 subRun: 1 event: 13096 at 15-Sep-2025 18:07:38 BST
Boundary wire vector sizes: 180, 245, 229
minwire 0: 70
minwire 1: 2284
minwire 2: 4
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0133059, 0.00217844, 0.984516, 
Output 1: 0.00574965, 0.0296687, 0.158748, 0.805833, 
Output 2: 0.96503, 0.0327486, 0.0019305, 0.00029106, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1074!
Begin processing the 1097th record. run: 6467376 subRun: 1 event: 13097 at 15-Sep-2025 18:07:42 BST
Boundary wire vector sizes: 507, 559, 489
minwire 0: 75
minwire 1: 2156
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00543203, 0.0458321, 0.948736, 
Output 1: 0.914865, 0.082607, 0.00213831, 0.000389507, 
Output 2: 0.979213, 0.0199442, 0.000745224, 9.74982e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1256!
Begin processing the 1098th record. run: 6467376 subRun: 1 event: 13098 at 15-Sep-2025 18:07:46 BST
Boundary wire vector sizes: 124, 192, 124
minwire 0: 1848
minwire 1: 1540
minwire 2: 1535
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000311496, 0.998622, 0.00106614, 
Output 1: 0.355182, 0.641416, 0.00321978, 0.000181586, 
Output 2: 0.999105, 0.000873781, 2.00523e-05, 8.82221e-07, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1811!
Begin processing the 1099th record. run: 6467376 subRun: 1 event: 13099 at 15-Sep-2025 18:07:50 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 500!
Begin processing the 1100th record. run: 6467376 subRun: 1 event: 13100 at 15-Sep-2025 18:07:53 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1480!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ee/57/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_119_20231201T181910Z_gen_g4_detsim_hitreco__20240507T210808Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/97/8b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_260_20231207T060419Z_gen_g4_detsim_hitreco__20240510T062821Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/76/ce/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_907_20231205T040030Z_gen_g4_detsim_hitreco__20240510T021300Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/57/90/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_297_20231120T140137Z_gen_g4_detsim_hitreco__20240507T181616Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_470_20231208T092948Z_gen_g4_detsim_hitreco__20240510T074028Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/bb/ed/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_238_20231201T104611Z_gen_g4_detsim_hitreco__20240507T183748Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/bd/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_560_20231207T112543Z_gen_g4_detsim_hitreco__20240510T053432Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/16/6c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_536_20231207T131243Z_gen_g4_detsim_hitreco__20240510T044747Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/43/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1185_20231205T112818Z_gen_g4_detsim_hitreco__20240509T222619Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/15/98/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_130_20231207T065732Z_gen_g4_detsim_hitreco__20240510T044715Z_reco2.root
.:
total 50408
-rw-r--r-- 1 dune020 dune 18947910 Sep 15 18:07 flatcaf.root
-rw-r--r-- 1 dune020 dune 15852352 Sep 15 18:07 caf.root
-rw-r--r-- 1 dune020 dune 15852352 Sep 15 18:07 caf_fd_hd_atmo_2501_20250915T150857Z.root
-rw-r--r-- 1 dune020 dune   648883 Sep 15 18:07 caf_20250915T150857Z.log
-rw-r--r-- 1 dune020 dune   232799 Sep 15 18:07 jobscript.log
-rw-r--r-- 1 dune020 dune     1956 Sep 15 18:07 caf_20250915T150857Z.file
-rw-r--r-- 1 dune020 dune     1956 Sep 15 18:07 caf_20250915T150857Z.pfns
-rw-r--r-- 1 dune020 dune     1956 Sep 15 16:08 file.list
-rw-r--r-- 1 dune020 dune     1956 Sep 15 18:07 justin-processed-pfns.txt
-rw-r--r-- 1 dune020 dune     1376 Sep 15 16:08 all-input-dids.txt
-rw-r--r-- 1 dune020 dune     1376 Sep 15 18:07 caf_20250915T150857Z.did
-rw-r--r-- 1 dune020 dune     1376 Sep 15 16:08 did.list
-rw-r--r-- 1 dune020 dune        0 Sep 15 16:09 debugprod.log