Jobsub ID 40471.8@dunegpschedd02.fnal.gov

Jobsub ID	40471.8@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-16 16:42:08
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce03
Last heartbeat	2025-09-16 19:08:42
From worker node	Hostname	dune001-8683472.0-lcg2386.gridpp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-16 16:43:12
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_613_20231202T070320Z_gen_g4_detsim_hitreco__20240508T035219Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_408_20231207T084010Z_gen_g4_detsim_hitreco__20240510T051205Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_480_20231207T084736Z_gen_g4_detsim_hitreco__20240510T033654Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_624_20231203T111936Z_gen_g4_detsim_hitreco__20240508T045455Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_138_20231208T033357Z_gen_g4_detsim_hitreco__20240510T034233Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_231_20231203T041941Z_gen_g4_detsim_hitreco__20240508T063647Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_888_20231204T045858Z_gen_g4_detsim_hitreco__20240509T211114Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_253_20231208T045036Z_gen_g4_detsim_hitreco__20240510T065413Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_975_20231204T010113Z_gen_g4_detsim_hitreco__20240509T211722Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_767_20231204T052545Z_gen_g4_detsim_hitreco__20240509T214030Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (8608s)
	CPU time	2h (7532s = 87%)
	Max RSS bytes	2200752128 (2098 MiB)
Outputting started	2025-09-16 19:06:41
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250916T164321Z.root
Finished	2025-09-16 19:08:42
Saved logs	justin-logs:40471.8-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00218668     0.107932      0.587513       0.14106      0.0909081      1000    
reco:energyrecnumurange:EnergyReco                    0.000325847    0.0835673      9.11699     0.00377317     0.485223       1000    
reco:energyrecnumumcs:EnergyReco                      0.000827715    0.0809743      8.89204     0.00448463     0.462431       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000683602    0.0795604      8.97013     0.00450438     0.455637       1000    
reco:opdec:Deconvolution                               0.0268258     0.341254      0.790811      0.335062      0.122963       1000    
reco:ophitspe:OpHitFinderDeco                         0.00486732     0.175167      0.249765      0.207279      0.0646233      1000    
reco:opflash:OpFlashFinder                            0.000235274    0.0020199     0.0149184    0.00190205    0.00097269      1000    
reco:opslicer:OpSlicer                                5.9719e-05     0.379703       1.87042      0.305738      0.312706       1000    
reco:rns:RandomNumberSaver                            2.4894e-05    4.12537e-05   0.00055073    3.67785e-05   2.03908e-05     1000    
reco:anglereconue:NuAngularReco                       0.000184024   0.00292234     0.228885     0.000578603    0.0119033      1000    
reco:anglereconumu:NuAngularReco                      8.3708e-05     0.0027097      0.23237     0.000367901    0.0119486      1000    
reco:anglereconuepfps:NuAngularReco                   0.000134703   0.00283518     0.229905     0.000488277    0.0119154      1000    
reco:anglereconumupfps:NuAngularReco                  0.000116835   0.00281395     0.229332     0.000465799    0.0118917      1000    
reco:anglerecohits:NuAngularReco                      0.000135617   0.00366779      0.24278     0.00101126     0.0128318      1000    
[art]:TriggerResults:TriggerResultInserter            1.0735e-05    1.75078e-05   7.4277e-05    1.6605e-05    3.92509e-06     1000    
end_path:cafmaker:CAFMaker                            0.00150208     0.0374482      2.30245     0.00844658     0.138407       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3922.37 MB
  Peak resident set size usage (VmHWM): 2200.75 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000557008, 0.99442, 0.00502336, 
Output 1: 0.996142, 0.00363156, 0.000121419, 0.000104877, 
Output 2: 0.997375, 0.00252461, 6.98299e-05, 3.07134e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1634!
Begin processing the 994th record. run: 50581281 subRun: 1 event: 76794 at 16-Sep-2025 19:06:07 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 847!
Begin processing the 995th record. run: 50581281 subRun: 1 event: 76795 at 16-Sep-2025 19:06:10 UTC
Boundary wire vector sizes: 112, 124, 130
minwire 0: 1604
minwire 1: 1597
minwire 2: 1270
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.994416, 0.000800633, 0.00478354, 
Output 1: 0.00853872, 0.988639, 0.00275564, 6.66644e-05, 
Output 2: 0.992584, 0.00722643, 0.000185953, 3.93076e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1603!
Begin processing the 996th record. run: 50581281 subRun: 1 event: 76796 at 16-Sep-2025 19:06:14 UTC
Boundary wire vector sizes: 69, 182, 111
minwire 0: 1929
minwire 1: 1469
minwire 2: 1457
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.989906, 0.00111127, 0.00898292, 
Output 1: 0.176862, 0.816335, 0.00659264, 0.00021043, 
Output 2: 0.976743, 0.0229339, 0.000316537, 6.00451e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1935!
Begin processing the 997th record. run: 50581281 subRun: 1 event: 76797 at 16-Sep-2025 19:06:19 UTC
Boundary wire vector sizes: 1130, 933, 290
minwire 0: 797
minwire 1: 1619
minwire 2: 485
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997453, 0.000523187, 0.00202385, 
Output 1: 0.206662, 0.751225, 0.0385443, 0.00356802, 
Output 2: 0.994995, 0.00479702, 0.000197414, 1.10352e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2765!
Begin processing the 998th record. run: 50581281 subRun: 1 event: 76798 at 16-Sep-2025 19:06:26 UTC
Boundary wire vector sizes: 191, 148, 101
minwire 0: 2182
minwire 1: 369
minwire 2: 2465
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995726, 0.000564666, 0.00370976, 
Output 1: 0.43876, 0.559062, 0.00209709, 8.11991e-05, 
Output 2: 0.996369, 0.00355661, 7.15925e-05, 2.58841e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 857!
Begin processing the 999th record. run: 50581281 subRun: 1 event: 76799 at 16-Sep-2025 19:06:30 UTC
Boundary wire vector sizes: 121, 161, 157
minwire 0: 723
minwire 1: 2886
minwire 2: 48
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.987626, 0.00270612, 0.00966812, 
Output 1: 0.0265449, 0.947667, 0.023879, 0.00190904, 
Output 2: 0.0539985, 0.934397, 0.0115641, 4.04356e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1294!
Begin processing the 1000th record. run: 50581281 subRun: 1 event: 76800 at 16-Sep-2025 19:06:34 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 975!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/f2/09/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_613_20231202T070320Z_gen_g4_detsim_hitreco__20240508T035219Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a8/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_408_20231207T084010Z_gen_g4_detsim_hitreco__20240510T051205Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e6/5e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_480_20231207T084736Z_gen_g4_detsim_hitreco__20240510T033654Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/59/e6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_624_20231203T111936Z_gen_g4_detsim_hitreco__20240508T045455Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ef/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_138_20231208T033357Z_gen_g4_detsim_hitreco__20240510T034233Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/83/aa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_231_20231203T041941Z_gen_g4_detsim_hitreco__20240508T063647Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8b/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_888_20231204T045858Z_gen_g4_detsim_hitreco__20240509T211114Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/b8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_253_20231208T045036Z_gen_g4_detsim_hitreco__20240510T065413Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/18/77/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_975_20231204T010113Z_gen_g4_detsim_hitreco__20240509T211722Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/22/97/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_767_20231204T052545Z_gen_g4_detsim_hitreco__20240509T214030Z_reco2.root
.:
total 45432
-rw-r--r-- 1 dune001 dune 17062940 Sep 16 19:06 flatcaf.root
-rw-r--r-- 1 dune001 dune 14280909 Sep 16 19:06 caf.root
-rw-r--r-- 1 dune001 dune 14280909 Sep 16 19:06 caf_fd_hd_atmo_2501_20250916T164321Z.root
-rw-r--r-- 1 dune001 dune   600719 Sep 16 19:06 caf_20250916T164321Z.log
-rw-r--r-- 1 dune001 dune   219135 Sep 16 19:06 jobscript.log
-rw-r--r-- 1 dune001 dune     1930 Sep 16 19:06 caf_20250916T164321Z.file
-rw-r--r-- 1 dune001 dune     1930 Sep 16 19:06 caf_20250916T164321Z.pfns
-rw-r--r-- 1 dune001 dune     1930 Sep 16 16:43 file.list
-rw-r--r-- 1 dune001 dune     1930 Sep 16 19:06 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune     1376 Sep 16 16:43 all-input-dids.txt
-rw-r--r-- 1 dune001 dune     1376 Sep 16 19:06 caf_20250916T164321Z.did
-rw-r--r-- 1 dune001 dune     1376 Sep 16 16:43 did.list
-rw-r--r-- 1 dune001 dune        0 Sep 16 16:43 debugprod.log