Jobsub ID 40063.39@dunegpschedd02.fnal.gov

Jobsub ID	40063.39@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 14:50:48
Site	UK_Bristol
Entry	CMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat	2025-09-15 16:19:00
From worker node	Hostname	hd87.dice.priv
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	259200 (72 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 14:52:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_639_20231202T121521Z_gen_g4_detsim_hitreco__20240507T212647Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_368_20231207T072056Z_gen_g4_detsim_hitreco__20240510T051500Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_594_20231204T014145Z_gen_g4_detsim_hitreco__20240509T213341Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_761_20231207T115635Z_gen_g4_detsim_hitreco__20240510T054430Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_1112_20231205T063616Z_gen_g4_detsim_hitreco__20240509T222135Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_598_20231205T053853Z_gen_g4_detsim_hitreco__20240509T221942Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_792_20231207T075407Z_gen_g4_detsim_hitreco__20240510T021738Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_741_20231205T074001Z_gen_g4_detsim_hitreco__20240510T021549Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_205_20231203T105724Z_gen_g4_detsim_hitreco__20240509T193746Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_124_20231205T073245Z_gen_g4_detsim_hitreco__20240510T033303Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3837s)
	CPU time	46m (2810s = 73%)
	Max RSS bytes	2066059264 (1970 MiB)
Outputting started	2025-09-15 15:56:34
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T145244Z.root
Finished	2025-09-15 16:19:00
Saved logs	justin-logs:40063.39-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

cnc:EnergyReco                           0.00175038     0.0933122     0.382482      0.0910617     0.0724938      1000    
reco:energyrecnumurange:EnergyReco                    0.000122795    0.0243904      2.55773      0.0018828     0.118738       1000    
reco:energyrecnumumcs:EnergyReco                      0.000323369    0.0250285      2.55312     0.00216556     0.118584       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000255569    0.0250316      2.5663      0.00214413     0.118959       1000    
reco:opdec:Deconvolution                               0.0447135     0.199177       0.44013      0.199392      0.0624034      1000    
reco:ophitspe:OpHitFinderDeco                          0.0952408     0.175606       1.45668      0.217285      0.0735197      1000    
reco:opflash:OpFlashFinder                            0.000140158   0.000879009   0.00341578    0.000846551   0.000384403     1000    
reco:opslicer:OpSlicer                                4.5948e-05     0.179451      0.852146      0.142506      0.144887       1000    
reco:rns:RandomNumberSaver                            1.3034e-05    1.70557e-05   0.000265278   1.59305e-05   8.43667e-06     1000    
reco:anglereconue:NuAngularReco                       7.7298e-05    0.00115569     0.045477     0.000216003   0.00403695      1000    
reco:anglereconumu:NuAngularReco                      5.6268e-05    0.00108906     0.0458429    0.000151781   0.00402567      1000    
reco:anglereconuepfps:NuAngularReco                   8.5043e-05    0.00115286     0.0454804    0.000214275   0.00403291      1000    
reco:anglereconumupfps:NuAngularReco                  8.0945e-05     0.0011353     0.0442193    0.00019592    0.00399634      1000    
reco:anglerecohits:NuAngularReco                      6.3672e-05    0.00154201     0.0486662    0.000472691    0.0044119      1000    
[art]:TriggerResults:TriggerResultInserter             7.374e-06    8.59439e-06   3.2743e-05     8.366e-06    1.75582e-06     1000    
end_path:cafmaker:CAFMaker                             0.0010247     0.0193383      3.92055     0.00297126     0.133829       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3732.8 MB
  Peak resident set size usage (VmHWM): 2066.06 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 945, 861, 660
minwire 0: 33
minwire 1: 1023
minwire 2: 14
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000759848, 0.992149, 0.00709079, 
Output 1: 0.807077, 0.189784, 0.00287432, 0.000264276, 
Output 2: 0.279514, 0.6922, 0.0274958, 0.000790588, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2707!
Begin processing the 995th record. run: 50602933 subRun: 1 event: 12495 at 15-Sep-2025 15:56:12 UTC
Boundary wire vector sizes: 2875, 3323, 2527
minwire 0: 440
minwire 1: 568
minwire 2: 346
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.459013, 0.475903, 0.065084, 
Output 1: 0.228657, 0.312363, 0.278602, 0.180379, 
Output 2: 0.194528, 0.522756, 0.225783, 0.0569327, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3104!
Begin processing the 996th record. run: 50602933 subRun: 1 event: 12496 at 15-Sep-2025 15:56:18 UTC
Boundary wire vector sizes: 571, 326, 411
minwire 0: 1925
minwire 1: 440
minwire 2: 2182
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000322707, 0.997659, 0.00201833, 
Output 1: 0.533854, 0.453935, 0.0116333, 0.000577714, 
Output 2: 0.988315, 0.0114726, 0.000206057, 6.74113e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1695!
Begin processing the 997th record. run: 50602933 subRun: 1 event: 12497 at 15-Sep-2025 15:56:21 UTC
Boundary wire vector sizes: 185, 394, 146
minwire 0: 192
minwire 1: 2365
minwire 2: 13
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996893, 0.000509334, 0.00259781, 
Output 1: 0.00195009, 0.993627, 0.00433734, 8.5334e-05, 
Output 2: 0.996305, 0.00362518, 6.85093e-05, 1.19382e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1025!
Begin processing the 998th record. run: 50602933 subRun: 1 event: 12498 at 15-Sep-2025 15:56:24 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1114!
Begin processing the 999th record. run: 50602933 subRun: 1 event: 12499 at 15-Sep-2025 15:56:26 UTC
Boundary wire vector sizes: 504, 469, 376
minwire 0: 1238
minwire 1: 821
minwire 2: 886
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0623411, 0.704443, 0.233216, 
Output 1: 0.235008, 0.5141, 0.20463, 0.0462622, 
Output 2: 0.022264, 0.0931768, 0.376271, 0.508289, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2492!
Begin processing the 1000th record. run: 50602933 subRun: 1 event: 12500 at 15-Sep-2025 15:56:29 UTC
Boundary wire vector sizes: 815, 773, 553
minwire 0: 1653
minwire 1: 736
minwire 2: 1114
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000423622, 0.995965, 0.00361168, 
Output 1: 0.932916, 0.0664443, 0.000606393, 3.32005e-05, 
Output 2: 0.990519, 0.00929175, 0.000179663, 9.21045e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2611!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/8d/2e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_639_20231202T121521Z_gen_g4_detsim_hitreco__20240507T212647Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/55/6e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_368_20231207T072056Z_gen_g4_detsim_hitreco__20240510T051500Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/66/f8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_594_20231204T014145Z_gen_g4_detsim_hitreco__20240509T213341Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/b2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_761_20231207T115635Z_gen_g4_detsim_hitreco__20240510T054430Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/71/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_1112_20231205T063616Z_gen_g4_detsim_hitreco__20240509T222135Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/67/cb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_598_20231205T053853Z_gen_g4_detsim_hitreco__20240509T221942Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ec/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_792_20231207T075407Z_gen_g4_detsim_hitreco__20240510T021738Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e7/93/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_741_20231205T074001Z_gen_g4_detsim_hitreco__20240510T021549Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/0a/8b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_205_20231203T105724Z_gen_g4_detsim_hitreco__20240509T193746Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d3/0e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_124_20231205T073245Z_gen_g4_detsim_hitreco__20240510T033303Z_reco2.root
.:
total 47319
-rw-r--r-- 1 dnpil001 users 17799548 Sep 15 15:56 flatcaf.root
-rw-r--r-- 1 dnpil001 users 14900095 Sep 15 15:56 caf.root
-rw-r--r-- 1 dnpil001 users 14900095 Sep 15 15:56 caf_fd_hd_atmo_2501_20250915T145244Z.root
-rw-r--r-- 1 dnpil001 users   602929 Sep 15 15:56 caf_20250915T145244Z.log
-rw-r--r-- 1 dnpil001 users   237882 Sep 15 15:56 jobscript.log
-rw-r--r-- 1 dnpil001 users     1991 Sep 15 15:56 caf_20250915T145244Z.file
-rw-r--r-- 1 dnpil001 users     1991 Sep 15 15:56 caf_20250915T145244Z.pfns
-rw-r--r-- 1 dnpil001 users     1991 Sep 15 14:52 file.list
-rw-r--r-- 1 dnpil001 users     1991 Sep 15 15:56 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users     1375 Sep 15 14:52 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users     1375 Sep 15 15:56 caf_20250915T145244Z.did
-rw-r--r-- 1 dnpil001 users     1375 Sep 15 14:52 did.list
-rw-r--r-- 1 dnpil001 users        0 Sep 15 14:52 debugprod.log