Jobsub ID 49216.20@dunegpschedd01.fnal.gov

Jobsub ID	49216.20@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-18 23:49:00
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw6
Last heartbeat	2025-09-19 03:40:05
From worker node	Hostname	mh-7662-7.t2.ucsd.edu
	cpuinfo	AMD EPYC 7662 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-18 23:50:28
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_66_20231121T152815Z_gen_g4_detsim_hitreco__20240503T053351Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_934_20231205T063057Z_gen_g4_detsim_hitreco__20240510T022935Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_236_20231201T071031Z_gen_g4_detsim_hitreco__20240507T184439Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_722_20231202T121954Z_gen_g4_detsim_hitreco__20240507T212425Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_709_20231207T024606Z_gen_g4_detsim_hitreco__20240510T042815Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_958_20231203T235355Z_gen_g4_detsim_hitreco__20240509T211817Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_708_20231207T040841Z_gen_g4_detsim_hitreco__20240510T050942Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_442_20231126T014826Z_gen_g4_detsim_hitreco__20240507T182808Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_212_20231202T075350Z_gen_g4_detsim_hitreco__20240507T210051Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_184_20231203T033407Z_gen_g4_detsim_hitreco__20240508T071726Z_reco2.root
Jobscript	Exit code	0
	Real time	3h (13750s)
	CPU time	1h (6052s = 44%)
	Max RSS bytes	2040029184 (1945 MiB)
Outputting started	2025-09-19 03:39:38
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250918T235037Z.root
Finished	2025-09-19 03:40:05
Saved logs	justin-logs:49216.20-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0835033   0.00876691      1200    
reco:energyrecnc:EnergyReco                           0.00254425     0.482432       1.52606      0.618523      0.331343       1200    
reco:energyrecnumurange:EnergyReco                    0.000242356    0.0458949      4.16467     0.00409861     0.200275       1200    
reco:energyrecnumumcs:EnergyReco                      0.000601074    0.0456686      4.14232     0.00460308     0.193158       1200    
reco:energyrecnumumcsllhd:EnergyReco                  0.000509341    0.0461343      4.25529     0.00457086     0.197328       1200    
reco:opdec:Deconvolution                               0.437459       0.77829       1.47935      0.800769      0.165142       1200    
reco:ophitspe:OpHitFinderDeco                           1.28578       1.34872       1.39043       1.36556      0.0311558      1200    
reco:opflash:OpFlashFinder                            0.000173284   0.00180037     0.0103833    0.00169955    0.000912848     1200    
reco:opslicer:OpSlicer                                3.9961e-05     0.338619       1.66328       0.25807      0.288146       1200    
reco:rns:RandomNumberSaver                            2.0661e-05    4.25176e-05   0.000631054   3.6755e-05    2.81552e-05     1200    
reco:anglereconue:NuAngularReco                       0.000165743   0.00204571      0.10059     0.000511316   0.00639091      1200    
reco:anglereconumu:NuAngularReco                      8.1022e-05    0.00183296     0.114765     0.000297866   0.00661577      1200    
reco:anglereconuepfps:NuAngularReco                   0.000124423    0.0019668     0.111426     0.000429674   0.00665638      1200    
reco:anglereconumupfps:NuAngularReco                  0.000112523   0.00194347     0.114054     0.000382123   0.00676557      1200    
reco:anglerecohits:NuAngularReco                      9.1912e-05    0.00275866     0.128404     0.000946195   0.00758027      1200    
[art]:TriggerResults:TriggerResultInserter            1.0631e-05    1.65715e-05   0.000144503   1.54455e-05   1.10915e-05     1200    
end_path:cafmaker:CAFMaker                            0.00145288     0.0266224      1.39714     0.00608278     0.0851045      1200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3786.12 MB
  Peak resident set size usage (VmHWM): 2040.03 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 802!
Begin processing the 1195th record. run: 66006611 subRun: 1 event: 18495 at 18-Sep-2025 20:38:36 PDT
Boundary wire vector sizes: 489, 591, 345
minwire 0: 808
minwire 1: 1392
minwire 2: 1020
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00876229, 0.263501, 0.727736, 
Output 1: 0.213347, 0.695159, 0.077217, 0.0142763, 
Output 2: 0.949336, 0.0457105, 0.00434351, 0.000610387, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2267!
Begin processing the 1196th record. run: 66006611 subRun: 1 event: 18496 at 18-Sep-2025 20:38:45 PDT
Boundary wire vector sizes: 432, 553, 587
minwire 0: 511
minwire 1: 1480
minwire 2: 546
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996849, 0.000589734, 0.00256102, 
Output 1: 0.0181921, 0.974099, 0.00754233, 0.000166246, 
Output 2: 0.997524, 0.00241458, 6.0505e-05, 1.24649e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1327!
Begin processing the 1197th record. run: 66006611 subRun: 1 event: 18497 at 18-Sep-2025 20:38:54 PDT
Boundary wire vector sizes: 228, 190, 179
minwire 0: 2189
minwire 1: 991
minwire 2: 1983
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00817004, 0.0142585, 0.977571, 
Output 1: 0.207248, 0.764744, 0.0261918, 0.00181704, 
Output 2: 0.965811, 0.0326386, 0.00147334, 7.69485e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2025!
Begin processing the 1198th record. run: 66006611 subRun: 1 event: 18498 at 18-Sep-2025 20:39:03 PDT
Boundary wire vector sizes: 1404, 1284, 967
minwire 0: 2258
minwire 1: 573
minwire 2: 2311
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00065318, 0.996959, 0.00238784, 
Output 1: 0.00862251, 0.227521, 0.61738, 0.146476, 
Output 2: 0.148804, 0.715497, 0.132762, 0.00293616, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2406!
Begin processing the 1199th record. run: 66006611 subRun: 1 event: 18499 at 18-Sep-2025 20:39:18 PDT
Boundary wire vector sizes: 158, 194, 146
minwire 0: 1151
minwire 1: 1205
minwire 2: 1371
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996532, 0.000571074, 0.00289705, 
Output 1: 0.0100956, 0.987091, 0.00272875, 8.49894e-05, 
Output 2: 0.994372, 0.00549376, 0.000131443, 3.09429e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1295!
Begin processing the 1200th record. run: 66006611 subRun: 1 event: 18500 at 18-Sep-2025 20:39:27 PDT
Boundary wire vector sizes: 89, 121, 104
minwire 0: 918
minwire 1: 1486
minwire 2: 978
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000285356, 0.998245, 0.00146997, 
Output 1: 0.908129, 0.0914013, 0.000432557, 3.73413e-05, 
Output 2: 0.998842, 0.0011363, 2.0538e-05, 1.55004e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1150!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/dd/41/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_66_20231121T152815Z_gen_g4_detsim_hitreco__20240503T053351Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/2e/60/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_934_20231205T063057Z_gen_g4_detsim_hitreco__20240510T022935Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c5/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_236_20231201T071031Z_gen_g4_detsim_hitreco__20240507T184439Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/85/92/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_722_20231202T121954Z_gen_g4_detsim_hitreco__20240507T212425Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/70/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_709_20231207T024606Z_gen_g4_detsim_hitreco__20240510T042815Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/63/14/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_958_20231203T235355Z_gen_g4_detsim_hitreco__20240509T211817Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/05/f1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_708_20231207T040841Z_gen_g4_detsim_hitreco__20240510T050942Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ff/c1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6171714_442_20231126T014826Z_gen_g4_detsim_hitreco__20240507T182808Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/34/1e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_212_20231202T075350Z_gen_g4_detsim_hitreco__20240507T210051Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/53/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_184_20231203T033407Z_gen_g4_detsim_hitreco__20240508T071726Z_reco2.root
.:
total 53256
-rw-r--r--. 1 cuser cuser 20023851 Sep 18 20:39 flatcaf.root
-rw-r--r--. 1 cuser cuser 16744103 Sep 18 20:39 caf.root
-rw-r--r--. 1 cuser cuser 16744103 Sep 18 20:39 caf_fd_hd_atmo_2501_20250918T235037Z.root
-rw-r--r--. 1 cuser cuser   709449 Sep 18 20:39 caf_20250918T235037Z.log
-rw-r--r--. 1 cuser cuser   260776 Sep 18 20:39 jobscript.log
-rw-r--r--. 1 cuser cuser     1944 Sep 18 20:39 caf_20250918T235037Z.file
-rw-r--r--. 1 cuser cuser     1944 Sep 18 20:39 caf_20250918T235037Z.pfns
-rw-r--r--. 1 cuser cuser     1944 Sep 18 16:50 file.list
-rw-r--r--. 1 cuser cuser     1944 Sep 18 20:39 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser     1376 Sep 18 16:50 all-input-dids.txt
-rw-r--r--. 1 cuser cuser     1376 Sep 18 20:39 caf_20250918T235037Z.did
-rw-r--r--. 1 cuser cuser     1376 Sep 18 16:50 did.list
-rw-r--r--. 1 cuser cuser        0 Sep 18 16:50 debugprod.log