Jobsub ID 39988.121@dunegpschedd02.fnal.gov

Jobsub ID	39988.121@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:33:41
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_condce_mcore
Last heartbeat	2025-09-15 15:10:52
From worker node	Hostname	mwt2-c061.campuscluster.illinois.edu
	cpuinfo	AMD EPYC 7443 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 12:35:31
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_372_20231202T142042Z_gen_g4_detsim_hitreco__20240508T055329Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_629_20231207T085330Z_gen_g4_detsim_hitreco__20240510T035004Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_152_20231201T212704Z_gen_g4_detsim_hitreco__20240508T044430Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_729_20231207T073918Z_gen_g4_detsim_hitreco__20240510T054951Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_1_20231122T083054Z_gen_g4_detsim_hitreco__20240503T054823Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_523_20231206T231708Z_gen_g4_detsim_hitreco__20240510T043431Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6454800_291_20231207T101829Z_gen_g4_detsim_hitreco__20240510T050310Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_338_20231207T033425Z_gen_g4_detsim_hitreco__20240510T034020Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_210_20231207T161958Z_gen_g4_detsim_hitreco__20240510T051711Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_213_20231207T010119Z_gen_g4_detsim_hitreco__20240510T043458Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (9248s)
	CPU time	1h (4864s = 52%)
	Max RSS bytes	2104758272 (2007 MiB)
Outputting started	2025-09-15 15:09:39
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T123542Z.root
Finished	2025-09-15 15:10:52
Saved logs	justin-logs:39988.121-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gyReco                           0.00150695     0.373413       1.14924      0.449949      0.250502       1000    
reco:energyrecnumurange:EnergyReco                    0.000273774    0.0447064      2.86912     0.00289385     0.204902       1000    
reco:energyrecnumumcs:EnergyReco                      0.000633239    0.0440351      2.90333     0.00335604     0.195052       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000454434    0.043657       2.83539     0.00335029     0.192236       1000    
reco:opdec:Deconvolution                               0.288928      0.553558       1.2738        0.56808       0.12068       1000    
reco:ophitspe:OpHitFinderDeco                          0.851366      0.957419       1.07237      0.983915      0.064744       1000    
reco:opflash:OpFlashFinder                            0.000171372   0.00160312     0.0046914    0.00153773    0.00068693      1000    
reco:opslicer:OpSlicer                                3.0858e-05     0.265554       1.20266      0.204563      0.228269       1000    
reco:rns:RandomNumberSaver                            1.9246e-05    7.57597e-05   0.000945546   6.5318e-05    5.94917e-05     1000    
reco:anglereconue:NuAngularReco                       0.000145964   0.00195162     0.0696206    0.000539654   0.00531773      1000    
reco:anglereconumu:NuAngularReco                      8.0881e-05    0.00158285     0.0683427    0.000277631   0.00492913      1000    
reco:anglereconuepfps:NuAngularReco                   0.000107441   0.00173101     0.0694705    0.000423675   0.00497338      1000    
reco:anglereconumupfps:NuAngularReco                  8.5591e-05    0.00164491     0.0605643    0.000341532    0.0048739      1000    
reco:anglerecohits:NuAngularReco                       7.474e-05    0.00246483     0.062749     0.000853164   0.00565475      1000    
[art]:TriggerResults:TriggerResultInserter             8.496e-06    1.78473e-05   0.000319289   1.5795e-05    1.42759e-05     1000    
end_path:cafmaker:CAFMaker                            0.00122873     0.0255182      1.15246     0.00665649     0.0721613      1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3810.35 MB
  Peak resident set size usage (VmHWM): 2104.76 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2967!
Begin processing the 995th record. run: 74530544 subRun: 1 event: 21395 at 15-Sep-2025 10:08:55 CDT
Boundary wire vector sizes: 312, 195, 227
minwire 0: 201
minwire 1: 2594
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00366821, 0.97121, 0.0251217, 
Output 1: 0.718969, 0.271147, 0.00903375, 0.000850367, 
Output 2: 0.174459, 0.813857, 0.0115101, 0.000174121, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1087!
Begin processing the 996th record. run: 74530544 subRun: 1 event: 21396 at 15-Sep-2025 10:09:02 CDT
Boundary wire vector sizes: 408, 539, 562
minwire 0: 334
minwire 1: 1911
minwire 2: 167
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997434, 0.000410179, 0.00215604, 
Output 1: 0.0334608, 0.878622, 0.0847859, 0.00313109, 
Output 2: 0.997311, 0.00259192, 9.50312e-05, 2.30803e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1346!
Begin processing the 997th record. run: 74530544 subRun: 1 event: 21397 at 15-Sep-2025 10:09:10 CDT
Boundary wire vector sizes: 53, 63, 53
minwire 0: 286
minwire 1: 2585
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.452032, 0.308728, 0.23924, 
Output 1: 0.0527863, 0.361674, 0.526775, 0.058765, 
Output 2: 0.723746, 0.265092, 0.0108891, 0.000272572, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 741!
Begin processing the 998th record. run: 74530544 subRun: 1 event: 21398 at 15-Sep-2025 10:09:16 CDT
Boundary wire vector sizes: 719, 820, 470
minwire 0: 2567
minwire 1: 824
minwire 2: 1940
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000768898, 0.995248, 0.00398344, 
Output 1: 0.765397, 0.22658, 0.00730629, 0.000716307, 
Output 2: 0.108849, 0.884557, 0.0065458, 4.83445e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2411!
Begin processing the 999th record. run: 74530544 subRun: 1 event: 21399 at 15-Sep-2025 10:09:25 CDT
Boundary wire vector sizes: 50, 66, 40
minwire 0: 440
minwire 1: 2173
minwire 2: 378
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00244573, 0.994845, 0.00270948, 
Output 1: 0.28229, 0.690473, 0.0255969, 0.00164054, 
Output 2: 0.998296, 0.00164863, 5.23159e-05, 2.59277e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 881!
Begin processing the 1000th record. run: 74530544 subRun: 1 event: 21400 at 15-Sep-2025 10:09:32 CDT
Boundary wire vector sizes: 86, 105, 56
minwire 0: 1081
minwire 1: 2077
minwire 2: 816
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.991109, 0.000575726, 0.00831533, 
Output 1: 0.988857, 0.0106801, 0.000301174, 0.000162194, 
Output 2: 0.992755, 0.00707509, 0.000143971, 2.61793e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1472!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/73/ef/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_372_20231202T142042Z_gen_g4_detsim_hitreco__20240508T055329Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ed/54/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_629_20231207T085330Z_gen_g4_detsim_hitreco__20240510T035004Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/c9/a9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_152_20231201T212704Z_gen_g4_detsim_hitreco__20240508T044430Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e0/62/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_729_20231207T073918Z_gen_g4_detsim_hitreco__20240510T054951Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/d7/b7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_1_20231122T083054Z_gen_g4_detsim_hitreco__20240503T054823Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/96/85/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_523_20231206T231708Z_gen_g4_detsim_hitreco__20240510T043431Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cd/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6454800_291_20231207T101829Z_gen_g4_detsim_hitreco__20240510T050310Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0d/37/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_338_20231207T033425Z_gen_g4_detsim_hitreco__20240510T034020Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/68/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_210_20231207T161958Z_gen_g4_detsim_hitreco__20240510T051711Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/7e/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_213_20231207T010119Z_gen_g4_detsim_hitreco__20240510T043458Z_reco2.root
.:
total 51476
-rw-r--r-- 1 dune osgvo 19390967 Sep 15 10:09 flatcaf.root
-rw-r--r-- 1 dune osgvo 16231639 Sep 15 10:09 caf.root
-rw-r--r-- 1 dune osgvo 16231639 Sep 15 10:09 caf_fd_hd_atmo_2501_20250915T123542Z.root
-rw-r--r-- 1 dune osgvo   574475 Sep 15 10:09 caf_20250915T123542Z.log
-rw-r--r-- 1 dune osgvo   228107 Sep 15 10:09 jobscript.log
-rw-r--r-- 1 dune osgvo     1962 Sep 15 10:09 caf_20250915T123542Z.file
-rw-r--r-- 1 dune osgvo     1962 Sep 15 10:09 caf_20250915T123542Z.pfns
-rw-r--r-- 1 dune osgvo     1962 Sep 15 07:35 file.list
-rw-r--r-- 1 dune osgvo     1962 Sep 15 10:09 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo     1376 Sep 15 07:35 all-input-dids.txt
-rw-r--r-- 1 dune osgvo     1376 Sep 15 10:09 caf_20250915T123542Z.did
-rw-r--r-- 1 dune osgvo     1376 Sep 15 07:35 did.list
-rw-r--r-- 1 dune osgvo        0 Sep 15 07:35 debugprod.log