Jobsub ID 40009.96@dunegpschedd02.fnal.gov

Jobsub ID	40009.96@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 13:43:44
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-15 18:18:16
From worker node	Hostname	wn39
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 13:44:59
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_717_20231203T025054Z_gen_g4_detsim_hitreco__20240508T071257Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_615_20231207T044028Z_gen_g4_detsim_hitreco__20240510T060706Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_493_20231207T072608Z_gen_g4_detsim_hitreco__20240510T054026Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_756_20231202T160947Z_gen_g4_detsim_hitreco__20240508T043546Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_1140_20231204T140441Z_gen_g4_detsim_hitreco__20240509T215223Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_629_20231208T010244Z_gen_g4_detsim_hitreco__20240510T031341Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_250_20231202T200401Z_gen_g4_detsim_hitreco__20240508T065324Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_856_20231202T161013Z_gen_g4_detsim_hitreco__20240508T043846Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_92_20231201T220544Z_gen_g4_detsim_hitreco__20240507T205346Z_reco2.root
Jobscript	Exit code	0
	Real time	4h (15618s)
	CPU time	1h (6429s = 41%)
	Max RSS bytes	2142158848 (2042 MiB)
Outputting started	2025-09-15 18:05:18
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T134513Z.root
Finished	2025-09-15 18:18:16
Saved logs	justin-logs:40009.96-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.301104     0.00107642     0.0166602      1000    
reco:energyrecnc:EnergyReco                           0.00221515     0.947835       7.07071       1.23847      0.696862       1000    
reco:energyrecnumurange:EnergyReco                    0.000390263    0.0931196      5.44316     0.00378342     0.402141       1000    
reco:energyrecnumumcs:EnergyReco                      0.000816886    0.0899798      5.15601     0.00461102     0.378334       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000727421    0.0893728       5.078      0.00452642     0.382927       1000    
reco:opdec:Deconvolution                               0.484191       1.0683        5.01654       1.0359       0.324338       1000    
reco:ophitspe:OpHitFinderDeco                           1.38493       1.78225        2.306        1.84429      0.172476       1000    
reco:opflash:OpFlashFinder                            0.000217894   0.00210459     0.0215609    0.00198548    0.00116917      1000    
reco:opslicer:OpSlicer                                6.9996e-05     0.374855       1.56518      0.298418      0.315991       1000    
reco:rns:RandomNumberSaver                            2.4114e-05    6.24864e-05   0.00072193    5.68765e-05   4.24327e-05     1000    
reco:anglereconue:NuAngularReco                       0.000180327   0.00312109      0.19736     0.000672983    0.0116692      1000    
reco:anglereconumu:NuAngularReco                      0.000125036   0.00270026      0.18503     0.000419074    0.0105767      1000    
reco:anglereconuepfps:NuAngularReco                   0.000159011   0.00287883      0.18627     0.000586795    0.0108791      1000    
reco:anglereconumupfps:NuAngularReco                  0.000151568   0.00284295     0.187521     0.000538899    0.0109502      1000    
reco:anglerecohits:NuAngularReco                      0.00017813    0.00386961     0.201787     0.00123829     0.0116653      1000    
[art]:TriggerResults:TriggerResultInserter            1.2184e-05    2.55505e-05   0.00204935    2.24795e-05   6.43249e-05     1000    
end_path:cafmaker:CAFMaker                            0.00167721     0.0395906      2.2876      0.00934299     0.142281       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3803.44 MB
  Peak resident set size usage (VmHWM): 2142.16 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1369!
Begin processing the 992nd record. run: 50542214 subRun: 1 event: 9292 at 15-Sep-2025 15:03:39 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 170!
Begin processing the 993rd record. run: 50542214 subRun: 1 event: 9293 at 15-Sep-2025 15:03:48 -03
Boundary wire vector sizes: 205, 142, 165
minwire 0: 1437
minwire 1: 685
minwire 2: 1877
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.33115, 0.00329884, 0.665551, 
Output 1: 0.0707204, 0.811538, 0.11387, 0.00387178, 
Output 2: 0.232385, 0.689468, 0.0765285, 0.00161835, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2039!
Begin processing the 994th record. run: 50542214 subRun: 1 event: 9294 at 15-Sep-2025 15:03:58 -03
Boundary wire vector sizes: 2004, 2859, 1564
minwire 0: 1320
minwire 1: 62
minwire 2: 946
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996253, 0.000750731, 0.00299586, 
Output 1: 0.152488, 0.844047, 0.00317989, 0.000285113, 
Output 2: 0.609462, 0.374506, 0.0156124, 0.000420084, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2635!
Begin processing the 995th record. run: 50542214 subRun: 1 event: 9295 at 15-Sep-2025 15:04:17 -03
Boundary wire vector sizes: 560, 334, 484
minwire 0: 620
minwire 1: 1656
minwire 2: 565
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.420986, 0.046183, 0.532831, 
Output 1: 0.433188, 0.46509, 0.0960924, 0.00562956, 
Output 2: 0.0938291, 0.405338, 0.427815, 0.0730175, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2029!
Begin processing the 996th record. run: 50542214 subRun: 1 event: 9296 at 15-Sep-2025 15:04:29 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 747!
Begin processing the 997th record. run: 50542214 subRun: 1 event: 9297 at 15-Sep-2025 15:04:38 -03
Boundary wire vector sizes: 78, 87, 63
minwire 0: 233
minwire 1: 2309
minwire 2: 44
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000434094, 0.99744, 0.00212556, 
Output 1: 0.859511, 0.139822, 0.000604277, 6.29292e-05, 
Output 2: 0.999173, 0.000808074, 1.70339e-05, 1.4995e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 629!
Begin processing the 998th record. run: 50542214 subRun: 1 event: 9298 at 15-Sep-2025 15:04:47 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 157!
Begin processing the 999th record. run: 50542214 subRun: 1 event: 9299 at 15-Sep-2025 15:04:55 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 252!
Begin processing the 1000th record. run: 50542214 subRun: 1 event: 9300 at 15-Sep-2025 15:05:03 -03
Boundary wire vector sizes: 560, 509, 395
minwire 0: 1800
minwire 1: 756
minwire 2: 1589
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.979489, 0.00258971, 0.0179213, 
Output 1: 0.227475, 0.513184, 0.183359, 0.0759823, 
Output 2: 0.778885, 0.197971, 0.020911, 0.00223285, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2531!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/97/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_373_20231205T012752Z_gen_g4_detsim_hitreco__20240509T222349Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ec/c5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_717_20231203T025054Z_gen_g4_detsim_hitreco__20240508T071257Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/20/c4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_615_20231207T044028Z_gen_g4_detsim_hitreco__20240510T060706Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a9/46/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_493_20231207T072608Z_gen_g4_detsim_hitreco__20240510T054026Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/65/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_756_20231202T160947Z_gen_g4_detsim_hitreco__20240508T043546Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/aa/6e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_1140_20231204T140441Z_gen_g4_detsim_hitreco__20240509T215223Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ad/64/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_629_20231208T010244Z_gen_g4_detsim_hitreco__20240510T031341Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a1/61/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_250_20231202T200401Z_gen_g4_detsim_hitreco__20240508T065324Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ce/ca/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_856_20231202T161013Z_gen_g4_detsim_hitreco__20240508T043846Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/fd/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_92_20231201T220544Z_gen_g4_detsim_hitreco__20240507T205346Z_reco2.root
.:
total 44680
-rw-r--r-- 1 nobody nobody 16798548 Sep 15 15:05 flatcaf.root
-rw-r--r-- 1 nobody nobody 14045277 Sep 15 15:05 caf.root
-rw-r--r-- 1 nobody nobody 14045277 Sep 15 15:05 caf_fd_hd_atmo_2501_20250915T134513Z.root
-rw-r--r-- 1 nobody nobody   596662 Sep 15 15:05 caf_20250915T134513Z.log
-rw-r--r-- 1 nobody nobody   207303 Sep 15 15:05 jobscript.log
-rw-r--r-- 1 nobody nobody     1902 Sep 15 15:05 caf_20250915T134513Z.file
-rw-r--r-- 1 nobody nobody     1902 Sep 15 15:05 caf_20250915T134513Z.pfns
-rw-r--r-- 1 nobody nobody     1902 Sep 15 10:45 file.list
-rw-r--r-- 1 nobody nobody     1902 Sep 15 15:05 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1378 Sep 15 10:45 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1378 Sep 15 15:05 caf_20250915T134513Z.did
-rw-r--r-- 1 nobody nobody     1378 Sep 15 10:45 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 15 10:45 debugprod.log