Jobsub ID 46788.1@dunegpschedd01.fnal.gov

Jobsub ID	46788.1@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 18:19:58
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce16-multicore
Last heartbeat	2025-09-15 20:28:44
From worker node	Hostname	tds443.pic.es
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 18:21:18
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_233_20231202T202524Z_gen_g4_detsim_hitreco__20240508T061527Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_630_20231208T025524Z_gen_g4_detsim_hitreco__20240510T041342Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_218_20231208T035249Z_gen_g4_detsim_hitreco__20240510T034430Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_770_20231208T024415Z_gen_g4_detsim_hitreco__20240510T034307Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_104_20231207T161741Z_gen_g4_detsim_hitreco__20240510T064056Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_396_20231204T212623Z_gen_g4_detsim_hitreco__20240510T025233Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_363_20231203T035027Z_gen_g4_detsim_hitreco__20240508T071506Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_326_20231207T134538Z_gen_g4_detsim_hitreco__20240510T062427Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_618_20231207T150310Z_gen_g4_detsim_hitreco__20240510T061821Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_374_20231201T205720Z_gen_g4_detsim_hitreco__20240507T220536Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (7608s)
	CPU time	1h (5740s = 75%)
	Max RSS bytes	2121003008 (2022 MiB)
Outputting started	2025-09-15 20:28:07
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T182132Z.root
Finished	2025-09-15 20:28:44
Saved logs	justin-logs:46788.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0351568     0.222096     0.000712374    0.0126055      1000    
reco:energyrecnc:EnergyReco                           0.00224536     0.157015      0.727418      0.197587      0.123097       1000    
reco:energyrecnumurange:EnergyReco                    0.000281758    0.0632201      6.94736     0.00345631     0.378587       1000    
reco:energyrecnumumcs:EnergyReco                      0.000602187    0.0627859      6.34903     0.00404952     0.363998       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000483633    0.0623343      6.56495     0.00415398     0.364848       1000    
reco:opdec:Deconvolution                               0.0179023     0.360353      0.809351      0.368689       0.1205        1000    
reco:ophitspe:OpHitFinderDeco                          0.0026766     0.371361      0.780225      0.420413      0.126303       1000    
reco:opflash:OpFlashFinder                            0.000174474   0.00159824    0.00457111    0.00153683    0.000704613     1000    
reco:opslicer:OpSlicer                                3.6771e-05     0.327495       1.34567      0.261583      0.269788       1000    
reco:rns:RandomNumberSaver                             2.189e-05    3.53847e-05   0.000428781   3.3426e-05    1.50005e-05     1000    
reco:anglereconue:NuAngularReco                       0.000162275   0.00242586     0.198557     0.000440622    0.0100118      1000    
reco:anglereconumu:NuAngularReco                      7.2252e-05    0.00223867     0.194602     0.000254867   0.00991638      1000    
reco:anglereconuepfps:NuAngularReco                   0.000117503    0.0023407      0.19068     0.000360555   0.00981383      1000    
reco:anglereconumupfps:NuAngularReco                  0.000102333   0.00232842     0.191932     0.000330164    0.009902       1000    
reco:anglerecohits:NuAngularReco                      0.000109683    0.0031439     0.203035     0.000821837    0.0109691      1000    
[art]:TriggerResults:TriggerResultInserter             9.95e-06     1.50717e-05   6.9032e-05    1.40205e-05   5.93296e-06     1000    
end_path:cafmaker:CAFMaker                            0.00164298     0.0308715      1.7831      0.00558875     0.117748       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3794 MB
  Peak resident set size usage (VmHWM): 2121 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.419073, 0.395412, 0.185515, 
Output 1: 0.131856, 0.612865, 0.223294, 0.0319844, 
Output 2: 0.27976, 0.690972, 0.0287407, 0.000527751, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1765!
Begin processing the 994th record. run: 6390196 subRun: 1 event: 37494 at 15-Sep-2025 22:27:36 CEST
Boundary wire vector sizes: 410, 343, 283
minwire 0: 809
minwire 1: 1557
minwire 2: 908
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0385061, 0.73785, 0.223644, 
Output 1: 0.82701, 0.167533, 0.00518927, 0.000267245, 
Output 2: 0.255999, 0.733804, 0.010083, 0.000113945, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1787!
Begin processing the 995th record. run: 6390196 subRun: 1 event: 37495 at 15-Sep-2025 22:27:41 CEST
Boundary wire vector sizes: 590, 451, 484
minwire 0: 968
minwire 1: 1530
minwire 2: 901
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00232223, 0.990107, 0.00757037, 
Output 1: 0.284586, 0.541107, 0.159301, 0.0150063, 
Output 2: 0.530979, 0.433931, 0.0337135, 0.00137637, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2180!
Begin processing the 996th record. run: 6390196 subRun: 1 event: 37496 at 15-Sep-2025 22:27:45 CEST
Boundary wire vector sizes: 84, 120, 85
minwire 0: 1426
minwire 1: 869
minwire 2: 1585
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00105724, 0.989413, 0.00952959, 
Output 1: 0.992691, 0.00709177, 0.000113496, 0.000103854, 
Output 2: 0.998373, 0.0015635, 4.33407e-05, 1.984e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 859!
Begin processing the 997th record. run: 6390196 subRun: 1 event: 37497 at 15-Sep-2025 22:27:49 CEST
Boundary wire vector sizes: 925, 1100, 496
minwire 0: 586
minwire 1: 2000
minwire 2: 331
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.978567, 0.0082219, 0.0132113, 
Output 1: 0.13098, 0.724651, 0.135246, 0.00912239, 
Output 2: 0.445657, 0.516478, 0.0368441, 0.00102118, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2155!
Begin processing the 998th record. run: 6390196 subRun: 1 event: 37498 at 15-Sep-2025 22:27:54 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 454!
Begin processing the 999th record. run: 6390196 subRun: 1 event: 37499 at 15-Sep-2025 22:27:57 CEST
Boundary wire vector sizes: 44, 63, 48
minwire 0: 1011
minwire 1: 1980
minwire 2: 779
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.396533, 0.307357, 0.296111, 
Output 1: 0.121228, 0.672247, 0.194819, 0.011707, 
Output 2: 0.649599, 0.341631, 0.00861873, 0.00015113, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1867!
Begin processing the 1000th record. run: 6390196 subRun: 1 event: 37500 at 15-Sep-2025 22:28:01 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 823!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/22/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_233_20231202T202524Z_gen_g4_detsim_hitreco__20240508T061527Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/14/af/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_630_20231208T025524Z_gen_g4_detsim_hitreco__20240510T041342Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d5/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_218_20231208T035249Z_gen_g4_detsim_hitreco__20240510T034430Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/50/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_770_20231208T024415Z_gen_g4_detsim_hitreco__20240510T034307Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a1/4a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_104_20231207T161741Z_gen_g4_detsim_hitreco__20240510T064056Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9a/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_396_20231204T212623Z_gen_g4_detsim_hitreco__20240510T025233Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/94/97/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_363_20231203T035027Z_gen_g4_detsim_hitreco__20240508T071506Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2e/ab/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_326_20231207T134538Z_gen_g4_detsim_hitreco__20240510T062427Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/50/e6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_618_20231207T150310Z_gen_g4_detsim_hitreco__20240510T061821Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/ed/c0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_374_20231201T205720Z_gen_g4_detsim_hitreco__20240507T220536Z_reco2.root
.:
total 49876
-rw-r--r-- 1 dune001 dune 18757666 Sep 15 22:28 flatcaf.root
-rw-r--r-- 1 dune001 dune 15715286 Sep 15 22:28 caf.root
-rw-r--r-- 1 dune001 dune 15715286 Sep 15 22:28 caf_fd_hd_atmo_2501_20250915T182132Z.root
-rw-r--r-- 1 dune001 dune   604133 Sep 15 22:28 caf_20250915T182132Z.log
-rw-r--r-- 1 dune001 dune   225887 Sep 15 22:28 jobscript.log
-rw-r--r-- 1 dune001 dune     1893 Sep 15 22:28 caf_20250915T182132Z.file
-rw-r--r-- 1 dune001 dune     1893 Sep 15 22:28 caf_20250915T182132Z.pfns
-rw-r--r-- 1 dune001 dune     1893 Sep 15 20:21 file.list
-rw-r--r-- 1 dune001 dune     1893 Sep 15 22:28 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune     1374 Sep 15 20:21 all-input-dids.txt
-rw-r--r-- 1 dune001 dune     1374 Sep 15 22:28 caf_20250915T182132Z.did
-rw-r--r-- 1 dune001 dune     1374 Sep 15 20:21 did.list
-rw-r--r-- 1 dune001 dune        0 Sep 15 20:21 debugprod.log