Jobsub ID 47907.6@dunegpschedd01.fnal.gov

Jobsub ID	47907.6@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 23:57:47
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce16-multicore
Last heartbeat	2025-09-18 01:51:16
From worker node	Hostname	hnode65.pic.es
	cpuinfo	AMD EPYC 7402P 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-17 23:58:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_321_20231203T045737Z_gen_g4_detsim_hitreco__20240508T074006Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481093_71_20231201T072642Z_gen_g4_detsim_hitreco__20240507T192747Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_364_20231202T203028Z_gen_g4_detsim_hitreco__20240508T062030Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_1086_20231203T120227Z_gen_g4_detsim_hitreco__20240508T041051Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_374_20231201T105700Z_gen_g4_detsim_hitreco__20240507T193254Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_565_20231207T025526Z_gen_g4_detsim_hitreco__20240510T050919Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_541_20231207T073126Z_gen_g4_detsim_hitreco__20240510T062902Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_367_20231204T051408Z_gen_g4_detsim_hitreco__20240509T212843Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6709s)
	CPU time	1h (4935s = 73%)
	Max RSS bytes	2136473600 (2037 MiB)
Outputting started	2025-09-18 01:50:24
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250917T235846Z.root
Finished	2025-09-18 01:51:16
Saved logs	justin-logs:47907.6-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

13     0.0854626     0.0229489     0.0160607       800    
reco:cvneva:CVNEvaluator                               1.711e-05     0.439928       3.55296      0.538202      0.317279        800    
reco:energyrecnumu:EnergyReco                         0.000223707     0.14673       13.9278      0.0270679     0.572673        800    
reco:energyrecnue:EnergyReco                          0.000115904   0.00427348      0.41135     0.000915058    0.0185513       800    
reco:energyrecnc:EnergyReco                           0.00193082      0.17358       1.58406      0.211256      0.137001        800    
reco:energyrecnumurange:EnergyReco                    0.000292288    0.0671205      4.05998     0.00386617     0.313863        800    
reco:energyrecnumumcs:EnergyReco                      0.000506735    0.0737147      5.48587     0.00455548     0.379532        800    
reco:energyrecnumumcsllhd:EnergyReco                  0.000366561    0.0686835      4.17676     0.00461684     0.331107        800    
reco:opdec:Deconvolution                               0.145271      0.393158      0.795702      0.397665      0.119429        800    
reco:ophitspe:OpHitFinderDeco                          0.339404      0.443998      0.515016      0.454631      0.0379084       800    
reco:opflash:OpFlashFinder                            0.000221216   0.00226845     0.0261649    0.00191262    0.00181976       800    
reco:opslicer:OpSlicer                                4.2711e-05      0.33344       1.76829      0.248712      0.295361        800    
reco:rns:RandomNumberSaver                             2.228e-05    6.04268e-05   0.00110241    4.6696e-05     5.181e-05       800    
reco:anglereconue:NuAngularReco                       0.000186386   0.00240559      0.14292     0.00065279    0.00825191       800    
reco:anglereconumu:NuAngularReco                      6.6992e-05    0.00204665     0.128545     0.000273303   0.00792001       800    
reco:anglereconuepfps:NuAngularReco                   0.000110084   0.00215931     0.0898917    0.000436833   0.00702031       800    
reco:anglereconumupfps:NuAngularReco                  0.000109113   0.00215544     0.102484     0.000350475   0.00743448       800    
reco:anglerecohits:NuAngularReco                      9.7423e-05    0.00298036     0.0966145    0.000980674   0.00782755       800    
[art]:TriggerResults:TriggerResultInserter             9.56e-06     2.38723e-05   0.000162325    1.832e-05    1.61031e-05      800    
end_path:cafmaker:CAFMaker                            0.00196961     0.0301846      1.04189     0.00872843     0.0859871       800    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3807.09 MB
  Peak resident set size usage (VmHWM): 2136.47 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Output 2: 0.0564763, 0.22815, 0.3451, 0.370273, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2680!
Begin processing the 793rd record. run: 50581281 subRun: 1 event: 36793 at 18-Sep-2025 03:48:19 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 36!
Begin processing the 794th record. run: 50581281 subRun: 1 event: 36794 at 18-Sep-2025 03:48:23 CEST
Boundary wire vector sizes: 59, 56, 50
minwire 0: 1540
minwire 1: 1052
minwire 2: 1257
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 584, 883
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00589733, 0.00219017, 0.991912, 
Output 1: 0.949429, 0.0491398, 0.00115152, 0.00027976, 
Output 2: 0.997985, 0.00194603, 5.93269e-05, 9.16023e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1873!
Begin processing the 795th record. run: 50581281 subRun: 1 event: 36795 at 18-Sep-2025 03:48:30 CEST
Boundary wire vector sizes: 252, 113, 175
minwire 0: 2195
minwire 1: 153
minwire 2: 2635
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995829, 0.000679758, 0.00349116, 
Output 1: 0.00613442, 0.990484, 0.00331097, 7.0599e-05, 
Output 2: 0.995997, 0.00391173, 8.95658e-05, 1.79852e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 916!
Begin processing the 796th record. run: 50581281 subRun: 1 event: 36796 at 18-Sep-2025 03:48:37 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 131!
Begin processing the 797th record. run: 50581281 subRun: 1 event: 36797 at 18-Sep-2025 03:48:41 CEST
Boundary wire vector sizes: 2410, 2247, 1475
minwire 0: 529
minwire 1: 903
minwire 2: 485
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.075067, 0.849652, 0.0752809, 
Output 1: 0.257867, 0.433742, 0.219213, 0.089179, 
Output 2: 0.0550652, 0.176845, 0.386821, 0.381268, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3209!
Begin processing the 798th record. run: 50581281 subRun: 1 event: 36798 at 18-Sep-2025 03:48:52 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1091!
Begin processing the 799th record. run: 50581281 subRun: 1 event: 36799 at 18-Sep-2025 03:48:58 CEST
Boundary wire vector sizes: 994, 914, 503
minwire 0: 463
minwire 1: 710
minwire 2: 962
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0297298, 0.661894, 0.308376, 
Output 1: 0.0275714, 0.0335582, 0.064483, 0.874387, 
Output 2: 0.68213, 0.287922, 0.0251871, 0.00476015, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3052!
Begin processing the 800th record. run: 50581281 subRun: 1 event: 36800 at 18-Sep-2025 03:49:06 CEST
Boundary wire vector sizes: 7167, 6029, 5281
minwire 0: 189
minwire 1: 1527
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0957355, 0.867989, 0.0362755, 
Output 1: 0.306426, 0.527006, 0.091907, 0.0746607, 
Output 2: 0.0150003, 0.0681709, 0.241097, 0.675731, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3302!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/a3/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_321_20231203T045737Z_gen_g4_detsim_hitreco__20240508T074006Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/1c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481093_71_20231201T072642Z_gen_g4_detsim_hitreco__20240507T192747Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a5/67/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_364_20231202T203028Z_gen_g4_detsim_hitreco__20240508T062030Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/26/aa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_1086_20231203T120227Z_gen_g4_detsim_hitreco__20240508T041051Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a1/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_374_20231201T105700Z_gen_g4_detsim_hitreco__20240507T193254Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/28/89/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_565_20231207T025526Z_gen_g4_detsim_hitreco__20240510T050919Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3f/29/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_541_20231207T073126Z_gen_g4_detsim_hitreco__20240510T062902Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/36/04/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_367_20231204T051408Z_gen_g4_detsim_hitreco__20240509T212843Z_reco2.root
.:
total 36488
-rw-r--r-- 1 dune001 dune 13710379 Sep 18 03:50 flatcaf.root
-rw-r--r-- 1 dune001 dune 11463526 Sep 18 03:50 caf.root
-rw-r--r-- 1 dune001 dune 11463526 Sep 18 03:50 caf_fd_hd_atmo_2501_20250917T235846Z.root
-rw-r--r-- 1 dune001 dune   483295 Sep 18 03:50 caf_20250917T235846Z.log
-rw-r--r-- 1 dune001 dune   191346 Sep 18 03:50 jobscript.log
-rw-r--r-- 1 dune001 dune     1508 Sep 18 03:50 caf_20250917T235846Z.file
-rw-r--r-- 1 dune001 dune     1508 Sep 18 03:50 caf_20250917T235846Z.pfns
-rw-r--r-- 1 dune001 dune     1508 Sep 18 01:58 file.list
-rw-r--r-- 1 dune001 dune     1508 Sep 18 03:50 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune     1102 Sep 18 01:58 all-input-dids.txt
-rw-r--r-- 1 dune001 dune     1102 Sep 18 03:50 caf_20250917T235846Z.did
-rw-r--r-- 1 dune001 dune     1102 Sep 18 01:58 did.list
-rw-r--r-- 1 dune001 dune        0 Sep 18 01:59 debugprod.log