Jobsub ID 40197.0@dunegpschedd02.fnal.gov

Jobsub ID	40197.0@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 17:31:55
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2025-09-15 19:30:36
From worker node	Hostname	hnode61.pic.es
	cpuinfo	AMD EPYC 7402P 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 17:32:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_339_20231201T183853Z_gen_g4_detsim_hitreco__20240507T212509Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_898_20231202T091810Z_gen_g4_detsim_hitreco__20240507T210640Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_563_20231207T031938Z_gen_g4_detsim_hitreco__20240510T043930Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_403_20231201T170249Z_gen_g4_detsim_hitreco__20240507T191254Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_422_20231202T170942Z_gen_g4_detsim_hitreco__20240507T204933Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_311_20231202T080205Z_gen_g4_detsim_hitreco__20240507T210248Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_995_20231202T182947Z_gen_g4_detsim_hitreco__20240508T070101Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_643_20231202T121521Z_gen_g4_detsim_hitreco__20240507T212512Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_919_20231203T133857Z_gen_g4_detsim_hitreco__20240508T053740Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (7057s)
	CPU time	1h (4882s = 69%)
	Max RSS bytes	2135605248 (2036 MiB)
Outputting started	2025-09-15 19:30:11
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T173243Z.root
Finished	2025-09-15 19:30:36
Saved logs	justin-logs:40197.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

20977    0.109047       2.80735      0.0206971     0.194188        900    
reco:energyrecnue:EnergyReco                          0.000120443   0.00269728     0.151484     0.000579263   0.00780783       900    
reco:energyrecnc:EnergyReco                            0.0018789     0.163908      0.548503      0.207864      0.114786        900    
reco:energyrecnumurange:EnergyReco                    0.000186296    0.0435637      2.08666     0.00290944     0.167196        900    
reco:energyrecnumumcs:EnergyReco                      0.000547957    0.0434781      1.72458     0.00329908     0.161401        900    
reco:energyrecnumumcsllhd:EnergyReco                  0.000406833    0.0424761      1.73209     0.00330429     0.155577        900    
reco:opdec:Deconvolution                               0.115358      0.344074      0.767706      0.346186      0.0920364       900    
reco:ophitspe:OpHitFinderDeco                          0.331188      0.540321       125.936      0.389725       4.18254        900    
reco:opflash:OpFlashFinder                            0.000175486   0.00142795     0.0156812     0.0013753    0.000781455      900    
reco:opslicer:OpSlicer                                3.7531e-05     0.289698       1.13967      0.232052       0.23596        900    
reco:rns:RandomNumberSaver                            1.7741e-05    3.17483e-05   0.000398742   3.0731e-05    1.3424e-05       900    
reco:anglereconue:NuAngularReco                       0.000140705   0.00193793     0.150589     0.000379807   0.00680282       900    
reco:anglereconumu:NuAngularReco                      6.4422e-05    0.00176026     0.153218     0.000212777   0.00683705       900    
reco:anglereconuepfps:NuAngularReco                   0.000101224   0.00187882     0.156965     0.000313134   0.00696383       900    
reco:anglereconumupfps:NuAngularReco                  8.9403e-05    0.00184272     0.151643     0.000279779   0.00685901       900    
reco:anglerecohits:NuAngularReco                      7.4032e-05    0.00251892     0.161573     0.000724372   0.00748861       900    
[art]:TriggerResults:TriggerResultInserter             8.381e-06    1.28817e-05   4.6291e-05    1.20305e-05   3.94051e-06      900    
end_path:cafmaker:CAFMaker                            0.00129585     0.0241015      1.73432     0.00452853     0.0823772       900    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3798.67 MB
  Peak resident set size usage (VmHWM): 2135.61 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 772!
Begin processing the 895th record. run: 50574832 subRun: 1 event: 91995 at 15-Sep-2025 21:29:40 CEST
Boundary wire vector sizes: 1758, 1424, 1383
minwire 0: 1544
minwire 1: 629
minwire 2: 1455
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0063483, 0.729797, 0.263855, 
Output 1: 0.0499451, 0.94427, 0.00545007, 0.000334779, 
Output 2: 0.923003, 0.0745739, 0.00234003, 8.35524e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2245!
Begin processing the 896th record. run: 50574832 subRun: 1 event: 91996 at 15-Sep-2025 21:29:46 CEST
Boundary wire vector sizes: 855, 1290, 738
minwire 0: 966
minwire 1: 1159
minwire 2: 1094
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0724003, 0.858929, 0.0686707, 
Output 1: 0.139559, 0.756011, 0.0969382, 0.00749172, 
Output 2: 0.0433903, 0.369273, 0.482065, 0.105272, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2389!
Begin processing the 897th record. run: 50574832 subRun: 1 event: 91997 at 15-Sep-2025 21:29:52 CEST
Boundary wire vector sizes: 136, 132, 118
minwire 0: 1638
minwire 1: 704
minwire 2: 1966
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.989829, 0.000773235, 0.00939723, 
Output 1: 0.00346324, 0.993943, 0.00253076, 6.31487e-05, 
Output 2: 0.988371, 0.0113087, 0.000313292, 7.40503e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 601!
Begin processing the 898th record. run: 50574832 subRun: 1 event: 91998 at 15-Sep-2025 21:29:56 CEST
Boundary wire vector sizes: 77, 67, 67
minwire 0: 1528
minwire 1: 1811
minwire 2: 1143
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0160105, 0.00326879, 0.980721, 
Output 1: 0.0278508, 0.147169, 0.778475, 0.0465047, 
Output 2: 0.978889, 0.0200029, 0.00107792, 3.06243e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2123!
Begin processing the 899th record. run: 50574832 subRun: 1 event: 91999 at 15-Sep-2025 21:30:01 CEST
Boundary wire vector sizes: 371, 381, 72
minwire 0: 1206
minwire 1: 2150
minwire 2: 834
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995983, 0.000429737, 0.00358724, 
Output 1: 0.985567, 0.0138734, 0.000375123, 0.000184611, 
Output 2: 0.996211, 0.00368904, 8.28041e-05, 1.74202e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2122!
Begin processing the 900th record. run: 50574832 subRun: 1 event: 92000 at 15-Sep-2025 21:30:05 CEST
Boundary wire vector sizes: 43, 81, 64
minwire 0: 2292
minwire 1: 310
minwire 2: 2601
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.317027, 0.00490073, 0.678072, 
Output 1: 0.00975513, 0.960017, 0.0294038, 0.000824135, 
Output 2: 0.785706, 0.208734, 0.00548737, 7.3205e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 657!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ce/74/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_339_20231201T183853Z_gen_g4_detsim_hitreco__20240507T212509Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/99/79/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_898_20231202T091810Z_gen_g4_detsim_hitreco__20240507T210640Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/00/9a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_563_20231207T031938Z_gen_g4_detsim_hitreco__20240510T043930Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/46/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_403_20231201T170249Z_gen_g4_detsim_hitreco__20240507T191254Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6c/2c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_422_20231202T170942Z_gen_g4_detsim_hitreco__20240507T204933Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/37/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_311_20231202T080205Z_gen_g4_detsim_hitreco__20240507T210248Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fb/71/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_995_20231202T182947Z_gen_g4_detsim_hitreco__20240508T070101Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/c1/2c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_643_20231202T121521Z_gen_g4_detsim_hitreco__20240507T212512Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7a/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_919_20231203T133857Z_gen_g4_detsim_hitreco__20240508T053740Z_reco2.root
.:
total 43244
-rw-r--r-- 1 dune001 dune 16248535 Sep 15 21:30 flatcaf.root
-rw-r--r-- 1 dune001 dune 13603548 Sep 15 21:30 caf.root
-rw-r--r-- 1 dune001 dune 13603548 Sep 15 21:30 caf_fd_hd_atmo_2501_20250915T173243Z.root
-rw-r--r-- 1 dune001 dune   550852 Sep 15 21:30 caf_20250915T173243Z.log
-rw-r--r-- 1 dune001 dune   217252 Sep 15 21:30 jobscript.log
-rw-r--r-- 1 dune001 dune     1776 Sep 15 21:30 caf_20250915T173243Z.file
-rw-r--r-- 1 dune001 dune     1776 Sep 15 21:30 caf_20250915T173243Z.pfns
-rw-r--r-- 1 dune001 dune     1776 Sep 15 19:32 file.list
-rw-r--r-- 1 dune001 dune     1776 Sep 15 21:30 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune     1237 Sep 15 19:32 all-input-dids.txt
-rw-r--r-- 1 dune001 dune     1237 Sep 15 21:30 caf_20250915T173243Z.did
-rw-r--r-- 1 dune001 dune     1237 Sep 15 19:32 did.list
-rw-r--r-- 1 dune001 dune        0 Sep 15 19:32 debugprod.log