Jobsub ID 46601.148@dunegpschedd01.fnal.gov

Jobsub ID	46601.148@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:17:41
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-09-15 13:59:07
From worker node	Hostname	comp05-26
	cpuinfo	Intel(R) Xeon(R) Platinum 8352Y CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 12:19:47
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_735_20231203T083248Z_gen_g4_detsim_hitreco__20240508T040200Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_727_20231207T075043Z_gen_g4_detsim_hitreco__20240510T024235Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_397_20231203T173857Z_gen_g4_detsim_hitreco__20240509T215330Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_1175_20231204T011233Z_gen_g4_detsim_hitreco__20240509T214358Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_173_20231207T082910Z_gen_g4_detsim_hitreco__20240510T044144Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_418_20231208T064252Z_gen_g4_detsim_hitreco__20240510T055240Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_321_20231202T124737Z_gen_g4_detsim_hitreco__20240507T220649Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_895_20231202T071752Z_gen_g4_detsim_hitreco__20240507T204533Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_892_20231203T161427Z_gen_g4_detsim_hitreco__20240508T060739Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_31_20231207T151324Z_gen_g4_detsim_hitreco__20240510T030319Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4622s)
	CPU time	55m (3354s = 72%)
	Max RSS bytes	2202304512 (2100 MiB)
Outputting started	2025-09-15 13:36:51
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T121954Z.root
Finished	2025-09-15 13:59:07
Saved logs	justin-logs:46601.148-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00704258      1000    
reco:energyrecnc:EnergyReco                            0.0020781      0.14491       3.89953      0.182277      0.156048       1000    
reco:energyrecnumurange:EnergyReco                    0.000287799    0.0356682      2.71561     0.00260504     0.157405       1000    
reco:energyrecnumumcs:EnergyReco                      0.000481386    0.0362289      2.68373     0.00291602      0.15549       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.00033796     0.0361398      2.67209     0.00297991     0.155297       1000    
reco:opdec:Deconvolution                               0.0638929     0.283296       1.2458       0.280403      0.0916528      1000    
reco:ophitspe:OpHitFinderDeco                          0.141062       0.2529       0.323465      0.266832      0.0367868      1000    
reco:opflash:OpFlashFinder                            0.000213791   0.00143739    0.00442902    0.00139953    0.000599064     1000    
reco:opslicer:OpSlicer                                4.3968e-05     0.213033      0.832113      0.164724      0.177613       1000    
reco:rns:RandomNumberSaver                            1.8399e-05    2.94253e-05   0.000352523   2.6263e-05    1.29215e-05     1000    
reco:anglereconue:NuAngularReco                       0.000154354   0.00167062     0.0748797    0.000379268   0.00493559      1000    
reco:anglereconumu:NuAngularReco                      7.2408e-05    0.00148491     0.0731058    0.000201207   0.00485477      1000    
reco:anglereconuepfps:NuAngularReco                   0.000115278    0.0015757     0.0716656    0.000291179   0.00486157      1000    
reco:anglereconumupfps:NuAngularReco                  9.4168e-05    0.00155145     0.0723919    0.000252045   0.00487957      1000    
reco:anglerecohits:NuAngularReco                      8.5151e-05     0.0020991     0.0766544    0.00062858    0.00536522      1000    
[art]:TriggerResults:TriggerResultInserter             8.882e-06    1.22132e-05   6.2963e-05    1.1665e-05    2.40226e-06     1000    
end_path:cafmaker:CAFMaker                            0.00109875     0.0208959     0.797312     0.00471629     0.0592516      1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3878.61 MB
  Peak resident set size usage (VmHWM): 2202.3 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Output 1: 0.283885, 0.485289, 0.159292, 0.0715345, 
Output 2: 0.310031, 0.491333, 0.164659, 0.0339767, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2868!
Begin processing the 993rd record. run: 74568471 subRun: 1 event: 3193 at 15-Sep-2025 14:36:24 BST
Boundary wire vector sizes: 415, 365, 293
minwire 0: 1859
minwire 1: 669
minwire 2: 1970
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.278113, 0.438267, 0.28362, 
Output 1: 0.0614644, 0.538183, 0.361996, 0.0383569, 
Output 2: 0.236316, 0.584234, 0.168994, 0.0104561, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1961!
Begin processing the 994th record. run: 74568471 subRun: 1 event: 3194 at 15-Sep-2025 14:36:27 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 504!
Begin processing the 995th record. run: 74568471 subRun: 1 event: 3195 at 15-Sep-2025 14:36:30 BST
Boundary wire vector sizes: 891, 836, 723
minwire 0: 803
minwire 1: 1630
minwire 2: 420
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00291191, 0.974357, 0.022731, 
Output 1: 0.050162, 0.345227, 0.518447, 0.086164, 
Output 2: 0.877456, 0.116725, 0.00562723, 0.000191947, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2601!
Begin processing the 996th record. run: 74568471 subRun: 1 event: 3196 at 15-Sep-2025 14:36:34 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 728!
Begin processing the 997th record. run: 74568471 subRun: 1 event: 3197 at 15-Sep-2025 14:36:37 BST
Boundary wire vector sizes: 69, 31, 36
minwire 0: 1385
minwire 1: 1814
minwire 2: 1178
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992105, 0.00497197, 0.00292257, 
Output 1: 0.00497553, 0.990133, 0.0047824, 0.000109536, 
Output 2: 0.996092, 0.00378917, 0.000115592, 3.4615e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1347!
Begin processing the 998th record. run: 74568471 subRun: 1 event: 3198 at 15-Sep-2025 14:36:40 BST
Boundary wire vector sizes: 66, 52, 63
minwire 0: 893
minwire 1: 1339
minwire 2: 825
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.02885, 0.00437752, 0.966773, 
Output 1: 0.0287773, 0.0632805, 0.253069, 0.654873, 
Output 2: 0.984911, 0.0142679, 0.000719691, 0.000101632, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1712!
Begin processing the 999th record. run: 74568471 subRun: 1 event: 3199 at 15-Sep-2025 14:36:43 BST
Boundary wire vector sizes: 122, 142, 150
minwire 0: 1332
minwire 1: 1776
minwire 2: 1098
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997572, 0.000608359, 0.00181988, 
Output 1: 0.00262639, 0.987182, 0.00997513, 0.000216937, 
Output 2: 0.997066, 0.00285594, 7.65676e-05, 1.49383e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1809!
Begin processing the 1000th record. run: 74568471 subRun: 1 event: 3200 at 15-Sep-2025 14:36:46 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 934!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/11/79/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_735_20231203T083248Z_gen_g4_detsim_hitreco__20240508T040200Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/87/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_727_20231207T075043Z_gen_g4_detsim_hitreco__20240510T024235Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c7/cd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_397_20231203T173857Z_gen_g4_detsim_hitreco__20240509T215330Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/e7/12/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_1175_20231204T011233Z_gen_g4_detsim_hitreco__20240509T214358Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/94/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_173_20231207T082910Z_gen_g4_detsim_hitreco__20240510T044144Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ae/da/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_418_20231208T064252Z_gen_g4_detsim_hitreco__20240510T055240Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/62/c4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_321_20231202T124737Z_gen_g4_detsim_hitreco__20240507T220649Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a5/2c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_895_20231202T071752Z_gen_g4_detsim_hitreco__20240507T204533Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ff/ea/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_892_20231203T161427Z_gen_g4_detsim_hitreco__20240508T060739Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/71/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_31_20231207T151324Z_gen_g4_detsim_hitreco__20240510T030319Z_reco2.root
.:
total 45512
-rw-r--r-- 1 pltdune002 pltdune 17090089 Sep 15 14:36 flatcaf.root
-rw-r--r-- 1 pltdune002 pltdune 14304327 Sep 15 14:36 caf.root
-rw-r--r-- 1 pltdune002 pltdune 14304327 Sep 15 14:36 caf_fd_hd_atmo_2501_20250915T121954Z.root
-rw-r--r-- 1 pltdune002 pltdune   590414 Sep 15 14:36 caf_20250915T121954Z.log
-rw-r--r-- 1 pltdune002 pltdune   225626 Sep 15 14:36 jobscript.log
-rw-r--r-- 1 pltdune002 pltdune     1916 Sep 15 14:36 caf_20250915T121954Z.file
-rw-r--r-- 1 pltdune002 pltdune     1916 Sep 15 14:36 caf_20250915T121954Z.pfns
-rw-r--r-- 1 pltdune002 pltdune     1916 Sep 15 13:19 file.list
-rw-r--r-- 1 pltdune002 pltdune     1916 Sep 15 14:36 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune002 pltdune     1374 Sep 15 13:19 all-input-dids.txt
-rw-r--r-- 1 pltdune002 pltdune     1374 Sep 15 14:36 caf_20250915T121954Z.did
-rw-r--r-- 1 pltdune002 pltdune     1374 Sep 15 13:19 did.list
-rw-r--r-- 1 pltdune002 pltdune        0 Sep 15 13:20 debugprod.log