Jobsub ID 47837.8@dunegpschedd01.fnal.gov

Jobsub ID	47837.8@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 22:59:44
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2025-09-18 01:42:07
From worker node	Hostname	hnode61.pic.es
	cpuinfo	AMD EPYC 7402P 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-17 23:00:23
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_324_20231205T024928Z_gen_g4_detsim_hitreco__20240509T223018Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_752_20231202T181829Z_gen_g4_detsim_hitreco__20240508T064350Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_309_20231202T141758Z_gen_g4_detsim_hitreco__20240508T055615Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_496_20231120T135424Z_gen_g4_detsim_hitreco__20240503T061919Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_831_20231202T204934Z_gen_g4_detsim_hitreco__20240508T070423Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_667_20231207T112857Z_gen_g4_detsim_hitreco__20240510T054416Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_998_20231203T181156Z_gen_g4_detsim_hitreco__20240509T202528Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_370_20231201T123019Z_gen_g4_detsim_hitreco__20240507T190540Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_827_20231202T120545Z_gen_g4_detsim_hitreco__20240507T215930Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_163_20231028T022256Z_gen_g4_detsim_hitreco__20240503T043913Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (9674s)
	CPU time	1h (7083s = 73%)
	Max RSS bytes	2149486592 (2049 MiB)
Outputting started	2025-09-18 01:41:39
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250917T230033Z.root
Finished	2025-09-18 01:42:07
Saved logs	justin-logs:47837.8-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

range:EnergyReco                    0.000164485    0.0464848      4.23268     0.00295411     0.221835       1200    
reco:energyrecnumumcs:EnergyReco                      0.000504126    0.0470094      4.13964     0.00349608     0.220393       1200    
reco:energyrecnumumcsllhd:EnergyReco                  0.000348941    0.0455704      4.08799     0.00356092     0.208368       1200    
reco:opdec:Deconvolution                               0.114257      0.368517       1.22343      0.368675      0.106239       1200    
reco:ophitspe:OpHitFinderDeco                          0.330881      0.437137      0.550247      0.450981      0.0446229      1200    
reco:opflash:OpFlashFinder                            0.000115544   0.00144447    0.00654444    0.00140206    0.000678062     1200    
reco:opslicer:OpSlicer                                3.0861e-05     0.273561       1.09962      0.214632      0.227821       1200    
reco:rns:RandomNumberSaver                             1.59e-05     3.37305e-05   0.00033937    3.1636e-05    1.36608e-05     1200    
reco:anglereconue:NuAngularReco                       0.000130905   0.00189124     0.225432     0.000419098   0.00856364      1200    
reco:anglereconumu:NuAngularReco                      6.0062e-05    0.00167165     0.216438     0.000217407   0.00833946      1200    
reco:anglereconuepfps:NuAngularReco                   9.5033e-05    0.00178399     0.217819     0.00033155    0.00837966      1200    
reco:anglereconumupfps:NuAngularReco                  8.3482e-05    0.00175332     0.215175     0.000281464   0.00834814      1200    
reco:anglerecohits:NuAngularReco                      6.7512e-05     0.0023925     0.222581     0.000709487   0.00888676      1200    
[art]:TriggerResults:TriggerResultInserter             7.97e-06     1.50486e-05   0.000128294   1.35555e-05   7.15654e-06     1200    
end_path:cafmaker:CAFMaker                            0.00121047     0.0243628      2.68689     0.00530679     0.104261       1200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3810.1 MB
  Peak resident set size usage (VmHWM): 2149.49 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00924296, 0.979901, 0.0108564, 
Output 1: 0.0975181, 0.550234, 0.298518, 0.0537297, 
Output 2: 0.0512803, 0.318384, 0.500867, 0.129468, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1890!
Begin processing the 1195th record. run: 1499 subRun: 1 event: 32795 at 18-Sep-2025 03:41:08 CEST
Boundary wire vector sizes: 21, 56, 34
minwire 0: 948
minwire 1: 1469
minwire 2: 1108
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.852573, 0.143047, 0.00437935, 
Output 1: 0.0351351, 0.961386, 0.00339124, 8.76703e-05, 
Output 2: 0.998361, 0.00157609, 5.91938e-05, 3.32031e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 820!
Begin processing the 1196th record. run: 1499 subRun: 1 event: 32796 at 18-Sep-2025 03:41:12 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 497!
Begin processing the 1197th record. run: 1499 subRun: 1 event: 32797 at 18-Sep-2025 03:41:15 CEST
Boundary wire vector sizes: 432, 505, 508
minwire 0: 1527
minwire 1: 939
minwire 2: 1333
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0202538, 0.018975, 0.960771, 
Output 1: 0.0620968, 0.874272, 0.0606645, 0.00296649, 
Output 2: 0.0775459, 0.791035, 0.127605, 0.00381364, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2316!
Begin processing the 1198th record. run: 1499 subRun: 1 event: 32798 at 18-Sep-2025 03:41:19 CEST
Boundary wire vector sizes: 1627, 2191, 1907
minwire 0: 1098
minwire 1: 892
minwire 2: 1081
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.929927, 0.0642632, 0.00580965, 
Output 1: 0.280332, 0.551342, 0.154396, 0.0139302, 
Output 2: 0.115072, 0.405899, 0.380751, 0.098278, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2616!
Begin processing the 1199th record. run: 1499 subRun: 1 event: 32799 at 18-Sep-2025 03:41:26 CEST
Boundary wire vector sizes: 970, 1524, 1084
minwire 0: 515
minwire 1: 1767
minwire 2: 165
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.407553, 0.0587327, 0.533714, 
Output 1: 0.298677, 0.3133, 0.198136, 0.189886, 
Output 2: 0.0417465, 0.111627, 0.288372, 0.558254, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2735!
Begin processing the 1200th record. run: 1499 subRun: 1 event: 32800 at 18-Sep-2025 03:41:32 CEST
Boundary wire vector sizes: 51, 48, 39
minwire 0: 1580
minwire 1: 781
minwire 2: 1830
Used alternate method to get min and max wires due to vertex determination failure: 54, 353
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 19, 318
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.196067, 0.00408057, 0.799853, 
Output 1: 0.328369, 0.550259, 0.118303, 0.00306938, 
Output 2: 0.946959, 0.0514844, 0.00151724, 3.91797e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1205!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c9/56/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_324_20231205T024928Z_gen_g4_detsim_hitreco__20240509T223018Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/51/15/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_752_20231202T181829Z_gen_g4_detsim_hitreco__20240508T064350Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/dc/5a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_309_20231202T141758Z_gen_g4_detsim_hitreco__20240508T055615Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/1e/61/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_496_20231120T135424Z_gen_g4_detsim_hitreco__20240503T061919Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/69/95/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_831_20231202T204934Z_gen_g4_detsim_hitreco__20240508T070423Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6f/00/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_667_20231207T112857Z_gen_g4_detsim_hitreco__20240510T054416Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/47/0e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_998_20231203T181156Z_gen_g4_detsim_hitreco__20240509T202528Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/0f/bf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_370_20231201T123019Z_gen_g4_detsim_hitreco__20240507T190540Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/a6/2c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_827_20231202T120545Z_gen_g4_detsim_hitreco__20240507T215930Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/48/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_163_20231028T022256Z_gen_g4_detsim_hitreco__20240503T043913Z_reco2.root
.:
total 53844
-rw-r--r-- 1 dune001 dune 20222021 Sep 18 03:41 flatcaf.root
-rw-r--r-- 1 dune001 dune 16938020 Sep 18 03:41 caf.root
-rw-r--r-- 1 dune001 dune 16938020 Sep 18 03:41 caf_fd_hd_atmo_2501_20250917T230033Z.root
-rw-r--r-- 1 dune001 dune   717136 Sep 18 03:41 caf_20250917T230033Z.log
-rw-r--r-- 1 dune001 dune   260254 Sep 18 03:41 jobscript.log
-rw-r--r-- 1 dune001 dune     1916 Sep 18 03:41 caf_20250917T230033Z.file
-rw-r--r-- 1 dune001 dune     1916 Sep 18 03:41 caf_20250917T230033Z.pfns
-rw-r--r-- 1 dune001 dune     1916 Sep 18 01:00 file.list
-rw-r--r-- 1 dune001 dune     1916 Sep 18 03:41 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune     1374 Sep 18 01:00 all-input-dids.txt
-rw-r--r-- 1 dune001 dune     1374 Sep 18 03:41 caf_20250917T230033Z.did
-rw-r--r-- 1 dune001 dune     1374 Sep 18 01:00 did.list
-rw-r--r-- 1 dune001 dune        0 Sep 18 01:00 debugprod.log