Jobsub ID 40120.148@dunegpschedd02.fnal.gov

Jobsub ID	40120.148@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 16:45:54
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce04
Last heartbeat	2025-09-15 17:47:19
From worker node	Hostname	ccwcondor0385
	cpuinfo	AMD EPYC 9334 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	106200 (29 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 16:46:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_615_20231207T161443Z_gen_g4_detsim_hitreco__20240510T031631Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_1_20231207T041035Z_gen_g4_detsim_hitreco__20240510T044851Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_213_20231201T103117Z_gen_g4_detsim_hitreco__20240507T203025Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_203_20231203T125452Z_gen_g4_detsim_hitreco__20240508T074443Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_31_20231121T154806Z_gen_g4_detsim_hitreco__20240503T055530Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_159_20231207T041539Z_gen_g4_detsim_hitreco__20240510T045303Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_958_20231205T044046Z_gen_g4_detsim_hitreco__20240510T021741Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_565_20231202T160904Z_gen_g4_detsim_hitreco__20240508T043653Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_972_20231203T124346Z_gen_g4_detsim_hitreco__20240509T201907Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1158_20231203T073411Z_gen_g4_detsim_hitreco__20240508T073316Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3612s)
	CPU time	43m (2624s = 72%)
	Max RSS bytes	2324635648 (2216 MiB)
Outputting started	2025-09-15 17:46:49
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T164646Z.root
Finished	2025-09-15 17:47:19
Saved logs	justin-logs:40120.148-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

6.7503e-05    0.00152249     0.0609666    0.000398108   0.00402592      1100    
reco:energyrecnc:EnergyReco                           0.000926296    0.0826452     0.393344      0.0940178     0.0639973      1100    
reco:energyrecnumurange:EnergyReco                    0.000125721    0.0245876      2.33239     0.00149983     0.132346       1100    
reco:energyrecnumumcs:EnergyReco                      0.000335869    0.0245777      2.21659     0.00216894     0.130262       1100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000205803    0.0247795      2.36385     0.00215342      0.13352       1100    
reco:opdec:Deconvolution                               0.057206      0.179857      0.418042      0.172393      0.0600121      1100    
reco:ophitspe:OpHitFinderDeco                           0.15971      0.170307      0.188604      0.170785     0.00508674      1100    
reco:opflash:OpFlashFinder                            0.000113061   0.000961416   0.00871617    0.000853915   0.000560164     1100    
reco:opslicer:OpSlicer                                2.6731e-05     0.173713      0.869385      0.133437      0.157899       1100    
reco:rns:RandomNumberSaver                             1.333e-05    3.47125e-05   0.000192212   3.1578e-05    1.37358e-05     1100    
reco:anglereconue:NuAngularReco                       0.000101905   0.00102846     0.0452028    0.00033337    0.00269409      1100    
reco:anglereconumu:NuAngularReco                      3.6114e-05    0.000813018    0.0444474    0.000133302   0.00264419      1100    
reco:anglereconuepfps:NuAngularReco                   6.3927e-05    0.000911629    0.0450188    0.000223784   0.00268448      1100    
reco:anglereconumupfps:NuAngularReco                  4.7432e-05    0.000856151    0.0417638    0.000170449   0.00264069      1100    
reco:anglerecohits:NuAngularReco                      5.3502e-05    0.00132599     0.0488747    0.000493278   0.00307502      1100    
[art]:TriggerResults:TriggerResultInserter             6.84e-06     1.05489e-05   3.3321e-05     9.144e-06    3.34924e-06     1100    
end_path:cafmaker:CAFMaker                            0.000786935    0.0129072     0.596923     0.00345278     0.0376176      1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4008.46 MB
  Peak resident set size usage (VmHWM): 2324.64 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 311, 285, 255
minwire 0: 2171
minwire 1: 869
minwire 2: 2016
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.061671, 0.204275, 0.734054, 
Output 1: 0.0604107, 0.44824, 0.447154, 0.0441953, 
Output 2: 0.358876, 0.606617, 0.0338234, 0.000683606, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2001!
Begin processing the 1095th record. run: 50572032 subRun: 1 event: 115895 at 15-Sep-2025 19:45:58 CEST
Boundary wire vector sizes: 104, 87, 69
minwire 0: 2007
minwire 1: 1475
minwire 2: 1655
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000472339, 0.995994, 0.00353403, 
Output 1: 0.951936, 0.0474265, 0.0005151, 0.000122825, 
Output 2: 0.99829, 0.00166708, 3.72038e-05, 5.59044e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1572!
Begin processing the 1096th record. run: 50572032 subRun: 1 event: 115896 at 15-Sep-2025 19:46:01 CEST
Boundary wire vector sizes: 128, 98, 54
minwire 0: 1406
minwire 1: 1012
minwire 2: 1683
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.962605, 0.000584368, 0.0368104, 
Output 1: 0.99436, 0.00510678, 0.00032792, 0.00020525, 
Output 2: 0.963751, 0.0353233, 0.00078159, 0.00014431, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1081!
Begin processing the 1097th record. run: 50572032 subRun: 1 event: 115897 at 15-Sep-2025 19:46:04 CEST
Boundary wire vector sizes: 6054, 5811, 2494
minwire 0: 1094
minwire 1: 322
minwire 2: 645
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000964334, 0.992433, 0.00660311, 
Output 1: 0.501463, 0.340147, 0.0855951, 0.0727945, 
Output 2: 0.44266, 0.411275, 0.114712, 0.031353, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3297!
Begin processing the 1098th record. run: 50572032 subRun: 1 event: 115898 at 15-Sep-2025 19:46:36 CEST
Boundary wire vector sizes: 1025, 1884, 1179
minwire 0: 698
minwire 1: 1694
minwire 2: 37
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992169, 0.00208962, 0.00574103, 
Output 1: 0.00547504, 0.0352593, 0.0739798, 0.885286, 
Output 2: 0.142027, 0.489339, 0.262073, 0.106561, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2438!
Begin processing the 1099th record. run: 50572032 subRun: 1 event: 115899 at 15-Sep-2025 19:46:40 CEST
Boundary wire vector sizes: 797, 564, 611
minwire 0: 441
minwire 1: 1778
minwire 2: 345
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000573135, 0.997143, 0.00228342, 
Output 1: 0.0369152, 0.694382, 0.241369, 0.0273345, 
Output 2: 0.0507547, 0.607565, 0.321513, 0.0201668, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2022!
Begin processing the 1100th record. run: 50572032 subRun: 1 event: 115900 at 15-Sep-2025 19:46:44 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 809!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c7/7a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_615_20231207T161443Z_gen_g4_detsim_hitreco__20240510T031631Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_1_20231207T041035Z_gen_g4_detsim_hitreco__20240510T044851Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/27/00/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_213_20231201T103117Z_gen_g4_detsim_hitreco__20240507T203025Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9f/06/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_203_20231203T125452Z_gen_g4_detsim_hitreco__20240508T074443Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2b/7a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_31_20231121T154806Z_gen_g4_detsim_hitreco__20240503T055530Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/26/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_159_20231207T041539Z_gen_g4_detsim_hitreco__20240510T045303Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a3/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_958_20231205T044046Z_gen_g4_detsim_hitreco__20240510T021741Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/bd/20/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_565_20231202T160904Z_gen_g4_detsim_hitreco__20240508T043653Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d0/3c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_972_20231203T124346Z_gen_g4_detsim_hitreco__20240509T201907Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7a/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1158_20231203T073411Z_gen_g4_detsim_hitreco__20240508T073316Z_reco2.root
.:
total 51652
-rw-r--r-- 1 dune001 lbno 19407468 Sep 15 19:46 flatcaf.root
-rw-r--r-- 1 dune001 lbno 16265040 Sep 15 19:46 caf.root
-rw-r--r-- 1 dune001 lbno 16265040 Sep 15 19:46 caf_fd_hd_atmo_2501_20250915T164646Z.root
-rw-r--r-- 1 dune001 lbno   642927 Sep 15 19:46 caf_20250915T164646Z.log
-rw-r--r-- 1 dune001 lbno   255077 Sep 15 19:46 jobscript.log
-rw-r--r-- 1 dune001 lbno     1896 Sep 15 19:46 caf_20250915T164646Z.file
-rw-r--r-- 1 dune001 lbno     1896 Sep 15 19:46 caf_20250915T164646Z.pfns
-rw-r--r-- 1 dune001 lbno     1896 Sep 15 18:46 file.list
-rw-r--r-- 1 dune001 lbno     1896 Sep 15 19:46 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 lbno     1374 Sep 15 18:46 all-input-dids.txt
-rw-r--r-- 1 dune001 lbno     1374 Sep 15 19:46 caf_20250915T164646Z.did
-rw-r--r-- 1 dune001 lbno     1374 Sep 15 18:46 did.list
-rw-r--r-- 1 dune001 lbno        0 Sep 15 18:46 debugprod.log