Jobsub ID 40362.2@dunegpschedd02.fnal.gov

Jobsub ID	40362.2@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-16 07:52:42
Site	US_Wisconsin
Entry	HCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat	2025-09-16 09:52:08
From worker node	Hostname	e2616
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	82800 (23 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-16 07:53:28
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_608_20231207T120529Z_gen_g4_detsim_hitreco__20240510T055437Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_332_20231204T010921Z_gen_g4_detsim_hitreco__20240509T214937Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_402_20231201T105705Z_gen_g4_detsim_hitreco__20240507T193503Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1124_20231204T100539Z_gen_g4_detsim_hitreco__20240509T212557Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_630_20231207T145423Z_gen_g4_detsim_hitreco__20240510T063651Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_6_20231202T122927Z_gen_g4_detsim_hitreco__20240507T222108Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_366_20231122T071348Z_gen_g4_detsim_hitreco__20240507T174003Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_673_20231208T082228Z_gen_g4_detsim_hitreco__20240510T062726Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (7109s)
	CPU time	53m (3234s = 45%)
	Max RSS bytes	2177572864 (2076 MiB)
Outputting started	2025-09-16 09:51:58
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250916T075332Z.root
Finished	2025-09-16 09:52:08
Saved logs	justin-logs:40362.2-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

reco:cvneva:CVNEvaluator                               1.576e-05     0.222988       1.72746      0.299603      0.142014        900    
reco:energyrecnumu:EnergyReco                         0.000231476    0.172851       7.14418      0.0118572     0.338738        900    
reco:energyrecnue:EnergyReco                          0.000105779   0.00214305     0.176736     0.000396678   0.00862593       900    
reco:energyrecnc:EnergyReco                           0.00134237     0.378743       1.48186      0.448942      0.285856        900    
reco:energyrecnumurange:EnergyReco                    0.000134744    0.0331808      6.04814     0.00162013     0.238738        900    
reco:energyrecnumumcs:EnergyReco                      0.000338598    0.0336802      6.06594      0.0019433     0.239011        900    
reco:energyrecnumumcsllhd:EnergyReco                  0.00029138     0.0334934      5.96931     0.00188614     0.236279        900    
reco:opdec:Deconvolution                               0.307292      0.512691       1.11099      0.522389      0.100721        900    
reco:ophitspe:OpHitFinderDeco                          0.908051      0.967503       1.00878      0.995408      0.0396941       900    
reco:opflash:OpFlashFinder                            0.000155553   0.000991183   0.00433962    0.000941716   0.000442867      900    
reco:opslicer:OpSlicer                                3.2982e-05      0.17134      0.768297      0.139098      0.141701        900    
reco:rns:RandomNumberSaver                            1.6191e-05    3.48528e-05   0.000423389   3.4505e-05    1.73999e-05      900    
reco:anglereconue:NuAngularReco                       9.4668e-05    0.00131984     0.104996     0.000291385   0.00528714       900    
reco:anglereconumu:NuAngularReco                      5.7418e-05    0.00116757     0.104511     0.000152062   0.00525979       900    
reco:anglereconuepfps:NuAngularReco                   9.4278e-05    0.00125421     0.104443     0.000248404   0.00523436       900    
reco:anglereconumupfps:NuAngularReco                  7.7987e-05    0.00120614     0.104464     0.000200989   0.00522519       900    
reco:anglerecohits:NuAngularReco                      7.0693e-05    0.00172416     0.110466     0.00055364    0.00569241       900    
[art]:TriggerResults:TriggerResultInserter             7.454e-06    9.64014e-06   4.5365e-05     9.198e-06    2.59542e-06      900    
end_path:cafmaker:CAFMaker                             0.0010189     0.0185327      1.05677     0.00515948     0.0614992       900    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3846.88 MB
  Peak resident set size usage (VmHWM): 2177.57 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 743!
Begin processing the 895th record. run: 74581430 subRun: 1 event: 67395 at 16-Sep-2025 04:50:34 CDT
Boundary wire vector sizes: 1341, 1174, 1018
minwire 0: 609
minwire 1: 1331
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.985926, 0.00589889, 0.00817497, 
Output 1: 0.245465, 0.733196, 0.0206877, 0.000651584, 
Output 2: 0.028924, 0.373567, 0.531921, 0.0655881, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2224!
Begin processing the 896th record. run: 74581430 subRun: 1 event: 67396 at 16-Sep-2025 04:50:40 CDT
Boundary wire vector sizes: 105, 74, 74
minwire 0: 1669
minwire 1: 1408
minwire 2: 1495
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.873747, 0.00182068, 0.124433, 
Output 1: 0.010227, 0.98563, 0.00402215, 0.000120689, 
Output 2: 0.909983, 0.0873618, 0.00260466, 4.99861e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1698!
Begin processing the 897th record. run: 74581430 subRun: 1 event: 67397 at 16-Sep-2025 04:50:46 CDT
Boundary wire vector sizes: 8673, 9599, 7416
minwire 0: 31
minwire 1: 1523
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.210835, 0.683338, 0.105827, 
Output 1: 0.328183, 0.457614, 0.137899, 0.0763047, 
Output 2: 0.0132363, 0.0470307, 0.201409, 0.738324, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3852!
Begin processing the 898th record. run: 74581430 subRun: 1 event: 67398 at 16-Sep-2025 04:51:37 CDT
Boundary wire vector sizes: 229, 175, 212
minwire 0: 1625
minwire 1: 812
minwire 2: 1852
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.9958, 0.000363272, 0.00383632, 
Output 1: 0.990224, 0.00960833, 9.7216e-05, 6.99326e-05, 
Output 2: 0.998068, 0.00187997, 3.98897e-05, 1.20927e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1261!
Begin processing the 899th record. run: 74581430 subRun: 1 event: 67399 at 16-Sep-2025 04:51:42 CDT
Boundary wire vector sizes: 547, 898, 468
minwire 0: 1705
minwire 1: 685
minwire 2: 1881
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.991364, 0.00334253, 0.00529317, 
Output 1: 0.911521, 0.0862498, 0.00202288, 0.000206076, 
Output 2: 0.935263, 0.063278, 0.00138266, 7.58832e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2185!
Begin processing the 900th record. run: 74581430 subRun: 1 event: 67400 at 16-Sep-2025 04:51:48 CDT
Boundary wire vector sizes: 1510, 1916, 1549
minwire 0: 498
minwire 1: 1060
minwire 2: 15
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0229085, 0.890661, 0.0864311, 
Output 1: 0.354624, 0.555561, 0.0734885, 0.0163266, 
Output 2: 0.11674, 0.33895, 0.367436, 0.176874, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2962!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9e/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_608_20231207T120529Z_gen_g4_detsim_hitreco__20240510T055437Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/05/4c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_332_20231204T010921Z_gen_g4_detsim_hitreco__20240509T214937Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/2a/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_402_20231201T105705Z_gen_g4_detsim_hitreco__20240507T193503Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bb/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1124_20231204T100539Z_gen_g4_detsim_hitreco__20240509T212557Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/54/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_630_20231207T145423Z_gen_g4_detsim_hitreco__20240510T063651Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/40/af/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_6_20231202T122927Z_gen_g4_detsim_hitreco__20240507T222108Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/1d/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_366_20231122T071348Z_gen_g4_detsim_hitreco__20240507T174003Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4c/9b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_673_20231208T082228Z_gen_g4_detsim_hitreco__20240510T062726Z_reco2.root
.:
total 48412
-rw-r--r-- 1 slot1_1 slot1_1 18181918 Sep 16 04:51 flatcaf.root
-rw-r--r-- 1 slot1_1 slot1_1 15295144 Sep 16 04:51 caf.root
-rw-r--r-- 1 slot1_1 slot1_1 15295144 Sep 16 04:51 caf_fd_hd_atmo_2501_20250916T075332Z.root
-rw-r--r-- 1 slot1_1 slot1_1   541811 Sep 16 04:51 caf_20250916T075332Z.log
-rw-r--r-- 1 slot1_1 slot1_1   208531 Sep 16 04:51 jobscript.log
-rw-r--r-- 1 slot1_1 slot1_1     1564 Sep 16 04:51 caf_20250916T075332Z.file
-rw-r--r-- 1 slot1_1 slot1_1     1564 Sep 16 04:51 caf_20250916T075332Z.pfns
-rw-r--r-- 1 slot1_1 slot1_1     1564 Sep 16 02:53 file.list
-rw-r--r-- 1 slot1_1 slot1_1     1564 Sep 16 04:51 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_1 slot1_1     1102 Sep 16 02:53 all-input-dids.txt
-rw-r--r-- 1 slot1_1 slot1_1     1102 Sep 16 04:51 caf_20250916T075332Z.did
-rw-r--r-- 1 slot1_1 slot1_1     1102 Sep 16 02:53 did.list
-rw-r--r-- 1 slot1_1 slot1_1        0 Sep 16 02:53 debugprod.log