Jobsub ID 38122.38@dunegpschedd02.fnal.gov

Jobsub ID	38122.38@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:02:34
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-11 15:25:13
From worker node	Hostname	wn58
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-11 10:47:06
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_736_20231202T190941Z_gen_g4_detsim_hitreco__20240508T062256Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_557_20231202T050023Z_gen_g4_detsim_hitreco__20240507T202115Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_209_20231203T122733Z_gen_g4_detsim_hitreco__20240508T045418Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_245_20231207T023924Z_gen_g4_detsim_hitreco__20240510T034800Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_291_20231120T140134Z_gen_g4_detsim_hitreco__20240507T181241Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_373_20231202T150421Z_gen_g4_detsim_hitreco__20240508T024413Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_984_20231204T073710Z_gen_g4_detsim_hitreco__20240509T211916Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_284_20231202T102013Z_gen_g4_detsim_hitreco__20240507T210750Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_439_20231202T051220Z_gen_g4_detsim_hitreco__20240507T193120Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_324_20231203T131236Z_gen_g4_detsim_hitreco__20240508T074544Z_reco2.root
Jobscript	Exit code	0
	Real time	4h (15911s)
	CPU time	2h (7319s = 45%)
	Max RSS bytes	2152579072 (2052 MiB)
Outputting started	2025-09-11 15:12:18
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250911T104717Z.root
Finished	2025-09-11 15:25:13
Saved logs	justin-logs:38122.38-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00481268     0.469252     0.00100934     0.0222689      1100    
reco:energyrecnc:EnergyReco                           0.00221117      0.79691       72.4754      0.941474       2.22019       1100    
reco:energyrecnumurange:EnergyReco                    0.000299439    0.073423       9.95372     0.00377891     0.497503       1100    
reco:energyrecnumumcs:EnergyReco                      0.000780974    0.0742601      10.1549     0.00455522      0.49763       1100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000584997    0.0698847      9.80302      0.0045793     0.461671       1100    
reco:opdec:Deconvolution                               0.429824      0.864138       2.44394      0.837113      0.224503       1100    
reco:ophitspe:OpHitFinderDeco                           1.2831        1.37684       1.91232       1.39837      0.0473817      1100    
reco:opflash:OpFlashFinder                            0.000197949   0.00204028    0.00574192    0.00198225    0.000917901     1100    
reco:opslicer:OpSlicer                                6.3393e-05     0.346702       1.76719      0.279307      0.298027       1100    
reco:rns:RandomNumberSaver                            2.5537e-05    6.05668e-05   0.000790627   5.5871e-05    4.8527e-05      1100    
reco:anglereconue:NuAngularReco                       0.000171657   0.00338253     0.296978     0.000593462    0.0162257      1100    
reco:anglereconumu:NuAngularReco                      0.000111103   0.00306941     0.293203     0.000379774    0.0159482      1100    
reco:anglereconuepfps:NuAngularReco                   0.00015378     0.0032111      0.2995      0.00052012     0.0159273      1100    
reco:anglereconumupfps:NuAngularReco                  0.000143505   0.00318835     0.296218     0.000485646    0.0159744      1100    
reco:anglerecohits:NuAngularReco                      0.000156884   0.00410479     0.312011     0.00113555     0.0168574      1100    
[art]:TriggerResults:TriggerResultInserter            1.2255e-05    2.21362e-05    8.171e-05    2.17425e-05   5.05717e-06     1100    
end_path:cafmaker:CAFMaker                            0.00171381     0.0705436      34.1259     0.00796295      1.04286       1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3829.04 MB
  Peak resident set size usage (VmHWM): 2152.58 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Output 1: 0.00765817, 0.030688, 0.0682357, 0.893418, 
Output 2: 0.46569, 0.42034, 0.0875897, 0.0263804, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2503!
Begin processing the 1093rd record. run: 74511302 subRun: 1 event: 32493 at 11-Sep-2025 12:08:56 -03
Boundary wire vector sizes: 44, 31, 43
minwire 0: 886
minwire 1: 2634
minwire 2: 353
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00596463, 0.00269668, 0.991339, 
Output 1: 0.0651317, 0.929563, 0.0051048, 0.000200727, 
Output 2: 0.99789, 0.00202011, 8.49958e-05, 4.61362e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1257!
Begin processing the 1094th record. run: 74511302 subRun: 1 event: 32494 at 11-Sep-2025 12:09:06 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 261!
Begin processing the 1095th record. run: 74511302 subRun: 1 event: 32495 at 11-Sep-2025 12:09:14 -03
Boundary wire vector sizes: 357, 553, 466
minwire 0: 2122
minwire 1: 114
minwire 2: 2353
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000509229, 0.996949, 0.00254125, 
Output 1: 0.990026, 0.00960412, 0.0001784, 0.000191945, 
Output 2: 0.998518, 0.00140954, 4.9087e-05, 2.30671e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1310!
Begin processing the 1096th record. run: 74511302 subRun: 1 event: 32496 at 11-Sep-2025 12:09:25 -03
Boundary wire vector sizes: 73, 76, 63
minwire 0: 1365
minwire 1: 1603
minwire 2: 1135
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 43, 342
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00124539, 0.993442, 0.00531242, 
Output 1: 0.971965, 0.02771, 0.000245723, 7.91276e-05, 
Output 2: 0.998788, 0.00116584, 3.65793e-05, 9.22562e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1641!
Begin processing the 1097th record. run: 74511302 subRun: 1 event: 32497 at 11-Sep-2025 12:09:35 -03
Boundary wire vector sizes: 44, 33, 31
minwire 0: 1223
minwire 1: 2316
minwire 2: 693
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.300245, 0.00519441, 0.694561, 
Output 1: 0.273015, 0.672118, 0.0528473, 0.00201931, 
Output 2: 0.52544, 0.468231, 0.00625317, 7.55555e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1498!
Begin processing the 1098th record. run: 74511302 subRun: 1 event: 32498 at 11-Sep-2025 12:09:45 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 524!
Begin processing the 1099th record. run: 74511302 subRun: 1 event: 32499 at 11-Sep-2025 12:09:54 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 719!
Begin processing the 1100th record. run: 74511302 subRun: 1 event: 32500 at 11-Sep-2025 12:10:02 -03
Boundary wire vector sizes: 9086, 7877, 4359
minwire 0: 1514
minwire 1: 115
minwire 2: 1505
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000368775, 0.997884, 0.00174687, 
Output 1: 0.119891, 0.666619, 0.176602, 0.0368883, 
Output 2: 0.214898, 0.622799, 0.15107, 0.0112327, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3850!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cd/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_736_20231202T190941Z_gen_g4_detsim_hitreco__20240508T062256Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f2/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_557_20231202T050023Z_gen_g4_detsim_hitreco__20240507T202115Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/45/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_209_20231203T122733Z_gen_g4_detsim_hitreco__20240508T045418Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/27/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_245_20231207T023924Z_gen_g4_detsim_hitreco__20240510T034800Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d1/08/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_291_20231120T140134Z_gen_g4_detsim_hitreco__20240507T181241Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/91/eb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_373_20231202T150421Z_gen_g4_detsim_hitreco__20240508T024413Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/44/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_984_20231204T073710Z_gen_g4_detsim_hitreco__20240509T211916Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c7/d7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_284_20231202T102013Z_gen_g4_detsim_hitreco__20240507T210750Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_439_20231202T051220Z_gen_g4_detsim_hitreco__20240507T193120Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/46/e2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_324_20231203T131236Z_gen_g4_detsim_hitreco__20240508T074544Z_reco2.root
.:
total 48884
-rw-r--r-- 1 nobody nobody 18357334 Sep 11 12:12 flatcaf.root
-rw-r--r-- 1 nobody nobody 15378332 Sep 11 12:12 caf.root
-rw-r--r-- 1 nobody nobody 15378332 Sep 11 12:12 caf_fd_hd_atmo_2501_20250911T104717Z.root
-rw-r--r-- 1 nobody nobody   648435 Sep 11 12:12 caf_20250911T104717Z.log
-rw-r--r-- 1 nobody nobody   238724 Sep 11 12:12 jobscript.log
-rw-r--r-- 1 nobody nobody     1911 Sep 11 12:12 caf_20250911T104717Z.file
-rw-r--r-- 1 nobody nobody     1911 Sep 11 12:12 caf_20250911T104717Z.pfns
-rw-r--r-- 1 nobody nobody     1911 Sep 11 07:47 file.list
-rw-r--r-- 1 nobody nobody     1911 Sep 11 12:12 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1375 Sep 11 07:47 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1375 Sep 11 12:12 caf_20250911T104717Z.did
-rw-r--r-- 1 nobody nobody     1375 Sep 11 07:47 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 11 07:47 debugprod.log