Jobsub ID 38122.93@dunegpschedd02.fnal.gov

Jobsub ID	38122.93@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:02:34
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-11 15:43:23
From worker node	Hostname	wn58
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-11 10:55:20
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_677_20231202T151129Z_gen_g4_detsim_hitreco__20240508T030057Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_949_20231204T140425Z_gen_g4_detsim_hitreco__20240509T204106Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_25_20231121T152358Z_gen_g4_detsim_hitreco__20240503T065552Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_524_20231201T185819Z_gen_g4_detsim_hitreco__20240507T214222Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_721_20231208T000904Z_gen_g4_detsim_hitreco__20240510T055751Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_243_20231207T041650Z_gen_g4_detsim_hitreco__20240510T051421Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_557_20231207T031935Z_gen_g4_detsim_hitreco__20240510T043657Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_77_20231201T110749Z_gen_g4_detsim_hitreco__20240507T194220Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_991_20231204T041352Z_gen_g4_detsim_hitreco__20240509T212045Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_814_20231202T082032Z_gen_g4_detsim_hitreco__20240507T200106Z_reco2.root
Jobscript	Exit code	0
	Real time	4h (16517s)
	CPU time	1h (6508s = 39%)
	Max RSS bytes	2104717312 (2007 MiB)
Outputting started	2025-09-11 15:30:38
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250911T105532Z.root
Finished	2025-09-11 15:43:23
Saved logs	justin-logs:38122.93-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

503      15.9586     0.00380339      0.64398       1100    
reco:energyrecnumumcs:EnergyReco                      0.000775573    0.0839459      13.4072     0.00442145      0.57284       1100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000675259    0.0832965      16.3455     0.00446455     0.611866       1100    
reco:opdec:Deconvolution                                0.46174      0.872325       3.17433      0.832419      0.240647       1100    
reco:ophitspe:OpHitFinderDeco                           1.29877       2.10494       802.486       1.39083       24.1434       1100    
reco:opflash:OpFlashFinder                            0.00019676    0.00205474    0.00825393    0.00197907    0.000949296     1100    
reco:opslicer:OpSlicer                                7.0807e-05     0.348396       1.64776      0.267374      0.298627       1100    
reco:rns:RandomNumberSaver                            2.4857e-05    5.63653e-05   0.00126157    4.8647e-05    6.3318e-05      1100    
reco:anglereconue:NuAngularReco                       0.000213008   0.00303358      0.15621     0.000614835    0.0115878      1100    
reco:anglereconumu:NuAngularReco                      0.000110182   0.00275611     0.153045     0.000388486    0.011401       1100    
reco:anglereconuepfps:NuAngularReco                   0.000146612   0.00292526     0.153856     0.000546032    0.011435       1100    
reco:anglereconumupfps:NuAngularReco                  0.000140743   0.00290866     0.152901     0.000512491    0.0114029      1100    
reco:anglerecohits:NuAngularReco                      0.000171268    0.0037868     0.163863     0.00116956     0.0118236      1100    
[art]:TriggerResults:TriggerResultInserter             1.172e-05    2.27023e-05   0.000600603   2.1534e-05    1.8386e-05      1100    
end_path:cafmaker:CAFMaker                            0.00162422     0.0351943      1.97499     0.00742127     0.123875       1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3768.23 MB
  Peak resident set size usage (VmHWM): 2104.72 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00195242, 0.889751, 0.108297, 
Output 1: 0.985906, 0.013545, 0.000426433, 0.000122648, 
Output 2: 0.995921, 0.00392113, 0.000116777, 4.13116e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1411!
Begin processing the 1095th record. run: 50546594 subRun: 1 event: 81495 at 11-Sep-2025 12:27:11 -03
Boundary wire vector sizes: 160, 232, 158
minwire 0: 1346
minwire 1: 1224
minwire 2: 1244
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0577877, 0.00917493, 0.933037, 
Output 1: 0.0093929, 0.0777997, 0.717053, 0.195755, 
Output 2: 0.980803, 0.0183575, 0.000805588, 3.38216e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2172!
Begin processing the 1096th record. run: 50546594 subRun: 1 event: 81496 at 11-Sep-2025 12:27:22 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 557!
Begin processing the 1097th record. run: 50546594 subRun: 1 event: 81497 at 11-Sep-2025 12:27:32 -03
Boundary wire vector sizes: 9363, 6078, 5012
minwire 0: 925
minwire 1: 397
minwire 2: 726
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.057324, 0.781946, 0.16073, 
Output 1: 0.255306, 0.394312, 0.187816, 0.162566, 
Output 2: 0.0242034, 0.0747478, 0.242404, 0.658644, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3168!
Begin processing the 1098th record. run: 50546594 subRun: 1 event: 81498 at 11-Sep-2025 12:28:56 -03
Boundary wire vector sizes: 86, 107, 49
minwire 0: 2579
minwire 1: 299
minwire 2: 2773
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00161881, 0.986971, 0.0114101, 
Output 1: 0.978589, 0.0200115, 0.00109495, 0.000304588, 
Output 2: 0.995252, 0.00460591, 0.000121203, 2.03886e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 623!
Begin processing the 1099th record. run: 50546594 subRun: 1 event: 81499 at 11-Sep-2025 12:29:06 -03
Boundary wire vector sizes: 69, 64, 51
minwire 0: 1567
minwire 1: 721
minwire 2: 1878
Used alternate method to get min and max wires due to vertex determination failure: 716, 1015
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 348, 647
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.529113, 0.177985, 0.292902, 
Output 1: 0.785185, 0.208048, 0.00632767, 0.000439394, 
Output 2: 0.753726, 0.240095, 0.00596935, 0.000210498, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1175!
Begin processing the 1100th record. run: 50546594 subRun: 1 event: 81500 at 11-Sep-2025 12:29:16 -03
Boundary wire vector sizes: 8875, 7571, 4974
minwire 0: 198
minwire 1: 1415
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00369054, 0.983965, 0.0123446, 
Output 1: 0.236037, 0.277642, 0.150775, 0.335546, 
Output 2: 0.0357911, 0.103071, 0.310177, 0.550961, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3481!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1b/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_677_20231202T151129Z_gen_g4_detsim_hitreco__20240508T030057Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/22/8c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_949_20231204T140425Z_gen_g4_detsim_hitreco__20240509T204106Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/55/51/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_25_20231121T152358Z_gen_g4_detsim_hitreco__20240503T065552Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/69/7f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_524_20231201T185819Z_gen_g4_detsim_hitreco__20240507T214222Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/88/96/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_721_20231208T000904Z_gen_g4_detsim_hitreco__20240510T055751Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/75/2b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_243_20231207T041650Z_gen_g4_detsim_hitreco__20240510T051421Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/db/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_557_20231207T031935Z_gen_g4_detsim_hitreco__20240510T043657Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/32/a6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481118_77_20231201T110749Z_gen_g4_detsim_hitreco__20240507T194220Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f2/7d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_991_20231204T041352Z_gen_g4_detsim_hitreco__20240509T212045Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f0/60/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_814_20231202T082032Z_gen_g4_detsim_hitreco__20240507T200106Z_reco2.root
.:
total 46860
-rw-r--r-- 1 nobody nobody 17631662 Sep 11 12:30 flatcaf.root
-rw-r--r-- 1 nobody nobody 14711648 Sep 11 12:30 caf.root
-rw-r--r-- 1 nobody nobody 14711648 Sep 11 12:30 caf_fd_hd_atmo_2501_20250911T105532Z.root
-rw-r--r-- 1 nobody nobody   643234 Sep 11 12:30 caf_20250911T105532Z.log
-rw-r--r-- 1 nobody nobody   230640 Sep 11 12:30 jobscript.log
-rw-r--r-- 1 nobody nobody     1923 Sep 11 12:30 caf_20250911T105532Z.file
-rw-r--r-- 1 nobody nobody     1923 Sep 11 12:30 caf_20250911T105532Z.pfns
-rw-r--r-- 1 nobody nobody     1923 Sep 11 07:55 file.list
-rw-r--r-- 1 nobody nobody     1923 Sep 11 12:30 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1373 Sep 11 07:55 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1373 Sep 11 12:30 caf_20250911T105532Z.did
-rw-r--r-- 1 nobody nobody     1373 Sep 11 07:55 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 11 07:55 debugprod.log