Jobsub ID 39988.119@dunegpschedd02.fnal.gov

Jobsub ID	39988.119@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:33:41
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_condce_mcore
Last heartbeat	2025-09-15 15:15:48
From worker node	Hostname	cit2-c056.mwt2.org
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-15 12:36:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_313_20231208T085734Z_gen_g4_detsim_hitreco__20240510T061357Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_469_20231201T132505Z_gen_g4_detsim_hitreco__20240507T195232Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_699_20231203T024733Z_gen_g4_detsim_hitreco__20240508T071251Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_525_20231208T080839Z_gen_g4_detsim_hitreco__20240510T062228Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_895_20231203T084115Z_gen_g4_detsim_hitreco__20240509T211836Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_999_20231202T075105Z_gen_g4_detsim_hitreco__20240507T204751Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_367_20231203T113006Z_gen_g4_detsim_hitreco__20240509T194417Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_555_20231203T081444Z_gen_g4_detsim_hitreco__20240509T205759Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_726_20231207T233337Z_gen_g4_detsim_hitreco__20240510T050627Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_357_20231204T051406Z_gen_g4_detsim_hitreco__20240509T212924Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (9552s)
	CPU time	1h (5031s = 52%)
	Max RSS bytes	2085470208 (1988 MiB)
Outputting started	2025-09-15 15:15:23
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/001/caf_fd_hd_atmo_2501_20250915T123615Z.root
Finished	2025-09-15 15:15:48
Saved logs	justin-logs:39988.119-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0    
reco:energyrecnc:EnergyReco                           0.00213586     0.354464       1.0613       0.449689      0.238408       1000    
reco:energyrecnumurange:EnergyReco                    0.000295032    0.0653527      10.0392     0.00347614     0.400605       1000    
reco:energyrecnumumcs:EnergyReco                      0.000674526    0.0668023      10.1435      0.0042033     0.407898       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.00059173     0.0668902      10.4796     0.00423029     0.413696       1000    
reco:opdec:Deconvolution                               0.281642      0.614671       1.41475      0.637503      0.150333       1000    
reco:ophitspe:OpHitFinderDeco                          0.802851      0.964239       1.40089      0.988602      0.0879394      1000    
reco:opflash:OpFlashFinder                            0.000146131   0.00192168    0.00512519    0.00189058    0.000854434     1000    
reco:opslicer:OpSlicer                                4.8083e-05     0.319982       1.25665      0.262052      0.259568       1000    
reco:rns:RandomNumberSaver                            1.9792e-05    4.83693e-05   0.000596674   4.54895e-05   2.5272e-05      1000    
reco:anglereconue:NuAngularReco                       0.000162896   0.00251988     0.205376     0.00053023     0.0097253      1000    
reco:anglereconumu:NuAngularReco                      0.00010563    0.00229269     0.149569     0.000348213   0.00883757      1000    
reco:anglereconuepfps:NuAngularReco                   0.000149209   0.00242864     0.168557     0.000471862   0.00907562      1000    
reco:anglereconumupfps:NuAngularReco                  0.00014519    0.00239923     0.169281     0.000450874   0.00902657      1000    
reco:anglerecohits:NuAngularReco                      0.000114989   0.00321635     0.179134     0.00101741    0.00980714      1000    
[art]:TriggerResults:TriggerResultInserter            1.2068e-05    1.87187e-05   5.7358e-05    1.83925e-05   5.40645e-06     1000    
end_path:cafmaker:CAFMaker                            0.00128312     0.031672       1.85699     0.00730357     0.112041       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3751.34 MB
  Peak resident set size usage (VmHWM): 2085.47 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2525!
Begin processing the 995th record. run: 50581281 subRun: 1 event: 35795 at 15-Sep-2025 11:14:31 EDT
Boundary wire vector sizes: 304, 145, 214
minwire 0: 1086
minwire 1: 1939
minwire 2: 866
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995255, 0.00147102, 0.00327394, 
Output 1: 0.00915674, 0.987077, 0.00367545, 9.11527e-05, 
Output 2: 0.994189, 0.00569214, 0.00011614, 2.36761e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1974!
Begin processing the 996th record. run: 50581281 subRun: 1 event: 35796 at 15-Sep-2025 11:14:39 EDT
Boundary wire vector sizes: 464, 497, 290
minwire 0: 2821
minwire 1: 700
minwire 2: 2679
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99537, 0.00122204, 0.00340816, 
Output 1: 0.0389471, 0.587639, 0.353943, 0.0194709, 
Output 2: 0.752984, 0.232009, 0.0146672, 0.000339502, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1676!
Begin processing the 997th record. run: 50581281 subRun: 1 event: 35797 at 15-Sep-2025 11:14:47 EDT
Boundary wire vector sizes: 78, 78, 71
minwire 0: 613
minwire 1: 2824
minwire 2: 55
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00163592, 0.989639, 0.00872549, 
Output 1: 0.98212, 0.0175077, 0.000284873, 8.75447e-05, 
Output 2: 0.997471, 0.00246094, 5.7364e-05, 1.11695e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 858!
Begin processing the 998th record. run: 50581281 subRun: 1 event: 35798 at 15-Sep-2025 11:14:55 EDT
Boundary wire vector sizes: 108, 162, 104
minwire 0: 837
minwire 1: 1789
minwire 2: 723
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.948366, 0.00776951, 0.043864, 
Output 1: 0.0491971, 0.870391, 0.0775983, 0.00281332, 
Output 2: 0.886999, 0.10993, 0.00301851, 5.25311e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1365!
Begin processing the 999th record. run: 50581281 subRun: 1 event: 35799 at 15-Sep-2025 11:15:02 EDT
Boundary wire vector sizes: 1404, 1483, 1364
minwire 0: 115
minwire 1: 1566
minwire 2: 79
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0372168, 0.374805, 0.587978, 
Output 1: 0.243165, 0.366836, 0.21734, 0.172658, 
Output 2: 0.07212, 0.209315, 0.35172, 0.366845, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2180!
Begin processing the 1000th record. run: 50581281 subRun: 1 event: 35800 at 15-Sep-2025 11:15:13 EDT
Boundary wire vector sizes: 70, 72, 50
minwire 0: 2266
minwire 1: 712
minwire 2: 2324
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00427609, 0.992758, 0.00296625, 
Output 1: 0.0136644, 0.828001, 0.151725, 0.00661004, 
Output 2: 0.997642, 0.00223215, 0.000120956, 4.46069e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1151!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/9f/22/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_313_20231208T085734Z_gen_g4_detsim_hitreco__20240510T061357Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/de/82/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_469_20231201T132505Z_gen_g4_detsim_hitreco__20240507T195232Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b3/49/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_699_20231203T024733Z_gen_g4_detsim_hitreco__20240508T071251Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/78/0a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_525_20231208T080839Z_gen_g4_detsim_hitreco__20240510T062228Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/f1/44/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_895_20231203T084115Z_gen_g4_detsim_hitreco__20240509T211836Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/16/b0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_999_20231202T075105Z_gen_g4_detsim_hitreco__20240507T204751Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/e0/3a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_367_20231203T113006Z_gen_g4_detsim_hitreco__20240509T194417Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6c/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_555_20231203T081444Z_gen_g4_detsim_hitreco__20240509T205759Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9b/01/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_726_20231207T233337Z_gen_g4_detsim_hitreco__20240510T050627Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/45/24/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50581281_357_20231204T051406Z_gen_g4_detsim_hitreco__20240509T212924Z_reco2.root
.:
total 44640
-rw-r--r--. 1 dune osgvo 16793894 Sep 15 11:15 flatcaf.root
-rw-r--r--. 1 dune osgvo 14031865 Sep 15 11:15 caf.root
-rw-r--r--. 1 dune osgvo 14031865 Sep 15 11:15 caf_fd_hd_atmo_2501_20250915T123615Z.root
-rw-r--r--. 1 dune osgvo   593147 Sep 15 11:15 caf_20250915T123615Z.log
-rw-r--r--. 1 dune osgvo   212274 Sep 15 11:15 jobscript.log
-rw-r--r--. 1 dune osgvo     1925 Sep 15 11:15 caf_20250915T123615Z.file
-rw-r--r--. 1 dune osgvo     1925 Sep 15 11:15 caf_20250915T123615Z.pfns
-rw-r--r--. 1 dune osgvo     1925 Sep 15 08:36 file.list
-rw-r--r--. 1 dune osgvo     1925 Sep 15 11:15 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo     1375 Sep 15 08:36 all-input-dids.txt
-rw-r--r--. 1 dune osgvo     1375 Sep 15 11:15 caf_20250915T123615Z.did
-rw-r--r--. 1 dune osgvo     1375 Sep 15 08:36 did.list
-rw-r--r--. 1 dune osgvo        0 Sep 15 08:36 debugprod.log