Jobsub ID 47733.1@dunegpschedd01.fnal.gov

Jobsub ID	47733.1@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 19:01:32
Site	US_PuertoRico
Entry	CMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat	2025-09-17 21:36:39
From worker node	Hostname	compute-0-5.local
	cpuinfo	Intel(R) Xeon(R) CPU W3550 @ 3.07GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-17 19:02:55
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_256_20231207T124229Z_gen_g4_detsim_hitreco__20240510T042435Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_742_20231207T045153Z_gen_g4_detsim_hitreco__20240510T061558Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_812_20231202T182107Z_gen_g4_detsim_hitreco__20240508T065049Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_306_20231207T113856Z_gen_g4_detsim_hitreco__20240510T053334Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_964_20231202T140205Z_gen_g4_detsim_hitreco__20240507T223604Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_749_20231204T120445Z_gen_g4_detsim_hitreco__20240509T202718Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_165_20231203T164416Z_gen_g4_detsim_hitreco__20240509T193750Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (9197s)
	CPU time	1h (4393s = 47%)
	Max RSS bytes	1901441024 (1813 MiB)
Outputting started	2025-09-17 21:36:14
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250917T190307Z.root
Finished	2025-09-17 21:36:39
Saved logs	justin-logs:47733.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

55       4.58147       1.14356      0.691052        700    
reco:energyrecnumu:EnergyReco                         0.000287737     0.34835       2.33265      0.0303379     0.461413        700    
reco:energyrecnue:EnergyReco                          0.000207047   0.00293801     0.0990217    0.000784372   0.00793875       700    
reco:energyrecnc:EnergyReco                           0.00209574     0.740014       1.83216      0.985935      0.510902        700    
reco:energyrecnumurange:EnergyReco                    0.000281959    0.0487189      2.19522      0.0025852     0.189281        700    
reco:energyrecnumumcs:EnergyReco                      0.000568424    0.0503772      2.1968      0.00346035     0.193878        700    
reco:energyrecnumumcsllhd:EnergyReco                  0.000481205    0.0504197      2.20305     0.00347635     0.195291        700    
reco:opdec:Deconvolution                               0.462499      0.846596       1.7523       0.836369      0.174316        700    
reco:ophitspe:OpHitFinderDeco                           1.35066       1.42968       2.61355       1.46947      0.0839562       700    
reco:opflash:OpFlashFinder                            0.000156652   0.00148517    0.00817796    0.00138013    0.000792549      700    
reco:opslicer:OpSlicer                                4.8173e-05     0.301594       1.40159       0.23632      0.256676        700    
reco:rns:RandomNumberSaver                            1.9835e-05    4.72676e-05    0.0138644    2.5554e-05    0.000522763      700    
reco:anglereconue:NuAngularReco                       0.000153773   0.00192739     0.100699     0.000371432   0.00649357       700    
reco:anglereconumu:NuAngularReco                      0.000105113   0.00180842     0.100283     0.000275524    0.0064422       700    
reco:anglereconuepfps:NuAngularReco                   0.000147728   0.00190564     0.100316     0.000367537   0.00644716       700    
reco:anglereconumupfps:NuAngularReco                  0.000148709    0.0019065     0.100333     0.000355359   0.00646999       700    
reco:anglerecohits:NuAngularReco                      0.00011907    0.00264343     0.108473     0.000808625   0.00724087       700    
[art]:TriggerResults:TriggerResultInserter            1.2513e-05    1.51102e-05   6.1024e-05    1.33705e-05   4.09515e-06      700    
end_path:cafmaker:CAFMaker                            0.00122293     0.023035         1.2       0.00426373     0.0798148       700    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3815.46 MB
  Peak resident set size usage (VmHWM): 1901.44 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99756, 0.000480425, 0.00196002, 
Output 1: 0.309489, 0.503547, 0.116815, 0.0701502, 
Output 2: 0.709411, 0.23871, 0.0448918, 0.00698727, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1922!
Begin processing the 695th record. run: 74511854 subRun: 1 event: 16595 at 17-Sep-2025 17:35:00 AST
Boundary wire vector sizes: 279, 225, 197
minwire 0: 2225
minwire 1: 979
minwire 2: 2168
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0025363, 0.834813, 0.162651, 
Output 1: 0.674693, 0.306021, 0.0173447, 0.00194188, 
Output 2: 0.987632, 0.0120101, 0.000321344, 3.62526e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1986!
Begin processing the 696th record. run: 74511854 subRun: 1 event: 16596 at 17-Sep-2025 17:35:11 AST
Boundary wire vector sizes: 542, 502, 281
minwire 0: 1145
minwire 1: 1453
minwire 2: 834
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00172478, 0.982127, 0.0161479, 
Output 1: 0.876453, 0.0932948, 0.0205983, 0.00965432, 
Output 2: 0.914887, 0.0814805, 0.00323016, 0.000402799, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2804!
Begin processing the 697th record. run: 74511854 subRun: 1 event: 16597 at 17-Sep-2025 17:35:23 AST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 441!
Begin processing the 698th record. run: 74511854 subRun: 1 event: 16598 at 17-Sep-2025 17:35:31 AST
Boundary wire vector sizes: 627, 416, 452
minwire 0: 1219
minwire 1: 808
minwire 2: 1373
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.708105, 0.0404994, 0.251395, 
Output 1: 0.0434741, 0.935756, 0.0198972, 0.000872714, 
Output 2: 0.310698, 0.623647, 0.0637798, 0.00187588, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1949!
Begin processing the 699th record. run: 74511854 subRun: 1 event: 16599 at 17-Sep-2025 17:35:42 AST
Boundary wire vector sizes: 543, 730, 560
minwire 0: 291
minwire 1: 1947
minwire 2: 307
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.971917, 0.0060756, 0.0220079, 
Output 1: 0.214356, 0.742907, 0.0394547, 0.00328167, 
Output 2: 0.0432458, 0.443668, 0.429248, 0.0838384, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1467!
Begin processing the 700th record. run: 74511854 subRun: 1 event: 16600 at 17-Sep-2025 17:35:53 AST
Boundary wire vector sizes: 53, 54, 50
minwire 0: 2739
minwire 1: 678
minwire 2: 2644
Used alternate method to get min and max wires due to vertex determination failure: 2, 301
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.157803, 0.00516037, 0.837037, 
Output 1: 0.33108, 0.577703, 0.0884639, 0.00275333, 
Output 2: 0.965466, 0.0333324, 0.00116406, 3.76497e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 931!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/13/60/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_256_20231207T124229Z_gen_g4_detsim_hitreco__20240510T042435Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/40/77/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_742_20231207T045153Z_gen_g4_detsim_hitreco__20240510T061558Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2e/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_812_20231202T182107Z_gen_g4_detsim_hitreco__20240508T065049Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/4d/ce/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_306_20231207T113856Z_gen_g4_detsim_hitreco__20240510T053334Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/80/75/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_964_20231202T140205Z_gen_g4_detsim_hitreco__20240507T223604Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/f8/9f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_749_20231204T120445Z_gen_g4_detsim_hitreco__20240509T202718Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1a/6d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_165_20231203T164416Z_gen_g4_detsim_hitreco__20240509T193750Z_reco2.root
.:
total 31076
-rw-r--r-- 1 dunepilot gridusers 11731202 Sep 17 17:36 flatcaf.root
-rw-r--r-- 1 dunepilot gridusers  9728175 Sep 17 17:36 caf.root
-rw-r--r-- 1 dunepilot gridusers  9728175 Sep 17 17:36 caf_fd_hd_atmo_2501_20250917T190307Z.root
-rw-r--r-- 1 dunepilot gridusers   416628 Sep 17 17:36 caf_20250917T190307Z.log
-rw-r--r-- 1 dunepilot gridusers   158795 Sep 17 17:36 jobscript.log
-rw-r--r-- 1 dunepilot gridusers     1398 Sep 17 17:36 caf_20250917T190307Z.file
-rw-r--r-- 1 dunepilot gridusers     1398 Sep 17 17:36 caf_20250917T190307Z.pfns
-rw-r--r-- 1 dunepilot gridusers     1398 Sep 17 15:03 file.list
-rw-r--r-- 1 dunepilot gridusers     1398 Sep 17 17:36 justin-processed-pfns.txt
-rw-r--r-- 1 dunepilot gridusers      963 Sep 17 15:03 all-input-dids.txt
-rw-r--r-- 1 dunepilot gridusers      963 Sep 17 17:36 caf_20250917T190307Z.did
-rw-r--r-- 1 dunepilot gridusers      963 Sep 17 15:03 did.list
-rw-r--r-- 1 dunepilot gridusers        0 Sep 17 15:03 debugprod.log