Jobsub ID 47760.9@dunegpschedd01.fnal.gov

Jobsub ID	47760.9@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 19:57:35
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce03
Last heartbeat	2025-09-17 21:42:52
From worker node	Hostname	dune001-8710192.0-lcg2668.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-17 19:58:54
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_744_20231202T151316Z_gen_g4_detsim_hitreco__20240508T030200Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1067_20231203T135356Z_gen_g4_detsim_hitreco__20240508T043122Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_592_20231201T192544Z_gen_g4_detsim_hitreco__20240507T214511Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_43_20231120T123826Z_gen_g4_detsim_hitreco__20240503T062046Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_355_20231207T134750Z_gen_g4_detsim_hitreco__20240510T062955Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_662_20231207T162527Z_gen_g4_detsim_hitreco__20240510T031552Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_515_20231202T162339Z_gen_g4_detsim_hitreco__20240508T043754Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_554_20231206T233324Z_gen_g4_detsim_hitreco__20240510T063337Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_608_20231205T104609Z_gen_g4_detsim_hitreco__20240510T040308Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6209s)
	CPU time	1h (4592s = 73%)
	Max RSS bytes	2274971648 (2169 MiB)
Outputting started	2025-09-17 21:42:24
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250917T195907Z.root
Finished	2025-09-17 21:42:52
Saved logs	justin-logs:47760.9-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Reco                         0.000328989    0.113262       10.3666      0.0252657     0.368756       1000    
reco:energyrecnue:EnergyReco                          0.000144181   0.00320646     0.0883741    0.000804517   0.00812174      1000    
reco:energyrecnc:EnergyReco                            0.0017758      0.15952       1.10089      0.181721      0.114352       1000    
reco:energyrecnumurange:EnergyReco                    0.000316926    0.0501771      9.41756     0.00367944      0.33003       1000    
reco:energyrecnumumcs:EnergyReco                      0.000688154    0.0500907      9.35454     0.00472681     0.326132       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000485653    0.0497271      9.54765      0.0046111     0.330972       1000    
reco:opdec:Deconvolution                               0.0871283     0.315126      0.748633      0.305545      0.102298       1000    
reco:ophitspe:OpHitFinderDeco                          0.241899      0.271014      0.400184       0.26071      0.0326931      1000    
reco:opflash:OpFlashFinder                            0.000251663   0.00223865     0.030315     0.00197237    0.00157516      1000    
reco:opslicer:OpSlicer                                5.1237e-05     0.324341       1.29383      0.251766      0.267417       1000    
reco:rns:RandomNumberSaver                            2.5718e-05    6.59518e-05   0.00122184    6.1395e-05    4.70876e-05     1000    
reco:anglereconue:NuAngularReco                       0.00020761    0.00205601     0.0589566    0.00059336    0.00537928      1000    
reco:anglereconumu:NuAngularReco                      9.2494e-05    0.00172215     0.060422     0.000345284   0.00519757      1000    
reco:anglereconuepfps:NuAngularReco                   0.000141376    0.0019363     0.100751     0.000488187   0.00590741      1000    
reco:anglereconumupfps:NuAngularReco                  0.000111509   0.00186488     0.126556     0.000426632   0.00627357      1000    
reco:anglerecohits:NuAngularReco                      0.000114525   0.00273062     0.0823249    0.000971135    0.0063312      1000    
[art]:TriggerResults:TriggerResultInserter            1.1051e-05    2.17318e-05   0.000208963   1.9858e-05    1.04823e-05     1000    
end_path:cafmaker:CAFMaker                            0.00160243     0.0272047     0.961353     0.00872265     0.070676       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3709.09 MB
  Peak resident set size usage (VmHWM): 2274.97 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1099!
Begin processing the 995th record. run: 50602933 subRun: 1 event: 60895 at 17-Sep-2025 21:41:57 UTC
Boundary wire vector sizes: 154, 73, 81
minwire 0: 1935
minwire 1: 1135
minwire 2: 1900
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99298, 0.000309349, 0.00671072, 
Output 1: 0.99572, 0.00403766, 0.000136127, 0.000105912, 
Output 2: 0.991859, 0.0079371, 0.000160639, 4.34318e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1674!
Begin processing the 996th record. run: 50602933 subRun: 1 event: 60896 at 17-Sep-2025 21:42:01 UTC
Boundary wire vector sizes: 206, 400, 284
minwire 0: 920
minwire 1: 2749
minwire 2: 245
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00242451, 0.828992, 0.168584, 
Output 1: 0.00524742, 0.990235, 0.00441229, 0.000105651, 
Output 2: 0.99705, 0.00287237, 7.4125e-05, 3.33404e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1810!
Begin processing the 997th record. run: 50602933 subRun: 1 event: 60897 at 17-Sep-2025 21:42:05 UTC
Boundary wire vector sizes: 1005, 1289, 1121
minwire 0: 255
minwire 1: 1157
minwire 2: 222
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00881009, 0.954164, 0.0370258, 
Output 1: 0.0778936, 0.316815, 0.259557, 0.345735, 
Output 2: 0.344262, 0.540637, 0.101584, 0.0135173, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2548!
Begin processing the 998th record. run: 50602933 subRun: 1 event: 60898 at 17-Sep-2025 21:42:10 UTC
Boundary wire vector sizes: 557, 490, 235
minwire 0: 2082
minwire 1: 638
minwire 2: 2215
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.984477, 0.00157118, 0.0139517, 
Output 1: 0.289396, 0.675002, 0.0338223, 0.00178024, 
Output 2: 0.0198987, 0.933542, 0.0460465, 0.000513244, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1811!
Begin processing the 999th record. run: 50602933 subRun: 1 event: 60899 at 17-Sep-2025 21:42:14 UTC
Boundary wire vector sizes: 379, 300, 261
minwire 0: 1510
minwire 1: 2087
minwire 2: 962
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00275724, 0.0024736, 0.994769, 
Output 1: 0.0300424, 0.965418, 0.0043149, 0.00022465, 
Output 2: 0.998458, 0.00148679, 5.27569e-05, 2.90477e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2417!
Begin processing the 1000th record. run: 50602933 subRun: 1 event: 60900 at 17-Sep-2025 21:42:18 UTC
Boundary wire vector sizes: 138, 94, 85
minwire 0: 2891
minwire 1: 780
minwire 2: 2712
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000480675, 0.998263, 0.00125662, 
Output 1: 0.0163056, 0.889131, 0.0906407, 0.00392246, 
Output 2: 0.998822, 0.00111994, 5.63707e-05, 1.73019e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1126!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f7/ff/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_744_20231202T151316Z_gen_g4_detsim_hitreco__20240508T030200Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f3/9b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1067_20231203T135356Z_gen_g4_detsim_hitreco__20240508T043122Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/fd/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_592_20231201T192544Z_gen_g4_detsim_hitreco__20240507T214511Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7a/51/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_43_20231120T123826Z_gen_g4_detsim_hitreco__20240503T062046Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/19/8a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_355_20231207T134750Z_gen_g4_detsim_hitreco__20240510T062955Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f0/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_662_20231207T162527Z_gen_g4_detsim_hitreco__20240510T031552Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5f/46/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_515_20231202T162339Z_gen_g4_detsim_hitreco__20240508T043754Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1d/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_554_20231206T233324Z_gen_g4_detsim_hitreco__20240510T063337Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fb/9f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_608_20231205T104609Z_gen_g4_detsim_hitreco__20240510T040308Z_reco2.root
.:
total 44516
-rw-r--r-- 1 dune001 dune 16704385 Sep 17 21:42 flatcaf.root
-rw-r--r-- 1 dune001 dune 13953291 Sep 17 21:42 caf.root
-rw-r--r-- 1 dune001 dune 13953291 Sep 17 21:42 caf_fd_hd_atmo_2501_20250917T195907Z.root
-rw-r--r-- 1 dune001 dune   601126 Sep 17 21:42 caf_20250917T195907Z.log
-rw-r--r-- 1 dune001 dune   217290 Sep 17 21:42 jobscript.log
-rw-r--r-- 1 dune001 dune     1742 Sep 17 21:42 caf_20250917T195907Z.file
-rw-r--r-- 1 dune001 dune     1742 Sep 17 21:42 caf_20250917T195907Z.pfns
-rw-r--r-- 1 dune001 dune     1742 Sep 17 19:59 file.list
-rw-r--r-- 1 dune001 dune     1742 Sep 17 21:42 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune     1239 Sep 17 19:59 all-input-dids.txt
-rw-r--r-- 1 dune001 dune     1239 Sep 17 21:42 caf_20250917T195907Z.did
-rw-r--r-- 1 dune001 dune     1239 Sep 17 19:59 did.list
-rw-r--r-- 1 dune001 dune        0 Sep 17 19:59 debugprod.log