Jobsub ID 47877.33@dunegpschedd01.fnal.gov

Jobsub ID	47877.33@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 23:13:45
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2025-09-18 01:57:19
From worker node	Hostname	wn2203291.tier2.hep.manchester.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-09-17 23:14:26
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_388_20231208T092631Z_gen_g4_detsim_hitreco__20240510T063717Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_150_20231203T022425Z_gen_g4_detsim_hitreco__20240508T042530Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_726_20231203T110356Z_gen_g4_detsim_hitreco__20240509T195827Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_609_20231208T050602Z_gen_g4_detsim_hitreco__20240510T040919Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_8_20231207T233507Z_gen_g4_detsim_hitreco__20240510T042526Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_51_20231203T071007Z_gen_g4_detsim_hitreco__20240508T055251Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_609_20231208T100639Z_gen_g4_detsim_hitreco__20240510T064950Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_162_20231120T132911Z_gen_g4_detsim_hitreco__20240503T061710Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_738_20231207T063245Z_gen_g4_detsim_hitreco__20240510T045613Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_534_20231203T041545Z_gen_g4_detsim_hitreco__20240508T071939Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (9743s)
	CPU time	2h (7803s = 80%)
	Max RSS bytes	2085269504 (1988 MiB)
Outputting started	2025-09-18 01:56:50
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/pgranger/cafs/fnal/02501/1/002/caf_fd_hd_atmo_2501_20250917T231437Z.root
Finished	2025-09-18 01:57:19
Saved logs	justin-logs:47877.33-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

reco:energyrecnc:EnergyReco                           0.00231646     0.168502       7.62183       0.17457      0.300818       1100    
reco:energyrecnumurange:EnergyReco                    0.000311442    0.0887755      5.89657     0.00465962     0.425593       1100    
reco:energyrecnumumcs:EnergyReco                      0.000689601    0.0890776      6.33297     0.00594881     0.432469       1100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000597581    0.088625       6.17398     0.00595428     0.430209       1100    
reco:opdec:Deconvolution                               0.046902      0.458859       8.23557      0.440598      0.286935       1100    
reco:ophitspe:OpHitFinderDeco                          0.0892925     0.182506      0.451681      0.178018      0.0536584      1100    
reco:opflash:OpFlashFinder                            0.000223121   0.00310327     0.0207761     0.0023284    0.00299901      1100    
reco:opslicer:OpSlicer                                6.3301e-05     0.464236       2.08192      0.367721      0.388411       1100    
reco:rns:RandomNumberSaver                            2.3156e-05    5.7087e-05    0.000429063   5.09215e-05   2.86924e-05     1100    
reco:anglereconue:NuAngularReco                       0.000194262   0.00359155     0.184935     0.000690826    0.0124421      1100    
reco:anglereconumu:NuAngularReco                      0.000113067   0.00317782     0.181901     0.000415029    0.0119359      1100    
reco:anglereconuepfps:NuAngularReco                   0.000170566   0.00329486     0.170135     0.000548442    0.0115199      1100    
reco:anglereconumupfps:NuAngularReco                  0.000149905   0.00339656      0.17096     0.000533002    0.0119271      1100    
reco:anglerecohits:NuAngularReco                      0.000140038   0.00451589     0.183412     0.00126246     0.0137237      1100    
[art]:TriggerResults:TriggerResultInserter            1.1043e-05    2.04164e-05   0.000109861   1.8343e-05    7.48334e-06     1100    
end_path:cafmaker:CAFMaker                            0.00177966     0.0451637      1.98141      0.0114354     0.142278       1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3748.36 MB
  Peak resident set size usage (VmHWM): 2085.27 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1288!
Begin processing the 1095th record. run: 66006611 subRun: 1 event: 53495 at 18-Sep-2025 02:53:46 BST
Boundary wire vector sizes: 9821, 11801, 10600
minwire 0: 124
minwire 1: 1416
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.541608, 0.0832402, 0.375152, 
Output 1: 0.409638, 0.44276, 0.0827699, 0.064833, 
Output 2: 0.0268501, 0.0783769, 0.218944, 0.675829, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3593!
Begin processing the 1096th record. run: 66006611 subRun: 1 event: 53496 at 18-Sep-2025 02:56:17 BST
Boundary wire vector sizes: 363, 352, 338
minwire 0: 1515
minwire 1: 558
minwire 2: 1779
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000233421, 0.998845, 0.000921428, 
Output 1: 0.00380892, 0.991966, 0.00409569, 0.00012908, 
Output 2: 0.998291, 0.00166527, 4.22476e-05, 1.23498e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1588!
Begin processing the 1097th record. run: 66006611 subRun: 1 event: 53497 at 18-Sep-2025 02:56:23 BST
Boundary wire vector sizes: 238, 112, 173
minwire 0: 874
minwire 1: 2358
minwire 2: 510
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00157714, 0.983836, 0.0145865, 
Output 1: 0.0122733, 0.919605, 0.0653814, 0.00273985, 
Output 2: 0.842337, 0.153265, 0.00432373, 7.44916e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1822!
Begin processing the 1098th record. run: 66006611 subRun: 1 event: 53498 at 18-Sep-2025 02:56:29 BST
Boundary wire vector sizes: 84, 114, 106
minwire 0: 2883
minwire 1: 695
minwire 2: 2659
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000851556, 0.980972, 0.0181761, 
Output 1: 0.961333, 0.0384283, 0.000180557, 5.80872e-05, 
Output 2: 0.999332, 0.000646239, 1.61963e-05, 5.04621e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 813!
Begin processing the 1099th record. run: 66006611 subRun: 1 event: 53499 at 18-Sep-2025 02:56:34 BST
Boundary wire vector sizes: 589, 654, 420
minwire 0: 1791
minwire 1: 1655
minwire 2: 970
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00315941, 0.955763, 0.0410772, 
Output 1: 0.0824304, 0.876753, 0.0385893, 0.00222753, 
Output 2: 0.744589, 0.247867, 0.00737129, 0.000172177, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2992!
Begin processing the 1100th record. run: 66006611 subRun: 1 event: 53500 at 18-Sep-2025 02:56:43 BST
Boundary wire vector sizes: 164, 230, 159
minwire 0: 1736
minwire 1: 522
minwire 2: 2061
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00210348, 0.988697, 0.00919949, 
Output 1: 0.131922, 0.486299, 0.327866, 0.0539131, 
Output 2: 0.873729, 0.122674, 0.00349117, 0.000106216, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1251!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/22/22/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_388_20231208T092631Z_gen_g4_detsim_hitreco__20240510T063717Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/8b/22/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_150_20231203T022425Z_gen_g4_detsim_hitreco__20240508T042530Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/72/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_726_20231203T110356Z_gen_g4_detsim_hitreco__20240509T195827Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/e2/7d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_609_20231208T050602Z_gen_g4_detsim_hitreco__20240510T040919Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/0c/ef/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_8_20231207T233507Z_gen_g4_detsim_hitreco__20240510T042526Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/c2/9c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_51_20231203T071007Z_gen_g4_detsim_hitreco__20240508T055251Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/c7/6f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_609_20231208T100639Z_gen_g4_detsim_hitreco__20240510T064950Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/0a/28/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_162_20231120T132911Z_gen_g4_detsim_hitreco__20240503T061710Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/1f/bb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_738_20231207T063245Z_gen_g4_detsim_hitreco__20240510T045613Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/cd/48/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_534_20231203T041545Z_gen_g4_detsim_hitreco__20240508T071939Z_reco2.root
.:
total 50092
-rw-r--r-- 1 dune01 dune01 18846848 Sep 18 02:56 flatcaf.root
-rw-r--r-- 1 dune01 dune01 15743703 Sep 18 02:56 caf.root
-rw-r--r-- 1 dune01 dune01 15743703 Sep 18 02:56 caf_fd_hd_atmo_2501_20250917T231437Z.root
-rw-r--r-- 1 dune01 dune01   676709 Sep 18 02:56 caf_20250917T231437Z.log
-rw-r--r-- 1 dune01 dune01   228071 Sep 18 02:56 jobscript.log
-rw-r--r-- 1 dune01 dune01     1970 Sep 18 02:56 caf_20250917T231437Z.file
-rw-r--r-- 1 dune01 dune01     1970 Sep 18 02:56 caf_20250917T231437Z.pfns
-rw-r--r-- 1 dune01 dune01     1970 Sep 18 00:14 file.list
-rw-r--r-- 1 dune01 dune01     1970 Sep 18 02:56 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01     1370 Sep 18 00:14 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01     1370 Sep 18 02:56 caf_20250917T231437Z.did
-rw-r--r-- 1 dune01 dune01     1370 Sep 18 00:14 did.list
-rw-r--r-- 1 dune01 dune01        0 Sep 18 00:15 debugprod.log