Jobsub ID 45030.58@dunegpschedd02.fnal.gov

Jobsub ID	45030.58@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-21 15:42:23
Site	UK_Bristol
Entry	CMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat	2025-09-21 17:56:55
From worker node	Hostname	hd81.dice.priv
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	259200 (72 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-21 15:43:48
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_847_20231202T091437Z_gen_g4_detsim_hitreco__20240507T210639Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_30_20231207T022309Z_gen_g4_detsim_hitreco__20240510T035819Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_124_20231207T123644Z_gen_g4_detsim_hitreco__20240510T042353Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_833_20231204T044102Z_gen_g4_detsim_hitreco__20240509T201259Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_279_20231202T190829Z_gen_g4_detsim_hitreco__20240508T062958Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_665_20231204T220718Z_gen_g4_detsim_hitreco__20240510T032520Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_327_20231207T111934Z_gen_g4_detsim_hitreco__20240510T053248Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_609_20231203T082915Z_gen_g4_detsim_hitreco__20240508T035044Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_453_20231205T015920Z_gen_g4_detsim_hitreco__20240509T222936Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_145_20231121T154938Z_gen_g4_detsim_hitreco__20240503T073223Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (7955s)
	CPU time	1h (6590s = 82%)
	Max RSS bytes	2040532992 (1946 MiB)
Outputting started	2025-09-21 17:56:24
Output files
Finished	2025-09-21 17:56:55
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eco:energyrecnumurange:EnergyReco                    0.000332498    0.0416307      3.45027     0.00323615     0.195429       1100    
reco:energyrecnumumcs:EnergyReco                      0.000655988    0.0426656      3.47199     0.00407981     0.195906       1100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000514711    0.0427327      3.45118     0.00403929     0.196427       1100    
reco:opdec:Deconvolution                               0.0377406      0.28358      0.609398       0.28463      0.0949487      1100    
reco:ophitspe:OpHitFinderDeco                          0.0961039     0.191406      0.262987      0.221545      0.0531369      1100    
reco:opflash:OpFlashFinder                            0.000192323   0.00158638    0.00711796    0.00153427    0.000713057     1100    
reco:opslicer:OpSlicer                                5.0535e-05     0.294837       1.26496      0.238081      0.241589       1100    
reco:rns:RandomNumberSaver                             2.092e-05    3.90277e-05   0.000399694   3.3233e-05    1.73112e-05     1100    
reco:anglereconue:NuAngularReco                       0.000184629   0.00193902     0.110507     0.000466039   0.00704853      1100    
reco:anglereconumu:NuAngularReco                      0.000105149   0.00173454     0.108639     0.000281762   0.00697932      1100    
reco:anglereconuepfps:NuAngularReco                   0.000157377    0.0018529     0.114664     0.000396693    0.0070874      1100    
reco:anglereconumupfps:NuAngularReco                  0.000149472   0.00182557     0.109214     0.000359883   0.00704539      1100    
reco:anglerecohits:NuAngularReco                      0.000118273   0.00257383     0.116714     0.000859381   0.00776948      1100    
[art]:TriggerResults:TriggerResultInserter            1.1522e-05    1.50787e-05   6.5744e-05    1.3506e-05    4.67243e-06     1100    
end_path:cafmaker:CAFMaker                             0.0014226     0.0267589      1.27576     0.00660067     0.0932395      1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3722.72 MB
  Peak resident set size usage (VmHWM): 2040.53 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
minwire 2: 1255
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.081976, 0.0949749, 0.823049, 
Output 1: 0.158627, 0.455968, 0.26007, 0.125336, 
Output 2: 0.0361997, 0.142391, 0.42088, 0.40053, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3214!
Begin processing the 1092nd record. run: 6208838 subRun: 1 event: 29192 at 21-Sep-2025 17:55:41 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 286!
Begin processing the 1093rd record. run: 6208838 subRun: 1 event: 29193 at 21-Sep-2025 17:55:44 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1099!
Begin processing the 1094th record. run: 6208838 subRun: 1 event: 29194 at 21-Sep-2025 17:55:47 UTC
Boundary wire vector sizes: 47, 45, 44
minwire 0: 1046
minwire 1: 1876
minwire 2: 876
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0164227, 0.00250785, 0.981069, 
Output 1: 0.0622845, 0.51729, 0.399686, 0.0207399, 
Output 2: 0.990253, 0.00928881, 0.000443844, 1.45904e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1537!
Begin processing the 1095th record. run: 6208838 subRun: 1 event: 29195 at 21-Sep-2025 17:55:52 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 412!
Begin processing the 1096th record. run: 6208838 subRun: 1 event: 29196 at 21-Sep-2025 17:55:55 UTC
Boundary wire vector sizes: 2791, 2575, 1605
minwire 0: 348
minwire 1: 2348
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.200183, 0.789697, 0.0101202, 
Output 1: 0.150733, 0.253744, 0.250382, 0.345141, 
Output 2: 0.0891544, 0.299727, 0.352967, 0.258152, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2547!
Begin processing the 1097th record. run: 6208838 subRun: 1 event: 29197 at 21-Sep-2025 17:56:04 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 335!
Begin processing the 1098th record. run: 6208838 subRun: 1 event: 29198 at 21-Sep-2025 17:56:07 UTC
Boundary wire vector sizes: 1220, 835, 862
minwire 0: 680
minwire 1: 1738
minwire 2: 762
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.975046, 0.0058214, 0.0191327, 
Output 1: 0.938676, 0.0607775, 0.000451855, 9.48401e-05, 
Output 2: 0.992999, 0.00683675, 0.000146127, 1.79793e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2307!
Begin processing the 1099th record. run: 6208838 subRun: 1 event: 29199 at 21-Sep-2025 17:56:13 UTC
Boundary wire vector sizes: 654, 600, 386
minwire 0: 1110
minwire 1: 1976
minwire 2: 810
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00345035, 0.83956, 0.156989, 
Output 1: 0.00728965, 0.986672, 0.00584318, 0.000195596, 
Output 2: 0.992526, 0.00733381, 0.000136138, 4.3302e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2990!
Begin processing the 1100th record. run: 6208838 subRun: 1 event: 29200 at 21-Sep-2025 17:56:18 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 967!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/03/fb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_847_20231202T091437Z_gen_g4_detsim_hitreco__20240507T210639Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/58/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_30_20231207T022309Z_gen_g4_detsim_hitreco__20240510T035819Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/26/96/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_124_20231207T123644Z_gen_g4_detsim_hitreco__20240510T042353Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/d7/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_833_20231204T044102Z_gen_g4_detsim_hitreco__20240509T201259Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9b/8f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_279_20231202T190829Z_gen_g4_detsim_hitreco__20240508T062958Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/71/d4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_665_20231204T220718Z_gen_g4_detsim_hitreco__20240510T032520Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a8/e4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_327_20231207T111934Z_gen_g4_detsim_hitreco__20240510T053248Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/15/d9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_609_20231203T082915Z_gen_g4_detsim_hitreco__20240508T035044Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/60/b0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_453_20231205T015920Z_gen_g4_detsim_hitreco__20240509T222936Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/96/9e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_145_20231121T154938Z_gen_g4_detsim_hitreco__20240503T073223Z_reco2.root
.:
total 47044
-rw-r--r-- 1 dnpil001 users 17672246 Sep 21 17:56 flatcaf.root
-rw-r--r-- 1 dnpil001 users 14768126 Sep 21 17:56 caf.root
-rw-r--r-- 1 dnpil001 users 14768126 Sep 21 17:56 caf_fd_hd_atmo_2501_20250921T154359Z.root
-rw-r--r-- 1 dnpil001 users   658429 Sep 21 17:56 caf_20250921T154359Z.log
-rw-r--r-- 1 dnpil001 users   251751 Sep 21 17:56 jobscript.log
-rw-r--r-- 1 dnpil001 users     1959 Sep 21 17:56 caf_20250921T154359Z.file
-rw-r--r-- 1 dnpil001 users     1959 Sep 21 17:56 caf_20250921T154359Z.pfns
-rw-r--r-- 1 dnpil001 users     1959 Sep 21 15:43 file.list
-rw-r--r-- 1 dnpil001 users     1959 Sep 21 17:56 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users     1375 Sep 21 15:43 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users     1375 Sep 21 17:56 caf_20250921T154359Z.did
-rw-r--r-- 1 dnpil001 users     1375 Sep 21 15:43 did.list
-rw-r--r-- 1 dnpil001 users        0 Sep 21 15:44 debugprod.log