Jobsub ID 50596.10@dunegpschedd01.fnal.gov

Jobsub ID	50596.10@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-20 11:26:56
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-09-20 12:33:35
From worker node	Hostname	wn-snel-029.farm.nikhef.nl
	cpuinfo	AMD EPYC 7H12 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-20 11:28:20
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_239_20231201T083045Z_gen_g4_detsim_hitreco__20240507T202854Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_749_20231202T130439Z_gen_g4_detsim_hitreco__20240507T221940Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_138_20231202T092209Z_gen_g4_detsim_hitreco__20240507T212811Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_221_20231202T124157Z_gen_g4_detsim_hitreco__20240507T220556Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_474_20231205T100842Z_gen_g4_detsim_hitreco__20240510T035421Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_746_20231204T045457Z_gen_g4_detsim_hitreco__20240509T211006Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_198_20231204T005852Z_gen_g4_detsim_hitreco__20240509T213421Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_992_20231202T173341Z_gen_g4_detsim_hitreco__20240507T210931Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3891s)
	CPU time	1h (3772s = 96%)
	Max RSS bytes	2029113344 (1935 MiB)
Outputting started	2025-09-20 12:33:12
Output files
Finished	2025-09-20 12:33:35
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

46      0.0112357     0.211128        800    
reco:energyrecnue:EnergyReco                          0.000131065   0.00275261     0.136909     0.000626593   0.00925025       800    
reco:energyrecnc:EnergyReco                           0.00189977     0.0164039     0.241905      0.0121757     0.018013        800    
reco:energyrecnumurange:EnergyReco                    0.000267421    0.0367776      1.99437     0.00259405     0.162252        800    
reco:energyrecnumumcs:EnergyReco                      0.000478776    0.0369538      1.96807     0.00300341     0.159231        800    
reco:energyrecnumumcsllhd:EnergyReco                  0.000316592    0.037025       1.93431     0.00310749     0.159117        800    
reco:opdec:Deconvolution                              0.00506046     0.157539      0.418361      0.155301      0.0632163       800    
reco:ophitspe:OpHitFinderDeco                         0.00250863    0.00844053     0.0279645    0.00831716    0.00231267       800    
reco:opflash:OpFlashFinder                            0.000208931   0.00145162    0.00809712    0.00137939    0.000596498      800    
reco:opslicer:OpSlicer                                4.1107e-05     0.195139      0.751785      0.152029      0.157448        800    
reco:rns:RandomNumberSaver                            2.5037e-05    4.81179e-05   0.000546574   4.63565e-05   2.0543e-05       800    
reco:anglereconue:NuAngularReco                       0.000183714   0.00179421     0.0846735    0.000535517   0.00532846       800    
reco:anglereconumu:NuAngularReco                      6.8528e-05    0.00145066     0.0864334    0.000201116   0.00531761       800    
reco:anglereconuepfps:NuAngularReco                   0.000116137   0.00160499     0.0901523    0.000351313   0.00545745       800    
reco:anglereconumupfps:NuAngularReco                  9.7272e-05    0.00155605     0.081398     0.000255489    0.0054315       800    
reco:anglerecohits:NuAngularReco                      8.0079e-05    0.00221665     0.076262     0.00075562    0.00591466       800    
[art]:TriggerResults:TriggerResultInserter            1.0841e-05    1.80828e-05   0.000111909   1.68265e-05   6.06271e-06      800    
end_path:cafmaker:CAFMaker                            0.00225723     0.0233554     0.752293     0.00636661     0.0636941       800    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3686.09 MB
  Peak resident set size usage (VmHWM): 2029.11 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1745!
Begin processing the 792nd record. run: 50568178 subRun: 1 event: 99292 at 20-Sep-2025 14:32:37 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 273!
Begin processing the 793rd record. run: 50568178 subRun: 1 event: 99293 at 20-Sep-2025 14:32:39 CEST
Boundary wire vector sizes: 84, 113, 78
minwire 0: 887
minwire 1: 1404
minwire 2: 1014
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00638547, 0.00772509, 0.985889, 
Output 1: 0.0915644, 0.891212, 0.0166865, 0.000536571, 
Output 2: 0.980999, 0.0185059, 0.000481943, 1.31462e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1200!
Begin processing the 794th record. run: 50568178 subRun: 1 event: 99294 at 20-Sep-2025 14:32:42 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 804!
Begin processing the 795th record. run: 50568178 subRun: 1 event: 99295 at 20-Sep-2025 14:32:45 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1055!
Begin processing the 796th record. run: 50568178 subRun: 1 event: 99296 at 20-Sep-2025 14:32:47 CEST
Boundary wire vector sizes: 1244, 1402, 442
minwire 0: 1655
minwire 1: 763
minwire 2: 1455
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995789, 0.00061824, 0.00359243, 
Output 1: 0.00568292, 0.0217519, 0.191194, 0.781371, 
Output 2: 0.991524, 0.00800556, 0.000396898, 7.34233e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3124!
Begin processing the 797th record. run: 50568178 subRun: 1 event: 99297 at 20-Sep-2025 14:32:52 CEST
Boundary wire vector sizes: 1755, 3104, 2526
minwire 0: 1202
minwire 1: 219
minwire 2: 774
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.985438, 0.0055065, 0.00905553, 
Output 1: 0.919392, 0.0771384, 0.00254101, 0.000928394, 
Output 2: 0.922847, 0.0750647, 0.0018755, 0.000212839, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3040!
Begin processing the 798th record. run: 50568178 subRun: 1 event: 99298 at 20-Sep-2025 14:32:57 CEST
Boundary wire vector sizes: 152, 332, 132
minwire 0: 1392
minwire 1: 1087
minwire 2: 1478
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.989171, 0.000479397, 0.0103495, 
Output 1: 0.97649, 0.0224665, 0.000930119, 0.00011354, 
Output 2: 0.981791, 0.0178548, 0.000328657, 2.57998e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1525!
Begin processing the 799th record. run: 50568178 subRun: 1 event: 99299 at 20-Sep-2025 14:33:01 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 222!
Begin processing the 800th record. run: 50568178 subRun: 1 event: 99300 at 20-Sep-2025 14:33:03 CEST
Boundary wire vector sizes: 3273, 2327, 1987
minwire 0: 223
minwire 1: 1595
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0673395, 0.876406, 0.0562542, 
Output 1: 0.481672, 0.359168, 0.105883, 0.0532764, 
Output 2: 0.049262, 0.190414, 0.365224, 0.3951, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2683!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/95/79/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_239_20231201T083045Z_gen_g4_detsim_hitreco__20240507T202854Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/1c/e2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_749_20231202T130439Z_gen_g4_detsim_hitreco__20240507T221940Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/7c/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_138_20231202T092209Z_gen_g4_detsim_hitreco__20240507T212811Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/8a/28/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_221_20231202T124157Z_gen_g4_detsim_hitreco__20240507T220556Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/36/85/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_474_20231205T100842Z_gen_g4_detsim_hitreco__20240510T035421Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/5e/f2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_746_20231204T045457Z_gen_g4_detsim_hitreco__20240509T211006Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/b3/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_198_20231204T005852Z_gen_g4_detsim_hitreco__20240509T213421Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e3/12/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_992_20231202T173341Z_gen_g4_detsim_hitreco__20240507T210931Z_reco2.root
.:
total 43980
-rw-r--r--. 1 dune012 dune 16529392 Sep 20 14:33 flatcaf.root
-rw-r--r--. 1 dune012 dune 13891715 Sep 20 14:33 caf.root
-rw-r--r--. 1 dune012 dune 13891715 Sep 20 14:33 caf_fd_hd_atmo_2501_20250920T112831Z.root
-rw-r--r--. 1 dune012 dune   485116 Sep 20 14:33 caf_20250920T112831Z.log
-rw-r--r--. 1 dune012 dune   183514 Sep 20 14:33 jobscript.log
-rw-r--r--. 1 dune012 dune     1732 Sep 20 14:33 caf_20250920T112831Z.file
-rw-r--r--. 1 dune012 dune     1732 Sep 20 14:33 caf_20250920T112831Z.pfns
-rw-r--r--. 1 dune012 dune     1732 Sep 20 13:28 file.list
-rw-r--r--. 1 dune012 dune     1732 Sep 20 14:33 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     1100 Sep 20 13:28 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     1100 Sep 20 14:33 caf_20250920T112831Z.did
-rw-r--r--. 1 dune012 dune     1100 Sep 20 13:28 did.list
-rw-r--r--. 1 dune012 dune        0 Sep 20 13:28 debugprod.log