Jobsub ID 44184.14@dunegpschedd02.fnal.gov

Jobsub ID	44184.14@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-20 10:04:53
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-09-20 11:25:12
From worker node	Hostname	wn-snel-029.farm.nikhef.nl
	cpuinfo	AMD EPYC 7H12 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-20 10:05:07
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_239_20231201T083045Z_gen_g4_detsim_hitreco__20240507T202854Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_749_20231202T130439Z_gen_g4_detsim_hitreco__20240507T221940Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_138_20231202T092209Z_gen_g4_detsim_hitreco__20240507T212811Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_221_20231202T124157Z_gen_g4_detsim_hitreco__20240507T220556Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_474_20231205T100842Z_gen_g4_detsim_hitreco__20240510T035421Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_746_20231204T045457Z_gen_g4_detsim_hitreco__20240509T211006Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_198_20231204T005852Z_gen_g4_detsim_hitreco__20240509T213421Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_159_20231204T100452Z_gen_g4_detsim_hitreco__20240509T213840Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_992_20231202T173341Z_gen_g4_detsim_hitreco__20240507T210931Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_866_20231204T075513Z_gen_g4_detsim_hitreco__20240509T220142Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4779s)
	CPU time	1h (4610s = 96%)
	Max RSS bytes	2059632640 (1964 MiB)
Outputting started	2025-09-20 11:24:46
Output files
Finished	2025-09-20 11:25:07
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000322444    0.0348535      1.91927     0.00261524     0.154452       1000    
reco:energyrecnumumcs:EnergyReco                      0.000682758    0.0355623      1.87631     0.00308642     0.154844       1000    
reco:energyrecnumumcsllhd:EnergyReco                  0.000297807    0.0360157      1.84715     0.00323042     0.157923       1000    
reco:opdec:Deconvolution                              0.00469808     0.156758      0.559786      0.154548      0.0641077      1000    
reco:ophitspe:OpHitFinderDeco                         0.00296859    0.00898423     0.309343     0.00849074    0.00991549      1000    
reco:opflash:OpFlashFinder                            0.000179997   0.00151952     0.0115836    0.00148037    0.000626327     1000    
reco:opslicer:OpSlicer                                3.8302e-05     0.192235      0.746261      0.151471       0.15525       1000    
reco:rns:RandomNumberSaver                            2.7642e-05    5.02559e-05   0.000631522   4.77145e-05   2.1457e-05      1000    
reco:anglereconue:NuAngularReco                       0.000198712   0.00175558     0.0810779    0.000561702   0.00493213      1000    
reco:anglereconumu:NuAngularReco                      7.1815e-05    0.00135972     0.0706519    0.000204543   0.00466342      1000    
reco:anglereconuepfps:NuAngularReco                   0.000129402   0.00153115     0.0744154    0.000365264   0.00488518      1000    
reco:anglereconumupfps:NuAngularReco                  9.4747e-05    0.00146397     0.0801624    0.00025653     0.0050117      1000    
reco:anglerecohits:NuAngularReco                      8.4869e-05    0.00216612     0.0840377    0.000759416    0.0056302      1000    
[art]:TriggerResults:TriggerResultInserter             1.085e-05    1.81077e-05   8.3777e-05    1.7117e-05    4.84881e-06     1000    
end_path:cafmaker:CAFMaker                             0.0021069     0.0232739     0.746309     0.00642962     0.0645188      1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3717.47 MB
  Peak resident set size usage (VmHWM): 2059.63 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 264, 272, 265
minwire 0: 1871
minwire 1: 905
minwire 2: 1863
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000215283, 0.998913, 0.00087153, 
Output 1: 0.00960776, 0.986195, 0.00406957, 0.000127471, 
Output 2: 0.998743, 0.00122287, 3.31987e-05, 9.32616e-07, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1696!
Begin processing the 995th record. run: 50584512 subRun: 1 event: 86695 at 20-Sep-2025 13:24:17 CEST
Boundary wire vector sizes: 3097, 4834, 3199
minwire 0: 967
minwire 1: 977
minwire 2: 940
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00296809, 0.835408, 0.161624, 
Output 1: 0.345939, 0.644909, 0.00792987, 0.00122182, 
Output 2: 0.864515, 0.124659, 0.0095427, 0.00128332, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3296!
Begin processing the 996th record. run: 50584512 subRun: 1 event: 86696 at 20-Sep-2025 13:24:28 CEST
Boundary wire vector sizes: 405, 310, 411
minwire 0: 2121
minwire 1: 441
minwire 2: 2165
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0288127, 0.0055581, 0.965629, 
Output 1: 0.022421, 0.361853, 0.521924, 0.0938018, 
Output 2: 0.296705, 0.652219, 0.0502231, 0.000852956, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1735!
Begin processing the 997th record. run: 50584512 subRun: 1 event: 86697 at 20-Sep-2025 13:24:32 CEST
Boundary wire vector sizes: 565, 663, 503
minwire 0: 1043
minwire 1: 864
minwire 2: 652
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00213111, 0.991815, 0.00605371, 
Output 1: 0.00943404, 0.0278392, 0.0622856, 0.900441, 
Output 2: 0.646488, 0.307989, 0.0375578, 0.0079646, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2738!
Begin processing the 998th record. run: 50584512 subRun: 1 event: 86698 at 20-Sep-2025 13:24:36 CEST
Boundary wire vector sizes: 137, 150, 80
minwire 0: 2419
minwire 1: 983
minwire 2: 2089
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.215994, 0.0144484, 0.769557, 
Output 1: 0.748305, 0.16435, 0.0772477, 0.0100968, 
Output 2: 0.21659, 0.767097, 0.0160957, 0.000217485, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1913!
Begin processing the 999th record. run: 50584512 subRun: 1 event: 86699 at 20-Sep-2025 13:24:39 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 137!
Begin processing the 1000th record. run: 50584512 subRun: 1 event: 86700 at 20-Sep-2025 13:24:42 CEST
Boundary wire vector sizes: 158, 50, 112
minwire 0: 2167
minwire 1: 371
minwire 2: 2411
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.989268, 0.00055227, 0.0101801, 
Output 1: 0.654071, 0.343587, 0.00221609, 0.000126691, 
Output 2: 0.993145, 0.00671366, 0.000134378, 6.85092e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 630!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/95/79/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_239_20231201T083045Z_gen_g4_detsim_hitreco__20240507T202854Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/1c/e2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_749_20231202T130439Z_gen_g4_detsim_hitreco__20240507T221940Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/7c/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_138_20231202T092209Z_gen_g4_detsim_hitreco__20240507T212811Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/8a/28/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_221_20231202T124157Z_gen_g4_detsim_hitreco__20240507T220556Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/36/85/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_474_20231205T100842Z_gen_g4_detsim_hitreco__20240510T035421Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/5e/f2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_746_20231204T045457Z_gen_g4_detsim_hitreco__20240509T211006Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/b3/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_198_20231204T005852Z_gen_g4_detsim_hitreco__20240509T213421Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/d8/da/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_159_20231204T100452Z_gen_g4_detsim_hitreco__20240509T213840Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e3/12/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50568178_992_20231202T173341Z_gen_g4_detsim_hitreco__20240507T210931Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/32/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_866_20231204T075513Z_gen_g4_detsim_hitreco__20240509T220142Z_reco2.root
.:
total 52160
-rw-r--r--. 1 dune012 dune 19592787 Sep 20 13:24 flatcaf.root
-rw-r--r--. 1 dune012 dune 16465788 Sep 20 13:24 caf.root
-rw-r--r--. 1 dune012 dune 16465788 Sep 20 13:24 caf_fd_hd_atmo_2501_20250920T100518Z.root
-rw-r--r--. 1 dune012 dune   603276 Sep 20 13:24 caf_20250920T100518Z.log
-rw-r--r--. 1 dune012 dune   229384 Sep 20 13:24 jobscript.log
-rw-r--r--. 1 dune012 dune     2166 Sep 20 13:24 caf_20250920T100518Z.file
-rw-r--r--. 1 dune012 dune     2166 Sep 20 13:24 caf_20250920T100518Z.pfns
-rw-r--r--. 1 dune012 dune     2166 Sep 20 12:05 file.list
-rw-r--r--. 1 dune012 dune     2166 Sep 20 13:24 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     1376 Sep 20 12:05 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     1376 Sep 20 13:24 caf_20250920T100518Z.did
-rw-r--r--. 1 dune012 dune     1376 Sep 20 12:05 did.list
-rw-r--r--. 1 dune012 dune        0 Sep 20 12:05 debugprod.log