Jobsub ID 50475.0@dunegpschedd01.fnal.gov

Jobsub ID	50475.0@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-20 03:32:31
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-09-20 03:39:40
From worker node	Hostname	wn-lot-028.farm.nikhef.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-20 03:33:05
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_588_20231208T045943Z_gen_g4_detsim_hitreco__20240510T040936Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (378s)
	CPU time	5m (324s = 85%)
	Max RSS bytes	1988018176 (1895 MiB)
Outputting started	2025-09-20 03:39:24
Output files
Finished	2025-09-20 03:39:40
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

===================================================================================
Full event                                              0.24877       3.03272       65.9329       2.00551       6.5258         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00921214     0.0140001     0.0197919     0.0103647    0.00442224       100    
reco:pandora:StandardPandora                           0.0652355      1.95343       50.4259       1.15599       4.99473        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000102692   0.00467849     0.108598     0.00111961     0.0137773       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000145672   0.00717683     0.129206      0.0024955     0.015818        100    
reco:pandoracalo:Calorimetry                          8.3767e-05     0.0149707     0.689715     0.000800562    0.072764        100    
reco:pandorapid:Chi2ParticleID                        3.1159e-05    0.000113831   0.00112954    8.56355e-05   0.000130541      100    
reco:cvnmap:CVNMapper                                 1.5158e-05    0.00725125     0.0323166    0.00853445    0.00511351       100    
reco:cvneva:CVNEvaluator                              1.3906e-05     0.273906       2.11628      0.337449       0.2371         100    
reco:energyrecnumu:EnergyReco                         0.000216525    0.0728145      3.23574     0.00927701     0.335488        100    
reco:energyrecnue:EnergyReco                          0.00012321    0.00163266     0.0396756    0.000355661   0.00489778       100    
reco:energyrecnc:EnergyReco                            0.0019575     0.0494731     0.166799      0.0607942     0.0337758       100    
reco:energyrecnumurange:EnergyReco                    0.000182882    0.0510556      3.01306     0.00214276     0.312977        100    
reco:energyrecnumumcs:EnergyReco                      0.000462776    0.0516075      3.00531      0.0023379     0.312151        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000353091    0.0517967      3.00563     0.00230212     0.312233        100    
reco:opdec:Deconvolution                               0.0371141     0.173381      0.364676      0.167343      0.0614327       100    
reco:ophitspe:OpHitFinderDeco                          0.0791215     0.0836852     0.102571      0.0834348    0.00261554       100    
reco:opflash:OpFlashFinder                            0.00015514    0.000973889   0.00270538    0.000892278   0.000502722      100    
reco:opslicer:OpSlicer                                0.00018715     0.195881      0.852999      0.150927      0.170772        100    
reco:rns:RandomNumberSaver                             1.592e-05    2.45629e-05   0.000263473   2.26575e-05   2.40875e-05      100    
reco:anglereconue:NuAngularReco                       0.000124904   0.00111505     0.0291128    0.000263703   0.00367582       100    
reco:anglereconumu:NuAngularReco                      6.6103e-05    0.00100875     0.0296157    0.000148933    0.003746        100    
reco:anglereconuepfps:NuAngularReco                   9.8855e-05     0.0010978     0.0296095    0.000219661   0.00374167       100    
reco:anglereconumupfps:NuAngularReco                   9.081e-05     0.0010529     0.0293971    0.000199327   0.00371093       100    
reco:anglerecohits:NuAngularReco                      7.0522e-05    0.00151056     0.0329393    0.000511637   0.00409269       100    
[art]:TriggerResults:TriggerResultInserter             7.684e-06    9.86476e-06   3.5607e-05     9.483e-06    2.80139e-06      100    
end_path:cafmaker:CAFMaker                            0.00118168     0.0181397      0.52323     0.00288235     0.0656454       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3654.97 MB
  Peak resident set size usage (VmHWM): 1988.02 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000383038, 0.998037, 0.00157979, 
Output 1: 0.0141112, 0.981608, 0.0039791, 0.000301401, 
Output 2: 0.973938, 0.025354, 0.000689264, 1.84717e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1833!
Begin processing the 94th record. run: 6492179 subRun: 1 event: 58894 at 20-Sep-2025 05:39:08 CEST
Boundary wire vector sizes: 102, 170, 148
minwire 0: 621
minwire 1: 1863
minwire 2: 519
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.97477, 0.000434138, 0.0247961, 
Output 1: 0.0210273, 0.676939, 0.297451, 0.00458257, 
Output 2: 0.978906, 0.0203617, 0.000718189, 1.42648e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1174!
Begin processing the 95th record. run: 6492179 subRun: 1 event: 58895 at 20-Sep-2025 05:39:10 CEST
Boundary wire vector sizes: 300, 201, 169
minwire 0: 2051
minwire 1: 903
minwire 2: 2025
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000708966, 0.99629, 0.00300067, 
Output 1: 0.205144, 0.725579, 0.0653415, 0.00393564, 
Output 2: 0.993626, 0.00618564, 0.000181494, 6.88876e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2118!
Begin processing the 96th record. run: 6492179 subRun: 1 event: 58896 at 20-Sep-2025 05:39:12 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1136!
Begin processing the 97th record. run: 6492179 subRun: 1 event: 58897 at 20-Sep-2025 05:39:14 CEST
Boundary wire vector sizes: 70, 58, 44
minwire 0: 996
minwire 1: 2257
minwire 2: 596
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0019462, 0.995993, 0.00206022, 
Output 1: 0.974413, 0.0253439, 0.00012205, 0.000121592, 
Output 2: 0.999576, 0.00040707, 1.09055e-05, 5.81726e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1224!
Begin processing the 98th record. run: 6492179 subRun: 1 event: 58898 at 20-Sep-2025 05:39:16 CEST
Boundary wire vector sizes: 152, 161, 165
minwire 0: 1936
minwire 1: 1310
minwire 2: 1699
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99445, 0.00043109, 0.00511902, 
Output 1: 0.969207, 0.0304942, 0.000266545, 3.17244e-05, 
Output 2: 0.99614, 0.00378315, 7.0402e-05, 6.59313e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1875!
Begin processing the 99th record. run: 6492179 subRun: 1 event: 58899 at 20-Sep-2025 05:39:18 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 990!
Begin processing the 100th record. run: 6492179 subRun: 1 event: 58900 at 20-Sep-2025 05:39:20 CEST
Boundary wire vector sizes: 207, 173, 199
minwire 0: 1293
minwire 1: 1458
minwire 2: 1027
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00329438, 0.00195597, 0.99475, 
Output 1: 0.89303, 0.105671, 0.000879908, 0.000418701, 
Output 2: 0.998894, 0.001066, 3.27128e-05, 7.23568e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2096!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/5e/bb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_588_20231208T045943Z_gen_g4_detsim_hitreco__20240510T040936Z_reco2.root
.:
total 4256
-rw-r--r--. 1 dune012 dune 1686976 Sep 20 05:39 flatcaf.root
-rw-r--r--. 1 dune012 dune 1254968 Sep 20 05:39 caf.root
-rw-r--r--. 1 dune012 dune 1254968 Sep 20 05:39 caf_fd_hd_atmo_2501_20250920T033312Z.root
-rw-r--r--. 1 dune012 dune   75210 Sep 20 05:39 caf_20250920T033312Z.log
-rw-r--r--. 1 dune012 dune   35158 Sep 20 05:39 jobscript.log
-rw-r--r--. 1 dune012 dune     200 Sep 20 05:39 caf_20250920T033312Z.file
-rw-r--r--. 1 dune012 dune     200 Sep 20 05:39 caf_20250920T033312Z.pfns
-rw-r--r--. 1 dune012 dune     200 Sep 20 05:33 file.list
-rw-r--r--. 1 dune012 dune     200 Sep 20 05:39 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     137 Sep 20 05:33 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     137 Sep 20 05:39 caf_20250920T033312Z.did
-rw-r--r--. 1 dune012 dune     137 Sep 20 05:33 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 20 05:33 debugprod.log