Jobsub ID 43947.0@dunegpschedd02.fnal.gov

Jobsub ID	43947.0@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-19 22:14:11
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp
Last heartbeat	2025-09-19 22:22:47
From worker node	Hostname	wn-sate-036.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-19 22:15:44
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_443_20231202T105844Z_gen_g4_detsim_hitreco__20240507T210756Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (403s)
	CPU time	6m (382s = 94%)
	Max RSS bytes	2017955840 (1924 MiB)
Outputting started	2025-09-19 22:22:28
Output files
Finished	2025-09-19 22:22:47
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

==================================================================================================================
Full event                                              1.63922       3.26924       22.3201       2.75153       2.77358        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000809671   0.00147837    0.00243749    0.00144065    0.000353798      100    
reco:pandora:StandardPandora                            1.55454       2.19101       18.7059       1.68092       2.28307        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000131196   0.00568745     0.135961     0.00151314     0.0151581       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000184917    0.0151851     0.311525     0.00291127     0.0385044       100    
reco:pandoracalo:Calorimetry                          0.000107832    0.0110265     0.263991     0.00124291     0.0359137       100    
reco:pandorapid:Chi2ParticleID                        3.9785e-05    0.000154343   0.00194857    0.000111199   0.000204688      100    
reco:cvnmap:CVNMapper                                 2.5548e-05     0.0114123     0.0354729     0.0114718    0.00884492       100    
reco:cvneva:CVNEvaluator                              1.8976e-05     0.452513       2.64256      0.604755      0.352783        100    
reco:energyrecnumu:EnergyReco                         0.000266391    0.0308545     0.326796      0.0114746     0.0605346       100    
reco:energyrecnue:EnergyReco                          0.000188765   0.00205972     0.0572525    0.000586883   0.00598566       100    
reco:energyrecnc:EnergyReco                            0.002656      0.013063      0.0938152     0.0124664     0.0114547       100    
reco:energyrecnumurange:EnergyReco                    0.000283262    0.0244453     0.298276     0.00291503     0.0550125       100    
reco:energyrecnumumcs:EnergyReco                      0.000552378    0.0251911     0.297537      0.0033145     0.0547144       100    
reco:energyrecnumumcsllhd:EnergyReco                  0.00045246     0.025186      0.298955     0.00330393     0.0547361       100    
reco:opdec:Deconvolution                               0.0256106     0.173667      0.312536      0.167554      0.0685876       100    
reco:ophitspe:OpHitFinderDeco                         0.00364178     0.0086291     0.025683     0.00825032    0.00289444       100    
reco:opflash:OpFlashFinder                            0.000240382   0.00121094    0.00431903    0.00113598    0.000617097      100    
reco:opslicer:OpSlicer                                0.00115804     0.244044      0.663173      0.191895      0.189622        100    
reco:rns:RandomNumberSaver                            1.8575e-05    2.66154e-05   0.000298381   2.3349e-05    2.74268e-05      100    
reco:anglereconue:NuAngularReco                       0.000156724   0.00132641     0.0460718    0.000333928   0.00473006       100    
reco:anglereconumu:NuAngularReco                      9.4888e-05    0.00122416     0.0457471    0.000233373   0.00470344       100    
reco:anglereconuepfps:NuAngularReco                   0.000137058    0.0013452     0.0470776    0.00031977    0.00484675       100    
reco:anglereconumupfps:NuAngularReco                  0.000135034   0.00131178     0.0467482    0.000303611    0.0047996       100    
reco:anglerecohits:NuAngularReco                      0.000363974   0.00181694     0.0489418    0.000669457   0.00508819       100    
[art]:TriggerResults:TriggerResultInserter            1.0469e-05    1.23489e-05   5.1847e-05    1.1627e-05    4.22567e-06      100    
end_path:cafmaker:CAFMaker                            0.00107827     0.0241493     0.880846     0.00372802     0.100294        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3680.28 MB
  Peak resident set size usage (VmHWM): 2017.96 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
minwire 0: 1908
minwire 1: 984
minwire 2: 1726
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0011142, 0.993816, 0.00506985, 
Output 1: 0.995689, 0.0039136, 0.000214726, 0.000182373, 
Output 2: 0.996353, 0.00351194, 9.88409e-05, 3.61773e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1168!
Begin processing the 95th record. run: 6402377 subRun: 1 event: 44395 at 20-Sep-2025 00:22:10 CEST
Boundary wire vector sizes: 230, 201, 147
minwire 0: 1391
minwire 1: 2081
minwire 2: 902
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.843281, 0.0462783, 0.110441, 
Output 1: 0.958014, 0.040867, 0.00107652, 4.24168e-05, 
Output 2: 0.0786243, 0.899517, 0.0216619, 0.000197314, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1875!
Begin processing the 96th record. run: 6402377 subRun: 1 event: 44396 at 20-Sep-2025 00:22:13 CEST
Boundary wire vector sizes: 1540, 1580, 1317
minwire 0: 2157
minwire 1: 648
minwire 2: 1942
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.982487, 0.00527549, 0.0122377, 
Output 1: 0.439853, 0.402288, 0.10398, 0.0538796, 
Output 2: 0.1891, 0.433215, 0.265376, 0.112309, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2751!
Begin processing the 97th record. run: 6402377 subRun: 1 event: 44397 at 20-Sep-2025 00:22:17 CEST
Boundary wire vector sizes: 152, 164, 118
minwire 0: 1921
minwire 1: 1742
minwire 2: 1506
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0012026, 0.970794, 0.0280033, 
Output 1: 0.957072, 0.042297, 0.000543268, 8.7257e-05, 
Output 2: 0.997299, 0.0026326, 5.8745e-05, 9.22069e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1772!
Begin processing the 98th record. run: 6402377 subRun: 1 event: 44398 at 20-Sep-2025 00:22:20 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 507!
Begin processing the 99th record. run: 6402377 subRun: 1 event: 44399 at 20-Sep-2025 00:22:22 CEST
Boundary wire vector sizes: 66, 47, 58
minwire 0: 560
minwire 1: 2749
minwire 2: 113
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0106336, 0.988187, 0.00117958, 
Output 1: 0.00521984, 0.991091, 0.00361144, 7.75203e-05, 
Output 2: 0.998456, 0.00148488, 5.65365e-05, 2.64343e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 844!
Begin processing the 100th record. run: 6402377 subRun: 1 event: 44400 at 20-Sep-2025 00:22:25 CEST
Boundary wire vector sizes: 51, 101, 56
minwire 0: 2471
minwire 1: 311
minwire 2: 2320
Used alternate method to get min and max wires due to vertex determination failure: 29, 328
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.982178, 0.00049461, 0.0173271, 
Output 1: 0.0170115, 0.808817, 0.169863, 0.00430838, 
Output 2: 0.974525, 0.0244925, 0.000958526, 2.39906e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1394!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/63/a9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_443_20231202T105844Z_gen_g4_detsim_hitreco__20240507T210756Z_reco2.root
.:
total 4592
-rw-r--r--. 1 dune012 dune 1811048 Sep 20 00:22 flatcaf.root
-rw-r--r--. 1 dune012 dune 1365105 Sep 20 00:22 caf.root
-rw-r--r--. 1 dune012 dune 1365105 Sep 20 00:22 caf_fd_hd_atmo_2501_20250919T221553Z.root
-rw-r--r--. 1 dune012 dune   71504 Sep 20 00:22 caf_20250919T221553Z.log
-rw-r--r--. 1 dune012 dune   33923 Sep 20 00:22 jobscript.log
-rw-r--r--. 1 dune012 dune     216 Sep 20 00:22 caf_20250919T221553Z.file
-rw-r--r--. 1 dune012 dune     216 Sep 20 00:22 caf_20250919T221553Z.pfns
-rw-r--r--. 1 dune012 dune     216 Sep 20 00:15 file.list
-rw-r--r--. 1 dune012 dune     216 Sep 20 00:22 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     137 Sep 20 00:15 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     137 Sep 20 00:22 caf_20250919T221553Z.did
-rw-r--r--. 1 dune012 dune     137 Sep 20 00:15 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 20 00:16 debugprod.log