Jobsub ID 44825.18@dunegpschedd02.fnal.gov

Jobsub ID	44825.18@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-21 09:10:02
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-09-21 09:22:50
From worker node	Hostname	comp15-01
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-21 09:11:24
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_711_20231204T225152Z_gen_g4_detsim_hitreco__20240509T221952Z_reco2.root
Jobscript	Exit code	0
	Real time	10m (644s)
	CPU time	8m (491s = 76%)
	Max RSS bytes	2358001664 (2248 MiB)
Outputting started	2025-09-21 09:22:09
Output files
Finished	2025-09-21 09:22:50
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

0.67286       5.3412        153.209       2.56288       16.1757        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0290778     0.0582836     0.103502      0.0541465     0.016637        100    
reco:pandora:StandardPandora                           0.192488       3.83639       139.795       1.29525       14.8642        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000101061   0.00612393     0.221435     0.000980164    0.0231338       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000144201    0.0112458     0.123646     0.00248785     0.024477        100    
reco:pandoracalo:Calorimetry                           8.805e-05     0.0417481       2.309      0.000634421    0.242498        100    
reco:pandorapid:Chi2ParticleID                         3.046e-05    0.000145246   0.00103025    9.9656e-05    0.000163174      100    
reco:cvnmap:CVNMapper                                  1.637e-05     0.0076433     0.0430516     0.0050511    0.00874972       100    
reco:cvneva:CVNEvaluator                               1.638e-05     0.244373       2.00512      0.301475      0.224956        100    
reco:energyrecnumu:EnergyReco                         0.000242533    0.0934931      1.94784      0.0165978     0.247174        100    
reco:energyrecnue:EnergyReco                          0.000109921   0.00222335     0.0802393    0.000478045   0.00826384       100    
reco:energyrecnc:EnergyReco                           0.00149947     0.145072      0.505879      0.179731       0.10167        100    
reco:energyrecnumurange:EnergyReco                    0.000263962    0.0407777      1.69664     0.00169162     0.210802        100    
reco:energyrecnumumcs:EnergyReco                      0.000428994    0.0412669      1.69971      0.0018568     0.211105        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000325873    0.0411261      1.70032     0.00183721     0.211057        100    
reco:opdec:Deconvolution                               0.0882529     0.245437      0.536656      0.246955      0.0720849       100    
reco:ophitspe:OpHitFinderDeco                          0.255948      0.264003      0.293826       0.26365     0.00460255       100    
reco:opflash:OpFlashFinder                            0.000113471    0.0010103    0.00268301    0.000939854   0.000503072      100    
reco:opslicer:OpSlicer                                 3.949e-05     0.206611      0.887075       0.15756      0.179492        100    
reco:rns:RandomNumberSaver                             1.748e-05    3.82011e-05   0.000266493   3.42605e-05   2.68165e-05      100    
reco:anglereconue:NuAngularReco                       0.000139562    0.0013685     0.050899     0.000313948   0.00522329       100    
reco:anglereconumu:NuAngularReco                       5.636e-05    0.00120551     0.0507537    0.000165526    0.0052164       100    
reco:anglereconuepfps:NuAngularReco                   9.2751e-05    0.00130752     0.0509498    0.000248042   0.00522764       100    
reco:anglereconumupfps:NuAngularReco                  7.3251e-05    0.00125776     0.0516289    0.000203982   0.00529758       100    
reco:anglerecohits:NuAngularReco                       7.561e-05    0.00181411     0.0574142    0.000479395   0.00594235       100    
[art]:TriggerResults:TriggerResultInserter             9.21e-06     1.10848e-05    3.544e-05    1.0205e-05    3.36378e-06      100    
end_path:cafmaker:CAFMaker                            0.00103377     0.0462941      2.78891     0.00496989     0.284134        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4033.17 MB
  Peak resident set size usage (VmHWM): 2358 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1078!
Begin processing the 93rd record. run: 50598687 subRun: 1 event: 71193 at 21-Sep-2025 10:19:12 BST
Boundary wire vector sizes: 105, 64, 91
minwire 0: 669
minwire 1: 2088
minwire 2: 575
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995517, 0.000548676, 0.00393393, 
Output 1: 0.00311271, 0.991167, 0.00558222, 0.000137861, 
Output 2: 0.994392, 0.00548757, 0.000118371, 2.32508e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 872!
Begin processing the 94th record. run: 50598687 subRun: 1 event: 71194 at 21-Sep-2025 10:19:15 BST
Boundary wire vector sizes: 37, 38, 35
minwire 0: 54
minwire 1: 2277
minwire 2: 38
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00719818, 0.00184885, 0.990953, 
Output 1: 0.445882, 0.538538, 0.0151332, 0.00044722, 
Output 2: 0.998169, 0.00173627, 9.00645e-05, 4.59549e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 623!
Begin processing the 95th record. run: 50598687 subRun: 1 event: 71195 at 21-Sep-2025 10:19:17 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 746!
Begin processing the 96th record. run: 50598687 subRun: 1 event: 71196 at 21-Sep-2025 10:19:20 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 408!
Begin processing the 97th record. run: 50598687 subRun: 1 event: 71197 at 21-Sep-2025 10:19:22 BST
Boundary wire vector sizes: 18136, 14243, 7868
minwire 0: 747
minwire 1: 230
minwire 2: 10
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.389282, 0.56994, 0.0407777, 
Output 1: 0.258047, 0.40549, 0.186408, 0.150055, 
Output 2: 0.0136618, 0.0465502, 0.186065, 0.753723, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 4075!
Begin processing the 98th record. run: 50598687 subRun: 1 event: 71198 at 21-Sep-2025 10:21:59 BST
Boundary wire vector sizes: 93, 101, 20
minwire 0: 2765
minwire 1: 655
minwire 2: 2532
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0043925, 0.00219185, 0.993416, 
Output 1: 0.0331135, 0.957067, 0.00916419, 0.000655566, 
Output 2: 0.997802, 0.00211466, 7.89098e-05, 4.63999e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1267!
Begin processing the 99th record. run: 50598687 subRun: 1 event: 71199 at 21-Sep-2025 10:22:02 BST
Boundary wire vector sizes: 86, 107, 91
minwire 0: 826
minwire 1: 1915
minwire 2: 528
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000571904, 0.995054, 0.00437428, 
Output 1: 0.995839, 0.00392884, 0.000115294, 0.000117274, 
Output 2: 0.997767, 0.00214336, 6.11913e-05, 2.89538e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1302!
Begin processing the 100th record. run: 50598687 subRun: 1 event: 71200 at 21-Sep-2025 10:22:04 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 475!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d2/99/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_711_20231204T225152Z_gen_g4_detsim_hitreco__20240509T221952Z_reco2.root
.:
total 6800
-rw-r--r-- 1 pltdune002 pltdune 2634386 Sep 21 10:22 flatcaf.root
-rw-r--r-- 1 pltdune002 pltdune 2088098 Sep 21 10:22 caf.root
-rw-r--r-- 1 pltdune002 pltdune 2088098 Sep 21 10:22 caf_fd_hd_atmo_2501_20250921T091133Z.root
-rw-r--r-- 1 pltdune002 pltdune   75045 Sep 21 10:22 caf_20250921T091133Z.log
-rw-r--r-- 1 pltdune002 pltdune   37169 Sep 21 10:22 jobscript.log
-rw-r--r-- 1 pltdune002 pltdune     187 Sep 21 10:22 caf_20250921T091133Z.file
-rw-r--r-- 1 pltdune002 pltdune     187 Sep 21 10:22 caf_20250921T091133Z.pfns
-rw-r--r-- 1 pltdune002 pltdune     187 Sep 21 10:11 file.list
-rw-r--r-- 1 pltdune002 pltdune     187 Sep 21 10:22 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune002 pltdune     138 Sep 21 10:11 all-input-dids.txt
-rw-r--r-- 1 pltdune002 pltdune     138 Sep 21 10:22 caf_20250921T091133Z.did
-rw-r--r-- 1 pltdune002 pltdune     138 Sep 21 10:11 did.list
-rw-r--r-- 1 pltdune002 pltdune       0 Sep 21 10:11 debugprod.log