Jobsub ID 45204.1@dunegpschedd02.fnal.gov

Jobsub ID	45204.1@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-21 19:32:35
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-09-21 19:43:49
From worker node	Hostname	wn-sate-041.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-21 19:34:02
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_625_20231202T150939Z_gen_g4_detsim_hitreco__20240508T024146Z_reco2.root
Jobscript	Exit code	0
	Real time	9m (567s)
	CPU time	8m (488s = 86%)
	Max RSS bytes	2147131392 (2047 MiB)
Outputting started	2025-09-21 19:43:30
Output files
Finished	2025-09-21 19:43:49
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.92643       4.69413       125.101       2.94423       12.3996        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0152493     0.031074      0.0903978     0.032574      0.010398        100    
reco:pandora:StandardPandora                            1.59239       3.22005       103.802       1.66999       10.3505        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000135395   0.00998729     0.469325     0.00121501     0.0476907       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000190929    0.0137331     0.312795     0.00278023     0.0413522       100    
reco:pandoracalo:Calorimetry                          0.000110849    0.0323106      2.06128     0.00090026     0.208434        100    
reco:pandorapid:Chi2ParticleID                         4.214e-05    0.000159245   0.00112554    0.000113554   0.000171149      100    
reco:cvnmap:CVNMapper                                 2.8334e-05    0.00855657     0.045843      0.0109666    0.00725035       100    
reco:cvneva:CVNEvaluator                              1.9968e-05      0.42408       2.20584      0.604554       0.33725        100    
reco:energyrecnumu:EnergyReco                         0.000283474    0.0921611      3.94968      0.0151447     0.394277        100    
reco:energyrecnue:EnergyReco                          0.000184437   0.00387258     0.186055     0.000497135    0.0187536       100    
reco:energyrecnc:EnergyReco                           0.00253365     0.0821811     0.490837      0.0966152     0.0691635       100    
reco:energyrecnumurange:EnergyReco                    0.000259319    0.0504489      3.11801     0.00288741     0.313942        100    
reco:energyrecnumumcs:EnergyReco                      0.00057884     0.051473        3.125      0.00318016      0.31452        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000431653    0.0522879      3.22072     0.00325559     0.323878        100    
reco:opdec:Deconvolution                               0.069659      0.220121      0.496781      0.217144      0.0809848       100    
reco:ophitspe:OpHitFinderDeco                          0.131002      0.135634      0.145053       0.13551     0.00257858       100    
reco:opflash:OpFlashFinder                            0.00023921     0.0011407    0.00333724    0.00103993    0.000611848      100    
reco:opslicer:OpSlicer                                0.00106098     0.218363       0.89584      0.145894      0.206716        100    
reco:rns:RandomNumberSaver                            2.0469e-05    3.02329e-05   0.000343537   2.6941e-05    3.16417e-05      100    
reco:anglereconue:NuAngularReco                       0.000158107   0.00244337     0.120766     0.000314447    0.0122356       100    
reco:anglereconumu:NuAngularReco                      9.3857e-05    0.00234519     0.121374     0.00019429     0.0123073       100    
reco:anglereconuepfps:NuAngularReco                   0.00013816    0.00242239     0.118596     0.000290426    0.0120289       100    
reco:anglereconumupfps:NuAngularReco                  0.000133892   0.00242205     0.122778     0.000262013    0.0124409       100    
reco:anglerecohits:NuAngularReco                      0.00036663    0.00305223     0.129386     0.000620057    0.0131396       100    
[art]:TriggerResults:TriggerResultInserter            1.0741e-05    1.28516e-05   6.0624e-05    1.1787e-05    5.0738e-06       100    
end_path:cafmaker:CAFMaker                            0.00105578     0.0324348      1.59018     0.00320134     0.161423        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3807.44 MB
  Peak resident set size usage (VmHWM): 2147.13 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1150!
Begin processing the 92nd record. run: 6409602 subRun: 1 event: 62592 at 21-Sep-2025 21:42:58 CEST
Boundary wire vector sizes: 213, 268, 204
minwire 0: 811
minwire 1: 2481
minwire 2: 220
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000368768, 0.998384, 0.00124764, 
Output 1: 0.00437135, 0.9914, 0.00406583, 0.000162626, 
Output 2: 0.998, 0.00194924, 4.93735e-05, 1.56039e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1722!
Begin processing the 93rd record. run: 6409602 subRun: 1 event: 62593 at 21-Sep-2025 21:43:02 CEST
Boundary wire vector sizes: 4187, 2984, 2430
minwire 0: 174
minwire 1: 919
minwire 2: 209
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.950065, 0.011581, 0.0383544, 
Output 1: 0.188545, 0.454847, 0.239602, 0.117005, 
Output 2: 0.0232632, 0.0863298, 0.283739, 0.606668, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3210!
Begin processing the 94th record. run: 6409602 subRun: 1 event: 62594 at 21-Sep-2025 21:43:10 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 635!
Begin processing the 95th record. run: 6409602 subRun: 1 event: 62595 at 21-Sep-2025 21:43:13 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 830!
Begin processing the 96th record. run: 6409602 subRun: 1 event: 62596 at 21-Sep-2025 21:43:15 CEST
Boundary wire vector sizes: 173, 381, 231
minwire 0: 1594
minwire 1: 569
minwire 2: 1342
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.927019, 0.000787991, 0.0721931, 
Output 1: 0.0313256, 0.962931, 0.00546426, 0.000279071, 
Output 2: 0.984962, 0.0141864, 0.000797029, 5.46882e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2363!
Begin processing the 97th record. run: 6409602 subRun: 1 event: 62597 at 21-Sep-2025 21:43:19 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1030!
Begin processing the 98th record. run: 6409602 subRun: 1 event: 62598 at 21-Sep-2025 21:43:21 CEST
Boundary wire vector sizes: 104, 171, 109
minwire 0: 2059
minwire 1: 360
minwire 2: 2151
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00354082, 0.597589, 0.39887, 
Output 1: 0.939542, 0.0595533, 0.000782438, 0.000122008, 
Output 2: 0.996081, 0.00377605, 0.000115231, 2.81602e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 914!
Begin processing the 99th record. run: 6409602 subRun: 1 event: 62599 at 21-Sep-2025 21:43:24 CEST
Boundary wire vector sizes: 104, 65, 88
minwire 0: 1710
minwire 1: 950
minwire 2: 1854
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.926225, 0.00128715, 0.0724882, 
Output 1: 0.997155, 0.00252399, 0.000158494, 0.000162752, 
Output 2: 0.884412, 0.113028, 0.00221025, 0.000349671, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1072!
Begin processing the 100th record. run: 6409602 subRun: 1 event: 62600 at 21-Sep-2025 21:43:27 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 141!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0f/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_625_20231202T150939Z_gen_g4_detsim_hitreco__20240508T024146Z_reco2.root
.:
total 5624
-rw-r--r--. 1 dune012 dune 2197604 Sep 21 21:43 flatcaf.root
-rw-r--r--. 1 dune012 dune 1702168 Sep 21 21:43 caf.root
-rw-r--r--. 1 dune012 dune 1702168 Sep 21 21:43 caf_fd_hd_atmo_2501_20250921T193411Z.root
-rw-r--r--. 1 dune012 dune   69720 Sep 21 21:43 caf_20250921T193411Z.log
-rw-r--r--. 1 dune012 dune   34462 Sep 21 21:43 jobscript.log
-rw-r--r--. 1 dune012 dune     186 Sep 21 21:43 caf_20250921T193411Z.file
-rw-r--r--. 1 dune012 dune     186 Sep 21 21:43 caf_20250921T193411Z.pfns
-rw-r--r--. 1 dune012 dune     186 Sep 21 21:34 file.list
-rw-r--r--. 1 dune012 dune     186 Sep 21 21:43 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     137 Sep 21 21:34 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     137 Sep 21 21:43 caf_20250921T193411Z.did
-rw-r--r--. 1 dune012 dune     137 Sep 21 21:34 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 21 21:34 debugprod.log