Jobsub ID 51666.1@dunegpschedd01.fnal.gov

Jobsub ID	51666.1@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-21 23:08:46
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-09-21 23:15:58
From worker node	Hostname	wn-lot-027.farm.nikhef.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-21 23:09:14
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_1056_20231204T010301Z_gen_g4_detsim_hitreco__20240509T214117Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (386s)
	CPU time	6m (371s = 96%)
	Max RSS bytes	1989963776 (1897 MiB)
Outputting started	2025-09-21 23:15:41
Output files
Finished	2025-09-21 23:15:58
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

nEvts   
========================================================================================================================================
Full event                                             0.0460084      3.20659       42.0603       2.07132       5.67495        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00111011     0.0119151     0.0334113     0.0146231    0.00793737       100    
reco:pandora:StandardPandora                          0.00882021      2.24648       38.9942       1.28509       4.97336        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000120296   0.00752065     0.249206     0.000958995    0.0277171       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000169046    0.0183829      0.72514     0.00228955     0.078117        100    
reco:pandoracalo:Calorimetry                          9.8495e-05     0.0154914     0.587663     0.000732957    0.0679149       100    
reco:pandorapid:Chi2ParticleID                        3.6338e-05    0.000143801   0.00111668     9.629e-05    0.000166898      100    
reco:cvnmap:CVNMapper                                 1.8305e-05    0.00781965     0.037644     0.00879091    0.00673754       100    
reco:cvneva:CVNEvaluator                              1.7122e-05     0.289573       2.09881      0.382024      0.255446        100    
reco:energyrecnumu:EnergyReco                         0.00024574     0.051776       1.91999      0.0107221     0.213554        100    
reco:energyrecnue:EnergyReco                          0.000146093   0.00278623     0.0919968    0.000442724    0.0102019       100    
reco:energyrecnc:EnergyReco                           0.00231383     0.0140665     0.147245      0.0118657     0.0175997       100    
reco:energyrecnumurange:EnergyReco                    0.000199473    0.0436638      1.77991     0.00243049     0.197954        100    
reco:energyrecnumumcs:EnergyReco                      0.000455573    0.0445832      1.80048     0.00273425     0.199694        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000343032    0.0444232      1.7906      0.00271657     0.198758        100    
reco:opdec:Deconvolution                              0.00593303      0.15399       0.30666      0.154491      0.0663498       100    
reco:ophitspe:OpHitFinderDeco                         0.00240058    0.00800107     0.0142477    0.00795927    0.00224126       100    
reco:opflash:OpFlashFinder                            9.9666e-05     0.0010764    0.00365515    0.00103473    0.000572114      100    
reco:opslicer:OpSlicer                                3.4444e-05     0.208435      0.722382      0.149867      0.183005        100    
reco:rns:RandomNumberSaver                            1.6771e-05    2.89284e-05   0.000301134   2.5313e-05    3.02272e-05      100    
reco:anglereconue:NuAngularReco                       0.000142046   0.00202587     0.0692311    0.000313862   0.00777135       100    
reco:anglereconumu:NuAngularReco                      7.5682e-05     0.0019344     0.0717442    0.000186364   0.00804013       100    
reco:anglereconuepfps:NuAngularReco                   0.000117871   0.00203757     0.0695274    0.000274884   0.00785387       100    
reco:anglereconumupfps:NuAngularReco                  0.000110046   0.00194618     0.0702906    0.000245185   0.00786064       100    
reco:anglerecohits:NuAngularReco                      8.5711e-05    0.00252147     0.0752922    0.000578799    0.0084619       100    
[art]:TriggerResults:TriggerResultInserter             9.678e-06    1.16345e-05   4.6136e-05     1.095e-05    3.67971e-06      100    
end_path:cafmaker:CAFMaker                            0.00123174     0.0248452     0.827297     0.00343567     0.0925294       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3653.4 MB
  Peak resident set size usage (VmHWM): 1989.96 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Output 1: 0.0458391, 0.660295, 0.286487, 0.00737899, 
Output 2: 0.855671, 0.1392, 0.0050406, 8.88824e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2259!
Begin processing the 93rd record. run: 6419766 subRun: 1 event: 105693 at 22-Sep-2025 01:15:24 CEST
Boundary wire vector sizes: 52, 99, 75
minwire 0: 252
minwire 1: 2281
minwire 2: 30
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000989518, 0.987853, 0.0111576, 
Output 1: 0.923811, 0.0739596, 0.00200094, 0.000228169, 
Output 2: 0.998152, 0.00180102, 4.22123e-05, 4.40017e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 611!
Begin processing the 94th record. run: 6419766 subRun: 1 event: 105694 at 22-Sep-2025 01:15:26 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 652!
Begin processing the 95th record. run: 6419766 subRun: 1 event: 105695 at 22-Sep-2025 01:15:28 CEST
Boundary wire vector sizes: 541, 584, 447
minwire 0: 1872
minwire 1: 325
minwire 2: 2030
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992683, 0.00113505, 0.00618217, 
Output 1: 0.491531, 0.374519, 0.0908321, 0.0431173, 
Output 2: 0.655627, 0.291674, 0.0455875, 0.00711124, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1484!
Begin processing the 96th record. run: 6419766 subRun: 1 event: 105696 at 22-Sep-2025 01:15:30 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 849!
Begin processing the 97th record. run: 6419766 subRun: 1 event: 105697 at 22-Sep-2025 01:15:32 CEST
Boundary wire vector sizes: 317, 254, 217
minwire 0: 982
minwire 1: 1106
minwire 2: 863
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0209078, 0.960777, 0.0183149, 
Output 1: 0.0355848, 0.721705, 0.222738, 0.0199724, 
Output 2: 0.0512719, 0.913614, 0.0348514, 0.000262496, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1859!
Begin processing the 98th record. run: 6419766 subRun: 1 event: 105698 at 22-Sep-2025 01:15:34 CEST
Boundary wire vector sizes: 106, 28, 78
minwire 0: 290
minwire 1: 2249
minwire 2: 241
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996768, 0.00109859, 0.00213297, 
Output 1: 0.00501859, 0.991489, 0.00338625, 0.000106151, 
Output 2: 0.997909, 0.00203061, 5.83779e-05, 1.63382e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 790!
Begin processing the 99th record. run: 6419766 subRun: 1 event: 105699 at 22-Sep-2025 01:15:36 CEST
Boundary wire vector sizes: 292, 345, 287
minwire 0: 1980
minwire 1: 1143
minwire 2: 1669
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.991442, 0.000389653, 0.00816793, 
Output 1: 0.996545, 0.00317645, 0.000163588, 0.000114549, 
Output 2: 0.989877, 0.00987023, 0.000199605, 5.31608e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1872!
Begin processing the 100th record. run: 6419766 subRun: 1 event: 105700 at 22-Sep-2025 01:15:38 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 710!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ba/13/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_1056_20231204T010301Z_gen_g4_detsim_hitreco__20240509T214117Z_reco2.root
.:
total 4532
-rw-r--r--. 1 dune012 dune 1784921 Sep 22 01:15 flatcaf.root
-rw-r--r--. 1 dune012 dune 1353583 Sep 22 01:15 caf.root
-rw-r--r--. 1 dune012 dune 1353583 Sep 22 01:15 caf_fd_hd_atmo_2501_20250921T230922Z.root
-rw-r--r--. 1 dune012 dune   73107 Sep 22 01:15 caf_20250921T230922Z.log
-rw-r--r--. 1 dune012 dune   28661 Sep 22 01:15 jobscript.log
-rw-r--r--. 1 dune012 dune     217 Sep 22 01:15 caf_20250921T230922Z.file
-rw-r--r--. 1 dune012 dune     217 Sep 22 01:15 caf_20250921T230922Z.pfns
-rw-r--r--. 1 dune012 dune     217 Sep 22 01:09 file.list
-rw-r--r--. 1 dune012 dune     217 Sep 22 01:15 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     138 Sep 22 01:09 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     138 Sep 22 01:15 caf_20250921T230922Z.did
-rw-r--r--. 1 dune012 dune     138 Sep 22 01:09 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 22 01:09 debugprod.log