Jobsub ID 51897.1@dunegpschedd01.fnal.gov

Jobsub ID	51897.1@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-22 03:45:01
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-09-22 04:05:56
From worker node	Hostname	wn-sate-048.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-22 03:45:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_973_20231202T131420Z_gen_g4_detsim_hitreco__20240507T222328Z_reco2.root
Jobscript	Exit code	0
	Real time	20m (1221s)
	CPU time	19m (1188s = 97%)
	Max RSS bytes	2151059456 (2051 MiB)
Outputting started	2025-09-22 04:05:37
Output files
Finished	2025-09-22 04:05:56
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

==================================================
Full event                                              1.65293       11.3924       408.789       2.67569       52.712         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000849265   0.00163228    0.00656351    0.00155214    0.00065473       100    
reco:pandora:StandardPandora                            1.5174        10.1301       400.054       1.62551       51.8762        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000133852     0.01221      0.512959     0.00149626     0.0542816       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000187241    0.0108702      0.16658     0.00330329     0.0228496       100    
reco:pandoracalo:Calorimetry                          0.000111069    0.0491382      1.84784     0.00110398      0.22721        100    
reco:pandorapid:Chi2ParticleID                        4.0406e-05    0.000164369   0.00202605    0.000114496   0.000232024      100    
reco:cvnmap:CVNMapper                                 2.3705e-05     0.0100016     0.0417444     0.0110757    0.00692482       100    
reco:cvneva:CVNEvaluator                              1.8976e-05      0.49527       2.5863       0.605876      0.328325        100    
reco:energyrecnumu:EnergyReco                         0.000270388    0.0535236      2.51249     0.00606646     0.256741        100    
reco:energyrecnue:EnergyReco                          0.000184056   0.00437312     0.208737     0.000485616    0.0215776       100    
reco:energyrecnc:EnergyReco                           0.00267659     0.0180955     0.289789      0.0126859     0.0304759       100    
reco:energyrecnumurange:EnergyReco                    0.000289253    0.044707       2.24299     0.00287436     0.229088        100    
reco:energyrecnumumcs:EnergyReco                      0.000535034    0.0463614      2.25097     0.00336241     0.230111        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000419808    0.0482337      2.4364      0.00338833     0.247824        100    
reco:opdec:Deconvolution                               0.0196498     0.175276      0.336455       0.17133      0.0617541       100    
reco:ophitspe:OpHitFinderDeco                         0.00360747    0.00910919     0.0164907    0.00874095    0.00233206       100    
reco:opflash:OpFlashFinder                            0.000263745   0.00122437    0.00396524    0.00113323    0.00058915       100    
reco:opslicer:OpSlicer                                0.00124401     0.226935       1.07913      0.184257      0.182022        100    
reco:rns:RandomNumberSaver                            1.9987e-05    2.98506e-05   0.000331483   2.64895e-05   3.03834e-05      100    
reco:anglereconue:NuAngularReco                       0.000155021   0.00289039     0.138312     0.000343305    0.0143931       100    
reco:anglereconumu:NuAngularReco                      9.2593e-05    0.00281819     0.143337     0.000231279    0.0148723       100    
reco:anglereconuepfps:NuAngularReco                   0.000137858   0.00287154     0.137305     0.000317896    0.0142982       100    
reco:anglereconumupfps:NuAngularReco                  0.000135765   0.00284667     0.137444     0.000300624    0.0143361       100    
reco:anglerecohits:NuAngularReco                      0.000358303   0.00351422     0.145661     0.00065549     0.0152377       100    
[art]:TriggerResults:TriggerResultInserter            1.1151e-05    1.27893e-05   6.0955e-05    1.2083e-05    4.94908e-06      100    
end_path:cafmaker:CAFMaker                            0.00105869     0.0388293      1.68444     0.00358028     0.179841        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3811.24 MB
  Peak resident set size usage (VmHWM): 2151.06 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Found hits: 1626!
Begin processing the 95th record. run: 6406500 subRun: 1 event: 97395 at 22-Sep-2025 06:05:17 CEST
Boundary wire vector sizes: 386, 437, 253
minwire 0: 780
minwire 1: 1868
minwire 2: 684
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.015913, 0.97136, 0.0127273, 
Output 1: 0.0138608, 0.910796, 0.0721256, 0.00321732, 
Output 2: 0.128974, 0.835864, 0.0348387, 0.000323715, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1895!
Begin processing the 96th record. run: 6406500 subRun: 1 event: 97396 at 22-Sep-2025 06:05:20 CEST
Boundary wire vector sizes: 93, 131, 116
minwire 0: 988
minwire 1: 1148
minwire 2: 716
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.963538, 0.000707044, 0.035755, 
Output 1: 0.00234927, 0.995939, 0.00164764, 6.42869e-05, 
Output 2: 0.939412, 0.0592542, 0.00131369, 2.01598e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1979!
Begin processing the 97th record. run: 6406500 subRun: 1 event: 97397 at 22-Sep-2025 06:05:22 CEST
Boundary wire vector sizes: 76, 107, 76
minwire 0: 2239
minwire 1: 506
minwire 2: 2339
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.252503, 0.041666, 0.705831, 
Output 1: 0.0316262, 0.293239, 0.554232, 0.120903, 
Output 2: 0.372119, 0.550938, 0.0741982, 0.00274439, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1210!
Begin processing the 98th record. run: 6406500 subRun: 1 event: 97398 at 22-Sep-2025 06:05:25 CEST
Boundary wire vector sizes: 1915, 1522, 959
minwire 0: 998
minwire 1: 1952
minwire 2: 485
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0295903, 0.942438, 0.0279721, 
Output 1: 0.0682611, 0.406576, 0.339185, 0.185978, 
Output 2: 0.104261, 0.344821, 0.385247, 0.16567, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2951!
Begin processing the 99th record. run: 6406500 subRun: 1 event: 97399 at 22-Sep-2025 06:05:30 CEST
Boundary wire vector sizes: 47, 37, 36
minwire 0: 1905
minwire 1: 561
minwire 2: 1930
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 267, 566
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.134628, 0.0154062, 0.849966, 
Output 1: 0.25881, 0.735101, 0.00579744, 0.000290759, 
Output 2: 0.681846, 0.311516, 0.00652152, 0.000116097, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1035!
Begin processing the 100th record. run: 6406500 subRun: 1 event: 97400 at 22-Sep-2025 06:05:32 CEST
Boundary wire vector sizes: 189, 107, 94
minwire 0: 1203
minwire 1: 1793
minwire 2: 972
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.994178, 0.000458455, 0.00536315, 
Output 1: 0.97921, 0.020554, 0.000167042, 6.85623e-05, 
Output 2: 0.996894, 0.00302794, 6.55356e-05, 1.27194e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1758!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/5b/ae/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_973_20231202T131420Z_gen_g4_detsim_hitreco__20240507T222328Z_reco2.root
.:
total 6540
-rw-r--r--. 1 dune012 dune 2535891 Sep 22 06:05 flatcaf.root
-rw-r--r--. 1 dune012 dune 1999296 Sep 22 06:05 caf.root
-rw-r--r--. 1 dune012 dune 1999296 Sep 22 06:05 caf_fd_hd_atmo_2501_20250922T034524Z.root
-rw-r--r--. 1 dune012 dune   76468 Sep 22 06:05 caf_20250922T034524Z.log
-rw-r--r--. 1 dune012 dune   31481 Sep 22 06:05 jobscript.log
-rw-r--r--. 1 dune012 dune     216 Sep 22 06:05 caf_20250922T034524Z.file
-rw-r--r--. 1 dune012 dune     216 Sep 22 06:05 caf_20250922T034524Z.pfns
-rw-r--r--. 1 dune012 dune     216 Sep 22 05:45 file.list
-rw-r--r--. 1 dune012 dune     216 Sep 22 06:05 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     137 Sep 22 05:45 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     137 Sep 22 06:05 caf_20250922T034524Z.did
-rw-r--r--. 1 dune012 dune     137 Sep 22 05:45 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 22 05:45 debugprod.log