Jobsub ID 45164.0@dunegpschedd02.fnal.gov

Jobsub ID	45164.0@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-21 18:34:31
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-09-21 18:44:02
From worker node	Hostname	wn-lot-018.farm.nikhef.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-21 18:36:19
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_787_20231201T170354Z_gen_g4_detsim_hitreco__20240507T201522Z_reco2.root
Jobscript	Exit code	0
	Real time	7m (444s)
	CPU time	6m (368s = 82%)
	Max RSS bytes	2152734720 (2053 MiB)
Outputting started	2025-09-21 18:43:44
Output files
Finished	2025-09-21 18:44:02
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

g           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.385645       3.58226       58.5719       2.71964       5.80417        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0150153     0.0319051     0.074828      0.0320163     0.0109348       100    
reco:pandora:StandardPandora                           0.0967116      2.34799       45.6063       1.67196       4.52758        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000125044   0.00545261     0.121914     0.00159444     0.0143766       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000175348    0.016665      0.235552      0.0034823     0.0401314       100    
reco:pandoracalo:Calorimetry                          9.8194e-05     0.0208867      1.26779     0.00121672     0.127927        100    
reco:pandorapid:Chi2ParticleID                        3.6548e-05    0.000140496   0.000546993   0.00012288    9.71104e-05      100    
reco:cvnmap:CVNMapper                                  2.148e-05    0.00950364     0.0407205     0.0114569    0.00639151       100    
reco:cvneva:CVNEvaluator                               1.613e-05     0.267066       1.96675      0.331126      0.223858        100    
reco:energyrecnumu:EnergyReco                         0.000239789    0.0669731      2.24638     0.00895555     0.239018        100    
reco:energyrecnue:EnergyReco                          0.000132538   0.00194136     0.0482434    0.000622395   0.00542611       100    
reco:energyrecnc:EnergyReco                           0.00199889     0.0868317     0.264584      0.0952458     0.0487254       100    
reco:energyrecnumurange:EnergyReco                    0.000175358    0.0442239      2.10204     0.00258196     0.224317        100    
reco:energyrecnumumcs:EnergyReco                      0.000417382    0.0447708      2.09592     0.00286091     0.223686        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000266048    0.0452108      2.15478     0.00278099     0.229014        100    
reco:opdec:Deconvolution                               0.0514041     0.214802      0.454575      0.215837      0.0741404       100    
reco:ophitspe:OpHitFinderDeco                          0.129308      0.133555      0.154099      0.133323     0.00310609       100    
reco:opflash:OpFlashFinder                            0.000162605   0.00109154    0.00282569    0.00108303    0.000464069      100    
reco:opslicer:OpSlicer                                 6.375e-05     0.216057      0.892891       0.18755      0.167963        100    
reco:rns:RandomNumberSaver                            1.8846e-05    2.83941e-05   0.000239338   2.6535e-05    2.1395e-05       100    
reco:anglereconue:NuAngularReco                       0.000152565   0.00136695     0.0370269    0.000369466   0.00416967       100    
reco:anglereconumu:NuAngularReco                      7.1403e-05    0.00120718     0.0365514    0.000209246   0.00415701       100    
reco:anglereconuepfps:NuAngularReco                   0.000118692   0.00133631     0.0372179    0.000337772   0.00418756       100    
reco:anglereconumupfps:NuAngularReco                  9.8684e-05    0.00124747     0.0346749    0.000274459   0.00399758       100    
reco:anglerecohits:NuAngularReco                      0.000125264   0.00182225     0.0399238    0.000687807   0.00453118       100    
[art]:TriggerResults:TriggerResultInserter             9.758e-06    1.23789e-05   4.0976e-05    1.18225e-05   3.1713e-06       100    
end_path:cafmaker:CAFMaker                            0.00165003     0.0191102     0.504276     0.00453145     0.0573449       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3815.29 MB
  Peak resident set size usage (VmHWM): 2152.73 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 185, 273, 220
minwire 0: 2555
minwire 1: 798
minwire 2: 2285
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00724914, 0.0342577, 0.958493, 
Output 1: 0.895651, 0.0925846, 0.00788535, 0.00387918, 
Output 2: 0.981217, 0.0179203, 0.000721836, 0.000141128, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1691!
Begin processing the 95th record. run: 74494003 subRun: 1 event: 78795 at 21-Sep-2025 20:43:23 CEST
Boundary wire vector sizes: 572, 629, 426
minwire 0: 740
minwire 1: 2561
minwire 2: 65
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.016215, 0.857521, 0.126264, 
Output 1: 0.0421309, 0.726092, 0.222143, 0.00963448, 
Output 2: 0.232489, 0.668817, 0.0955306, 0.00316315, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2083!
Begin processing the 96th record. run: 74494003 subRun: 1 event: 78796 at 21-Sep-2025 20:43:26 CEST
Boundary wire vector sizes: 1779, 754, 1472
minwire 0: 1197
minwire 1: 538
minwire 2: 1405
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996619, 0.000572959, 0.00280851, 
Output 1: 0.882447, 0.116402, 0.000888232, 0.000262611, 
Output 2: 0.996215, 0.00365531, 0.000111952, 1.80703e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2283!
Begin processing the 97th record. run: 74494003 subRun: 1 event: 78797 at 21-Sep-2025 20:43:30 CEST
Boundary wire vector sizes: 566, 645, 498
minwire 0: 2183
minwire 1: 248
minwire 2: 2015
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00897599, 0.117998, 0.873026, 
Output 1: 0.708591, 0.277884, 0.0118485, 0.00167725, 
Output 2: 0.176746, 0.598804, 0.195298, 0.0291521, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2123!
Begin processing the 98th record. run: 74494003 subRun: 1 event: 78798 at 21-Sep-2025 20:43:34 CEST
Boundary wire vector sizes: 1104, 1140, 907
minwire 0: 149
minwire 1: 1779
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.113989, 0.0199056, 0.866106, 
Output 1: 0.345705, 0.478636, 0.115412, 0.0602481, 
Output 2: 0.040755, 0.128258, 0.388376, 0.442611, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2517!
Begin processing the 99th record. run: 74494003 subRun: 1 event: 78799 at 21-Sep-2025 20:43:38 CEST
Boundary wire vector sizes: 115, 192, 165
minwire 0: 753
minwire 1: 1655
minwire 2: 738
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000371861, 0.996727, 0.0029006, 
Output 1: 0.00620772, 0.98089, 0.01253, 0.000372554, 
Output 2: 0.997933, 0.00200788, 5.76643e-05, 1.83565e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1166!
Begin processing the 100th record. run: 74494003 subRun: 1 event: 78800 at 21-Sep-2025 20:43:41 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 919!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ad/94/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_787_20231201T170354Z_gen_g4_detsim_hitreco__20240507T201522Z_reco2.root
.:
total 5408
-rw-r--r--. 1 dune012 dune 2121542 Sep 21 20:43 flatcaf.root
-rw-r--r--. 1 dune012 dune 1628143 Sep 21 20:43 caf.root
-rw-r--r--. 1 dune012 dune 1628143 Sep 21 20:43 caf_fd_hd_atmo_2501_20250921T183628Z.root
-rw-r--r--. 1 dune012 dune   75775 Sep 21 20:43 caf_20250921T183628Z.log
-rw-r--r--. 1 dune012 dune   34833 Sep 21 20:43 jobscript.log
-rw-r--r--. 1 dune012 dune     187 Sep 21 20:43 caf_20250921T183628Z.file
-rw-r--r--. 1 dune012 dune     187 Sep 21 20:43 caf_20250921T183628Z.pfns
-rw-r--r--. 1 dune012 dune     187 Sep 21 20:36 file.list
-rw-r--r--. 1 dune012 dune     187 Sep 21 20:43 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     138 Sep 21 20:36 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     138 Sep 21 20:43 caf_20250921T183628Z.did
-rw-r--r--. 1 dune012 dune     138 Sep 21 20:36 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 21 20:36 debugprod.log