Jobsub ID 51624.0@dunegpschedd01.fnal.gov

Jobsub ID	51624.0@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-21 21:20:40
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-09-21 21:47:16
From worker node	Hostname	wn-sate-041.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-21 21:21:27
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_582_20231207T115103Z_gen_g4_detsim_hitreco__20240510T053512Z_reco2.root
Jobscript	Exit code	0
	Real time	25m (1530s)
	CPU time	23m (1427s = 93%)
	Max RSS bytes	2061864960 (1966 MiB)
Outputting started	2025-09-21 21:46:57
Output files
Finished	2025-09-21 21:47:16
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

=======================================================================================================================================
Full event                                              1.9691        14.2335       1006.14       3.14326       99.7673        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0153421     0.0429547      0.46914      0.0418953     0.044155        100    
reco:pandora:StandardPandora                            1.62599       12.579        988.321       1.73758       98.1234        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00013833     0.0106552     0.310334     0.00188284     0.0337234       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000192663    0.0123466     0.131215      0.0043417     0.0225578       100    
reco:pandoracalo:Calorimetry                          0.000113534    0.0393349      1.94063      0.0013283     0.200164        100    
reco:pandorapid:Chi2ParticleID                        4.1788e-05    0.000178308   0.00214883    0.000123823   0.000237674      100    
reco:cvnmap:CVNMapper                                 2.6711e-05    0.00954827     0.0438479     0.0110201    0.00750218       100    
reco:cvneva:CVNEvaluator                              2.0198e-05      0.45441       2.2409       0.604482      0.326532        100    
reco:energyrecnumu:EnergyReco                         0.000279677    0.096849       3.24348      0.0142018     0.335958        100    
reco:energyrecnue:EnergyReco                          0.000190128   0.00366672     0.112396     0.000694207    0.0119835       100    
reco:energyrecnc:EnergyReco                           0.00280268     0.0972013     0.394463      0.0985719     0.0682618       100    
reco:energyrecnumurange:EnergyReco                    0.000276671    0.0615112      2.62086     0.00297931      0.27628        100    
reco:energyrecnumumcs:EnergyReco                      0.000682925    0.0619503      2.63624     0.00342816     0.277115        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000510881    0.0612742      2.62602     0.00347949      0.27627        100    
reco:opdec:Deconvolution                               0.0661399     0.261484      0.531935      0.252427      0.0981483       100    
reco:ophitspe:OpHitFinderDeco                          0.131152      0.137247      0.179127      0.136418      0.0054212       100    
reco:opflash:OpFlashFinder                            0.000227648   0.00132141    0.00888178    0.00119018    0.000961165      100    
reco:opslicer:OpSlicer                                0.00061624     0.255296       1.0327       0.207422      0.204015        100    
reco:rns:RandomNumberSaver                            1.9587e-05    3.02703e-05   0.000374125   2.66905e-05   3.46305e-05      100    
reco:anglereconue:NuAngularReco                       0.000162216   0.00265258     0.0809957    0.00040629    0.00901683       100    
reco:anglereconumu:NuAngularReco                      9.4237e-05    0.00244893     0.0807969    0.000281014   0.00872727       100    
reco:anglereconuepfps:NuAngularReco                   0.000140886    0.0027443     0.080862     0.000372551   0.00913994       100    
reco:anglereconumupfps:NuAngularReco                  0.000134664   0.00254533     0.0808027    0.000346924   0.00883421       100    
reco:anglerecohits:NuAngularReco                      0.000397548   0.00357414     0.0879367    0.000770696    0.0103653       100    
[art]:TriggerResults:TriggerResultInserter             1.074e-05    1.28647e-05   7.3438e-05    1.1747e-05    6.30495e-06      100    
end_path:cafmaker:CAFMaker                             0.0011163     0.0318268     0.938299     0.00425329     0.104235        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3720.12 MB
  Peak resident set size usage (VmHWM): 2061.86 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.951705, 0.00100441, 0.0472905, 
Output 1: 0.489005, 0.497739, 0.0128665, 0.00038886, 
Output 2: 0.12667, 0.820398, 0.0522579, 0.000673479, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1262!
Begin processing the 94th record. run: 6455553 subRun: 1 event: 58294 at 21-Sep-2025 23:46:34 CEST
Boundary wire vector sizes: 81, 60, 69
minwire 0: 2265
minwire 1: 756
minwire 2: 2146
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.696993, 0.00777379, 0.295234, 
Output 1: 0.985027, 0.0144559, 0.000447817, 6.94902e-05, 
Output 2: 0.5542, 0.43747, 0.00806184, 0.000267921, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1037!
Begin processing the 95th record. run: 6455553 subRun: 1 event: 58295 at 21-Sep-2025 23:46:37 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 356!
Begin processing the 96th record. run: 6455553 subRun: 1 event: 58296 at 21-Sep-2025 23:46:39 CEST
Boundary wire vector sizes: 380, 329, 258
minwire 0: 2085
minwire 1: 1139
minwire 2: 1689
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00140605, 0.987122, 0.0114714, 
Output 1: 0.0383833, 0.876329, 0.0803461, 0.00494161, 
Output 2: 0.018683, 0.88712, 0.0920371, 0.00216005, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2531!
Begin processing the 97th record. run: 6455553 subRun: 1 event: 58297 at 21-Sep-2025 23:46:43 CEST
Boundary wire vector sizes: 275, 222, 177
minwire 0: 1550
minwire 1: 797
minwire 2: 1815
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.55907, 0.0234589, 0.417471, 
Output 1: 0.109933, 0.814609, 0.0732974, 0.00216061, 
Output 2: 0.026924, 0.775424, 0.19335, 0.00430229, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1616!
Begin processing the 98th record. run: 6455553 subRun: 1 event: 58298 at 21-Sep-2025 23:46:47 CEST
Boundary wire vector sizes: 94, 90, 82
minwire 0: 920
minwire 1: 1358
minwire 2: 907
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 258, 557
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.155411, 0.680683, 0.163906, 
Output 1: 0.963406, 0.0358064, 0.000699913, 8.76622e-05, 
Output 2: 0.939839, 0.0582156, 0.00177995, 0.00016573, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1272!
Begin processing the 99th record. run: 6455553 subRun: 1 event: 58299 at 21-Sep-2025 23:46:50 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 792!
Begin processing the 100th record. run: 6455553 subRun: 1 event: 58300 at 21-Sep-2025 23:46:52 CEST
Boundary wire vector sizes: 502, 536, 510
minwire 0: 931
minwire 1: 2440
minwire 2: 349
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000741585, 0.997545, 0.00171313, 
Output 1: 0.00648075, 0.0735142, 0.8126, 0.107405, 
Output 2: 0.973758, 0.0246904, 0.00150217, 4.88812e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2187!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/83/df/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_582_20231207T115103Z_gen_g4_detsim_hitreco__20240510T053512Z_reco2.root
.:
total 6644
-rw-r--r--. 1 dune012 dune 2572262 Sep 21 23:46 flatcaf.root
-rw-r--r--. 1 dune012 dune 2037928 Sep 21 23:46 caf.root
-rw-r--r--. 1 dune012 dune 2037928 Sep 21 23:46 caf_fd_hd_atmo_2501_20250921T212135Z.root
-rw-r--r--. 1 dune012 dune   72075 Sep 21 23:46 caf_20250921T212135Z.log
-rw-r--r--. 1 dune012 dune   34791 Sep 21 23:46 jobscript.log
-rw-r--r--. 1 dune012 dune     186 Sep 21 23:46 caf_20250921T212135Z.file
-rw-r--r--. 1 dune012 dune     186 Sep 21 23:46 caf_20250921T212135Z.pfns
-rw-r--r--. 1 dune012 dune     186 Sep 21 23:21 file.list
-rw-r--r--. 1 dune012 dune     186 Sep 21 23:46 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune     137 Sep 21 23:21 all-input-dids.txt
-rw-r--r--. 1 dune012 dune     137 Sep 21 23:46 caf_20250921T212135Z.did
-rw-r--r--. 1 dune012 dune     137 Sep 21 23:21 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 21 23:21 debugprod.log