justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258266.1@dunegpschedd02.fnal.gov

Jobsub ID258266.1@dunegpschedd02.fnal.gov
Workflow ID11026
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 23:00:12
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:31:19
From worker nodeHostnamedunegli-7989276-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:28
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T192642Z_gen_002291_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (31533s)
CPU time8h (30803s = 97%)
Max RSS bytes3035467776 (2894 MiB)
Outputting started2025-12-06 10:24:02
Output files
Finished2025-12-06 10:31:19
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0936799, 
Output 1: 0.0631515, 0.0216328, 0.0649475, 0.850268, 
Output 2: 0.115675, 0.308945, 0.538539, 0.0368413, 
Output 3: 0.974445, 0.0190954, 0.00163474, 0.00482521, 
Output 4: 0.297953, 0.64197, 0.0564755, 0.00360093, 
Output 5: 0.749899, 0.229661, 0.0171884, 0.0032522, 
Output 6: 0.0138922, 0.0503471, 0.0686319, 0.867129, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 22910 at 06-Dec-2025 09:33:19 UTC
Boundary wire vector sizes: 6835, 6822, 6771
minwire 0: 32
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.12174, 
Output 1: 0.0861597, 0.195969, 0.18955, 0.528322, 
Output 2: 0.0950951, 0.175016, 0.708045, 0.0218434, 
Output 3: 0.959429, 0.0370035, 0.0016051, 0.0019628, 
Output 4: 0.409475, 0.522156, 0.0605066, 0.00786295, 
Output 5: 0.435723, 0.452602, 0.0873222, 0.0243528, 
Output 6: 0.126349, 0.232843, 0.142649, 0.498159, 

06-Dec-2025 10:22:22 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:22:22 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/a4/15/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T192642Z_gen_002291_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2933.38       3119.48       3417.11       3077.67       171.398        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00425175     0.0168546     0.0416037     0.013219      0.0104518       10     
reco:pandora:StandardPandora                            2926.9        3113.33       3405.31       3071.73       169.934        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00468546     0.0367008     0.226297      0.0120601     0.0642461       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00272867     0.0168538     0.0755391     0.0048602     0.024614        10     
reco:pandoracalo:Calorimetry                          0.000495935    0.0304656     0.123231     0.00823575     0.0385842       10     
reco:pandorapid:Chi2ParticleID                        0.000142188   0.000879371   0.00433853    0.00022005    0.00133705       10     
reco:cvnmap:CVNMapper                                  0.0558565     0.104734      0.210611      0.0796391     0.0502395       10     
reco:cvneva:CVNEvaluator                                3.07836       4.94227       10.026        4.3206        1.86593        10     
reco:energyrecnumu:EnergyReco                          0.021499       0.12892      0.312918      0.139722      0.0850083       10     
reco:energyrecnue:EnergyReco                          0.00215542     0.0122986     0.0295196     0.0152755    0.00868337       10     
reco:energyrecnc:EnergyReco                            0.0150078     0.0496222      0.12772      0.0179042     0.0475919       10     
[art]:TriggerResults:TriggerResultInserter            3.5106e-05    6.07656e-05   0.000206019   3.79065e-05   5.13632e-05      10     
end_path:out1:RootOutput                               8.426e-06    1.8556e-05    6.0955e-05    1.3991e-05    1.43682e-05      10     
end_path:out1:RootOutput(write)                        0.592517      0.811521       1.77465      0.671247      0.350426        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21743.9 MB
  Peak resident set size usage (VmHWM): 3035.47 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Begin processing the 7th record. run: 5026 subRun: 0 event: 22907 at 06-Dec-2025 10:23:38 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Electron Energy: 9.39029
Completed MCParticle Loop
True Neutrino Energy: 14.769
 Size of all hits: 0
Begin processing the 8th record. run: 5026 subRun: 0 event: 22908 at 06-Dec-2025 10:23:44 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 14.0771
Completed MCParticle Loop
True Neutrino Energy: 17.8033
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 22909 at 06-Dec-2025 10:23:49 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 6.16903
Completed MCParticle Loop
True Neutrino Energy: 9.4542
 Size of all hits: 0
Begin processing the 10th record. run: 5026 subRun: 0 event: 22910 at 06-Dec-2025 10:23:55 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  47
Added hit list length: 47 | Max was set to: -3.40282e+38
True Neutron Energy: 939.579
True Electron Energy: 49.8456
Completed MCParticle Loop
True Neutrino Energy: 57.7223
 Size of all hits: 47
hit initial : 450.018 | hit correction : 1.18145
hit initial : 517.056 | hit correction : 1.18126
hit initial : 151.657 | hit correction : 1.18118
hit initial : 222.563 | hit correction : 1.18204
hit initial : 576.75 | hit correction : 1.18199
hit initial : 280.903 | hit correction : 1.18193
hit initial : 216.813 | hit correction : 1.18168
hit initial : 295.083 | hit correction : 1.18161
hit initial : 274.715 | hit correction : 1.18133
hit initial : 313.315 | hit correction : 1.18098
hit initial : 412.706 | hit correction : 1.18074
hit initial : 503.069 | hit correction : 1.18049
hit initial : 398.943 | hit correction : 1.18035
hit initial : 119.512 | hit correction : 1.18016
hit initial : 360.871 | hit correction : 1.18022
hit initial : 347.008 | hit correction : 1.18017
 *** Total Charge : 6426.34
 Looking at view: 1 | Number of hits in view: 16
hit initial : 20.6885 | hit correction : 1.1801
hit initial : 190.07 | hit correction : 1.18167
hit initial : 93.2882 | hit correction : 1.18181
hit initial : 76.3336 | hit correction : 1.18102
hit initial : 382.172 | hit correction : 1.1811
hit initial : 377.883 | hit correction : 1.18139
hit initial : 50.6308 | hit correction : 1.18211
hit initial : 203.021 | hit correction : 1.1814
hit initial : 395.905 | hit correction : 1.18199
hit initial : 284.169 | hit correction : 1.18202
hit initial : 218.566 | hit correction : 1.18202
hit initial : 101.919 | hit correction : 1.182
hit initial : 57.9766 | hit correction : 1.18167
hit initial : 311.176 | hit correction : 1.18164
hit initial : 175.986 | hit correction : 1.18138
hit initial : 165.146 | hit correction : 1.18115
hit initial : 128.497 | hit correction : 1.181
hit initial : 117.215 | hit correction : 1.1809
hit initial : 156.043 | hit correction : 1.18083
hit initial : 144.75 | hit correction : 1.18077
hit initial : 311.25 | hit correction : 1.1805
hit initial : 56.4523 | hit correction : 1.18181
hit initial : 183.962 | hit correction : 1.18053
hit initial : 227.554 | hit correction : 1.18043
hit initial : 168.1 | hit correction : 1.18036
hit initial : 123.406 | hit correction : 1.18026
hit initial : 118.152 | hit correction : 1.18021
hit initial : 158.76 | hit correction : 1.18019
hit initial : 170.244 | hit correction : 1.18017
hit initial : 252.372 | hit correction : 1.18019
hit initial : 118.018 | hit correction : 1.18011
 *** Total Charge : 6542.97
 Looking at view: 2 | Number of hits in view: 31
06-Dec-2025 10:24:01 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.137132      0.154286      0.200697      0.145251      0.0196697       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000560347   0.000886761   0.00188673    0.000789664   0.000359659      10     
end_path:ana:AnalyzeEvents         0.136258      0.153066      0.199772       0.14323      0.0197531       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19479.5 MB
  Peak resident set size usage (VmHWM): 1540.19 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 15:13:31 UTC       justIN version: 01.05.03