justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 223867.3@dunegpschedd02.fnal.gov

Jobsub ID223867.3@dunegpschedd02.fnal.gov
Workflow ID8430
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-03 20:46:36
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-10-03 22:31:10
From worker nodeHostnamedunegli-46729-0-cmswn4001.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-03 20:47:48
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T083206Z_gen_003403_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time1h (6185s)
CPU time1h (6091s = 98%)
Max RSS bytes3022049280 (2882 MiB)
Outputting started2025-10-03 22:30:53
Output files
Finished2025-10-03 22:31:10
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/5e/27/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T083206Z_gen_003403_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_92_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c
MRB_SOURCE=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/srcs
MRB_BUILDDIR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

PRODUCTS=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof

local product directory is /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b0d6c9a892b29184076e7176977fea37c4d4181c/v09_92_00.prof/localProducts_larsoft_v09_92_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6894, 6777, 6765
minwire 0: 62
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.150504, 
Output 1: 0.0721292, 0.0957284, 0.14099, 0.691153, 
Output 2: 0.0494354, 0.215936, 0.705649, 0.0289805, 
Output 3: 0.964579, 0.0191028, 0.00438675, 0.0119312, 
Output 4: 0.445026, 0.472813, 0.0729886, 0.00917253, 
Output 5: 0.310317, 0.510105, 0.152715, 0.0268633, 
Output 6: 0.0374376, 0.0821261, 0.0919514, 0.788485, 

Begin processing the 8th record. run: 4933 subRun: 0 event: 34028 at 03-Oct-2025 22:00:32 UTC
Boundary wire vector sizes: 6569, 6682, 6556
minwire 0: 85
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2277, 2776
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0543646, 
Output 1: 0.0495239, 0.0836865, 0.108891, 0.757899, 
Output 2: 0.0625942, 0.193113, 0.707686, 0.0366063, 
Output 3: 0.904445, 0.070425, 0.0133157, 0.0118141, 
Output 4: 0.901267, 0.07927, 0.0142319, 0.00523085, 
Output 5: 0.663546, 0.245167, 0.0705503, 0.020737, 
Output 6: 0.0224903, 0.0705356, 0.0722955, 0.834679, 

Begin processing the 9th record. run: 4933 subRun: 0 event: 34029 at 03-Oct-2025 22:10:34 UTC
Boundary wire vector sizes: 6734, 6717, 6651
minwire 0: 35
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2333, 2832
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0989814, 
Output 1: 0.0518127, 0.139502, 0.136555, 0.67213, 
Output 2: 0.0388684, 0.210655, 0.73249, 0.0179867, 
Output 3: 0.941906, 0.0437138, 0.00596588, 0.00841453, 
Output 4: 0.417615, 0.489325, 0.0834987, 0.00956137, 
Output 5: 0.395142, 0.47473, 0.112213, 0.0179148, 
Output 6: 0.0317429, 0.0875214, 0.0959461, 0.78479, 

Begin processing the 10th record. run: 4933 subRun: 0 event: 34030 at 03-Oct-2025 22:20:41 UTC
Boundary wire vector sizes: 6780, 6740, 6751
minwire 0: 46
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.156757, 
Output 1: 0.0744574, 0.0907352, 0.177465, 0.657342, 
Output 2: 0.0532328, 0.232386, 0.683934, 0.0304469, 
Output 3: 0.956193, 0.032431, 0.00511548, 0.00626054, 
Output 4: 0.371649, 0.526055, 0.0930858, 0.00921009, 
Output 5: 0.417286, 0.445031, 0.117902, 0.0197808, 
Output 6: 0.0371321, 0.0849139, 0.092149, 0.785805, 

03-Oct-2025 22:30:53 UTC  Closed output file "mydetsimreco1reco2_20251003T204754Z.root"
03-Oct-2025 22:30:53 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/5e/27/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T083206Z_gen_003403_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              599.777       608.86        617.735       608.246       6.30607        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00121371    0.00571904     0.0196177    0.00446398    0.00504324       10     
reco:pandora:StandardPandora                            598.834       607.749       616.045       607.288       6.09142        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00186639     0.0136276     0.0524027     0.011792      0.013937        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000616839   0.00386803     0.0130255    0.00341803    0.00343914       10     
reco:pandoracalo:Calorimetry                          0.00017972    0.00175134     0.0120269    0.000322611   0.00348024       10     
reco:pandorapid:Chi2ParticleID                        4.1539e-05    0.00621771     0.0609442    6.83995e-05    0.0182428       10     
reco:cvnmap:CVNMapper                                  0.0144899     0.0193591     0.0325673     0.018712     0.00511768       10     
reco:cvneva:CVNEvaluator                               0.662964      0.794883       1.90796      0.668507      0.371143        10     
reco:energyrecnumu:EnergyReco                         0.00325578     0.0408143     0.115958      0.0525396     0.0349002       10     
reco:energyrecnue:EnergyReco                          0.000736336   0.00197949    0.00472851    0.000848148   0.00173875       10     
reco:energyrecnc:EnergyReco                           0.00450719     0.0298596     0.0611585     0.0260026     0.0255795       10     
[art]:TriggerResults:TriggerResultInserter             8.166e-06    1.37382e-05   4.8512e-05    1.0014e-05    1.16723e-05      10     
end_path:out1:RootOutput                               5.611e-06    8.1234e-06    2.2022e-05     6.903e-06    4.66378e-06      10     
end_path:out1:RootOutput(write)                        0.187057      0.192714        0.232       0.188684      0.0131357       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21753 MB
  Peak resident set size usage (VmHWM): 3022.05 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-11-03 18:41:13 UTC       justIN version: 01.05.01