justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258147.4@dunegpschedd02.fnal.gov

Jobsub ID258147.4@dunegpschedd02.fnal.gov
Workflow ID11028
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 20:20:06
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 07:49:40
From worker nodeHostnamedunegli-7832021-0-fnpc17112.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-06 01:34:08
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T224409Z_gen_000539_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time6h (22522s)
CPU time6h (22334s = 99%)
Max RSS bytes3057319936 (2915 MiB)
Outputting started2025-12-06 07:49:30
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11028/1/001/analysisLowE_20251206T013409Z.root
Finished2025-12-06 07:49:40
Saved logsjustin-logs:258147.4-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nd->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6954, 6908, 6813
minwire 0: 18
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 903, 1402
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.133519, 
Output 1: 0.0472932, 0.145272, 0.127553, 0.679882, 
Output 2: 0.0398962, 0.310837, 0.622934, 0.026333, 
Output 3: 0.952045, 0.0308752, 0.00625886, 0.0108215, 
Output 4: 0.367785, 0.533296, 0.0919858, 0.00693266, 
Output 5: 0.451804, 0.448756, 0.0885932, 0.0108466, 
Output 6: 0.0182505, 0.0470723, 0.0687967, 0.86588, 

06-Dec-2025 07:48:43 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 07:48:43 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/88/84/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T224409Z_gen_000539_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1958.53       2228.42       2660.98       2149.7        206.976        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00179932    0.00997419     0.0319264    0.00770883    0.00767303       10     
reco:pandora:StandardPandora                            1956.81       2225.84       2653.33       2147.66       205.779        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00315757     0.0339175     0.169231      0.0123365     0.047003        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00235036     0.0113405     0.0384759    0.00553355     0.0123793       10     
reco:pandoracalo:Calorimetry                          0.000461445   0.00792363     0.0358197    0.000535844    0.0131982       10     
reco:pandorapid:Chi2ParticleID                        0.000103475   0.000414215   0.00272417    0.000150741   0.000772039      10     
reco:cvnmap:CVNMapper                                  0.0484678     0.0767353     0.194914      0.0655162     0.0408391       10     
reco:cvneva:CVNEvaluator                               0.909268       1.71367       6.22503       1.31763       1.51995        10     
reco:energyrecnumu:EnergyReco                         0.00647224     0.0913535     0.182154      0.110596      0.0573566       10     
reco:energyrecnue:EnergyReco                          0.00139896    0.00414335      0.01248     0.00221056    0.00386993       10     
reco:energyrecnc:EnergyReco                            0.0101226     0.0425015     0.134177      0.0143421     0.0467758       10     
[art]:TriggerResults:TriggerResultInserter            2.0001e-05    3.42683e-05   0.000105853   2.35655e-05   2.50587e-05      10     
end_path:out1:RootOutput                               7.553e-06    1.23583e-05   3.1354e-05    8.7765e-06    7.31611e-06      10     
end_path:out1:RootOutput(write)                        0.390394      0.587528      0.805107      0.591781      0.147009        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14272.7 MB
  Peak resident set size usage (VmHWM): 3057.32 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 159.684 | hit correction : 1.06233
hit initial : 36.5312 | hit correction : 1.0628
hit initial : 55.0125 | hit correction : 1.06293
hit initial : 43.5757 | hit correction : 1.06342
hit initial : 125.91 | hit correction : 1.0635
hit initial : 286.095 | hit correction : 1.06401
hit initial : 337.521 | hit correction : 1.0645
hit initial : 233.655 | hit correction : 1.06495
hit initial : 69.1412 | hit correction : 1.06509
 *** Total Charge : 2493.21
 Looking at view: 0 | Number of hits in view: 22
hit initial : 34.8159 | hit correction : 1.06514
hit initial : 143.746 | hit correction : 1.06054
hit initial : 122.09 | hit correction : 1.06054
hit initial : 30.6044 | hit correction : 1.06065
hit initial : 112.002 | hit correction : 1.06094
hit initial : 96.2524 | hit correction : 1.06095
hit initial : 72.0671 | hit correction : 1.06105
hit initial : 101.799 | hit correction : 1.06174
hit initial : 56.6376 | hit correction : 1.06186
hit initial : 451.222 | hit correction : 1.06258
hit initial : 392.685 | hit correction : 1.06344
hit initial : 325.888 | hit correction : 1.06409
hit initial : 334.16 | hit correction : 1.06462
hit initial : 193.798 | hit correction : 1.06497
 *** Total Charge : 2623.09
 Looking at view: 1 | Number of hits in view: 14
hit initial : 25.582 | hit correction : 1.06004
hit initial : 111.055 | hit correction : 1.06056
hit initial : 214.365 | hit correction : 1.06054
hit initial : 360.818 | hit correction : 1.06097
hit initial : 66.3806 | hit correction : 1.06105
hit initial : 337.724 | hit correction : 1.06171
hit initial : 171.465 | hit correction : 1.06217
hit initial : 325.002 | hit correction : 1.06264
hit initial : 127.788 | hit correction : 1.06321
hit initial : 760.765 | hit correction : 1.06391
hit initial : 91.7027 | hit correction : 1.06445
hit initial : 305.038 | hit correction : 1.06499
hit initial : 21.3645 | hit correction : 1.0653
 *** Total Charge : 3102.02
 Looking at view: 2 | Number of hits in view: 13
Begin processing the 9th record. run: 5027 subRun: 0 event: 5389 at 06-Dec-2025 07:49:24 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  19
Added hit list length: 19 | Max was set to: -3.40282e+38
True Electron Energy: 15.5315
Completed MCParticle Loop
True Neutrino Energy: 19.2616
 Size of all hits: 19
hit initial : 55.2135 | hit correction : 1.35281
hit initial : 203.861 | hit correction : 1.35301
hit initial : 69.5264 | hit correction : 1.35195
hit initial : 256.887 | hit correction : 1.35199
hit initial : 334.112 | hit correction : 1.35215
hit initial : 231.049 | hit correction : 1.35273
hit initial : 168.76 | hit correction : 1.35292
hit initial : 189.736 | hit correction : 1.35298
 *** Total Charge : 2041.17
 Looking at view: 0 | Number of hits in view: 8
hit initial : 133.904 | hit correction : 1.35304
hit initial : 111.679 | hit correction : 1.35305
hit initial : 141.874 | hit correction : 1.35298
hit initial : 178.543 | hit correction : 1.35194
hit initial : 181.971 | hit correction : 1.3518
hit initial : 116.35 | hit correction : 1.35216
hit initial : 118.255 | hit correction : 1.35236
hit initial : 138.05 | hit correction : 1.35262
hit initial : 151.915 | hit correction : 1.35288
hit initial : 71.4051 | hit correction : 1.35273
hit initial : 136.096 | hit correction : 1.35288
 *** Total Charge : 2001.82
 Looking at view: 2 | Number of hits in view: 11
Begin processing the 10th record. run: 5027 subRun: 0 event: 5390 at 06-Dec-2025 07:49:27 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.567
True Electron Energy: 2.62985
Completed MCParticle Loop
True Neutrino Energy: 8.00788
 Size of all hits: 0
06-Dec-2025 07:49:30 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0805453     0.0942143     0.112443      0.0948286     0.0107233       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000350481   0.00050872    0.000955196   0.00044595    0.000169659      10     
end_path:ana:AnalyzeEvents         0.079942      0.0935321     0.111967      0.0941745     0.0106066       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11962.8 MB
  Peak resident set size usage (VmHWM): 1540.88 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:08:57 UTC       justIN version: 01.06.00