justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257906.139@dunegpschedd02.fnal.gov

Jobsub ID257906.139@dunegpschedd02.fnal.gov
Workflow ID11024
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 11:19:42
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 09:30:05
From worker nodeHostnamedunegli-7832317-0-fnpc18010.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-05 22:34:38
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T165524Z_gen_000728_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time10h (38906s)
CPU time10h (38159s = 98%)
Max RSS bytes3031031808 (2890 MiB)
Outputting started2025-12-06 09:23:05
Output files
Finished2025-12-06 09:30:05
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6746, 6681, 6742
minwire 0: 22
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 865, 1364
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0725552, 
Output 1: 0.0352658, 0.032995, 0.0930108, 0.838728, 
Output 2: 0.10608, 0.273485, 0.575569, 0.0448662, 
Output 3: 0.939232, 0.0476913, 0.00577796, 0.00729893, 
Output 4: 0.31327, 0.618133, 0.0643136, 0.00428393, 
Output 5: 0.802059, 0.17487, 0.0188108, 0.0042604, 
Output 6: 0.0212758, 0.06291, 0.0713188, 0.844495, 

06-Dec-2025 09:21:40 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 09:21:40 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/03/1e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T165524Z_gen_000728_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2851.47       3851.44       4566.03       3835.8        504.864        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00578287     0.0125271     0.0264008    0.00965685     0.0064115       10     
reco:pandora:StandardPandora                            2847.41       3846.54       4555.59       3831.29       503.906        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00138701     0.0341458     0.222831      0.0128446     0.0637789       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000577549    0.0103949     0.0509938    0.00488719     0.0143662       10     
reco:pandoracalo:Calorimetry                          0.000847958    0.0148734     0.0621229    0.00670197     0.020008        10     
reco:pandorapid:Chi2ParticleID                        0.000114647   0.000778052   0.00574231    0.000232719    0.0016573       10     
reco:cvnmap:CVNMapper                                  0.0556304     0.0873488     0.136714      0.0815433     0.0237191       10     
reco:cvneva:CVNEvaluator                                2.55675       3.88115       8.78819       3.40967       1.6644         10     
reco:energyrecnumu:EnergyReco                          0.0778602     0.132233       0.17032      0.135032      0.0220326       10     
reco:energyrecnue:EnergyReco                          0.00281503    0.00850594     0.018229     0.00411229    0.00619699       10     
reco:energyrecnc:EnergyReco                            0.0138266     0.0352462     0.205887      0.0161813     0.0568902       10     
[art]:TriggerResults:TriggerResultInserter            3.0968e-05    6.14828e-05   0.00022305     4.788e-05    5.4342e-05       10     
end_path:out1:RootOutput                              1.0921e-05    2.33764e-05   6.9231e-05    1.4172e-05    1.86911e-05      10     
end_path:out1:RootOutput(write)                        0.585571      0.684083      0.904023      0.661563      0.0945967       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21778.7 MB
  Peak resident set size usage (VmHWM): 3031.03 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Begin processing the 6th record. run: 5026 subRun: 0 event: 7276 at 06-Dec-2025 09:22:38 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.575
True Electron Energy: 3.60086
Completed MCParticle Loop
True Neutrino Energy: 7.32586
 Size of all hits: 0
Begin processing the 7th record. run: 5026 subRun: 0 event: 7277 at 06-Dec-2025 09:22:43 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 19.2409
Completed MCParticle Loop
True Neutrino Energy: 26.3799
 Size of all hits: 0
Begin processing the 8th record. run: 5026 subRun: 0 event: 7278 at 06-Dec-2025 09:22:49 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 7.68899
Completed MCParticle Loop
True Neutrino Energy: 10.9746
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 7279 at 06-Dec-2025 09:22:54 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  35
Added hit list length: 35 | Max was set to: -3.40282e+38
True Neutron Energy: 941.712
True Neutron Energy: 939.739
True Electron Energy: 19.5732
Completed MCParticle Loop
True Neutrino Energy: 35.0673
 Size of all hits: 35
hit initial : 34.9413 | hit correction : 1.31362
hit initial : 36.707 | hit correction : 1.31354
hit initial : 78.3431 | hit correction : 1.31019
hit initial : 90.6747 | hit correction : 1.30583
hit initial : 51.8111 | hit correction : 1.3064
hit initial : 61.8683 | hit correction : 1.30581
hit initial : 29.6489 | hit correction : 1.30579
hit initial : 149.87 | hit correction : 1.30719
hit initial : 170.154 | hit correction : 1.30749
hit initial : 143.024 | hit correction : 1.30768
hit initial : 177.058 | hit correction : 1.3088
hit initial : 199.455 | hit correction : 1.3092
hit initial : 219.062 | hit correction : 1.30971
hit initial : 203.71 | hit correction : 1.31007
 *** Total Charge : 2154.41
 Looking at view: 0 | Number of hits in view: 14
hit initial : 98.7923 | hit correction : 1.30584
hit initial : 35.198 | hit correction : 1.30612
hit initial : 62.0194 | hit correction : 1.30578
hit initial : 28.5129 | hit correction : 1.30581
hit initial : 229.136 | hit correction : 1.30697
hit initial : 133.698 | hit correction : 1.30734
hit initial : 140.706 | hit correction : 1.30753
hit initial : 101.245 | hit correction : 1.30848
hit initial : 218.506 | hit correction : 1.30897
hit initial : 273.93 | hit correction : 1.30949
hit initial : 261.957 | hit correction : 1.30995
hit initial : 88.8397 | hit correction : 1.31011
 *** Total Charge : 2188.18
 Looking at view: 1 | Number of hits in view: 12
hit initial : 9.94285 | hit correction : 1.30831
hit initial : 89.0321 | hit correction : 1.30583
hit initial : 262.994 | hit correction : 1.30594
hit initial : 281.238 | hit correction : 1.30711
hit initial : 270.512 | hit correction : 1.30753
hit initial : 39.8132 | hit correction : 1.30826
hit initial : 518.62 | hit correction : 1.3092
hit initial : 307.276 | hit correction : 1.31002
hit initial : 20.6382 | hit correction : 1.31025
 *** Total Charge : 2354.68
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 10th record. run: 5026 subRun: 0 event: 7280 at 06-Dec-2025 09:22:59 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.567
True Electron Energy: 2.38953
Completed MCParticle Loop
True Neutrino Energy: 5.67327
 Size of all hits: 0
06-Dec-2025 09:23:04 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.121735      0.152358      0.197757      0.156201      0.0237219       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000574714   0.00105718    0.00383522    0.000803977   0.000938458      10     
end_path:ana:AnalyzeEvents         0.120951      0.150948      0.192577      0.155307      0.0228846       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19470.1 MB
  Peak resident set size usage (VmHWM): 1534.23 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:11:01 UTC       justIN version: 01.06.00