justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258185.1@dunegpschedd02.fnal.gov

Jobsub ID258185.1@dunegpschedd02.fnal.gov
Workflow ID11012
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 21:39:09
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:21:16
From worker nodeHostnamedunegli-7988599-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:41
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021743Z_gen_003870_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (30948s)
CPU time8h (30270s = 97%)
Max RSS bytes3046916096 (2905 MiB)
Outputting started2025-12-06 10:14:29
Output files
Finished2025-12-06 10:21:16
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

er->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6803, 6790, 6724
minwire 0: 64
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1020, 1519
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.13841, 
Output 1: 0.201059, 0.112705, 0.281319, 0.404917, 
Output 2: 0.0484778, 0.0621735, 0.8794, 0.00994857, 
Output 3: 0.966551, 0.0308337, 0.00106207, 0.00155271, 
Output 4: 0.377755, 0.545818, 0.0629692, 0.0134583, 
Output 5: 0.191284, 0.408532, 0.259984, 0.140199, 
Output 6: 0.232456, 0.311774, 0.13708, 0.318691, 

06-Dec-2025 10:13:30 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:13:30 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/57/41/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021743Z_gen_003870_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1754.56       3051.29       6038.74       2148.55       1609.96        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00423494    0.00946785     0.0251745    0.00736143    0.00630625       10     
reco:pandora:StandardPandora                            1752.24       3028.5        5956.99       2146.19       1579.23        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000395505    0.0580102     0.411344      0.0191701     0.118859        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000415733    0.0128194     0.0962638    0.00255639     0.0279729       10     
reco:pandoracalo:Calorimetry                          0.000170071   0.00867033     0.0391105    0.00124124     0.0123389       10     
reco:pandorapid:Chi2ParticleID                        6.1816e-05     0.0014488     0.0131153    0.000113509   0.00389073       10     
reco:cvnmap:CVNMapper                                  0.0367019     0.0710253     0.217949      0.0458251     0.0565561       10     
reco:cvneva:CVNEvaluator                                1.76285       3.24736       14.2409       1.89627       3.69271        10     
reco:energyrecnumu:EnergyReco                          0.0112092      12.8166       65.4592      0.094708       25.5294        10     
reco:energyrecnue:EnergyReco                          0.00130738     0.0057789     0.0124038     0.0037152    0.00441048       10     
reco:energyrecnc:EnergyReco                            0.0094153      6.06775       60.324       0.0195412      18.0855        10     
[art]:TriggerResults:TriggerResultInserter             2.122e-05    6.23101e-05   0.00035587    2.79175e-05   9.82152e-05      10     
end_path:out1:RootOutput                               8.627e-06    2.28171e-05   0.000118192   9.6735e-06    3.2155e-05       10     
end_path:out1:RootOutput(write)                        0.351132      0.495635       1.21543       0.3741       0.268363        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21787.2 MB
  Peak resident set size usage (VmHWM): 3046.92 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 135.463 | hit correction : 1.31872
hit initial : 11.2288 | hit correction : 1.31963
hit initial : 28.1066 | hit correction : 1.31965
hit initial : 53.3388 | hit correction : 1.31979
hit initial : 66.6435 | hit correction : 1.32055
hit initial : 41.0626 | hit correction : 1.33016
hit initial : 312.443 | hit correction : 1.33019
hit initial : 506.951 | hit correction : 1.33022
hit initial : 238.661 | hit correction : 1.33022
hit initial : 79.9504 | hit correction : 1.33022
hit initial : 146.615 | hit correction : 1.33971
hit initial : 224.079 | hit correction : 1.34028
hit initial : 186.146 | hit correction : 1.34095
hit initial : 128.28 | hit correction : 1.34135
hit initial : 217.438 | hit correction : 1.34174
hit initial : 120.36 | hit correction : 1.34201
 *** Total Charge : 4819.68
 Looking at view: 2 | Number of hits in view: 23
Begin processing the 7th record. run: 5027 subRun: 0 event: 38697 at 06-Dec-2025 10:14:14 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Electron Energy: 42.1816
Completed MCParticle Loop
True Neutrino Energy: 45.5561
 Size of all hits: 0
Begin processing the 8th record. run: 5027 subRun: 0 event: 38698 at 06-Dec-2025 10:14:18 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  27
Added hit list length: 27 | Max was set to: -3.40282e+38
True Electron Energy: 20.4444
Completed MCParticle Loop
True Neutrino Energy: 23.7468
 Size of all hits: 27
hit initial : 160.79 | hit correction : 1.22222
hit initial : 163.766 | hit correction : 1.22231
hit initial : 41.7562 | hit correction : 1.22235
hit initial : 176.061 | hit correction : 1.22141
hit initial : 422.459 | hit correction : 1.22127
hit initial : 381.153 | hit correction : 1.22118
hit initial : 297.93 | hit correction : 1.22116
hit initial : 224.99 | hit correction : 1.22116
hit initial : 85.0033 | hit correction : 1.22116
 *** Total Charge : 2386.54
 Looking at view: 0 | Number of hits in view: 9
hit initial : 121.669 | hit correction : 1.22227
hit initial : 282.53 | hit correction : 1.22211
hit initial : 83.4195 | hit correction : 1.2222
hit initial : 164.418 | hit correction : 1.22145
hit initial : 211.631 | hit correction : 1.22129
hit initial : 256.453 | hit correction : 1.22116
hit initial : 233.04 | hit correction : 1.22114
hit initial : 199.234 | hit correction : 1.22117
hit initial : 183.488 | hit correction : 1.22116
hit initial : 155.213 | hit correction : 1.22116
hit initial : 73.355 | hit correction : 1.22116
 *** Total Charge : 2399.47
 Looking at view: 1 | Number of hits in view: 11
hit initial : 234.454 | hit correction : 1.22232
hit initial : 308.203 | hit correction : 1.22177
hit initial : 205.179 | hit correction : 1.22141
hit initial : 251.473 | hit correction : 1.22119
hit initial : 469.674 | hit correction : 1.22114
hit initial : 399.883 | hit correction : 1.22116
hit initial : 129.919 | hit correction : 1.22116
 *** Total Charge : 2441.35
 Looking at view: 2 | Number of hits in view: 7
Begin processing the 9th record. run: 5027 subRun: 0 event: 38699 at 06-Dec-2025 10:14:21 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 1.40071
Completed MCParticle Loop
True Neutrino Energy: 5.12476
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 38700 at 06-Dec-2025 10:14:25 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.572
True Electron Energy: 12.9166
Completed MCParticle Loop
True Neutrino Energy: 18.2953
 Size of all hits: 0
06-Dec-2025 10:14:28 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0889405     0.119766      0.152378      0.117872      0.0203143       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000583089    0.0007788     0.0016283    0.00070033    0.000291294      10     
end_path:ana:AnalyzeEvents         0.0880569     0.118722      0.150007      0.117014      0.0200439       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19478 MB
  Peak resident set size usage (VmHWM): 1538.98 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 13:09:26 UTC       justIN version: 01.05.03