justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258328.1@dunegpschedd02.fnal.gov

Jobsub ID258328.1@dunegpschedd02.fnal.gov
Workflow ID11028
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-06 01:22:33
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 10:20:11
From worker nodeHostnamedunegli-7832542-0-fnpc18010.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:41:31
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T164321Z_gen_000576_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (30707s)
CPU time8h (30172s = 98%)
Max RSS bytes3032109056 (2891 MiB)
Outputting started2025-12-06 10:13:18
Output files
Finished2025-12-06 10:20:11
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ine#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6753, 6761, 6701
minwire 0: 25
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 525, 1024
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.131043, 
Output 1: 0.0964299, 0.0878734, 0.089231, 0.726466, 
Output 2: 0.101025, 0.317217, 0.525023, 0.0567354, 
Output 3: 0.973475, 0.007434, 0.0016449, 0.0174465, 
Output 4: 0.593538, 0.383981, 0.0206148, 0.00186615, 
Output 5: 0.636061, 0.305648, 0.0452555, 0.0130355, 
Output 6: 0.0129902, 0.0317926, 0.0507887, 0.904429, 

06-Dec-2025 10:12:06 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:12:06 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/dc/5e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T164321Z_gen_000576_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2500.6        3032.11       4476.72       2877.51       587.157        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00501056    0.00969311     0.0244709    0.00697028     0.006081        10     
reco:pandora:StandardPandora                            2498.06       3021.22       4394.02       2874.12       567.573        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000694309    0.0493118     0.223451      0.0303639     0.0622576       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000611083   0.00886136     0.0558221    0.00245373     0.0158756       10     
reco:pandoracalo:Calorimetry                          0.000337416   0.00375371     0.0201007    0.000706818   0.00639641       10     
reco:pandorapid:Chi2ParticleID                        0.000123343   0.000753321   0.00587704    0.000164796   0.00170888       10     
reco:cvnmap:CVNMapper                                  0.0453254     0.0699442     0.137285      0.0558729     0.0264606       10     
reco:cvneva:CVNEvaluator                                1.8881        2.81953       8.18511       2.07887       1.8315         10     
reco:energyrecnumu:EnergyReco                          0.0104702     0.0743135      0.14921      0.0966194     0.0480705       10     
reco:energyrecnue:EnergyReco                          0.00189698    0.00474071     0.0150008    0.00262802    0.00418207       10     
reco:energyrecnc:EnergyReco                            0.0100094      7.38344       73.3365      0.0555614      21.9844        10     
[art]:TriggerResults:TriggerResultInserter            1.9868e-05    4.31254e-05   0.000156835   3.07335e-05   3.83091e-05      10     
end_path:out1:RootOutput                               9.538e-06    1.98354e-05   6.4031e-05    1.45125e-05   1.55276e-05      10     
end_path:out1:RootOutput(write)                        0.370118      0.463718       0.63051      0.406584      0.0947257       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21786.2 MB
  Peak resident set size usage (VmHWM): 3032.11 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 186.15 | hit correction : 1.07229
hit initial : 200.736 | hit correction : 1.07212
hit initial : 28.9654 | hit correction : 1.07188
hit initial : 177.196 | hit correction : 1.07138
hit initial : 109.879 | hit correction : 1.0705
hit initial : 865.098 | hit correction : 1.0706
hit initial : 680.801 | hit correction : 1.07045
hit initial : 91.2212 | hit correction : 1.07034
hit initial : 450.011 | hit correction : 1.07052
hit initial : 102.114 | hit correction : 1.06987
hit initial : 339.747 | hit correction : 1.07011
hit initial : 191.135 | hit correction : 1.06981
hit initial : 106.646 | hit correction : 1.06981
hit initial : 294.122 | hit correction : 1.06992
hit initial : 98.6062 | hit correction : 1.06988
 *** Total Charge : 4350.78
 Looking at view: 0 | Number of hits in view: 18
hit initial : 61.6921 | hit correction : 1.0724
hit initial : 255.12 | hit correction : 1.07224
hit initial : 159.793 | hit correction : 1.07212
hit initial : 183.994 | hit correction : 1.07145
hit initial : 142.817 | hit correction : 1.07073
hit initial : 619.786 | hit correction : 1.07067
hit initial : 874.813 | hit correction : 1.07048
hit initial : 520.033 | hit correction : 1.07035
hit initial : 353.265 | hit correction : 1.06995
hit initial : 190.495 | hit correction : 1.06987
hit initial : 500.813 | hit correction : 1.06988
 *** Total Charge : 4135.35
 Looking at view: 1 | Number of hits in view: 11
hit initial : 71.0215 | hit correction : 1.07547
hit initial : 61.2533 | hit correction : 1.07547
hit initial : 431.832 | hit correction : 1.07225
hit initial : 1388.49 | hit correction : 1.07104
hit initial : 392.375 | hit correction : 1.07049
hit initial : 197.431 | hit correction : 1.07036
hit initial : 328.136 | hit correction : 1.0703
hit initial : 185.638 | hit correction : 1.06998
hit initial : 371.545 | hit correction : 1.06989
hit initial : 194.662 | hit correction : 1.06988
hit initial : 171.592 | hit correction : 1.06987
hit initial : 132.203 | hit correction : 1.06976
 *** Total Charge : 4204.38
 Looking at view: 2 | Number of hits in view: 12
Begin processing the 9th record. run: 5026 subRun: 0 event: 5759 at 06-Dec-2025 10:13:07 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  15
Added hit list length: 15 | Max was set to: -3.40282e+38
True Electron Energy: 6.27311
Completed MCParticle Loop
True Neutrino Energy: 11.6544
 Size of all hits: 15
hit initial : 74.7253 | hit correction : 1.1455
hit initial : 268.576 | hit correction : 1.14544
hit initial : 236.965 | hit correction : 1.14535
hit initial : 65.5114 | hit correction : 1.14534
hit initial : 37.0227 | hit correction : 1.14416
hit initial : 22.8873 | hit correction : 1.14408
 *** Total Charge : 808.219
 Looking at view: 0 | Number of hits in view: 6
hit initial : 63.2149 | hit correction : 1.14582
hit initial : 55.6009 | hit correction : 1.14536
hit initial : 260.02 | hit correction : 1.14541
hit initial : 218.883 | hit correction : 1.14532
hit initial : 61.5262 | hit correction : 1.14532
 *** Total Charge : 755.102
 Looking at view: 1 | Number of hits in view: 5
hit initial : 65.2382 | hit correction : 1.14739
hit initial : 156.347 | hit correction : 1.14528
hit initial : 52.5587 | hit correction : 1.14591
hit initial : 476.986 | hit correction : 1.1454
 *** Total Charge : 860.481
 Looking at view: 2 | Number of hits in view: 4
Begin processing the 10th record. run: 5026 subRun: 0 event: 5760 at 06-Dec-2025 10:13:12 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.574
True Electron Energy: 12.5332
Completed MCParticle Loop
True Neutrino Energy: 15.8256
 Size of all hits: 0
06-Dec-2025 10:13:17 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.105047       0.15421      0.200591      0.156807      0.023915        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000532525   0.000778943   0.00185937    0.000646384   0.000371507      10     
end_path:ana:AnalyzeEvents         0.104344      0.153158      0.197924      0.155897      0.0235195       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19474.6 MB
  Peak resident set size usage (VmHWM): 1540.03 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:09:43 UTC       justIN version: 01.06.00