justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257896.15@dunegpschedd02.fnal.gov

Jobsub ID257896.15@dunegpschedd02.fnal.gov
Workflow ID11025
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 10:55:41
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:13:54
From worker nodeHostnamedunegli-7988756-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-05 19:24:38
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T233809Z_gen_004789_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time14h (52938s)
CPU time14h (51338s = 96%)
Max RSS bytes2850234368 (2718 MiB)
Outputting started2025-12-06 10:06:57
Output files
Finished2025-12-06 10:13:54
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

S_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6605, 6462, 6573
minwire 0: 39
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2120, 2619
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.110493, 
Output 1: 0.0453783, 0.310189, 0.121493, 0.52294, 
Output 2: 0.0701825, 0.215975, 0.692844, 0.0209986, 
Output 3: 0.979412, 0.0169953, 0.000855993, 0.00273678, 
Output 4: 0.552997, 0.41631, 0.0278195, 0.00287298, 
Output 5: 0.475167, 0.419515, 0.0817411, 0.0235776, 
Output 6: 0.0224771, 0.0760358, 0.0874536, 0.814034, 

06-Dec-2025 10:05:10 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:05:10 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f9/2b/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T233809Z_gen_004789_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3612.97       5243.19       6384.96       5377.35       899.803        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00749268     0.0163626     0.0340086     0.015154     0.00743835       10     
reco:pandora:StandardPandora                            3608.72       5184.95       6302.64       5308.94       874.583        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000785671    0.0605056      0.38632     0.00807291     0.115497        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000595052    0.0110634     0.0359794    0.00812506     0.0107245       10     
reco:pandoracalo:Calorimetry                          0.00047726     0.0331881     0.110793      0.013129      0.0378282       10     
reco:pandorapid:Chi2ParticleID                        0.000110728   0.00105846    0.00455139    0.000242838   0.00161483       10     
reco:cvnmap:CVNMapper                                  0.0732717     0.131594       0.22742      0.131987      0.0453246       10     
reco:cvneva:CVNEvaluator                                3.24464       6.09478       14.972        4.37536       4.2171         10     
reco:energyrecnumu:EnergyReco                          0.028524       38.3361       70.4815       60.8139       31.3387        10     
reco:energyrecnue:EnergyReco                          0.00413208     0.0197105     0.055718      0.016799      0.0136201       10     
reco:energyrecnc:EnergyReco                            0.0137183      12.684        64.3347      0.020045       25.3341        10     
[art]:TriggerResults:TriggerResultInserter            2.4336e-05    0.000132279   0.000450118   6.28335e-05   0.000126261      10     
end_path:out1:RootOutput                              1.1232e-05    3.79076e-05   0.000155994   1.7613e-05     4.242e-05       10     
end_path:out1:RootOutput(write)                         0.49277      0.848622       1.35748      0.763799      0.241941        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21741.2 MB
  Peak resident set size usage (VmHWM): 2850.23 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 824.981 | hit correction : 1.17742
hit initial : 561.783 | hit correction : 1.17763
hit initial : 280.358 | hit correction : 1.17877
hit initial : 235.357 | hit correction : 1.17898
hit initial : 179.956 | hit correction : 1.17942
hit initial : 277.635 | hit correction : 1.18025
hit initial : 317.718 | hit correction : 1.1809
hit initial : 42.173 | hit correction : 1.18089
hit initial : 292.787 | hit correction : 1.18156
hit initial : 404.167 | hit correction : 1.18247
hit initial : 18.6465 | hit correction : 1.18261
hit initial : 378.35 | hit correction : 1.18323
hit initial : 396.734 | hit correction : 1.18413
hit initial : 300.078 | hit correction : 1.18471
hit initial : 93.5178 | hit correction : 1.18497
 *** Total Charge : 8716.03
 Looking at view: 1 | Number of hits in view: 29
hit initial : 52.3697 | hit correction : 1.18393
hit initial : 17.6514 | hit correction : 1.18393
hit initial : 100.732 | hit correction : 1.17244
hit initial : 392.778 | hit correction : 1.17273
hit initial : 442.53 | hit correction : 1.17339
hit initial : 116.74 | hit correction : 1.1741
hit initial : 297.876 | hit correction : 1.17424
hit initial : 164.274 | hit correction : 1.17467
hit initial : 134.353 | hit correction : 1.17491
hit initial : 98.3381 | hit correction : 1.17457
hit initial : 148.313 | hit correction : 1.17515
hit initial : 16.4859 | hit correction : 1.17457
hit initial : 201.035 | hit correction : 1.17551
hit initial : 233.95 | hit correction : 1.17601
hit initial : 251.859 | hit correction : 1.17647
hit initial : 108.801 | hit correction : 1.17708
hit initial : 608.072 | hit correction : 1.17727
hit initial : 308.901 | hit correction : 1.17748
hit initial : 167.027 | hit correction : 1.17791
hit initial : 206.019 | hit correction : 1.17796
hit initial : 186.891 | hit correction : 1.17841
hit initial : 253.644 | hit correction : 1.1789
hit initial : 250.368 | hit correction : 1.1793
hit initial : 229.523 | hit correction : 1.17988
hit initial : 250.938 | hit correction : 1.18046
hit initial : 289.454 | hit correction : 1.18097
hit initial : 238.17 | hit correction : 1.18161
hit initial : 247.394 | hit correction : 1.18222
hit initial : 336.128 | hit correction : 1.18274
hit initial : 359.135 | hit correction : 1.18357
hit initial : 456.498 | hit correction : 1.18443
hit initial : 230.054 | hit correction : 1.18498
 *** Total Charge : 8715.73
 Looking at view: 2 | Number of hits in view: 32
Begin processing the 7th record. run: 5026 subRun: 0 event: 47887 at 06-Dec-2025 10:06:31 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 6.13593
Completed MCParticle Loop
True Neutrino Energy: 9.86167
 Size of all hits: 0
Begin processing the 8th record. run: 5026 subRun: 0 event: 47888 at 06-Dec-2025 10:06:38 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 5.85888
Completed MCParticle Loop
True Neutrino Energy: 11.2374
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 47889 at 06-Dec-2025 10:06:44 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 48.6884
Completed MCParticle Loop
True Neutrino Energy: 55.374
 Size of all hits: 0
Begin processing the 10th record. run: 5026 subRun: 0 event: 47890 at 06-Dec-2025 10:06:50 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.579
True Neutron Energy: 942.388
True Neutron Energy: 939.847
True Electron Energy: 50.3476
Completed MCParticle Loop
True Neutrino Energy: 55.7884
 Size of all hits: 0
06-Dec-2025 10:06:56 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.141945      0.170178      0.241714       0.15983      0.0293269       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)             0.0006628    0.00105924    0.00239137    0.000975529   0.000461347      10     
end_path:ana:AnalyzeEvents         0.140732      0.168766      0.240768      0.158546      0.029161        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19470.3 MB
  Peak resident set size usage (VmHWM): 1532.88 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:11:00 UTC       justIN version: 01.06.00