justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262168.0@dunegpschedd02.fnal.gov

Jobsub ID262168.0@dunegpschedd02.fnal.gov
Workflow ID11168
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 10:38:31
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 20:25:56
From worker nodeHostnamedunegli-7890507-0-fnpc17010.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 10:39:21
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T215302Z_gen_004523_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time9h (35178s)
CPU time9h (34504s = 98%)
Max RSS bytes2839646208 (2708 MiB)
Outputting started2025-12-13 20:25:40
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11168/1/001/analysisLowE_20251213T103925Z.root
Finished2025-12-13 20:25:56
Saved logsjustin-logs:262168.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6627, 6681, 6615
minwire 0: 73
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1753, 2252
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0950624, 
Output 1: 0.0335035, 0.0424085, 0.0949331, 0.829155, 
Output 2: 0.123774, 0.32865, 0.49487, 0.0527058, 
Output 3: 0.962533, 0.0293654, 0.00300202, 0.00509997, 
Output 4: 0.194308, 0.739, 0.0639542, 0.0027373, 
Output 5: 0.869499, 0.118568, 0.00950248, 0.00242989, 
Output 6: 0.0128885, 0.0514867, 0.0684757, 0.867149, 

13-Dec-2025 20:24:09 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 20:24:09 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/14/f1/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T215302Z_gen_004523_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2718.24       3481.06       4266.14       3478.85       593.907        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00116217    0.00579515     0.0194172    0.00179011    0.00668289       10     
reco:pandora:StandardPandora                            2715.49       3449.85       4189.18       3475.54       563.342        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00320095     0.111818      0.297984      0.100967      0.0871795       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000597222   0.00981804     0.0364795    0.00563957     0.0101575       10     
reco:pandoracalo:Calorimetry                          0.00042527     0.0108964     0.0919383    0.00123897     0.0270808       10     
reco:pandorapid:Chi2ParticleID                        0.00011901    0.000531249   0.00305307    0.000262458   0.000846984      10     
reco:cvnmap:CVNMapper                                  0.0855096     0.133188      0.242667      0.119499      0.0469621       10     
reco:cvneva:CVNEvaluator                                1.51509       2.7031        7.58594       1.96794       1.80867        10     
reco:energyrecnumu:EnergyReco                          0.102207       27.4889       73.2827      0.130958       33.5841        10     
reco:energyrecnue:EnergyReco                          0.00234128     0.0044542     0.0117403    0.00366905    0.00274178       10     
reco:energyrecnc:EnergyReco                            0.0121829     0.0140615     0.0182842     0.0136626    0.00182315       10     
[art]:TriggerResults:TriggerResultInserter            2.6542e-05    4.86724e-05   0.000151943   3.89955e-05   3.5389e-05       10     
end_path:out1:RootOutput                               9.422e-06    1.48829e-05    4.514e-05    1.16215e-05   1.02644e-05      10     
end_path:out1:RootOutput(write)                        0.500796      0.733461       1.74954      0.585471      0.354175        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11902.8 MB
  Peak resident set size usage (VmHWM): 2839.65 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 197.03 | hit correction : 1.40922
hit initial : 177.431 | hit correction : 1.40887
hit initial : 130.658 | hit correction : 1.40848
hit initial : 388.042 | hit correction : 1.40809
hit initial : 151.883 | hit correction : 1.40819
hit initial : 171.144 | hit correction : 1.40811
hit initial : 106.125 | hit correction : 1.40747
hit initial : 149.293 | hit correction : 1.40715
hit initial : 194.663 | hit correction : 1.40669
hit initial : 171.365 | hit correction : 1.40641
hit initial : 147.33 | hit correction : 1.4059
hit initial : 166.904 | hit correction : 1.40552
hit initial : 158.913 | hit correction : 1.40511
hit initial : 144.447 | hit correction : 1.40468
hit initial : 158.126 | hit correction : 1.4042
hit initial : 120.464 | hit correction : 1.40382
 *** Total Charge : 7784.29
 Looking at view: 2 | Number of hits in view: 33
Begin processing the 9th record. run: 5026 subRun: 0 event: 45229 at 13-Dec-2025 20:25:28 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  40
Added hit list length: 40 | Max was set to: -3.40282e+38
This pfp has size:  10
True Electron Energy: 25.3212
Completed MCParticle Loop
True Neutrino Energy: 31.5313
 Size of all hits: 40
hit initial : 33.8574 | hit correction : 1.19728
hit initial : 202.942 | hit correction : 1.19738
hit initial : 97.6578 | hit correction : 1.19803
hit initial : 83.2743 | hit correction : 1.19744
hit initial : 117.45 | hit correction : 1.19798
hit initial : 266.647 | hit correction : 1.1974
hit initial : 115.103 | hit correction : 1.19718
hit initial : 321.423 | hit correction : 1.19718
hit initial : 635.98 | hit correction : 1.19705
hit initial : 641.409 | hit correction : 1.1969
hit initial : 193.968 | hit correction : 1.19682
 *** Total Charge : 3243.99
 Looking at view: 0 | Number of hits in view: 11
hit initial : 148.055 | hit correction : 1.19736
hit initial : 65.6576 | hit correction : 1.19803
hit initial : 142.91 | hit correction : 1.198
hit initial : 287.384 | hit correction : 1.19738
hit initial : 186.418 | hit correction : 1.19757
hit initial : 107.5 | hit correction : 1.19726
hit initial : 333.438 | hit correction : 1.19722
hit initial : 312.427 | hit correction : 1.19712
hit initial : 345.199 | hit correction : 1.19698
hit initial : 441.802 | hit correction : 1.19688
hit initial : 264.571 | hit correction : 1.19675
hit initial : 60.8631 | hit correction : 1.19667
 *** Total Charge : 3227.86
 Looking at view: 1 | Number of hits in view: 12
hit initial : 94.0078 | hit correction : 1.19726
hit initial : 193.415 | hit correction : 1.19733
hit initial : 101.956 | hit correction : 1.19803
hit initial : 197.219 | hit correction : 1.1974
hit initial : 267.651 | hit correction : 1.19781
hit initial : 89.7905 | hit correction : 1.19714
hit initial : 226.105 | hit correction : 1.19725
hit initial : 137.901 | hit correction : 1.19732
hit initial : 141.819 | hit correction : 1.19719
hit initial : 123.178 | hit correction : 1.19707
hit initial : 118.105 | hit correction : 1.19705
hit initial : 135.744 | hit correction : 1.19701
hit initial : 123.162 | hit correction : 1.19693
hit initial : 288.58 | hit correction : 1.19689
hit initial : 358.521 | hit correction : 1.19685
hit initial : 130.783 | hit correction : 1.19664
hit initial : 23.3264 | hit correction : 1.19667
 *** Total Charge : 3293.76
 Looking at view: 2 | Number of hits in view: 17
Begin processing the 10th record. run: 5026 subRun: 0 event: 45230 at 13-Dec-2025 20:25:33 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Neutron Energy: 939.575
True Electron Energy: 26.4541
Completed MCParticle Loop
True Neutrino Energy: 31.8534
 Size of all hits: 0
13-Dec-2025 20:25:38 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.131947      0.232891      0.363868      0.224396      0.0678127       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000456242   0.000840885   0.00173174    0.000733879   0.000332629      10     
end_path:ana:AnalyzeEvents         0.131324       0.23179      0.363032      0.223307      0.0678005       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9808.22 MB
  Peak resident set size usage (VmHWM): 1533.79 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:06:19 UTC       justIN version: 01.06.00