justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258185.0@dunegpschedd02.fnal.gov

Jobsub ID258185.0@dunegpschedd02.fnal.gov
Workflow ID11012
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 21:39:09
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 11:23:08
From worker nodeHostnamedunegli-7832317-0-fnpc18010.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:56
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T213118Z_gen_004110_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time9h (34641s)
CPU time9h (33849s = 97%)
Max RSS bytes3031822336 (2891 MiB)
Outputting started2025-12-06 11:16:17
Output files
Finished2025-12-06 11:23:08
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6641, 6594, 6518
minwire 0: 20
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1024, 1523
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.156721, 
Output 1: 0.0744189, 0.0940494, 0.170509, 0.661022, 
Output 2: 0.0532066, 0.239101, 0.680411, 0.027281, 
Output 3: 0.966596, 0.021457, 0.00376425, 0.00818294, 
Output 4: 0.230749, 0.660704, 0.100819, 0.00772759, 
Output 5: 0.502908, 0.409679, 0.0742762, 0.0131371, 
Output 6: 0.0278946, 0.0739145, 0.0880862, 0.810105, 

06-Dec-2025 11:14:42 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 11:14:42 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/50/60/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T213118Z_gen_004110_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2660.16       3423.29       4508.08       3411.05       551.04         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00221622     0.0102051     0.0246438     0.0076371    0.00679494       10     
reco:pandora:StandardPandora                            2656.43       3398.49       4425.14       3408.25       524.44         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00053571     0.0337106     0.160259     0.00125819     0.055362        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000454328   0.00782319     0.0518668    0.00135951     0.0150943       10     
reco:pandoracalo:Calorimetry                          0.000277834    0.0031936     0.0120594    0.000878091   0.00422377       10     
reco:pandorapid:Chi2ParticleID                        9.3356e-05     0.0007334    0.00388114    0.000147292    0.0012168       10     
reco:cvnmap:CVNMapper                                  0.0437392     0.0704972     0.114724      0.0708717     0.0196478       10     
reco:cvneva:CVNEvaluator                                1.93485       3.20871       7.49774       2.90228       1.52402        10     
reco:energyrecnumu:EnergyReco                         0.00764798      14.557        74.3935      0.123615       28.9154        10     
reco:energyrecnue:EnergyReco                           0.0020438     0.0102966     0.0173692     0.0104443    0.00548083       10     
reco:energyrecnc:EnergyReco                           0.00992372      6.3397        63.2683      0.0155885      18.9762        10     
[art]:TriggerResults:TriggerResultInserter            2.2833e-05    5.50799e-05   0.000251555   3.34825e-05   6.59628e-05      10     
end_path:out1:RootOutput                              1.0469e-05    2.13822e-05   6.9962e-05    1.57245e-05   1.71966e-05      10     
end_path:out1:RootOutput(write)                        0.372963       0.5543       0.735225      0.587068      0.122685        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21757.9 MB
  Peak resident set size usage (VmHWM): 3031.82 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 57.9196 | hit correction : 1.21535
hit initial : 73.9761 | hit correction : 1.21755
hit initial : 105.925 | hit correction : 1.21534
hit initial : 70.0387 | hit correction : 1.21757
hit initial : 48.0421 | hit correction : 1.21533
hit initial : 35.3413 | hit correction : 1.21419
hit initial : 43.4455 | hit correction : 1.2115
hit initial : 30.7774 | hit correction : 1.2142
hit initial : 40.0961 | hit correction : 1.21149
hit initial : 29.3243 | hit correction : 1.21664
hit initial : 72.146 | hit correction : 1.21666
hit initial : 155.495 | hit correction : 1.21725
hit initial : 114.447 | hit correction : 1.21668
hit initial : 90.2273 | hit correction : 1.21742
hit initial : 141.456 | hit correction : 1.21744
hit initial : 97.5357 | hit correction : 1.21744
hit initial : 33.4588 | hit correction : 1.21743
hit initial : 165.505 | hit correction : 1.21724
hit initial : 324.772 | hit correction : 1.21716
hit initial : 382.33 | hit correction : 1.21706
hit initial : 429.915 | hit correction : 1.21686
hit initial : 522.384 | hit correction : 1.2166
hit initial : 386.67 | hit correction : 1.21649
hit initial : 121.629 | hit correction : 1.21647
 *** Total Charge : 4346.96
 Looking at view: 0 | Number of hits in view: 24
hit initial : 131.968 | hit correction : 1.21744
hit initial : 61.2084 | hit correction : 1.21757
hit initial : 57.5884 | hit correction : 1.21758
hit initial : 99.922 | hit correction : 1.21722
hit initial : 253.917 | hit correction : 1.21719
hit initial : 221.28 | hit correction : 1.21714
hit initial : 42.2738 | hit correction : 1.21677
hit initial : 227.829 | hit correction : 1.21684
hit initial : 181.113 | hit correction : 1.21735
hit initial : 181.503 | hit correction : 1.21671
hit initial : 196.934 | hit correction : 1.21744
hit initial : 56.3956 | hit correction : 1.21677
hit initial : 47.5448 | hit correction : 1.21739
hit initial : 170.901 | hit correction : 1.21707
hit initial : 165.475 | hit correction : 1.21699
hit initial : 173.38 | hit correction : 1.21689
hit initial : 177.749 | hit correction : 1.21681
hit initial : 175.457 | hit correction : 1.21669
hit initial : 187.801 | hit correction : 1.2165
hit initial : 181.59 | hit correction : 1.21657
hit initial : 188.251 | hit correction : 1.21646
hit initial : 167.092 | hit correction : 1.21645
hit initial : 109.021 | hit correction : 1.21648
hit initial : 30.39 | hit correction : 1.21652
 *** Total Charge : 4242.92
 Looking at view: 1 | Number of hits in view: 24
hit initial : 31.8784 | hit correction : 1.21743
hit initial : 118.287 | hit correction : 1.21758
hit initial : 28.9844 | hit correction : 1.21759
hit initial : 397.431 | hit correction : 1.21719
hit initial : 244.401 | hit correction : 1.21666
hit initial : 589.963 | hit correction : 1.21732
hit initial : 145.93 | hit correction : 1.21737
hit initial : 158.609 | hit correction : 1.21715
hit initial : 230.934 | hit correction : 1.21707
hit initial : 198.031 | hit correction : 1.21696
hit initial : 206.649 | hit correction : 1.21686
hit initial : 191.434 | hit correction : 1.21675
hit initial : 186.902 | hit correction : 1.21661
hit initial : 183.501 | hit correction : 1.21651
hit initial : 55.5852 | hit correction : 1.21652
hit initial : 163.31 | hit correction : 1.21647
hit initial : 178.437 | hit correction : 1.21647
hit initial : 173.026 | hit correction : 1.21643
 *** Total Charge : 4238.99
 Looking at view: 2 | Number of hits in view: 18
Begin processing the 10th record. run: 5026 subRun: 0 event: 41100 at 06-Dec-2025 11:16:10 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.568
True Neutron Energy: 941.722
True Neutron Energy: 939.796
True Electron Energy: 52.3118
Completed MCParticle Loop
True Neutrino Energy: 69.6262
 Size of all hits: 0
06-Dec-2025 11:16:16 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.137013      0.188872      0.230329      0.189678      0.029135        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000697426   0.000965723    0.0017891    0.00088844    0.000300323      10     
end_path:ana:AnalyzeEvents         0.135955       0.18756      0.229093      0.188564      0.0289255       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19479.1 MB
  Peak resident set size usage (VmHWM): 1542.84 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 13:10:08 UTC       justIN version: 01.05.03