justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 249610.2@dunegpschedd02.fnal.gov

Jobsub ID249610.2@dunegpschedd02.fnal.gov
Workflow ID10431
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-22 09:42:28
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-11-22 11:49:31
From worker nodeHostnamedunegli-7849909-0-fnpc22007.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-22 09:51:15
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193343Z_gen_000343_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time1h (7086s)
CPU time1h (6946s = 98%)
Max RSS bytes3054252032 (2912 MiB)
Outputting started2025-11-22 11:49:21
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/10431/1/001/analysisLowE_20251122T095115Z.root
Finished2025-11-22 11:49:31
Saved logsjustin-logs:249610.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

hm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6900, 6808, 6690
minwire 0: 23
minwire 1: 1
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2195, 2694
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.116712, 
Output 1: 0.0579064, 0.129785, 0.276089, 0.536219, 
Output 2: 0.090196, 0.163347, 0.717624, 0.0288332, 
Output 3: 0.95613, 0.0357638, 0.00358842, 0.00451779, 
Output 4: 0.410482, 0.541751, 0.0433145, 0.0044526, 
Output 5: 0.527145, 0.405022, 0.0554518, 0.0123806, 
Output 6: 0.0334173, 0.0880766, 0.0802014, 0.798305, 

22-Nov-2025 11:48:49 UTC  Closed output file "matt_reco2.root"
22-Nov-2025 11:48:49 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/3d/f3/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193343Z_gen_000343_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              632.823       699.206       766.849       715.462       46.4538        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00138802    0.00793133     0.0258242    0.00663238     0.0063589       10     
reco:pandora:StandardPandora                            630.532       698.028       765.744       714.399       46.6083        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000620263    0.0148993     0.0600616    0.00577736      0.01838        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000230145   0.00269753     0.0108501    0.00168411    0.00299402       10     
reco:pandoracalo:Calorimetry                          0.000183637   0.00817269     0.0421078    0.00230481     0.0129409       10     
reco:pandorapid:Chi2ParticleID                        4.2991e-05    0.000253054   0.000883752   8.74355e-05   0.000292169      10     
reco:cvnmap:CVNMapper                                  0.019438      0.0288337     0.0397059     0.0290517     0.0050647       10     
reco:cvneva:CVNEvaluator                               0.697092      0.832186       1.8562       0.718813      0.341584        10     
reco:energyrecnumu:EnergyReco                         0.00607163     0.0525849     0.0974647     0.0582885     0.0259208       10     
reco:energyrecnue:EnergyReco                          0.000828797   0.00292827    0.00484082    0.00325006    0.00186173       10     
reco:energyrecnc:EnergyReco                           0.00455802     0.0165798     0.0717709    0.00465146     0.024123        10     
[art]:TriggerResults:TriggerResultInserter             9.518e-06    2.0625e-05    7.0984e-05    1.6381e-05    1.70731e-05      10     
end_path:out1:RootOutput                               6.853e-06    9.9679e-06    2.9997e-05    8.1705e-06    6.71693e-06      10     
end_path:out1:RootOutput(write)                        0.198122      0.210035      0.226151      0.207306     0.00887575       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21753.1 MB
  Peak resident set size usage (VmHWM): 3054.25 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 257.748 | hit correction : 1.29409
hit initial : 682.565 | hit correction : 1.29292
hit initial : 227.253 | hit correction : 1.29269
hit initial : 459.69 | hit correction : 1.29135
hit initial : 469.625 | hit correction : 1.29031
hit initial : 781.046 | hit correction : 1.28897
hit initial : 623.07 | hit correction : 1.28766
hit initial : 75.5346 | hit correction : 1.28739
 *** Total Charge : 6857.15
 Looking at view: 2 | Number of hits in view: 20
Begin processing the 9th record. run: 3799 subRun: 0 event: 3429 at 22-Nov-2025 11:49:17 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  47
Added hit list length: 47 | Max was set to: -3.40282e+38
True Electron Energy: 30.6506
Completed MCParticle Loop
True Neutrino Energy: 37.8048
 Size of all hits: 47
hit initial : 63.4273 | hit correction : 1.29173
hit initial : 68.3043 | hit correction : 1.29172
hit initial : 61.1785 | hit correction : 1.29309
hit initial : 59.3852 | hit correction : 1.29307
hit initial : 100.203 | hit correction : 1.28869
hit initial : 71.5178 | hit correction : 1.28508
hit initial : 19.8257 | hit correction : 1.28492
hit initial : 80.867 | hit correction : 1.28239
hit initial : 70.668 | hit correction : 1.28237
hit initial : 313.115 | hit correction : 1.2886
hit initial : 281.101 | hit correction : 1.28819
hit initial : 291.61 | hit correction : 1.28805
hit initial : 301.738 | hit correction : 1.28779
hit initial : 362.043 | hit correction : 1.28709
hit initial : 286.227 | hit correction : 1.28648
hit initial : 221.859 | hit correction : 1.28613
hit initial : 235.46 | hit correction : 1.28603
hit initial : 228.253 | hit correction : 1.286
hit initial : 235.089 | hit correction : 1.2859
hit initial : 206.677 | hit correction : 1.28567
hit initial : 142.613 | hit correction : 1.28542
 *** Total Charge : 4763.89
 Looking at view: 0 | Number of hits in view: 21
hit initial : 46.104 | hit correction : 1.28512
hit initial : 69.5073 | hit correction : 1.28884
hit initial : 212.289 | hit correction : 1.2884
hit initial : 403.998 | hit correction : 1.28838
hit initial : 221.417 | hit correction : 1.28805
hit initial : 188.266 | hit correction : 1.28799
hit initial : 213.658 | hit correction : 1.28768
hit initial : 229.301 | hit correction : 1.28718
hit initial : 244.006 | hit correction : 1.28671
hit initial : 207.738 | hit correction : 1.28636
hit initial : 186.96 | hit correction : 1.28612
hit initial : 184.63 | hit correction : 1.28603
hit initial : 191.509 | hit correction : 1.286
hit initial : 187.169 | hit correction : 1.28598
hit initial : 187.251 | hit correction : 1.28585
hit initial : 203.618 | hit correction : 1.2856
hit initial : 161.573 | hit correction : 1.2853
 *** Total Charge : 4297.08
 Looking at view: 1 | Number of hits in view: 17
hit initial : 148.749 | hit correction : 1.28824
hit initial : 198.425 | hit correction : 1.2889
hit initial : 278.272 | hit correction : 1.28813
hit initial : 585.337 | hit correction : 1.28811
hit initial : 420.851 | hit correction : 1.28752
hit initial : 555.592 | hit correction : 1.28646
hit initial : 674.343 | hit correction : 1.28601
hit initial : 433.772 | hit correction : 1.28572
hit initial : 117.331 | hit correction : 1.28507
 *** Total Charge : 4392.11
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 10th record. run: 3799 subRun: 0 event: 3430 at 22-Nov-2025 11:49:19 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.575
True Neutron Energy: 942.653
True Neutron Energy: 939.677
True Electron Energy: 39.9645
Completed MCParticle Loop
True Neutrino Energy: 59.5082
 Size of all hits: 0
22-Nov-2025 11:49:21 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0481777     0.055858      0.0670528     0.0574414    0.00563747       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000373898   0.000513647   0.000932544   0.000471352   0.000146687      10     
end_path:ana:AnalyzeEvents         0.0475385     0.055125      0.0663426     0.0568233    0.00555435       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19475 MB
  Peak resident set size usage (VmHWM): 1536.73 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-09 01:55:53 UTC       justIN version: 01.06.00