justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 249757.18@dunegpschedd02.fnal.gov

Jobsub ID249757.18@dunegpschedd02.fnal.gov
Workflow ID10521
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-22 19:12:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-23 01:38:04
From worker nodeHostnamedunegli-7694053-0-fnpc18001.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-22 19:23:48
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193343Z_gen_000343_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (20965s)
CPU time5h (20120s = 95%)
Max RSS bytes3121885184 (2977 MiB)
Outputting started2025-11-23 01:13:15
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/10521/1/001/analysisLowE_20251122T192349Z.root
Finished2025-11-23 01:38:04
Saved logsjustin-logs:249757.18-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

hm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6900, 6808, 6690
minwire 0: 23
minwire 1: 1
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2195, 2694
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.116712, 
Output 1: 0.0579064, 0.129785, 0.276089, 0.536219, 
Output 2: 0.090196, 0.163347, 0.717624, 0.0288332, 
Output 3: 0.95613, 0.0357638, 0.00358842, 0.00451779, 
Output 4: 0.410482, 0.541751, 0.0433145, 0.0044526, 
Output 5: 0.527145, 0.405022, 0.0554518, 0.0123806, 
Output 6: 0.0334173, 0.0880766, 0.0802014, 0.798305, 

23-Nov-2025 01:02:17 UTC  Closed output file "matt_reco2.root"
23-Nov-2025 01:02:17 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/3d/f3/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193343Z_gen_000343_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1649.62       2017.97       2483.14       1943.33       296.619        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00143951    0.00551293    0.00827325    0.00603296    0.00217249       10     
reco:pandora:StandardPandora                            1647.25       2014.63       2479.73       1939.23       296.401        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00230283     0.023226      0.113891      0.0101277     0.0321281       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00238358    0.00832119     0.0295334     0.0066888    0.00743841       10     
reco:pandoracalo:Calorimetry                          0.000210397   0.00886518     0.0416327     0.0019756     0.0135945       10     
reco:pandorapid:Chi2ParticleID                        9.7925e-05    0.000457958    0.0016346    0.000141366   0.000555774      10     
reco:cvnmap:CVNMapper                                  0.0330712     0.111139      0.399497      0.0477193     0.133535        10     
reco:cvneva:CVNEvaluator                                1.7739        2.46092       4.44281       1.95133      0.818997        10     
reco:energyrecnumu:EnergyReco                          0.0114218     0.0888227     0.158398      0.0938986     0.0425192       10     
reco:energyrecnue:EnergyReco                          0.00150256    0.00719791     0.0142173    0.00633168     0.0053157       10     
reco:energyrecnc:EnergyReco                           0.00940631     0.0253553     0.0850697     0.0140794     0.0255985       10     
[art]:TriggerResults:TriggerResultInserter            1.8415e-05    3.68644e-05   0.000128953   2.8193e-05    3.09908e-05      10     
end_path:out1:RootOutput                               7.283e-06    1.2241e-05    4.2541e-05     9.067e-06    1.01558e-05      10     
end_path:out1:RootOutput(write)                        0.355687      0.599686       1.05932      0.483908      0.257793        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21777.1 MB
  Peak resident set size usage (VmHWM): 3121.89 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 257.748 | hit correction : 1.29409
hit initial : 682.565 | hit correction : 1.29292
hit initial : 227.253 | hit correction : 1.29269
hit initial : 459.69 | hit correction : 1.29135
hit initial : 469.625 | hit correction : 1.29031
hit initial : 781.046 | hit correction : 1.28897
hit initial : 623.07 | hit correction : 1.28766
hit initial : 75.5346 | hit correction : 1.28739
 *** Total Charge : 6857.15
 Looking at view: 2 | Number of hits in view: 20
Begin processing the 9th record. run: 3799 subRun: 0 event: 3429 at 23-Nov-2025 01:04:16 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  47
Added hit list length: 47 | Max was set to: -3.40282e+38
True Electron Energy: 30.6506
Completed MCParticle Loop
True Neutrino Energy: 37.8048
 Size of all hits: 47
hit initial : 63.4273 | hit correction : 1.29173
hit initial : 68.3043 | hit correction : 1.29172
hit initial : 61.1785 | hit correction : 1.29309
hit initial : 59.3852 | hit correction : 1.29307
hit initial : 100.203 | hit correction : 1.28869
hit initial : 71.5178 | hit correction : 1.28508
hit initial : 19.8257 | hit correction : 1.28492
hit initial : 80.867 | hit correction : 1.28239
hit initial : 70.668 | hit correction : 1.28237
hit initial : 313.115 | hit correction : 1.2886
hit initial : 281.101 | hit correction : 1.28819
hit initial : 291.61 | hit correction : 1.28805
hit initial : 301.738 | hit correction : 1.28779
hit initial : 362.043 | hit correction : 1.28709
hit initial : 286.227 | hit correction : 1.28648
hit initial : 221.859 | hit correction : 1.28613
hit initial : 235.46 | hit correction : 1.28603
hit initial : 228.253 | hit correction : 1.286
hit initial : 235.089 | hit correction : 1.2859
hit initial : 206.677 | hit correction : 1.28567
hit initial : 142.613 | hit correction : 1.28542
 *** Total Charge : 4763.89
 Looking at view: 0 | Number of hits in view: 21
hit initial : 46.104 | hit correction : 1.28512
hit initial : 69.5073 | hit correction : 1.28884
hit initial : 212.289 | hit correction : 1.2884
hit initial : 403.998 | hit correction : 1.28838
hit initial : 221.417 | hit correction : 1.28805
hit initial : 188.266 | hit correction : 1.28799
hit initial : 213.658 | hit correction : 1.28768
hit initial : 229.301 | hit correction : 1.28718
hit initial : 244.006 | hit correction : 1.28671
hit initial : 207.738 | hit correction : 1.28636
hit initial : 186.96 | hit correction : 1.28612
hit initial : 184.63 | hit correction : 1.28603
hit initial : 191.509 | hit correction : 1.286
hit initial : 187.169 | hit correction : 1.28598
hit initial : 187.251 | hit correction : 1.28585
hit initial : 203.618 | hit correction : 1.2856
hit initial : 161.573 | hit correction : 1.2853
 *** Total Charge : 4297.08
 Looking at view: 1 | Number of hits in view: 17
hit initial : 148.749 | hit correction : 1.28824
hit initial : 198.425 | hit correction : 1.2889
hit initial : 278.272 | hit correction : 1.28813
hit initial : 585.337 | hit correction : 1.28811
hit initial : 420.851 | hit correction : 1.28752
hit initial : 555.592 | hit correction : 1.28646
hit initial : 674.343 | hit correction : 1.28601
hit initial : 433.772 | hit correction : 1.28572
hit initial : 117.331 | hit correction : 1.28507
 *** Total Charge : 4392.11
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 10th record. run: 3799 subRun: 0 event: 3430 at 23-Nov-2025 01:04:27 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.575
True Neutron Energy: 942.653
True Neutron Energy: 939.677
True Electron Energy: 39.9645
Completed MCParticle Loop
True Neutrino Energy: 59.5082
 Size of all hits: 0
23-Nov-2025 01:04:40 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.087141      0.563129       1.11567      0.518132       0.34433        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000428929    0.0885344     0.335192     0.000559361    0.135741        10     
end_path:ana:AnalyzeEvents         0.0794181     0.474395       1.11486      0.482272      0.322024        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19475 MB
  Peak resident set size usage (VmHWM): 1537.33 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-08 21:39:25 UTC       justIN version: 01.06.00