justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261337.0@dunegpschedd02.fnal.gov

Jobsub ID261337.0@dunegpschedd02.fnal.gov
Workflow ID11092
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 23:08:41
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-12 00:55:12
From worker nodeHostnamedunegli-8035300-0-fnpc22034.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-11 23:09:56
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T203338Z_gen_002980_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time1h (6302s)
CPU time1h (6245s = 99%)
Max RSS bytes3050708992 (2909 MiB)
Outputting started2025-12-12 00:54:58
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11092/1/001/analysisLowE_20251211T230956Z.root
Finished2025-12-12 00:55:12
Saved logsjustin-logs:261337.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

hm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6643, 6595, 6594
minwire 0: 32
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1858, 2357
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.078926, 
Output 1: 0.0514902, 0.0509175, 0.121022, 0.776571, 
Output 2: 0.0592174, 0.192674, 0.708862, 0.0392469, 
Output 3: 0.958045, 0.0241713, 0.00657453, 0.0112096, 
Output 4: 0.797756, 0.175551, 0.022117, 0.00457591, 
Output 5: 0.624229, 0.293658, 0.0666019, 0.0155118, 
Output 6: 0.0124658, 0.0468486, 0.0558198, 0.884866, 

12-Dec-2025 00:54:30 UTC  Closed output file "matt_reco2.root"
12-Dec-2025 00:54:30 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/c3/29/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T203338Z_gen_002980_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              612.539       621.521       634.475       622.37        6.34978        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00123081    0.00583144     0.0190684    0.00386619    0.00519378       10     
reco:pandora:StandardPandora                            611.497       620.384       633.479       620.941       6.31351        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00115376     0.0123797     0.052573     0.00770044     0.0144933       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000883179   0.00377494     0.0132732    0.00315349    0.00341006       10     
reco:pandoracalo:Calorimetry                          0.000218077   0.00285111     0.0199237    0.000639723   0.00576321       10     
reco:pandorapid:Chi2ParticleID                        4.8072e-05    0.000190412   0.000758711   7.4022e-05    0.000248098      10     
reco:cvnmap:CVNMapper                                  0.0167836     0.0234322      0.05242      0.0203029    0.00994628       10     
reco:cvneva:CVNEvaluator                                0.67211      0.795203       1.80058      0.682951      0.335205        10     
reco:energyrecnumu:EnergyReco                         0.00571629     0.0478565     0.0868932     0.060869      0.0282957       10     
reco:energyrecnue:EnergyReco                          0.000768089   0.00212527    0.00487325    0.000907976   0.00175544       10     
reco:energyrecnc:EnergyReco                            0.0044987     0.0193534     0.054242     0.00463188     0.0225406       10     
[art]:TriggerResults:TriggerResultInserter             7.946e-06    1.34368e-05   4.2652e-05    1.01445e-05   9.83264e-06      10     
end_path:out1:RootOutput                               3.437e-06    7.5314e-06    2.5469e-05    6.0965e-06    6.11638e-06      10     
end_path:out1:RootOutput(write)                        0.201484      0.223098      0.259023       0.22194      0.013415        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21753.1 MB
  Peak resident set size usage (VmHWM): 3050.71 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 21.1522 | hit correction : 1.46384
hit initial : 170.402 | hit correction : 1.46497
hit initial : 150.176 | hit correction : 1.46506
hit initial : 132.363 | hit correction : 1.46511
 *** Total Charge : 5293.52
 Looking at view: 2 | Number of hits in view: 22
Begin processing the 8th record. run: 5026 subRun: 0 event: 29798 at 12-Dec-2025 00:54:52 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 10.1397
Completed MCParticle Loop
True Neutrino Energy: 13.8691
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 29799 at 12-Dec-2025 00:54:54 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  45
Added hit list length: 45 | Max was set to: -3.40282e+38
True Neutron Energy: 940.95
True Electron Energy: 51.2358
Completed MCParticle Loop
True Neutrino Energy: 68.5734
 Size of all hits: 45
hit initial : 60.4641 | hit correction : 1.03614
hit initial : 78.971 | hit correction : 1.03521
hit initial : 35.6347 | hit correction : 1.03952
hit initial : 165.03 | hit correction : 1.03956
hit initial : 195.96 | hit correction : 1.03961
hit initial : 56.0965 | hit correction : 1.03963
hit initial : 237.167 | hit correction : 1.03839
hit initial : 244.984 | hit correction : 1.03835
hit initial : 273.59 | hit correction : 1.03826
hit initial : 311.969 | hit correction : 1.03797
hit initial : 430.256 | hit correction : 1.0379
hit initial : 102.271 | hit correction : 1.03783
hit initial : 190.042 | hit correction : 1.03774
hit initial : 302.207 | hit correction : 1.03773
hit initial : 350.557 | hit correction : 1.03758
hit initial : 293.227 | hit correction : 1.03701
hit initial : 159.512 | hit correction : 1.0361
hit initial : 413.789 | hit correction : 1.03603
hit initial : 415.731 | hit correction : 1.0352
hit initial : 392.54 | hit correction : 1.03468
hit initial : 328.798 | hit correction : 1.03421
hit initial : 353.773 | hit correction : 1.03365
hit initial : 351.49 | hit correction : 1.03304
hit initial : 126.711 | hit correction : 1.03281
 *** Total Charge : 6085.08
 Looking at view: 1 | Number of hits in view: 24
hit initial : 41.6992 | hit correction : 1.03847
hit initial : 124.944 | hit correction : 1.03949
hit initial : 190.978 | hit correction : 1.0396
hit initial : 146.773 | hit correction : 1.03963
hit initial : 256.085 | hit correction : 1.0384
hit initial : 323.118 | hit correction : 1.03838
hit initial : 224.816 | hit correction : 1.03819
hit initial : 237.196 | hit correction : 1.03786
hit initial : 916.149 | hit correction : 1.03782
hit initial : 419.675 | hit correction : 1.03767
hit initial : 246.574 | hit correction : 1.03719
hit initial : 419.511 | hit correction : 1.03651
hit initial : 363.127 | hit correction : 1.03597
hit initial : 383.712 | hit correction : 1.03536
hit initial : 332.378 | hit correction : 1.03494
hit initial : 268.459 | hit correction : 1.03452
hit initial : 363.657 | hit correction : 1.03416
hit initial : 267.732 | hit correction : 1.03368
hit initial : 261.564 | hit correction : 1.03323
hit initial : 274.762 | hit correction : 1.03278
hit initial : 35.9176 | hit correction : 1.03251
 *** Total Charge : 6321.31
 Looking at view: 2 | Number of hits in view: 21
Begin processing the 10th record. run: 5026 subRun: 0 event: 29800 at 12-Dec-2025 00:54:56 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.57
True Neutron Energy: 943.23
True Neutron Energy: 941.741
True Electron Energy: 23.6414
Completed MCParticle Loop
True Neutrino Energy: 36.2368
 Size of all hits: 0
12-Dec-2025 00:54:58 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0452773     0.0595079     0.0733441     0.0581338    0.00891364       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000297869   0.000566134   0.00217406    0.000380037   0.000542291      10     
end_path:ana:AnalyzeEvents         0.0448555     0.058811      0.0714072     0.0575898    0.00858635       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.5 MB
  Peak resident set size usage (VmHWM): 1537.51 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 16:28:56 UTC       justIN version: 01.06.00