justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 259603.0@dunegpschedd02.fnal.gov

Jobsub ID259603.0@dunegpschedd02.fnal.gov
Workflow ID11090
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 13:23:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-10 17:28:56
From worker nodeHostnamedunegli-7867974-0-fnpc19127.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-10 12:23:18
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001249Z_gen_001968_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (18287s)
CPU time4h (17762s = 97%)
Max RSS bytes3055980544 (2914 MiB)
Outputting started2025-12-10 17:28:05
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11090/1/001/analysisLowE_20251210T122318Z.root
Finished2025-12-10 17:28:56
Saved logsjustin-logs:259603.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6787, 6696, 6677
minwire 0: 12
minwire 1: 1
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2138, 2637
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.105029, 
Output 1: 0.0442141, 0.070844, 0.130469, 0.754473, 
Output 2: 0.0445686, 0.258501, 0.664403, 0.0325266, 
Output 3: 0.952613, 0.0281118, 0.00634182, 0.0129335, 
Output 4: 0.206889, 0.687802, 0.099704, 0.0056051, 
Output 5: 0.767097, 0.185057, 0.0381335, 0.00971194, 
Output 6: 0.0143878, 0.0409911, 0.0591041, 0.885517, 

10-Dec-2025 17:27:07 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 17:27:07 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/d0/0a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001249Z_gen_001968_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1482.21       1812.99       1965.77       1881.33       149.812        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00169798    0.00675728     0.0161724    0.00548228    0.00391498       10     
reco:pandora:StandardPandora                            1480.84       1810.61       1963.13       1878.88       149.862        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000823869    0.0216767      0.10485     0.00893799     0.0300652       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00119614    0.00570073     0.0347574    0.00217474    0.00976432       10     
reco:pandoracalo:Calorimetry                          0.000164982    0.0138345     0.0343547    0.00653121     0.0145536       10     
reco:pandorapid:Chi2ParticleID                        4.8151e-05    0.00156456     0.0147182    8.8751e-05    0.00438497       10     
reco:cvnmap:CVNMapper                                  0.0275554     0.0389295     0.0653343     0.0352038     0.0130957       10     
reco:cvneva:CVNEvaluator                               0.954221       1.80126       3.13962       1.6416       0.592794        10     
reco:energyrecnumu:EnergyReco                         0.00411808     0.0701615     0.126999      0.0753946     0.0362512       10     
reco:energyrecnue:EnergyReco                          0.000987921   0.00458693     0.0116282    0.00390483    0.00361843       10     
reco:energyrecnc:EnergyReco                           0.00614331     0.0757711     0.488253      0.009665      0.142637        10     
[art]:TriggerResults:TriggerResultInserter            1.1651e-05    2.30022e-05   9.3171e-05    1.5355e-05    2.34921e-05      10     
end_path:out1:RootOutput                               7.38e-06     1.12923e-05    2.889e-05     9.145e-06    5.97625e-06      10     
end_path:out1:RootOutput(write)                        0.271339      0.336642      0.527087      0.286847      0.0821507       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21786.7 MB
  Peak resident set size usage (VmHWM): 3055.98 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 306.147 | hit correction : 1.17991
hit initial : 402.787 | hit correction : 1.18002
hit initial : 253.924 | hit correction : 1.1804
hit initial : 183.046 | hit correction : 1.1805
hit initial : 146.679 | hit correction : 1.18058
hit initial : 153.182 | hit correction : 1.18066
hit initial : 65.7831 | hit correction : 1.18069
 *** Total Charge : 2074.48
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 9th record. run: 5027 subRun: 0 event: 19679 at 10-Dec-2025 17:27:57 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  51
Added hit list length: 51 | Max was set to: -3.40282e+38
True Electron Energy: 53.1956
Completed MCParticle Loop
True Neutrino Energy: 68.8233
 Size of all hits: 51
hit initial : 31.2078 | hit correction : 1.42766
hit initial : 33.7032 | hit correction : 1.4292
hit initial : 21.3301 | hit correction : 1.42924
hit initial : 96.9795 | hit correction : 1.42446
hit initial : 123.787 | hit correction : 1.42516
hit initial : 206.451 | hit correction : 1.42399
hit initial : 339.715 | hit correction : 1.42413
hit initial : 76.685 | hit correction : 1.42406
hit initial : 101.155 | hit correction : 1.426
hit initial : 195.891 | hit correction : 1.42627
hit initial : 241.927 | hit correction : 1.42648
hit initial : 275.711 | hit correction : 1.42655
hit initial : 265.415 | hit correction : 1.42662
hit initial : 185.647 | hit correction : 1.42678
hit initial : 138.047 | hit correction : 1.42695
hit initial : 137.233 | hit correction : 1.42704
hit initial : 146.559 | hit correction : 1.42714
hit initial : 153.923 | hit correction : 1.42725
hit initial : 165.54 | hit correction : 1.42734
hit initial : 176.273 | hit correction : 1.4274
hit initial : 183.912 | hit correction : 1.42744
hit initial : 192.109 | hit correction : 1.42751
hit initial : 192.12 | hit correction : 1.42759
hit initial : 130.912 | hit correction : 1.42764
 *** Total Charge : 5437.83
 Looking at view: 1 | Number of hits in view: 24
hit initial : 48.9588 | hit correction : 1.42351
hit initial : 64.2498 | hit correction : 1.42349
hit initial : 38.5366 | hit correction : 1.42773
hit initial : 24.893 | hit correction : 1.42928
hit initial : 37.0423 | hit correction : 1.42923
hit initial : 22.4297 | hit correction : 1.42399
hit initial : 274.081 | hit correction : 1.4243
hit initial : 248.202 | hit correction : 1.4251
hit initial : 145.689 | hit correction : 1.42381
hit initial : 166.392 | hit correction : 1.42402
hit initial : 22.2843 | hit correction : 1.42414
hit initial : 244.842 | hit correction : 1.42636
hit initial : 298.693 | hit correction : 1.42644
hit initial : 251.814 | hit correction : 1.42655
hit initial : 241.198 | hit correction : 1.42664
hit initial : 146.696 | hit correction : 1.42683
hit initial : 137.515 | hit correction : 1.42695
hit initial : 154.436 | hit correction : 1.42702
hit initial : 154.502 | hit correction : 1.42712
hit initial : 133.966 | hit correction : 1.42724
hit initial : 144.13 | hit correction : 1.42732
hit initial : 134.404 | hit correction : 1.42737
hit initial : 156.526 | hit correction : 1.42741
hit initial : 146.99 | hit correction : 1.42745
hit initial : 146.13 | hit correction : 1.42751
hit initial : 169.07 | hit correction : 1.42759
hit initial : 157.307 | hit correction : 1.42765
 *** Total Charge : 5578.43
 Looking at view: 2 | Number of hits in view: 27
Begin processing the 10th record. run: 5027 subRun: 0 event: 19680 at 10-Dec-2025 17:28:01 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.579
True Neutron Energy: 939.699
True Electron Energy: 53.9326
Completed MCParticle Loop
True Neutrino Energy: 62.9704
 Size of all hits: 0
10-Dec-2025 17:28:04 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0674195     0.101414      0.154749      0.0868176     0.0306499       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000445265   0.000708236   0.00147426    0.000647222   0.000292138      10     
end_path:ana:AnalyzeEvents         0.0668628     0.100526      0.152817      0.0859494     0.0303721       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19477.7 MB
  Peak resident set size usage (VmHWM): 1540.42 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 16:28:15 UTC       justIN version: 01.06.00