justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262005.0@dunegpschedd02.fnal.gov

Jobsub ID262005.0@dunegpschedd02.fnal.gov
Workflow ID11173
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 06:42:19
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-13 16:42:00
From worker nodeHostnamedunegli-8047001-0-fnpc19108.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 06:43:45
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T005556Z_gen_002852_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time9h (35853s)
CPU time6h (23546s = 65%)
Max RSS bytes2832072704 (2700 MiB)
Outputting started2025-12-13 16:41:19
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11173/1/001/analysisLowE_20251213T064347Z.root
Finished2025-12-13 16:42:00
Saved logsjustin-logs:262005.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e#: 64
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6626, 6595, 6638
minwire 0: 33
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1402, 1901
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.108712, 
Output 1: 0.0636114, 0.043609, 0.083898, 0.808882, 
Output 2: 0.0496315, 0.228774, 0.703217, 0.0183776, 
Output 3: 0.967212, 0.0272819, 0.00196203, 0.00354352, 
Output 4: 0.243354, 0.650281, 0.0989382, 0.00742719, 
Output 5: 0.448612, 0.436294, 0.0978544, 0.0172397, 
Output 6: 0.031574, 0.0967078, 0.107993, 0.763725, 

13-Dec-2025 16:40:01 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 16:40:01 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/12/3b/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T005556Z_gen_002852_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2568.64       3546.2        6301.39       2920.09       1210.29        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000856148    0.0075668     0.0221232    0.00170365    0.00839179       10     
reco:pandora:StandardPandora                            2566.55       3542.36       6297.16       2916.9        1209.17        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000300783    0.0772084      0.21762      0.0508843     0.0727598       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000280632    0.0129559     0.092043     0.00312122     0.0267524       10     
reco:pandoracalo:Calorimetry                          0.000151572   0.00317622     0.0282143    0.000398908   0.00834912       10     
reco:pandorapid:Chi2ParticleID                        5.3351e-05    0.00089682    0.00811564    7.85955e-05   0.00240699       10     
reco:cvnmap:CVNMapper                                  0.0737171     0.198848      0.418977      0.168569      0.105319        10     
reco:cvneva:CVNEvaluator                                1.22231       2.71488       8.70789       2.2566        2.04308        10     
reco:energyrecnumu:EnergyReco                          0.101364      0.154123      0.324104      0.135972      0.0623034       10     
reco:energyrecnue:EnergyReco                          0.00101663    0.00491077     0.0205161    0.00111718    0.00614498       10     
reco:energyrecnc:EnergyReco                           0.00647092     0.0166109     0.0307337     0.0180115    0.00770158       10     
[art]:TriggerResults:TriggerResultInserter             1.07e-05     3.74574e-05   0.000188262   1.33205e-05   5.25454e-05      10     
end_path:out1:RootOutput                                 7e-06      1.70032e-05    6.919e-05     8.82e-06     1.89699e-05      10     
end_path:out1:RootOutput(write)                        0.566524      0.650755      0.887105      0.579069      0.118315        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21571.8 MB
  Peak resident set size usage (VmHWM): 2832.07 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 15.6254 | hit correction : 1.12399
 *** Total Charge : 3831.89
 Looking at view: 2 | Number of hits in view: 22
Begin processing the 8th record. run: 5027 subRun: 0 event: 28518 at 13-Dec-2025 16:41:08 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Electron Energy: 25.9112
Completed MCParticle Loop
True Neutrino Energy: 30.4347
 Size of all hits: 0
Begin processing the 9th record. run: 5027 subRun: 0 event: 28519 at 13-Dec-2025 16:41:11 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  47
Added hit list length: 47 | Max was set to: -3.40282e+38
This pfp has size:  23
True Neutron Energy: 941.434
True Electron Energy: 33.1547
Completed MCParticle Loop
True Neutrino Energy: 50.4313
 Size of all hits: 47
hit initial : 61.3552 | hit correction : 1.43204
hit initial : 136.825 | hit correction : 1.43219
hit initial : 160.888 | hit correction : 1.43209
hit initial : 49.4261 | hit correction : 1.4322
hit initial : 108.753 | hit correction : 1.41143
hit initial : 203.631 | hit correction : 1.41143
hit initial : 96.5551 | hit correction : 1.41144
hit initial : 50.7555 | hit correction : 1.40778
hit initial : 169.723 | hit correction : 1.40799
hit initial : 405.53 | hit correction : 1.40873
hit initial : 284.602 | hit correction : 1.40861
hit initial : 465.199 | hit correction : 1.4086
hit initial : 53.8274 | hit correction : 1.40762
hit initial : 467.96 | hit correction : 1.40824
hit initial : 318.116 | hit correction : 1.40792
 *** Total Charge : 4282.73
 Looking at view: 0 | Number of hits in view: 15
hit initial : 85.3302 | hit correction : 1.43215
hit initial : 202.465 | hit correction : 1.43213
hit initial : 109.365 | hit correction : 1.43208
hit initial : 118.596 | hit correction : 1.41145
hit initial : 187.259 | hit correction : 1.41143
hit initial : 71.3253 | hit correction : 1.41142
hit initial : 67.9593 | hit correction : 1.40792
hit initial : 214.505 | hit correction : 1.40811
hit initial : 154.574 | hit correction : 1.40882
hit initial : 93.7842 | hit correction : 1.40801
hit initial : 166.031 | hit correction : 1.40883
hit initial : 187.948 | hit correction : 1.40874
hit initial : 233.271 | hit correction : 1.40864
hit initial : 180.887 | hit correction : 1.40857
hit initial : 198.829 | hit correction : 1.40834
hit initial : 269.651 | hit correction : 1.40813
hit initial : 244.954 | hit correction : 1.40788
 *** Total Charge : 3935.34
 Looking at view: 1 | Number of hits in view: 17
hit initial : 102.558 | hit correction : 1.43226
hit initial : 152.527 | hit correction : 1.43217
hit initial : 116.992 | hit correction : 1.4319
hit initial : 24.161 | hit correction : 1.41154
hit initial : 220.457 | hit correction : 1.41145
hit initial : 161.551 | hit correction : 1.41141
hit initial : 132.731 | hit correction : 1.4078
hit initial : 225.951 | hit correction : 1.40883
hit initial : 24.4858 | hit correction : 1.40844
hit initial : 227.861 | hit correction : 1.40876
hit initial : 324.012 | hit correction : 1.40844
hit initial : 307.086 | hit correction : 1.40861
hit initial : 237.611 | hit correction : 1.40847
hit initial : 262.164 | hit correction : 1.4082
hit initial : 278.293 | hit correction : 1.40792
 *** Total Charge : 3951.39
 Looking at view: 2 | Number of hits in view: 15
Begin processing the 10th record. run: 5027 subRun: 0 event: 28520 at 13-Dec-2025 16:41:15 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Neutron Energy: 939.577
True Electron Energy: 31.1992
Completed MCParticle Loop
True Neutrino Energy: 35.7445
 Size of all hits: 0
13-Dec-2025 16:41:18 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0748294     0.120926      0.226538      0.100516      0.0439639       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000458384   0.000879297   0.00177171    0.000729516   0.000391124      10     
end_path:ana:AnalyzeEvents         0.073954      0.119702      0.224002      0.0996735     0.0434627       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19472.5 MB
  Peak resident set size usage (VmHWM): 1529.92 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:04:28 UTC       justIN version: 01.06.00