justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261260.0@dunegpschedd02.fnal.gov

Jobsub ID261260.0@dunegpschedd02.fnal.gov
Workflow ID11091
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 22:00:38
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-12 01:59:26
From worker nodeHostnamedunegli-7878612-0-fnpc19122.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-11 22:02:15
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021920Z_gen_003922_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time3h (14218s)
CPU time3h (14116s = 99%)
Max RSS bytes3017662464 (2877 MiB)
Outputting started2025-12-12 01:59:14
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11091/1/001/analysisLowE_20251211T220216Z.root
Finished2025-12-12 01:59:26
Saved logsjustin-logs:261260.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

icial_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6597, 6586, 6565
minwire 0: 46
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1653, 2152
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.119673, 
Output 1: 0.0907832, 0.162464, 0.203456, 0.543298, 
Output 2: 0.0385005, 0.175821, 0.757275, 0.0284036, 
Output 3: 0.949101, 0.0337509, 0.00686288, 0.0102848, 
Output 4: 0.584917, 0.356623, 0.0511721, 0.00728862, 
Output 5: 0.478413, 0.320505, 0.152088, 0.0489935, 
Output 6: 0.0439345, 0.0911743, 0.0885646, 0.776327, 

12-Dec-2025 01:58:33 UTC  Closed output file "matt_reco2.root"
12-Dec-2025 01:58:33 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/66/3c/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021920Z_gen_003922_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1361.53       1409.39       1467.9        1414.98       32.552         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00116995    0.00523349     0.0182001    0.00359646    0.00461408       10     
reco:pandora:StandardPandora                            1360.15       1407.95       1466.57       1413.74       32.735         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000244222    0.0150745     0.0576238    0.00394123     0.017884        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000222622    0.0045355     0.0112584     0.0041077    0.00345188       10     
reco:pandoracalo:Calorimetry                          0.000124401   0.00348142     0.0199383    0.000748496   0.00583053       10     
reco:pandorapid:Chi2ParticleID                         4.09e-05     0.000467789   0.00295634    7.11655e-05   0.000858065      10     
reco:cvnmap:CVNMapper                                  0.0214453     0.0261905     0.0385274     0.024272     0.00522098       10     
reco:cvneva:CVNEvaluator                                0.78265      0.983971       2.44797       0.81251      0.488844        10     
reco:energyrecnumu:EnergyReco                         0.00633006     0.0598481     0.0991254     0.0711343     0.0314736       10     
reco:energyrecnue:EnergyReco                          0.000848046   0.00324233    0.00671055    0.00111951    0.00270726       10     
reco:energyrecnc:EnergyReco                           0.00622365     0.0346298     0.119325     0.00655296     0.0436758       10     
[art]:TriggerResults:TriggerResultInserter             9.78e-06     1.5883e-05     5.353e-05    1.1115e-05    1.27207e-05      10     
end_path:out1:RootOutput                               6.25e-06      9.28e-06      2.026e-05     7.895e-06    4.04993e-06      10     
end_path:out1:RootOutput(write)                         0.28182      0.299645      0.337575       0.29265      0.0187076       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21780.3 MB
  Peak resident set size usage (VmHWM): 3017.66 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 89.2318 | hit correction : 1.21067
hit initial : 44.963 | hit correction : 1.21812
hit initial : 62.0734 | hit correction : 1.20752
hit initial : 61.6518 | hit correction : 1.20754
hit initial : 157.349 | hit correction : 1.20801
hit initial : 488.46 | hit correction : 1.20853
hit initial : 401.691 | hit correction : 1.2096
hit initial : 199.603 | hit correction : 1.2102
hit initial : 205.964 | hit correction : 1.21038
hit initial : 197.157 | hit correction : 1.21057
hit initial : 195.655 | hit correction : 1.21089
hit initial : 32.3808 | hit correction : 1.21111
hit initial : 341.629 | hit correction : 1.21153
hit initial : 432.543 | hit correction : 1.21252
hit initial : 908.374 | hit correction : 1.2141
hit initial : 195.667 | hit correction : 1.21446
hit initial : 23.4264 | hit correction : 1.21521
hit initial : 41.6981 | hit correction : 1.21537
hit initial : 568.422 | hit correction : 1.21647
hit initial : 117.489 | hit correction : 1.21663
hit initial : 309.445 | hit correction : 1.21777
hit initial : 271.518 | hit correction : 1.21843
 *** Total Charge : 6484.21
 Looking at view: 2 | Number of hits in view: 22
Begin processing the 8th record. run: 5027 subRun: 0 event: 39218 at 12-Dec-2025 01:59:06 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  13
Added hit list length: 13 | Max was set to: -3.40282e+38
This pfp has size:  21
Added hit list length: 21 | Max was set to: 13
True Electron Energy: 10.3402
Completed MCParticle Loop
True Neutrino Energy: 15.7256
 Size of all hits: 21
hit initial : 52.4748 | hit correction : 1.11174
hit initial : 35.6132 | hit correction : 1.11178
hit initial : 29.8074 | hit correction : 1.11111
hit initial : 57.2249 | hit correction : 1.11039
hit initial : 103.883 | hit correction : 1.1095
hit initial : 34.5172 | hit correction : 1.11146
hit initial : 39.365 | hit correction : 1.11101
hit initial : 61.2611 | hit correction : 1.11042
hit initial : 157.941 | hit correction : 1.10949
hit initial : 110.513 | hit correction : 1.11142
hit initial : 192.404 | hit correction : 1.11016
hit initial : 66.8885 | hit correction : 1.10947
hit initial : 32.4998 | hit correction : 1.1117
 *** Total Charge : 1081.98
 Looking at view: 0 | Number of hits in view: 13
hit initial : 86.8623 | hit correction : 1.11161
hit initial : 150.532 | hit correction : 1.11146
hit initial : 15.2287 | hit correction : 1.11162
hit initial : 51.8909 | hit correction : 1.11115
hit initial : 54.3802 | hit correction : 1.11044
 *** Total Charge : 398.841
 Looking at view: 1 | Number of hits in view: 5
hit initial : 37.6913 | hit correction : 1.11168
hit initial : 54.8132 | hit correction : 1.1117
hit initial : 12.851 | hit correction : 1.11171
 *** Total Charge : 117.123
 Looking at view: 2 | Number of hits in view: 3
Begin processing the 9th record. run: 5027 subRun: 0 event: 39219 at 12-Dec-2025 01:59:08 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 941.333
True Neutron Energy: 939.823
True Electron Energy: 9.01997
Completed MCParticle Loop
True Neutrino Energy: 23.5355
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 39220 at 12-Dec-2025 01:59:11 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.568
True Neutron Energy: 943.383
True Neutron Energy: 941.653
True Neutron Energy: 939.881
True Electron Energy: 46.2692
Completed MCParticle Loop
True Neutrino Energy: 58.8996
 Size of all hits: 0
12-Dec-2025 01:59:14 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0663858     0.0776755     0.0998336     0.0788157    0.00958296       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000319503   0.000558046   0.00144203    0.000418923   0.000331993      10     
end_path:ana:AnalyzeEvents         0.0659449     0.0769729     0.097929      0.0783549    0.00927199       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19472.6 MB
  Peak resident set size usage (VmHWM): 1531.14 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 16:25:18 UTC       justIN version: 01.06.00