justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261937.1@dunegpschedd02.fnal.gov

Jobsub ID261937.1@dunegpschedd02.fnal.gov
Workflow ID11174
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 05:22:15
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 19:04:24
From worker nodeHostnamedunegli-7890499-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 05:24:04
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T231413Z_gen_001400_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time13h (49192s)
CPU time13h (47638s = 96%)
Max RSS bytes2844667904 (2712 MiB)
Outputting started2025-12-13 19:03:57
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11174/1/001/analysisLowE_20251213T052409Z.root
Finished2025-12-13 19:04:24
Saved logsjustin-logs:261937.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

uild-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 19:02:16 UTC run: 5027 subRun: 0 event: 14000
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6683, 6669, 6544
minwire 0: 15
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.102025, 
Output 1: 0.0748232, 0.0833524, 0.108031, 0.733793, 
Output 2: 0.0553234, 0.222566, 0.674065, 0.0480452, 
Output 3: 0.971893, 0.014389, 0.00387292, 0.00984498, 
Output 4: 0.905243, 0.0802737, 0.0112565, 0.00322652, 
Output 5: 0.637254, 0.24504, 0.0885435, 0.0291626, 
Output 6: 0.0105891, 0.0379703, 0.0519018, 0.899539, 

13-Dec-2025 19:02:27 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 19:02:27 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/b8/20/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T231413Z_gen_001400_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3566.11       4840.93       7451.18       3882.99       1609.28        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0012176     0.0091737     0.0343781    0.00260081     0.0119914       10     
reco:pandora:StandardPandora                            3561.82       4787.46       7375.69       3817.17       1589.8         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000393671    0.119235      0.349125      0.110424      0.106985        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000440409    0.0245837     0.174694     0.00719231     0.0506421       10     
reco:pandoracalo:Calorimetry                          0.000232467   0.00576909     0.0428736     0.0009322     0.0125889       10     
reco:pandorapid:Chi2ParticleID                        7.4901e-05    0.00188859     0.0157735    0.000258206   0.00463766       10     
reco:cvnmap:CVNMapper                                  0.0839968     0.168418      0.332728      0.120086      0.0917228       10     
reco:cvneva:CVNEvaluator                                2.47321       5.35584       17.6062       3.70009       4.26009        10     
reco:energyrecnumu:EnergyReco                          0.100237       46.9909       71.3128       63.8108       30.826         10     
reco:energyrecnue:EnergyReco                          0.00287948     0.0095583     0.0247308    0.00612949    0.00724956       10     
reco:energyrecnc:EnergyReco                            0.0114728     0.0182658     0.0316238     0.0177447    0.00548392       10     
[art]:TriggerResults:TriggerResultInserter            2.8584e-05    8.82441e-05   0.000263466   4.1989e-05    7.90783e-05      10     
end_path:out1:RootOutput                              1.2504e-05    2.95358e-05   7.4651e-05    1.8986e-05    2.12438e-05      10     
end_path:out1:RootOutput(write)                        0.423998      0.757823       1.66974      0.629068      0.342973        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21570.3 MB
  Peak resident set size usage (VmHWM): 2844.67 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 593.996 | hit correction : 1.02993
hit initial : 165.063 | hit correction : 1.02983
hit initial : 53.7523 | hit correction : 1.02941
 *** Total Charge : 3112.92
 Looking at view: 0 | Number of hits in view: 13
hit initial : 56.2834 | hit correction : 1.03273
hit initial : 151.465 | hit correction : 1.03256
hit initial : 39.6719 | hit correction : 1.0326
hit initial : 266.975 | hit correction : 1.03204
hit initial : 167.477 | hit correction : 1.0305
hit initial : 407.943 | hit correction : 1.02996
hit initial : 180.894 | hit correction : 1.02999
hit initial : 107.617 | hit correction : 1.03
hit initial : 354.714 | hit correction : 1.03039
hit initial : 456.072 | hit correction : 1.03042
hit initial : 270.719 | hit correction : 1.02964
hit initial : 105.078 | hit correction : 1.02944
 *** Total Charge : 2643.28
 Looking at view: 1 | Number of hits in view: 12
hit initial : 256.446 | hit correction : 1.03263
hit initial : 420.883 | hit correction : 1.03219
hit initial : 251.839 | hit correction : 1.03176
hit initial : 342.98 | hit correction : 1.03125
hit initial : 377.842 | hit correction : 1.03058
hit initial : 220.089 | hit correction : 1.0308
hit initial : 114.866 | hit correction : 1.03045
hit initial : 696.148 | hit correction : 1.03002
hit initial : 90.1961 | hit correction : 1.03031
hit initial : 680.116 | hit correction : 1.02986
hit initial : 135.3 | hit correction : 1.02937
 *** Total Charge : 3697.09
 Looking at view: 2 | Number of hits in view: 11
Begin processing the 10th record. run: 5027 subRun: 0 event: 14000 at 13-Dec-2025 19:03:51 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  32
Added hit list length: 32 | Max was set to: -3.40282e+38
This pfp has size:  17
This pfp has size:  7
True Neutron Energy: 939.569
True Electron Energy: 17.8317
Completed MCParticle Loop
True Neutrino Energy: 24.0186
 Size of all hits: 32
hit initial : 25.5066 | hit correction : 1.10495
hit initial : 71.6626 | hit correction : 1.10496
hit initial : 24.8201 | hit correction : 1.10497
hit initial : 281.813 | hit correction : 1.09958
hit initial : 422.324 | hit correction : 1.09935
hit initial : 115.705 | hit correction : 1.09921
hit initial : 234.272 | hit correction : 1.09861
hit initial : 373.604 | hit correction : 1.09864
hit initial : 348.997 | hit correction : 1.09886
hit initial : 158.632 | hit correction : 1.09913
 *** Total Charge : 2261.82
 Looking at view: 0 | Number of hits in view: 10
hit initial : 59.6993 | hit correction : 1.10497
hit initial : 52.002 | hit correction : 1.10495
hit initial : 263.725 | hit correction : 1.09954
hit initial : 105.747 | hit correction : 1.09957
hit initial : 187.596 | hit correction : 1.09931
hit initial : 235.339 | hit correction : 1.09861
hit initial : 200.431 | hit correction : 1.09861
hit initial : 384.269 | hit correction : 1.09883
hit initial : 214.105 | hit correction : 1.09917
 *** Total Charge : 1872.23
 Looking at view: 1 | Number of hits in view: 9
hit initial : 120.167 | hit correction : 1.10497
hit initial : 304.847 | hit correction : 1.09962
hit initial : 22.4382 | hit correction : 1.09956
hit initial : 359.549 | hit correction : 1.09943
hit initial : 255.915 | hit correction : 1.0989
hit initial : 144.984 | hit correction : 1.09859
hit initial : 136.125 | hit correction : 1.09853
hit initial : 139.931 | hit correction : 1.09855
hit initial : 153.187 | hit correction : 1.09864
hit initial : 144.972 | hit correction : 1.09912
hit initial : 151.881 | hit correction : 1.09877
hit initial : 144.721 | hit correction : 1.09892
hit initial : 112.908 | hit correction : 1.09925
 *** Total Charge : 2409.4
 Looking at view: 2 | Number of hits in view: 13
13-Dec-2025 19:03:56 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0992044     0.210099      0.410479      0.181559       0.10248        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000831346   0.00126774    0.00273348    0.00109133    0.000517147      10     
end_path:ana:AnalyzeEvents         0.0978804     0.208408      0.406746       0.18023      0.102006        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19472.2 MB
  Peak resident set size usage (VmHWM): 1533.19 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:05:55 UTC       justIN version: 01.06.00