justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262292.1@dunegpschedd02.fnal.gov

Jobsub ID262292.1@dunegpschedd02.fnal.gov
Workflow ID11168
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 19:26:59
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-14 03:07:15
From worker nodeHostnamedunegli-8047129-0-fnpc19101.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 19:29:28
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T181604Z_gen_001461_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time7h (27456s)
CPU time7h (27114s = 98%)
Max RSS bytes2794450944 (2664 MiB)
Outputting started2025-12-14 03:07:05
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11168/1/001/analysisLowE_20251213T192930Z.root
Finished2025-12-14 03:07:15
Saved logsjustin-logs:262292.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  14-Dec-2025 03:06:25 UTC run: 5026 subRun: 0 event: 14610
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6516, 6478, 6686
minwire 0: 29
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2275, 2774
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0924376, 
Output 1: 0.0623483, 0.0482249, 0.104242, 0.785185, 
Output 2: 0.101038, 0.334357, 0.52836, 0.0362442, 
Output 3: 0.936598, 0.0476619, 0.0064497, 0.00928979, 
Output 4: 0.348807, 0.558861, 0.0845929, 0.00773917, 
Output 5: 0.766479, 0.218706, 0.0129145, 0.00190048, 
Output 6: 0.0132956, 0.0528972, 0.0730668, 0.86074, 

14-Dec-2025 03:06:27 UTC  Closed output file "matt_reco2.root"
14-Dec-2025 03:06:27 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/67/41/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T181604Z_gen_001461_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1542.44       2724.12       3335.22       2897.09       552.539        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000770377   0.00655953     0.055834     0.00102512     0.0164268       10     
reco:pandora:StandardPandora                            1541.13       2722.05       3333.43       2893.25       552.159        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00236533     0.0545484     0.107449      0.0586204     0.0326704       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00130979    0.00875517     0.0260816     0.0055573    0.00811694       10     
reco:pandoracalo:Calorimetry                          0.000313773   0.00152921    0.00468093    0.000882174   0.00149909       10     
reco:pandorapid:Chi2ParticleID                        7.9641e-05    0.00054993    0.00239254    0.000143651   0.000825071      10     
reco:cvnmap:CVNMapper                                  0.0517716     0.0891333     0.168995      0.0844029     0.0302004       10     
reco:cvneva:CVNEvaluator                               0.822231       1.45943       5.10571       1.10937       1.22332        10     
reco:energyrecnumu:EnergyReco                          0.0644811     0.112303      0.157061      0.114561      0.0289937       10     
reco:energyrecnue:EnergyReco                          0.00109864    0.00273365    0.00931281     0.0021004    0.00231301       10     
reco:energyrecnc:EnergyReco                           0.00692254    0.00815681     0.0100896    0.00794485    0.000929752      10     
[art]:TriggerResults:TriggerResultInserter             1.401e-05    3.75753e-05   0.000170381   2.13405e-05   4.4636e-05       10     
end_path:out1:RootOutput                               6.48e-06     1.47043e-05   4.5621e-05     1.128e-05    1.08653e-05      10     
end_path:out1:RootOutput(write)                        0.277428      0.328652      0.512389      0.309705      0.0643865       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21578.4 MB
  Peak resident set size usage (VmHWM): 2794.45 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 622.6 | hit correction : 1.35537
hit initial : 690.909 | hit correction : 1.3549
hit initial : 147.639 | hit correction : 1.35489
hit initial : 54.1845 | hit correction : 1.35375
 *** Total Charge : 2430.19
 Looking at view: 1 | Number of hits in view: 8
hit initial : 56.6502 | hit correction : 1.3579
hit initial : 18.626 | hit correction : 1.3579
hit initial : 79.2375 | hit correction : 1.35584
hit initial : 189.703 | hit correction : 1.35573
hit initial : 170.712 | hit correction : 1.35549
hit initial : 179.553 | hit correction : 1.35546
hit initial : 160.396 | hit correction : 1.35534
hit initial : 153.117 | hit correction : 1.35516
hit initial : 147.554 | hit correction : 1.35498
hit initial : 121.757 | hit correction : 1.35476
hit initial : 142.768 | hit correction : 1.35451
hit initial : 206.144 | hit correction : 1.35422
hit initial : 170.885 | hit correction : 1.35378
 *** Total Charge : 2435.27
 Looking at view: 2 | Number of hits in view: 13
Begin processing the 9th record. run: 5026 subRun: 0 event: 14609 at 14-Dec-2025 03:07:00 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  9
Added hit list length: 9 | Max was set to: -3.40282e+38
True Electron Energy: 58.7472
Completed MCParticle Loop
True Neutrino Energy: 64.356
 Size of all hits: 9
hit initial : 50.8823 | hit correction : 1.2348
hit initial : 84.7948 | hit correction : 1.2348
 *** Total Charge : 167.534
 Looking at view: 0 | Number of hits in view: 2
hit initial : 35.6975 | hit correction : 1.23481
hit initial : 80.5383 | hit correction : 1.2348
hit initial : 35.1102 | hit correction : 1.23479
 *** Total Charge : 186.882
 Looking at view: 1 | Number of hits in view: 3
hit initial : 14.5967 | hit correction : 1.23565
hit initial : 30.7601 | hit correction : 1.23564
hit initial : 131.161 | hit correction : 1.23479
hit initial : 35.1944 | hit correction : 1.23474
 *** Total Charge : 261.457
 Looking at view: 2 | Number of hits in view: 4
Begin processing the 10th record. run: 5026 subRun: 0 event: 14610 at 14-Dec-2025 03:07:02 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  19
Added hit list length: 19 | Max was set to: -3.40282e+38
This pfp has size:  19
This pfp has size:  4
True Neutron Energy: 939.567
True Electron Energy: 13.1446
Completed MCParticle Loop
True Neutrino Energy: 19.8335
 Size of all hits: 19
hit initial : 51.6489 | hit correction : 1.46637
hit initial : 205.306 | hit correction : 1.46637
hit initial : 379.728 | hit correction : 1.46633
hit initial : 403.079 | hit correction : 1.46629
hit initial : 136.606 | hit correction : 1.46624
 *** Total Charge : 1724.92
 Looking at view: 0 | Number of hits in view: 5
hit initial : 14.7435 | hit correction : 1.4664
hit initial : 99.0452 | hit correction : 1.46639
hit initial : 181.399 | hit correction : 1.46641
hit initial : 184.328 | hit correction : 1.46641
hit initial : 165.552 | hit correction : 1.46638
hit initial : 169.831 | hit correction : 1.46631
hit initial : 183.631 | hit correction : 1.46617
 *** Total Charge : 1464.19
 Looking at view: 1 | Number of hits in view: 7
hit initial : 46.4664 | hit correction : 1.46641
hit initial : 327.271 | hit correction : 1.4664
hit initial : 240.347 | hit correction : 1.4664
hit initial : 112.334 | hit correction : 1.46636
hit initial : 100.758 | hit correction : 1.46631
hit initial : 103.122 | hit correction : 1.46623
hit initial : 110.523 | hit correction : 1.46608
 *** Total Charge : 1526.19
 Looking at view: 2 | Number of hits in view: 7
14-Dec-2025 03:07:05 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0772435     0.0863165     0.0947119     0.0862411     0.0054427       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000377134   0.000521893   0.000766918   0.00048964    0.000113007      10     
end_path:ana:AnalyzeEvents         0.0761154     0.0856055     0.0940581     0.0854035    0.00551509       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.8 MB
  Peak resident set size usage (VmHWM): 1537.53 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:04:37 UTC       justIN version: 01.06.00