justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262084.1@dunegpschedd02.fnal.gov

Jobsub ID262084.1@dunegpschedd02.fnal.gov
Workflow ID11172
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 08:42:25
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 19:52:22
From worker nodeHostnamedunegli-7890501-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 08:44:30
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205019Z_gen_003233_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time11h (40048s)
CPU time10h (36407s = 90%)
Max RSS bytes2836729856 (2705 MiB)
Outputting started2025-12-13 19:51:59
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11172/1/001/analysisLowE_20251213T084435Z.root
Finished2025-12-13 19:52:22
Saved logsjustin-logs:262084.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

entValidationBaseAlgorithm.cc line#: 64
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 19:49:34 UTC run: 5026 subRun: 0 event: 32330
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6749, 6705, 6643
minwire 0: 31
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.133756, 
Output 1: 0.0625217, 0.0570202, 0.0782866, 0.802172, 
Output 2: 0.081071, 0.337163, 0.544231, 0.0375349, 
Output 3: 0.974087, 0.0173285, 0.00202222, 0.006562, 
Output 4: 0.379876, 0.55861, 0.0574271, 0.00408686, 
Output 5: 0.5424, 0.390619, 0.0572699, 0.00971031, 
Output 6: 0.0151001, 0.0486382, 0.0732205, 0.863041, 

13-Dec-2025 19:50:40 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 19:50:40 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/70/62/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205019Z_gen_003233_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3226.41       3956.15       5392.81       3853.02       537.795        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00111029    0.00797413     0.0259611    0.00368198    0.00843587       10     
reco:pandora:StandardPandora                            3223.59       3897.67       5311.12       3777.35       521.031        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000824974    0.0858959     0.153914      0.120201      0.0570167       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000721378   0.00824697     0.0326424    0.00408323    0.00925909       10     
reco:pandoracalo:Calorimetry                          0.000439568   0.00152002    0.00516889    0.000888023   0.00145056       10     
reco:pandorapid:Chi2ParticleID                        0.00016483    0.000564969   0.00201817    0.000224092   0.000657864      10     
reco:cvnmap:CVNMapper                                  0.102241      0.137985      0.193536      0.139308      0.0269036       10     
reco:cvneva:CVNEvaluator                                2.0381        3.43086       11.7544       2.46799       2.79178        10     
reco:energyrecnumu:EnergyReco                          0.121871       54.202        74.3776       66.3263       27.3941        10     
reco:energyrecnue:EnergyReco                          0.00255827    0.00878796     0.0181227    0.00560677    0.00647444       10     
reco:energyrecnc:EnergyReco                            0.0105282     0.0152038     0.0249369     0.0134224    0.00484605       10     
[art]:TriggerResults:TriggerResultInserter            2.5618e-05    5.05551e-05   0.000176953   3.65185e-05   4.27201e-05      10     
end_path:out1:RootOutput                              1.1672e-05    2.10585e-05   6.8448e-05    1.5945e-05    1.61615e-05      10     
end_path:out1:RootOutput(write)                        0.407629      0.574109       1.17554      0.459183      0.228114        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21576.2 MB
  Peak resident set size usage (VmHWM): 2836.73 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Number of reconstructed particles:  1
True Neutron Energy: 940.401
True Electron Energy: 56.5959
Completed MCParticle Loop
True Neutrino Energy: 69.2812
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 32329 at 13-Dec-2025 19:51:48 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  28
Added hit list length: 28 | Max was set to: -3.40282e+38
This pfp has size:  23
True Electron Energy: 17.8438
Completed MCParticle Loop
True Neutrino Energy: 25.6346
 Size of all hits: 28
hit initial : 79.7081 | hit correction : 1.19041
hit initial : 161.306 | hit correction : 1.19039
hit initial : 99.2729 | hit correction : 1.19041
hit initial : 53.4679 | hit correction : 1.19105
hit initial : 89.5659 | hit correction : 1.19199
hit initial : 257.057 | hit correction : 1.19226
hit initial : 255.032 | hit correction : 1.19268
hit initial : 76.657 | hit correction : 1.19319
hit initial : 300.77 | hit correction : 1.19312
hit initial : 87 | hit correction : 1.19327
 *** Total Charge : 1740.31
 Looking at view: 0 | Number of hits in view: 10
hit initial : 117.99 | hit correction : 1.19041
hit initial : 227.431 | hit correction : 1.1904
hit initial : 50.9114 | hit correction : 1.19039
hit initial : 85.3537 | hit correction : 1.19145
hit initial : 299.807 | hit correction : 1.19156
hit initial : 179.893 | hit correction : 1.19233
hit initial : 301.607 | hit correction : 1.19269
hit initial : 339.159 | hit correction : 1.19304
hit initial : 185.131 | hit correction : 1.19322
hit initial : 34.2539 | hit correction : 1.19337
 *** Total Charge : 2171.35
 Looking at view: 1 | Number of hits in view: 10
hit initial : 11.9781 | hit correction : 1.19047
hit initial : 258.835 | hit correction : 1.19037
hit initial : 398.422 | hit correction : 1.19074
hit initial : 215.562 | hit correction : 1.19173
hit initial : 289.059 | hit correction : 1.19231
hit initial : 281.611 | hit correction : 1.19265
hit initial : 160.838 | hit correction : 1.19316
hit initial : 358.632 | hit correction : 1.19317
 *** Total Charge : 2354
 Looking at view: 2 | Number of hits in view: 8
Begin processing the 10th record. run: 5026 subRun: 0 event: 32330 at 13-Dec-2025 19:51:52 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  15
Added hit list length: 15 | Max was set to: -3.40282e+38
This pfp has size:  8
True Neutron Energy: 939.566
True Electron Energy: 8.24533
Completed MCParticle Loop
True Neutrino Energy: 11.9715
 Size of all hits: 15
hit initial : 161.286 | hit correction : 1.38211
hit initial : 271.127 | hit correction : 1.38182
hit initial : 220.354 | hit correction : 1.38165
hit initial : 86.3498 | hit correction : 1.38162
hit initial : 13.9456 | hit correction : 1.38164
 *** Total Charge : 1040.59
 Looking at view: 0 | Number of hits in view: 5
hit initial : 125.01 | hit correction : 1.38198
hit initial : 353.184 | hit correction : 1.38184
hit initial : 248.61 | hit correction : 1.3817
hit initial : 51.3556 | hit correction : 1.38166
 *** Total Charge : 1075.27
 Looking at view: 1 | Number of hits in view: 4
hit initial : 94.9198 | hit correction : 1.38232
hit initial : 187.548 | hit correction : 1.38212
hit initial : 132.245 | hit correction : 1.38175
hit initial : 125.897 | hit correction : 1.3816
hit initial : 130.285 | hit correction : 1.38159
hit initial : 84.0361 | hit correction : 1.38169
 *** Total Charge : 1043.2
 Looking at view: 2 | Number of hits in view: 6
13-Dec-2025 19:51:57 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.102593      0.148822      0.310788      0.131885      0.0558524       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000749001   0.00177171    0.00475571    0.00103152    0.00147237       10     
end_path:ana:AnalyzeEvents         0.101523      0.146656      0.309447       0.13059      0.0558496       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.6 MB
  Peak resident set size usage (VmHWM): 1537.15 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:04:37 UTC       justIN version: 01.06.00