justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267852.3@dunegpschedd01.fnal.gov

Jobsub ID267852.3@dunegpschedd01.fnal.gov
Workflow ID11177
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 06:56:20
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 19:52:15
From worker nodeHostnamedunegli-7890498-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 06:57:34
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T234809Z_gen_001728_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time12h (46459s)
CPU time12h (43688s = 94%)
Max RSS bytes2788835328 (2659 MiB)
Outputting started2025-12-13 19:51:54
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11177/1/001/analysisLowE_20251213T065738Z.root
Finished2025-12-13 19:52:15
Saved logsjustin-logs:267852.3-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ring/EventValidationBaseAlgorithm.cc line#: 64
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6937, 6780, 6816
minwire 0: 16
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2030, 2529
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.135886, 
Output 1: 0.0795972, 0.052187, 0.113382, 0.754834, 
Output 2: 0.112474, 0.273953, 0.581099, 0.0324738, 
Output 3: 0.956127, 0.0350179, 0.00315458, 0.00570023, 
Output 4: 0.458509, 0.479668, 0.055404, 0.00641922, 
Output 5: 0.338206, 0.588546, 0.0655669, 0.0076817, 
Output 6: 0.027487, 0.075701, 0.0910865, 0.805725, 

13-Dec-2025 19:50:47 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 19:50:47 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/1a/1d/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T234809Z_gen_001728_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3013.84       4602.75       6023.66       4784.44       969.479        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00141782    0.00676452     0.0231416    0.00333949    0.00725528       10     
reco:pandora:StandardPandora                            3011.05       4545.82       5945.42       4714.66       947.326        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000677306    0.0877637     0.284778      0.0664364     0.0925728       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000638492    0.0195732      0.10866      0.0107355     0.030703        10     
reco:pandoracalo:Calorimetry                          0.000508828    0.0165276     0.128476      0.0019067     0.0376838       10     
reco:pandorapid:Chi2ParticleID                        0.000118954   0.00131207    0.00568049    0.000335562   0.00186339       10     
reco:cvnmap:CVNMapper                                  0.110784      0.229509      0.748593      0.171389      0.179179        10     
reco:cvneva:CVNEvaluator                                2.04467       3.63301       10.5303       2.68202       2.45229        10     
reco:energyrecnumu:EnergyReco                          0.0939927      52.199        74.3972       61.8711       26.4586        10     
reco:energyrecnue:EnergyReco                          0.00264758     0.0115455     0.0210079     0.0122262     0.0059262       10     
reco:energyrecnc:EnergyReco                            0.0111523     0.0178154     0.0368342     0.0141549    0.00832897       10     
[art]:TriggerResults:TriggerResultInserter            2.9786e-05    8.19192e-05   0.000386738   3.79515e-05   0.000105332      10     
end_path:out1:RootOutput                              1.1813e-05    3.21356e-05   0.000125737   1.9872e-05    3.2892e-05       10     
end_path:out1:RootOutput(write)                        0.424973      0.697848       1.29752      0.639209      0.285087        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21574.3 MB
  Peak resident set size usage (VmHWM): 2788.84 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 477.957 | hit correction : 1.24311
hit initial : 226.562 | hit correction : 1.24318
hit initial : 31.8548 | hit correction : 1.24324
hit initial : 26.754 | hit correction : 1.24364
hit initial : 28.0983 | hit correction : 1.24364
hit initial : 128.138 | hit correction : 1.25564
hit initial : 129.274 | hit correction : 1.25558
hit initial : 48.6322 | hit correction : 1.25571
hit initial : 131.624 | hit correction : 1.25629
hit initial : 126.263 | hit correction : 1.25643
hit initial : 41.5301 | hit correction : 1.25648
 *** Total Charge : 4243.55
 Looking at view: 1 | Number of hits in view: 29
hit initial : 29.9094 | hit correction : 1.22812
hit initial : 42.6942 | hit correction : 1.22811
hit initial : 25.6561 | hit correction : 1.23008
hit initial : 25.9972 | hit correction : 1.23168
hit initial : 43.5445 | hit correction : 1.23168
hit initial : 27.1922 | hit correction : 1.23535
hit initial : 33.8982 | hit correction : 1.23577
hit initial : 14.9969 | hit correction : 1.23569
hit initial : 71.5701 | hit correction : 1.24079
hit initial : 17.4553 | hit correction : 1.24081
hit initial : 50.8148 | hit correction : 1.24195
hit initial : 256.608 | hit correction : 1.24205
hit initial : 198.205 | hit correction : 1.24422
hit initial : 25.925 | hit correction : 1.24434
hit initial : 260.097 | hit correction : 1.24384
hit initial : 377.596 | hit correction : 1.24362
hit initial : 186.18 | hit correction : 1.24225
hit initial : 377.949 | hit correction : 1.24277
hit initial : 331.314 | hit correction : 1.2433
hit initial : 540.287 | hit correction : 1.24317
hit initial : 64.4028 | hit correction : 1.24365
hit initial : 116.815 | hit correction : 1.25547
hit initial : 193.808 | hit correction : 1.25566
hit initial : 25.4021 | hit correction : 1.25574
hit initial : 165.88 | hit correction : 1.25631
hit initial : 152.754 | hit correction : 1.25644
 *** Total Charge : 4551.57
 Looking at view: 2 | Number of hits in view: 26
Begin processing the 9th record. run: 5027 subRun: 0 event: 17279 at 13-Dec-2025 19:51:44 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  15
Added hit list length: 15 | Max was set to: -3.40282e+38
This pfp has size:  6
True Neutron Energy: 943.1
True Neutron Energy: 941.46
True Electron Energy: 38.5259
Completed MCParticle Loop
True Neutrino Energy: 55.8443
 Size of all hits: 15
hit initial : 26.4436 | hit correction : 1.41193
hit initial : 134.008 | hit correction : 1.4121
hit initial : 234.652 | hit correction : 1.41226
hit initial : 182.875 | hit correction : 1.41257
hit initial : 49.4204 | hit correction : 1.41282
 *** Total Charge : 886.105
 Looking at view: 0 | Number of hits in view: 5
hit initial : 76.6124 | hit correction : 1.41214
hit initial : 298.241 | hit correction : 1.41221
hit initial : 282.603 | hit correction : 1.41231
hit initial : 50.74 | hit correction : 1.41264
 *** Total Charge : 1000.17
 Looking at view: 1 | Number of hits in view: 4
hit initial : 42.4318 | hit correction : 1.4119
hit initial : 128.327 | hit correction : 1.41201
hit initial : 170.015 | hit correction : 1.41221
hit initial : 143.45 | hit correction : 1.41255
hit initial : 113.29 | hit correction : 1.41288
hit initial : 30.1452 | hit correction : 1.41299
 *** Total Charge : 886.496
 Looking at view: 2 | Number of hits in view: 6
Begin processing the 10th record. run: 5027 subRun: 0 event: 17280 at 13-Dec-2025 19:51:49 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Neutron Energy: 939.573
True Electron Energy: 14.4786
Completed MCParticle Loop
True Neutrino Energy: 17.7725
 Size of all hits: 0
13-Dec-2025 19:51:53 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0987929     0.125243      0.167559      0.125262      0.0196874       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000684299   0.00125801     0.0044637    0.000826652   0.00109959       10     
end_path:ana:AnalyzeEvents         0.0977223     0.123612      0.166515      0.123861      0.0194448       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19474.5 MB
  Peak resident set size usage (VmHWM): 1533.94 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:12:33 UTC       justIN version: 01.06.00