justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262297.1@dunegpschedd02.fnal.gov

Jobsub ID262297.1@dunegpschedd02.fnal.gov
Workflow ID11170
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 20:09:00
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-14 09:13:21
From worker nodeHostnamedunegli-8066212-0-fnpc17126.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-14 03:30:52
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T180314Z_gen_001342_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (20536s)
CPU time5h (19840s = 96%)
Max RSS bytes2793066496 (2663 MiB)
Outputting started2025-12-14 09:13:09
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11170/1/001/analysisLowE_20251214T033053Z.root
Finished2025-12-14 09:13:21
Saved logsjustin-logs:262297.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

E/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  14-Dec-2025 09:11:45 UTC run: 5026 subRun: 0 event: 13420
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6656, 6554, 6644
minwire 0: 47
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1133, 1632
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.055637, 
Output 1: 0.0996304, 0.298827, 0.10681, 0.494732, 
Output 2: 0.044196, 0.206125, 0.73286, 0.01682, 
Output 3: 0.930793, 0.0608235, 0.00420984, 0.00417387, 
Output 4: 0.830302, 0.159158, 0.00925092, 0.00128865, 
Output 5: 0.204111, 0.577973, 0.184856, 0.0330597, 
Output 6: 0.0911527, 0.284245, 0.159622, 0.46498, 

14-Dec-2025 09:11:50 UTC  Closed output file "matt_reco2.root"
14-Dec-2025 09:11:50 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/1b/6d/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T180314Z_gen_001342_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1834.77       1983.06       2275.02       1887.81       163.726        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000814515    0.0162417     0.120113     0.000984329    0.0353046       10     
reco:pandora:StandardPandora                            1833.02       1980.88       2272.77       1884.28        163.7         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00240524     0.0583145     0.0770473     0.0637805     0.0201653       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00142249    0.00352038     0.0164115    0.00190982    0.00437471       10     
reco:pandoracalo:Calorimetry                          0.000264766   0.000903608   0.00431387    0.000512215   0.00115275       10     
reco:pandorapid:Chi2ParticleID                        5.9522e-05    0.000279973   0.00181107    0.000118464   0.000510787      10     
reco:cvnmap:CVNMapper                                   0.11419      0.128929      0.161356      0.123989      0.0137821       10     
reco:cvneva:CVNEvaluator                               0.926067       1.43073       4.6028       0.963649       1.07298        10     
reco:energyrecnumu:EnergyReco                          0.0839058     0.100868      0.128714      0.0947251     0.0152494       10     
reco:energyrecnue:EnergyReco                          0.00171996    0.00201857    0.00293469     0.0018609    0.000351825      10     
reco:energyrecnc:EnergyReco                           0.00936365     0.0108159     0.0149805    0.00999198    0.00183966       10     
[art]:TriggerResults:TriggerResultInserter            1.4875e-05    2.82655e-05   9.1773e-05    1.91725e-05   2.21962e-05      10     
end_path:out1:RootOutput                               6.847e-06    1.09729e-05   2.7178e-05    7.8175e-06    6.89961e-06      10     
end_path:out1:RootOutput(write)                        0.356762       0.42529      0.598677      0.371722      0.0915784       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14055.5 MB
  Peak resident set size usage (VmHWM): 2793.07 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 113.387 | hit correction : 1.35034
hit initial : 106.493 | hit correction : 1.35029
hit initial : 114.335 | hit correction : 1.35025
hit initial : 107.993 | hit correction : 1.35021
hit initial : 93.7819 | hit correction : 1.35014
hit initial : 106.208 | hit correction : 1.35007
hit initial : 121.873 | hit correction : 1.34995
hit initial : 109.695 | hit correction : 1.34981
hit initial : 123.866 | hit correction : 1.34969
hit initial : 129.818 | hit correction : 1.34954
hit initial : 125.612 | hit correction : 1.3494
hit initial : 110.717 | hit correction : 1.34922
hit initial : 110.839 | hit correction : 1.34905
 *** Total Charge : 8392.32
 Looking at view: 2 | Number of hits in view: 45
Begin processing the 8th record. run: 5026 subRun: 0 event: 13418 at 14-Dec-2025 09:12:50 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
True Electron Energy: 60.6474
Completed MCParticle Loop
True Neutrino Energy: 66.0941
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 13419 at 14-Dec-2025 09:12:55 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  6
This pfp has size:  17
Added hit list length: 17 | Max was set to: -3.40282e+38
This pfp has size:  7
This pfp has size:  6
True Electron Energy: 9.17665
Completed MCParticle Loop
True Neutrino Energy: 14.5561
 Size of all hits: 17
hit initial : 40.9773 | hit correction : 1.13142
hit initial : 161.511 | hit correction : 1.13139
hit initial : 216.183 | hit correction : 1.13131
hit initial : 276.865 | hit correction : 1.13106
hit initial : 249.757 | hit correction : 1.13073
hit initial : 106.1 | hit correction : 1.13055
 *** Total Charge : 1189.18
 Looking at view: 0 | Number of hits in view: 6
hit initial : 73.1229 | hit correction : 1.1314
hit initial : 335.258 | hit correction : 1.13127
hit initial : 91.9998 | hit correction : 1.13133
hit initial : 401.035 | hit correction : 1.13088
hit initial : 161.749 | hit correction : 1.1309
 *** Total Charge : 1202.53
 Looking at view: 1 | Number of hits in view: 5
hit initial : 44.102 | hit correction : 1.1314
hit initial : 346.019 | hit correction : 1.13137
hit initial : 302.673 | hit correction : 1.13114
hit initial : 166.398 | hit correction : 1.13084
hit initial : 198.291 | hit correction : 1.13057
hit initial : 47.7585 | hit correction : 1.13038
 *** Total Charge : 1250.07
 Looking at view: 2 | Number of hits in view: 6
Begin processing the 10th record. run: 5026 subRun: 0 event: 13420 at 14-Dec-2025 09:13:03 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  10
Added hit list length: 10 | Max was set to: -3.40282e+38
True Neutron Energy: 939.566
True Electron Energy: 24.5662
Completed MCParticle Loop
True Neutrino Energy: 27.8801
 Size of all hits: 10
hit initial : 35.3696 | hit correction : 1.01836
hit initial : 150.425 | hit correction : 1.02725
hit initial : 138.752 | hit correction : 1.02722
 *** Total Charge : 333.073
 Looking at view: 0 | Number of hits in view: 3
hit initial : 39.0408 | hit correction : 1.01837
hit initial : 181.462 | hit correction : 1.02723
hit initial : 95.7507 | hit correction : 1.0272
 *** Total Charge : 324.517
 Looking at view: 1 | Number of hits in view: 3
hit initial : 43.2189 | hit correction : 1.01836
hit initial : 12.0301 | hit correction : 1.01795
hit initial : 41.3327 | hit correction : 1.01835
hit initial : 347.827 | hit correction : 1.02722
 *** Total Charge : 455.646
 Looking at view: 2 | Number of hits in view: 4
14-Dec-2025 09:13:07 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.104752      0.158045      0.282082      0.129254      0.0540552       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000467442   0.00441456     0.0385987    0.000547876    0.0113961       10     
end_path:ana:AnalyzeEvents         0.104037      0.153423      0.281454      0.128419      0.0538186       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11956.2 MB
  Peak resident set size usage (VmHWM): 1535.73 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:07:51 UTC       justIN version: 01.06.00