justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268020.0@dunegpschedd01.fnal.gov

Jobsub ID268020.0@dunegpschedd01.fnal.gov
Workflow ID11177
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 12:10:36
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 19:26:24
From worker nodeHostnamedunegli-7890455-0-fnpc18002.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 12:11:56
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T131553Z_gen_000301_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time7h (26051s)
CPU time7h (25927s = 99%)
Max RSS bytes2851905536 (2719 MiB)
Outputting started2025-12-13 19:26:07
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11177/1/001/analysisLowE_20251213T121158Z.root
Finished2025-12-13 19:26:24
Saved logsjustin-logs:268020.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 19:25:02 UTC run: 4696 subRun: 0 event: 3010
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6643, 6554, 6625
minwire 0: 48
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 874, 1373
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0644671, 
Output 1: 0.116596, 0.120327, 0.0784475, 0.684629, 
Output 2: 0.0366641, 0.196989, 0.744154, 0.0221927, 
Output 3: 0.975255, 0.0154619, 0.00203209, 0.00725081, 
Output 4: 0.924045, 0.0640309, 0.00900465, 0.00291984, 
Output 5: 0.693473, 0.200828, 0.074661, 0.0310374, 
Output 6: 0.0172261, 0.0451403, 0.056235, 0.881399, 

13-Dec-2025 19:25:06 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 19:25:06 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/0a/94/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T131553Z_gen_000301_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1676.89       2575.42       4190.77       2337.66       761.212        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00103486    0.00513285     0.0215752    0.00121439    0.00785742       10     
reco:pandora:StandardPandora                            1674.4        2565.7        4120.48       2334.32       746.977        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00499485     0.055183      0.118875      0.0595044     0.0318443       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00171295     0.0147997     0.0485012     0.0102681     0.0142155       10     
reco:pandoracalo:Calorimetry                          0.000393981   0.00149764    0.00810326    0.000685434   0.00221994       10     
reco:pandorapid:Chi2ParticleID                        9.0865e-05    0.000639526   0.00405288    0.000231074   0.00114691       10     
reco:cvnmap:CVNMapper                                  0.0406579     0.102314      0.275399      0.0855449     0.0647557       10     
reco:cvneva:CVNEvaluator                                1.86643       2.72046       6.88797       2.01934       1.47084        10     
reco:energyrecnumu:EnergyReco                          0.0825402      6.34309       62.3178      0.096279       18.6584        10     
reco:energyrecnue:EnergyReco                          0.00144951    0.00351169     0.0151937    0.00193418    0.00400585       10     
reco:energyrecnc:EnergyReco                           0.00922632     0.0116462     0.0156356     0.010382     0.00229485       10     
[art]:TriggerResults:TriggerResultInserter            2.2769e-05    4.08508e-05   0.00014632    2.81115e-05   3.56216e-05      10     
end_path:out1:RootOutput                               9.988e-06    1.97237e-05   4.6306e-05    1.41435e-05   1.22165e-05      10     
end_path:out1:RootOutput(write)                        0.376876      0.460628      0.742841      0.393839      0.119942        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21578.4 MB
  Peak resident set size usage (VmHWM): 2851.91 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 211.138 | hit correction : 1.32065
hit initial : 49.4323 | hit correction : 1.31984
hit initial : 99.3829 | hit correction : 1.31992
hit initial : 87.4676 | hit correction : 1.31986
 *** Total Charge : 1462.15
 Looking at view: 1 | Number of hits in view: 7
hit initial : 157.826 | hit correction : 1.32136
hit initial : 291.512 | hit correction : 1.32116
hit initial : 221.841 | hit correction : 1.3208
hit initial : 115.92 | hit correction : 1.31984
hit initial : 180.231 | hit correction : 1.31995
hit initial : 238.517 | hit correction : 1.32051
 *** Total Charge : 1592.54
 Looking at view: 2 | Number of hits in view: 6
Begin processing the 9th record. run: 4696 subRun: 0 event: 3009 at 13-Dec-2025 19:25:59 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  32
Added hit list length: 32 | Max was set to: -3.40282e+38
This pfp has size:  6
This pfp has size:  6
True Electron Energy: 25.3087
Completed MCParticle Loop
True Neutrino Energy: 30.181
 Size of all hits: 32
hit initial : 305.215 | hit correction : 1.10171
hit initial : 452.857 | hit correction : 1.1014
hit initial : 390.507 | hit correction : 1.10055
hit initial : 400.259 | hit correction : 1.09983
hit initial : 136.468 | hit correction : 1.09958
hit initial : 120.243 | hit correction : 1.09851
hit initial : 597.944 | hit correction : 1.09847
hit initial : 453.338 | hit correction : 1.09755
 *** Total Charge : 3141.56
 Looking at view: 0 | Number of hits in view: 8
hit initial : 257.356 | hit correction : 1.10171
hit initial : 147.958 | hit correction : 1.1017
hit initial : 196.329 | hit correction : 1.10099
hit initial : 432.642 | hit correction : 1.10074
hit initial : 197.053 | hit correction : 1.09993
hit initial : 226.627 | hit correction : 1.09968
hit initial : 237.023 | hit correction : 1.09935
hit initial : 323.426 | hit correction : 1.09873
hit initial : 351.136 | hit correction : 1.09808
hit initial : 354.417 | hit correction : 1.09746
 *** Total Charge : 2995.34
 Looking at view: 1 | Number of hits in view: 10
hit initial : 33.5433 | hit correction : 1.11339
hit initial : 25.6414 | hit correction : 1.10673
hit initial : 155.277 | hit correction : 1.10178
hit initial : 388.251 | hit correction : 1.10169
hit initial : 165.347 | hit correction : 1.10129
hit initial : 293.188 | hit correction : 1.10083
hit initial : 230.675 | hit correction : 1.10026
hit initial : 376.576 | hit correction : 1.09985
hit initial : 247.026 | hit correction : 1.09946
hit initial : 263.693 | hit correction : 1.09902
hit initial : 54.9946 | hit correction : 1.09858
hit initial : 355.403 | hit correction : 1.0984
hit initial : 382.189 | hit correction : 1.09769
hit initial : 172.016 | hit correction : 1.09722
 *** Total Charge : 3457.81
 Looking at view: 2 | Number of hits in view: 14
Begin processing the 10th record. run: 4696 subRun: 0 event: 3010 at 13-Dec-2025 19:26:03 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  3
Added hit list length: 3 | Max was set to: -3.40282e+38
True Neutron Energy: 939.57
True Electron Energy: 25.416
Completed MCParticle Loop
True Neutrino Energy: 30.8112
 Size of all hits: 3
hit initial : 26.4624 | hit correction : 1.00706
 *** Total Charge : 26.6492
 Looking at view: 0 | Number of hits in view: 1
hit initial : 27.8118 | hit correction : 1.00708
 *** Total Charge : 28.0087
 Looking at view: 1 | Number of hits in view: 1
hit initial : 55.1934 | hit correction : 1.00707
 *** Total Charge : 55.5836
 Looking at view: 2 | Number of hits in view: 1
13-Dec-2025 19:26:07 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0920052     0.118972      0.149297       0.1122       0.017208        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000436166   0.000636942   0.00148597    0.000551231   0.000289321      10     
end_path:ana:AnalyzeEvents         0.0913231     0.118129      0.147219       0.1115       0.0169418       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19478.6 MB
  Peak resident set size usage (VmHWM): 1539.61 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:12:13 UTC       justIN version: 01.06.00