justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261935.0@dunegpschedd02.fnal.gov

Jobsub ID261935.0@dunegpschedd02.fnal.gov
Workflow ID11173
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 05:20:15
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-13 17:18:55
From worker nodeHostnamedunegli-8047001-0-fnpc19108.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 05:21:23
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031257Z_gen_004773_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time11h (43042s)
CPU time11h (42403s = 98%)
Max RSS bytes2843189248 (2711 MiB)
Outputting started2025-12-13 17:18:46
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11173/1/001/analysisLowE_20251213T052125Z.root
Finished2025-12-13 17:18:55
Saved logsjustin-logs:261935.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 17:18:03 UTC run: 5027 subRun: 0 event: 47730
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6650, 6603, 6683
minwire 0: 9
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1936, 2435
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.135988, 
Output 1: 0.0599578, 0.0712358, 0.163072, 0.705735, 
Output 2: 0.037998, 0.177858, 0.755732, 0.0284113, 
Output 3: 0.947817, 0.0375024, 0.00729289, 0.00738795, 
Output 4: 0.412285, 0.475704, 0.0994905, 0.0125199, 
Output 5: 0.367249, 0.37983, 0.204612, 0.0483078, 
Output 6: 0.0376305, 0.104838, 0.10535, 0.752182, 

13-Dec-2025 17:18:05 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 17:18:05 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/26/24/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T031257Z_gen_004773_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2400.65       4271.34       4747.15       4540.14       669.667        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000866347   0.00524297     0.0210244    0.00232062    0.00678607       10     
reco:pandora:StandardPandora                            2399.24       4267.93       4744.29       4537.32       669.048        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000531024    0.0958126     0.180252      0.131276      0.0695032       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000928618    0.0112785     0.0476778    0.00455035     0.0140095       10     
reco:pandoracalo:Calorimetry                          0.000223202   0.00307406     0.0128564    0.00115312    0.00401549       10     
reco:pandorapid:Chi2ParticleID                        7.2251e-05    0.000817809   0.00340282    0.000210367   0.00127922       10     
reco:cvnmap:CVNMapper                                  0.0594953     0.147187      0.184833      0.158234      0.0368055       10     
reco:cvneva:CVNEvaluator                                0.89885       2.46025       8.75932       1.84553       2.13511        10     
reco:energyrecnumu:EnergyReco                          0.081061      0.128702      0.245822      0.114336      0.047584        10     
reco:energyrecnue:EnergyReco                          0.00120118    0.00527564     0.0149831    0.00381879    0.00402961       10     
reco:energyrecnc:EnergyReco                           0.00719585     0.0133343     0.0169295     0.0132283    0.00255669       10     
[art]:TriggerResults:TriggerResultInserter             1.092e-05    8.45816e-05   0.000536194   4.00355e-05   0.000150962      10     
end_path:out1:RootOutput                               7.51e-06     3.39954e-05   0.000188091   1.8915e-05    5.16108e-05      10     
end_path:out1:RootOutput(write)                        0.318979      0.533203      0.819605       0.50201       0.12414        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21580.3 MB
  Peak resident set size usage (VmHWM): 2843.19 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 303.751 | hit correction : 1.42917
hit initial : 298.036 | hit correction : 1.42954
hit initial : 238.065 | hit correction : 1.42971
hit initial : 253.569 | hit correction : 1.42981
hit initial : 205.858 | hit correction : 1.42979
hit initial : 387.627 | hit correction : 1.42982
hit initial : 792.176 | hit correction : 1.42998
hit initial : 410.531 | hit correction : 1.43005
 *** Total Charge : 6678.84
 Looking at view: 2 | Number of hits in view: 24
Begin processing the 9th record. run: 5027 subRun: 0 event: 47729 at 13-Dec-2025 17:18:39 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  3
Added hit list length: 3 | Max was set to: -3.40282e+38
True Electron Energy: 4.75224
Completed MCParticle Loop
True Neutrino Energy: 8.03681
 Size of all hits: 3
hit initial : 96.719 | hit correction : 1.01393
 *** Total Charge : 98.0662
 Looking at view: 0 | Number of hits in view: 1
hit initial : 104.16 | hit correction : 1.01394
 *** Total Charge : 105.611
 Looking at view: 1 | Number of hits in view: 1
hit initial : 178.951 | hit correction : 1.01394
 *** Total Charge : 181.445
 Looking at view: 2 | Number of hits in view: 1
Begin processing the 10th record. run: 5027 subRun: 0 event: 47730 at 13-Dec-2025 17:18:42 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  37
Added hit list length: 37 | Max was set to: -3.40282e+38
This pfp has size:  8
True Neutron Energy: 939.566
True Electron Energy: 18.8718
Completed MCParticle Loop
True Neutrino Energy: 34.3453
 Size of all hits: 37
hit initial : 47.7265 | hit correction : 1.33497
hit initial : 29.3267 | hit correction : 1.33494
hit initial : 61.0332 | hit correction : 1.339
hit initial : 258.252 | hit correction : 1.33903
hit initial : 261.677 | hit correction : 1.33949
hit initial : 359.269 | hit correction : 1.33913
hit initial : 26.2409 | hit correction : 1.33963
hit initial : 171.824 | hit correction : 1.33984
hit initial : 352.604 | hit correction : 1.34005
hit initial : 220.198 | hit correction : 1.34052
hit initial : 114.002 | hit correction : 1.34067
hit initial : 31.3719 | hit correction : 1.34076
 *** Total Charge : 2589.97
 Looking at view: 0 | Number of hits in view: 12
hit initial : 38.9048 | hit correction : 1.33497
hit initial : 40.4988 | hit correction : 1.33499
hit initial : 223.91 | hit correction : 1.33899
hit initial : 243.32 | hit correction : 1.33915
hit initial : 190.852 | hit correction : 1.3394
hit initial : 277.215 | hit correction : 1.33922
hit initial : 51.534 | hit correction : 1.33955
hit initial : 345.957 | hit correction : 1.3399
hit initial : 306.264 | hit correction : 1.34039
hit initial : 154.121 | hit correction : 1.34058
 *** Total Charge : 2508.24
 Looking at view: 1 | Number of hits in view: 10
hit initial : 34.4115 | hit correction : 1.33494
hit initial : 71.2 | hit correction : 1.33496
hit initial : 62.6087 | hit correction : 1.33925
hit initial : 148.588 | hit correction : 1.33892
hit initial : 361.327 | hit correction : 1.33923
hit initial : 108.385 | hit correction : 1.33893
hit initial : 135.695 | hit correction : 1.33916
hit initial : 56.9494 | hit correction : 1.3396
hit initial : 194.952 | hit correction : 1.33955
hit initial : 147.971 | hit correction : 1.33996
hit initial : 106.79 | hit correction : 1.34019
hit initial : 90.9786 | hit correction : 1.34032
hit initial : 99.607 | hit correction : 1.34043
hit initial : 128.685 | hit correction : 1.34061
hit initial : 97.3733 | hit correction : 1.34082
 *** Total Charge : 2471.88
 Looking at view: 2 | Number of hits in view: 15
13-Dec-2025 17:18:45 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.069094      0.0873551     0.107359      0.0856173    0.00995086       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000316463   0.000511932   0.000814787   0.000458909   0.00015638       10     
end_path:ana:AnalyzeEvents         0.068166      0.0866983     0.106736      0.085086      0.0100323       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19478.5 MB
  Peak resident set size usage (VmHWM): 1537.19 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:05:52 UTC       justIN version: 01.06.00