justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262233.1@dunegpschedd02.fnal.gov

Jobsub ID262233.1@dunegpschedd02.fnal.gov
Workflow ID11175
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 15:40:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 21:17:40
From worker nodeHostnamedunegli-7890613-0-fnpc19133.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 15:42:07
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T004454Z_gen_002782_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (20118s)
CPU time5h (19842s = 98%)
Max RSS bytes2834808832 (2703 MiB)
Outputting started2025-12-13 21:17:26
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11175/1/001/analysisLowE_20251213T154208Z.root
Finished2025-12-13 21:17:40
Saved logsjustin-logs:262233.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 21:16:36 UTC run: 5027 subRun: 0 event: 27820
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6457, 6414, 6612
minwire 0: 67
minwire 1: 1
minwire 2: 5
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 131, 630
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.101317, 
Output 1: 0.0570296, 0.094629, 0.15891, 0.689431, 
Output 2: 0.0585149, 0.22292, 0.694146, 0.0244198, 
Output 3: 0.939084, 0.0462554, 0.00703926, 0.00762084, 
Output 4: 0.471116, 0.43164, 0.0851136, 0.0121304, 
Output 5: 0.545961, 0.391415, 0.0544451, 0.0081792, 
Output 6: 0.0265606, 0.0792359, 0.0857566, 0.808447, 

13-Dec-2025 21:16:38 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 21:16:38 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/d7/9a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T004454Z_gen_002782_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1686.48       1992.8        2402.39       1958.1        257.944        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000713677   0.00491837     0.0314059    0.000933008   0.00927839       10     
reco:pandora:StandardPandora                            1684.5        1990.67       2400.53       1956.53       257.649        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000371043    0.0540589     0.196805      0.0439805     0.0510472       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000352304    0.0123189     0.0434386     0.0056205     0.0148054       10     
reco:pandoracalo:Calorimetry                          0.000168861   0.00256744     0.0207741    0.00055878    0.00607406       10     
reco:pandorapid:Chi2ParticleID                         6.63e-05     0.000465958   0.00356132    0.000119761   0.00103249       10     
reco:cvnmap:CVNMapper                                  0.0617488     0.0900305      0.21163      0.0650136     0.0463161       10     
reco:cvneva:CVNEvaluator                               0.971484       1.44138       4.1259        1.15273      0.909316        10     
reco:energyrecnumu:EnergyReco                          0.0786684     0.113873      0.221299      0.0998628     0.0408239       10     
reco:energyrecnue:EnergyReco                          0.00120734    0.00242374    0.00992684    0.00152822    0.00252101       10     
reco:energyrecnc:EnergyReco                            0.0077582     0.0102936     0.021073     0.00820574    0.00404861       10     
[art]:TriggerResults:TriggerResultInserter            1.6211e-05    2.90713e-05    8.495e-05    2.04005e-05   2.05409e-05      10     
end_path:out1:RootOutput                               9.31e-06     1.32511e-05    2.717e-05     1.078e-05    5.48262e-06      10     
end_path:out1:RootOutput(write)                        0.340112       0.39469      0.486502      0.354264      0.0596467       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21571 MB
  Peak resident set size usage (VmHWM): 2834.81 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 223.711 | hit correction : 1.10086
 *** Total Charge : 3463.6
 Looking at view: 1 | Number of hits in view: 10
hit initial : 27.323 | hit correction : 1.09498
hit initial : 63.4502 | hit correction : 1.11028
hit initial : 324.292 | hit correction : 1.11022
hit initial : 179.198 | hit correction : 1.11
hit initial : 199.103 | hit correction : 1.10992
hit initial : 15.168 | hit correction : 1.10274
hit initial : 466.523 | hit correction : 1.10263
hit initial : 555.733 | hit correction : 1.10258
hit initial : 269.178 | hit correction : 1.1024
hit initial : 52.7851 | hit correction : 1.10231
hit initial : 531.988 | hit correction : 1.10168
hit initial : 488.987 | hit correction : 1.10138
hit initial : 295.669 | hit correction : 1.10093
 *** Total Charge : 3829.24
 Looking at view: 2 | Number of hits in view: 13
Begin processing the 9th record. run: 5027 subRun: 0 event: 27819 at 13-Dec-2025 21:17:19 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  6
This pfp has size:  29
Added hit list length: 29 | Max was set to: -3.40282e+38
This pfp has size:  6
True Electron Energy: 16.5725
Completed MCParticle Loop
True Neutrino Energy: 22.0857
 Size of all hits: 29
hit initial : 81.0021 | hit correction : 1.34659
hit initial : 116.702 | hit correction : 1.35018
hit initial : 201.862 | hit correction : 1.35017
hit initial : 51.0579 | hit correction : 1.35019
hit initial : 138.437 | hit correction : 1.34942
hit initial : 246.879 | hit correction : 1.34931
hit initial : 40.4003 | hit correction : 1.34897
hit initial : 172.106 | hit correction : 1.34806
hit initial : 51.028 | hit correction : 1.34793
hit initial : 99.0854 | hit correction : 1.34783
 *** Total Charge : 1616.9
 Looking at view: 0 | Number of hits in view: 10
hit initial : 49.4555 | hit correction : 1.3465
hit initial : 107.197 | hit correction : 1.35018
hit initial : 252.291 | hit correction : 1.35011
hit initial : 43.2484 | hit correction : 1.35015
hit initial : 312.318 | hit correction : 1.34938
hit initial : 209.453 | hit correction : 1.34905
hit initial : 9.00279 | hit correction : 1.34873
hit initial : 149.779 | hit correction : 1.34794
hit initial : 298.462 | hit correction : 1.34769
hit initial : 82.3804 | hit correction : 1.34755
 *** Total Charge : 2041.62
 Looking at view: 1 | Number of hits in view: 10
hit initial : 140.571 | hit correction : 1.34647
hit initial : 237.022 | hit correction : 1.34661
hit initial : 161.603 | hit correction : 1.35018
hit initial : 87.7088 | hit correction : 1.3502
hit initial : 294.876 | hit correction : 1.34992
hit initial : 17.0123 | hit correction : 1.34931
hit initial : 412.329 | hit correction : 1.34914
hit initial : 159.39 | hit correction : 1.34802
hit initial : 296.709 | hit correction : 1.34765
 *** Total Charge : 2437.09
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 10th record. run: 5027 subRun: 0 event: 27820 at 13-Dec-2025 21:17:22 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  3
Added hit list length: 3 | Max was set to: -3.40282e+38
True Neutron Energy: 939.57
True Electron Energy: 18.6312
Completed MCParticle Loop
True Neutrino Energy: 35.8645
 Size of all hits: 3
hit initial : 81.2072 | hit correction : 1.00602
 *** Total Charge : 81.6958
 Looking at view: 0 | Number of hits in view: 1
hit initial : 78.2483 | hit correction : 1.00603
 *** Total Charge : 78.7201
 Looking at view: 1 | Number of hits in view: 1
hit initial : 158.978 | hit correction : 1.006
 *** Total Charge : 159.931
 Looking at view: 2 | Number of hits in view: 1
13-Dec-2025 21:17:25 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0916411     0.103875      0.127793      0.0995938     0.0110683       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000414734   0.000777279   0.00298397    0.000538425   0.000738808      10     
end_path:ana:AnalyzeEvents         0.0910606      0.10289      0.124212      0.0989026     0.0104476       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19471.6 MB
  Peak resident set size usage (VmHWM): 1529.96 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:22:22 UTC       justIN version: 01.06.00