justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262169.2@dunegpschedd02.fnal.gov

Jobsub ID262169.2@dunegpschedd02.fnal.gov
Workflow ID11175
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 10:44:31
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 20:24:50
From worker nodeHostnamedunegli-7890507-0-fnpc17010.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 10:46:11
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T020840Z_gen_003638_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time9h (34701s)
CPU time9h (33992s = 97%)
Max RSS bytes2798768128 (2669 MiB)
Outputting started2025-12-13 20:24:33
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11175/1/001/analysisLowE_20251213T104612Z.root
Finished2025-12-13 20:24:50
Saved logsjustin-logs:262169.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 20:22:55 UTC run: 5027 subRun: 0 event: 36380
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 20:22:55 UTC run: 5027 subRun: 0 event: 36380
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6785, 6788, 6737
minwire 0: 58
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.152223, 
Output 1: 0.0573251, 0.124536, 0.195788, 0.622351, 
Output 2: 0.0370044, 0.256216, 0.677539, 0.0292406, 
Output 3: 0.941356, 0.0409291, 0.00875535, 0.00895921, 
Output 4: 0.313997, 0.565539, 0.111687, 0.00877768, 
Output 5: 0.539523, 0.348907, 0.0952917, 0.0162784, 
Output 6: 0.0155081, 0.0501179, 0.0709775, 0.863396, 

13-Dec-2025 20:22:59 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 20:23:00 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/03/77/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T020840Z_gen_003638_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2569.44       3433.41       4207.21       3565.65       585.588        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00105357     0.0055492     0.0240599    0.00149425    0.00827135       10     
reco:pandora:StandardPandora                            2566.79       3395.03       4129.46       3527.22       551.843        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000494342    0.098269       0.17903      0.102853      0.0550762       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000454341    0.0113799     0.0607442    0.00701736     0.0166918       10     
reco:pandoracalo:Calorimetry                          0.000298277   0.00210325     0.0138182    0.00081144    0.00391367       10     
reco:pandorapid:Chi2ParticleID                        0.000100421   0.000718629   0.00540294    0.000207929   0.00156263       10     
reco:cvnmap:CVNMapper                                  0.0842184     0.113711       0.17725      0.112015      0.0261483       10     
reco:cvneva:CVNEvaluator                                1.79394       2.6476        8.09947       1.88375       1.85308        10     
reco:energyrecnumu:EnergyReco                          0.114892       34.9194       71.2855       34.2614       34.7955        10     
reco:energyrecnue:EnergyReco                          0.00214814    0.00412468     0.0122282    0.00330871    0.00282119       10     
reco:energyrecnc:EnergyReco                            0.010275      0.0130196     0.0175586     0.0127298    0.00204227       10     
[art]:TriggerResults:TriggerResultInserter            2.8802e-05    4.36136e-05   0.000127066   3.27645e-05   2.8117e-05       10     
end_path:out1:RootOutput                               9.194e-06    1.48493e-05    4.248e-05    1.13815e-05   9.40286e-06      10     
end_path:out1:RootOutput(write)                        0.405213      0.554289      0.802781       0.54454      0.104426        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11911.9 MB
  Peak resident set size usage (VmHWM): 2798.77 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
True Electron Energy: 28.7068
Completed MCParticle Loop
True Neutrino Energy: 32.0295
 Size of all hits: 41
hit initial : 483.926 | hit correction : 1.14802
hit initial : 797.367 | hit correction : 1.14795
hit initial : 657.35 | hit correction : 1.14766
hit initial : 367.785 | hit correction : 1.14731
hit initial : 250.177 | hit correction : 1.14683
hit initial : 246.046 | hit correction : 1.14638
hit initial : 275.389 | hit correction : 1.1459
hit initial : 255.142 | hit correction : 1.14546
hit initial : 77.6019 | hit correction : 1.14527
 *** Total Charge : 3912.94
 Looking at view: 0 | Number of hits in view: 9
hit initial : 65.9539 | hit correction : 1.14749
hit initial : 292.646 | hit correction : 1.14759
hit initial : 357.656 | hit correction : 1.14785
hit initial : 278.249 | hit correction : 1.148
hit initial : 217.509 | hit correction : 1.14813
hit initial : 204.032 | hit correction : 1.14812
hit initial : 195.668 | hit correction : 1.14801
hit initial : 190.815 | hit correction : 1.14785
hit initial : 195.08 | hit correction : 1.14762
hit initial : 225.706 | hit correction : 1.14736
hit initial : 251.791 | hit correction : 1.14706
hit initial : 276.126 | hit correction : 1.14664
hit initial : 304.981 | hit correction : 1.14618
hit initial : 276.87 | hit correction : 1.14564
hit initial : 159.797 | hit correction : 1.1453
 *** Total Charge : 4007.18
 Looking at view: 1 | Number of hits in view: 15
hit initial : 213.33 | hit correction : 1.1474
hit initial : 11.217 | hit correction : 1.14739
hit initial : 142.522 | hit correction : 1.14758
hit initial : 159.409 | hit correction : 1.14786
hit initial : 121.776 | hit correction : 1.14798
hit initial : 170.862 | hit correction : 1.14799
hit initial : 137.894 | hit correction : 1.14789
hit initial : 199.967 | hit correction : 1.14817
hit initial : 212.814 | hit correction : 1.14816
hit initial : 206.593 | hit correction : 1.14803
hit initial : 230.769 | hit correction : 1.14788
hit initial : 316.86 | hit correction : 1.14757
hit initial : 323.581 | hit correction : 1.14712
hit initial : 34.2135 | hit correction : 1.14701
hit initial : 542.912 | hit correction : 1.14637
hit initial : 172.964 | hit correction : 1.14576
hit initial : 312.582 | hit correction : 1.14534
 *** Total Charge : 4027.03
 Looking at view: 2 | Number of hits in view: 17
Begin processing the 10th record. run: 5027 subRun: 0 event: 36380 at 13-Dec-2025 20:24:26 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  13
Added hit list length: 13 | Max was set to: -3.40282e+38
This pfp has size:  5
This pfp has size:  5
True Neutron Energy: 939.572
True Electron Energy: 8.88365
Completed MCParticle Loop
True Neutrino Energy: 12.1727
 Size of all hits: 13
hit initial : 145.849 | hit correction : 1.04181
hit initial : 305.13 | hit correction : 1.0419
hit initial : 100.145 | hit correction : 1.04191
hit initial : 137.386 | hit correction : 1.04272
hit initial : 229.954 | hit correction : 1.04283
hit initial : 78.9922 | hit correction : 1.04331
hit initial : 36.4205 | hit correction : 1.04329
 *** Total Charge : 1077.67
 Looking at view: 0 | Number of hits in view: 7
hit initial : 248.87 | hit correction : 1.04186
hit initial : 225.924 | hit correction : 1.04194
hit initial : 251.973 | hit correction : 1.04275
hit initial : 111.681 | hit correction : 1.04322
 *** Total Charge : 873.939
 Looking at view: 1 | Number of hits in view: 4
hit initial : 664.73 | hit correction : 1.04214
hit initial : 346.241 | hit correction : 1.04314
 *** Total Charge : 1053.92
 Looking at view: 2 | Number of hits in view: 2
13-Dec-2025 20:24:31 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          0.12787      0.231567       0.55952      0.192235      0.118954        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000566171   0.00129722    0.00539451    0.000807262   0.00138996       10     
end_path:ana:AnalyzeEvents         0.127024      0.230014      0.557275      0.191373      0.118724        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9811.79 MB
  Peak resident set size usage (VmHWM): 1537.09 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:22:22 UTC       justIN version: 01.06.00