justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262356.0@dunegpschedd02.fnal.gov

Jobsub ID262356.0@dunegpschedd02.fnal.gov
Workflow ID11170
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 22:57:08
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-14 09:01:48
From worker nodeHostnamedunegli-8066212-0-fnpc17126.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-14 03:30:45
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001354Z_gen_002088_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (19849s)
CPU time5h (19148s = 96%)
Max RSS bytes2802196480 (2672 MiB)
Outputting started2025-12-14 09:01:35
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11170/1/001/analysisLowE_20251214T033048Z.root
Finished2025-12-14 09:01:48
Saved logsjustin-logs:262356.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  14-Dec-2025 09:00:16 UTC run: 5027 subRun: 0 event: 20880
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6531, 6546, 6669
minwire 0: 20
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 216, 715
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.114804, 
Output 1: 0.070278, 0.146394, 0.201003, 0.582324, 
Output 2: 0.090611, 0.277123, 0.592891, 0.0393749, 
Output 3: 0.929587, 0.0527998, 0.0081794, 0.00943378, 
Output 4: 0.473201, 0.473069, 0.049111, 0.00461992, 
Output 5: 0.568745, 0.3824, 0.0421871, 0.00666812, 
Output 6: 0.0308126, 0.0770404, 0.082064, 0.810083, 

14-Dec-2025 09:00:20 UTC  Closed output file "matt_reco2.root"
14-Dec-2025 09:00:20 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/88/ab/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001354Z_gen_002088_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1649.72       1913.33       2287.75       1888.26       186.963        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000722045    0.0109944     0.0643989    0.000930731    0.0193246       10     
reco:pandora:StandardPandora                            1647.8        1911.1        2285.7        1886.59       186.855        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00632653     0.0577261     0.0863119     0.0582102     0.0198424       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000535868   0.00363918     0.0140634    0.00181552    0.00407184       10     
reco:pandoracalo:Calorimetry                          0.000380193   0.00783936     0.0675399    0.000490009    0.0199456       10     
reco:pandorapid:Chi2ParticleID                        9.3011e-05    0.000275602   0.00152097    0.00011666    0.000419463      10     
reco:cvnmap:CVNMapper                                  0.110534      0.140103      0.259771      0.122896      0.0427489       10     
reco:cvneva:CVNEvaluator                               0.934412       1.38073       4.05185       1.02308      0.904454        10     
reco:energyrecnumu:EnergyReco                          0.085259      0.114254      0.208684      0.105065      0.0341406       10     
reco:energyrecnue:EnergyReco                          0.00162181    0.00323666     0.0146572    0.00181419    0.00382344       10     
reco:energyrecnc:EnergyReco                           0.00909912     0.010808      0.014584     0.00989205    0.00196432       10     
[art]:TriggerResults:TriggerResultInserter            1.4096e-05    2.49939e-05   7.3346e-05    1.9033e-05    1.65165e-05      10     
end_path:out1:RootOutput                               7.298e-06    1.11478e-05   2.2349e-05    8.3915e-06    5.51487e-06      10     
end_path:out1:RootOutput(write)                        0.356495      0.502148      0.731339      0.489415      0.134071        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14065.9 MB
  Peak resident set size usage (VmHWM): 2802.2 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 284.328 | hit correction : 1.05841
hit initial : 324.464 | hit correction : 1.05872
hit initial : 356.067 | hit correction : 1.05904
hit initial : 185.439 | hit correction : 1.05925
hit initial : 207.841 | hit correction : 1.0601
hit initial : 362.104 | hit correction : 1.06017
 *** Total Charge : 4493.4
 Looking at view: 0 | Number of hits in view: 21
hit initial : 31.0984 | hit correction : 1.05168
hit initial : 173.817 | hit correction : 1.05423
hit initial : 74.7775 | hit correction : 1.05422
hit initial : 315.678 | hit correction : 1.05625
hit initial : 248.851 | hit correction : 1.05628
hit initial : 126.233 | hit correction : 1.05634
hit initial : 307.53 | hit correction : 1.05633
hit initial : 35.8618 | hit correction : 1.05646
hit initial : 72.8369 | hit correction : 1.0576
hit initial : 24.4589 | hit correction : 1.05674
hit initial : 67.8735 | hit correction : 1.05758
hit initial : 263.599 | hit correction : 1.05832
hit initial : 266.76 | hit correction : 1.05834
hit initial : 69.8195 | hit correction : 1.05908
hit initial : 104.668 | hit correction : 1.05916
hit initial : 45.6932 | hit correction : 1.05903
hit initial : 186.969 | hit correction : 1.06015
hit initial : 375.259 | hit correction : 1.06014
 *** Total Charge : 2952.34
 Looking at view: 1 | Number of hits in view: 18
hit initial : 59.3564 | hit correction : 1.05166
hit initial : 143.901 | hit correction : 1.05428
hit initial : 176.839 | hit correction : 1.05419
hit initial : 333.94 | hit correction : 1.05615
hit initial : 325.152 | hit correction : 1.05632
hit initial : 428.962 | hit correction : 1.05637
hit initial : 25.1616 | hit correction : 1.0564
hit initial : 29.8399 | hit correction : 1.05673
hit initial : 92.5572 | hit correction : 1.05758
hit initial : 184.252 | hit correction : 1.05759
hit initial : 87.5037 | hit correction : 1.0577
hit initial : 125.555 | hit correction : 1.05807
hit initial : 304.038 | hit correction : 1.05823
hit initial : 192.388 | hit correction : 1.05836
hit initial : 160.036 | hit correction : 1.05834
hit initial : 165.956 | hit correction : 1.05836
hit initial : 200.564 | hit correction : 1.05852
hit initial : 222.897 | hit correction : 1.0588
hit initial : 282.542 | hit correction : 1.05904
hit initial : 242.847 | hit correction : 1.05933
hit initial : 170.778 | hit correction : 1.0599
hit initial : 500.82 | hit correction : 1.06021
 *** Total Charge : 4713.25
 Looking at view: 2 | Number of hits in view: 22
Begin processing the 10th record. run: 5027 subRun: 0 event: 20880 at 14-Dec-2025 09:01:28 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  12
Added hit list length: 12 | Max was set to: -3.40282e+38
This pfp has size:  11
This pfp has size:  4
True Neutron Energy: 939.579
True Electron Energy: 7.04611
Completed MCParticle Loop
True Neutrino Energy: 12.4244
 Size of all hits: 12
hit initial : 148.49 | hit correction : 1.2991
hit initial : 291.28 | hit correction : 1.29922
hit initial : 233.63 | hit correction : 1.29932
hit initial : 70.443 | hit correction : 1.29937
 *** Total Charge : 966.433
 Looking at view: 0 | Number of hits in view: 4
hit initial : 32.8114 | hit correction : 1.29901
hit initial : 169.088 | hit correction : 1.29909
hit initial : 267.015 | hit correction : 1.29922
hit initial : 191.521 | hit correction : 1.29932
hit initial : 79.8635 | hit correction : 1.29937
 *** Total Charge : 961.811
 Looking at view: 1 | Number of hits in view: 5
hit initial : 122.635 | hit correction : 1.29916
hit initial : 500.749 | hit correction : 1.29923
hit initial : 123.911 | hit correction : 1.29935
 *** Total Charge : 970.915
 Looking at view: 2 | Number of hits in view: 3
14-Dec-2025 09:01:33 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.118899       0.17181      0.403513      0.133354      0.0870983       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000466363   0.000674537   0.00108608    0.000643348   0.000171437      10     
end_path:ana:AnalyzeEvents         0.118009      0.170885       0.40277      0.132408      0.0870225       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11963.7 MB
  Peak resident set size usage (VmHWM): 1541.53 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:10:04 UTC       justIN version: 01.06.00