justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262243.0@dunegpschedd02.fnal.gov

Jobsub ID262243.0@dunegpschedd02.fnal.gov
Workflow ID11172
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 16:30:49
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 22:26:01
From worker nodeHostnamedunegli-7890610-0-fnpc9038.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 16:31:29
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T210817Z_gen_003803_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (21262s)
CPU time5h (21096s = 99%)
Max RSS bytes2806689792 (2676 MiB)
Outputting started2025-12-13 22:25:51
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11172/1/001/analysisLowE_20251213T163131Z.root
Finished2025-12-13 22:26:01
Saved logsjustin-logs:262243.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

E/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 22:25:05 UTC run: 5026 subRun: 0 event: 38030
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6880, 6866, 6669
minwire 0: 14
minwire 1: 1
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.162956, 
Output 1: 0.0650081, 0.0688617, 0.19372, 0.67241, 
Output 2: 0.0553906, 0.178446, 0.739494, 0.0266699, 
Output 3: 0.965352, 0.0267629, 0.0033244, 0.00456069, 
Output 4: 0.290942, 0.614937, 0.0861698, 0.00795101, 
Output 5: 0.346356, 0.478764, 0.146513, 0.0283669, 
Output 6: 0.0334433, 0.0903262, 0.0950118, 0.781219, 

13-Dec-2025 22:25:08 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 22:25:08 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/9e/34/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T210817Z_gen_003803_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1450.66       2091.63       3633.27       2009.83       563.028        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000649583   0.00505011     0.0219956    0.00106308    0.00767108       10     
reco:pandora:StandardPandora                            1448.43       2089.16       3627.11       2007.86       561.908        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00564534     0.0522252     0.0889067     0.0543907     0.022576        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00206646     0.0131609     0.0638692    0.00619889     0.0179438       10     
reco:pandoracalo:Calorimetry                          0.000443742   0.00144882    0.00593001    0.000736759   0.00169808       10     
reco:pandorapid:Chi2ParticleID                        0.000114658   0.000574972   0.00242076    0.000150188   0.00076183       10     
reco:cvnmap:CVNMapper                                  0.0619234     0.114678      0.254033      0.100374      0.0522112       10     
reco:cvneva:CVNEvaluator                                1.09715       1.65177       5.20789       1.20936       1.1913         10     
reco:energyrecnumu:EnergyReco                          0.0788829     0.161475      0.570953      0.119419      0.138132        10     
reco:energyrecnue:EnergyReco                          0.00144143    0.00201754    0.00319168     0.0019443    0.000453213      10     
reco:energyrecnc:EnergyReco                           0.00805752     0.0108471     0.0131507     0.0114232     0.0018058       10     
[art]:TriggerResults:TriggerResultInserter            1.5268e-05    3.51306e-05   0.000110235   2.88495e-05   2.63348e-05      10     
end_path:out1:RootOutput                               5.517e-06    1.16281e-05   3.0458e-05     9.165e-06    7.03114e-06      10     
end_path:out1:RootOutput(write)                        0.341392      0.447318      0.531214       0.47214      0.0671167       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11917.3 MB
  Peak resident set size usage (VmHWM): 2806.69 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 39.465 | hit correction : 1.41764
hit initial : 59.3087 | hit correction : 1.41768
hit initial : 24.7929 | hit correction : 1.41621
hit initial : 86.2823 | hit correction : 1.41689
hit initial : 185.689 | hit correction : 1.41759
hit initial : 83.7274 | hit correction : 1.41773
hit initial : 44.2597 | hit correction : 1.41579
hit initial : 116.704 | hit correction : 1.41561
hit initial : 200.245 | hit correction : 1.41554
hit initial : 175.186 | hit correction : 1.41542
hit initial : 148.431 | hit correction : 1.41521
hit initial : 184.019 | hit correction : 1.41486
hit initial : 270.462 | hit correction : 1.41453
hit initial : 263.547 | hit correction : 1.41423
hit initial : 20.4158 | hit correction : 1.41376
 *** Total Charge : 3206.47
 Looking at view: 1 | Number of hits in view: 19
hit initial : 38.4844 | hit correction : 1.43484
hit initial : 19.5346 | hit correction : 1.43485
hit initial : 69.5236 | hit correction : 1.41745
hit initial : 229.889 | hit correction : 1.4175
hit initial : 53.1682 | hit correction : 1.41761
hit initial : 187.195 | hit correction : 1.41766
hit initial : 35.0148 | hit correction : 1.41628
hit initial : 140.294 | hit correction : 1.41717
hit initial : 87.4092 | hit correction : 1.41682
hit initial : 79.8209 | hit correction : 1.41782
hit initial : 48.5131 | hit correction : 1.41547
hit initial : 309.679 | hit correction : 1.41549
hit initial : 346.863 | hit correction : 1.41539
hit initial : 356.639 | hit correction : 1.41472
hit initial : 366.142 | hit correction : 1.41425
 *** Total Charge : 3354.16
 Looking at view: 2 | Number of hits in view: 15
Begin processing the 10th record. run: 5026 subRun: 0 event: 38030 at 13-Dec-2025 22:25:48 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  30
Added hit list length: 30 | Max was set to: -3.40282e+38
This pfp has size:  15
This pfp has size:  8
True Neutron Energy: 939.571
True Electron Energy: 18.3088
Completed MCParticle Loop
True Neutrino Energy: 23.8297
 Size of all hits: 30
hit initial : 50.5023 | hit correction : 1.31861
hit initial : 213.543 | hit correction : 1.31856
hit initial : 302.851 | hit correction : 1.31847
hit initial : 346.208 | hit correction : 1.31852
hit initial : 394.044 | hit correction : 1.31851
hit initial : 129.11 | hit correction : 1.3185
hit initial : 198.393 | hit correction : 1.31768
hit initial : 123.883 | hit correction : 1.31751
 *** Total Charge : 2318.36
 Looking at view: 0 | Number of hits in view: 8
hit initial : 14.0723 | hit correction : 1.31863
hit initial : 32.2439 | hit correction : 1.31865
hit initial : 77.4247 | hit correction : 1.31863
hit initial : 198.02 | hit correction : 1.31858
hit initial : 204.741 | hit correction : 1.31845
hit initial : 192.346 | hit correction : 1.31843
hit initial : 178.61 | hit correction : 1.31858
hit initial : 181.59 | hit correction : 1.31863
hit initial : 212.576 | hit correction : 1.31855
hit initial : 220.312 | hit correction : 1.31834
hit initial : 221.569 | hit correction : 1.31785
hit initial : 146.668 | hit correction : 1.31753
hit initial : 39.7302 | hit correction : 1.31742
 *** Total Charge : 2531.08
 Looking at view: 1 | Number of hits in view: 13
hit initial : 37.6466 | hit correction : 1.31866
hit initial : 112.342 | hit correction : 1.31864
hit initial : 217.377 | hit correction : 1.31864
hit initial : 355.233 | hit correction : 1.3185
hit initial : 326.869 | hit correction : 1.31846
hit initial : 213.483 | hit correction : 1.31856
hit initial : 223.862 | hit correction : 1.31832
hit initial : 337.227 | hit correction : 1.31766
hit initial : 44.3451 | hit correction : 1.31753
 *** Total Charge : 2463.15
 Looking at view: 2 | Number of hits in view: 9
13-Dec-2025 22:25:51 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0864083     0.0909167     0.108791      0.0884019    0.00649521       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000334369   0.000432342   0.000736126   0.000406603   0.000112571      10     
end_path:ana:AnalyzeEvents         0.0858575     0.0903478     0.107663      0.0878395    0.00632407       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9813.43 MB
  Peak resident set size usage (VmHWM): 1539.96 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:04:26 UTC       justIN version: 01.06.00