justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262312.0@dunegpschedd02.fnal.gov

Jobsub ID262312.0@dunegpschedd02.fnal.gov
Workflow ID11170
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 21:41:04
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-14 09:16:41
From worker nodeHostnamedunegli-8066212-0-fnpc17126.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-14 03:30:41
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T204914Z_gen_003188_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (20746s)
CPU time5h (20029s = 96%)
Max RSS bytes2797035520 (2667 MiB)
Outputting started2025-12-14 09:16:28
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11170/1/001/analysisLowE_20251214T033042Z.root
Finished2025-12-14 09:16:41
Saved logsjustin-logs:262312.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  14-Dec-2025 09:15:08 UTC run: 5026 subRun: 0 event: 31880
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6706, 6768, 6632
minwire 0: 49
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2188, 2687
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.147143, 
Output 1: 0.0937172, 0.0815153, 0.113246, 0.711521, 
Output 2: 0.0452101, 0.136242, 0.800687, 0.0178601, 
Output 3: 0.986429, 0.00979198, 0.000742392, 0.00303717, 
Output 4: 0.38175, 0.537019, 0.0714013, 0.00982953, 
Output 5: 0.267694, 0.400322, 0.239196, 0.0927879, 
Output 6: 0.0803909, 0.158077, 0.130243, 0.631289, 

14-Dec-2025 09:15:13 UTC  Closed output file "matt_reco2.root"
14-Dec-2025 09:15:13 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/44/d7/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T204914Z_gen_003188_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1851.19       2002.96       2307.88       1906.74       170.366        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000709984   0.00768976     0.0412922    0.00102467     0.0124426       10     
reco:pandora:StandardPandora                            1849.62       2000.74       2305.39       1903.48       170.143        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00034253     0.0624868     0.0906187     0.067425      0.0247212       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000325756   0.00292786     0.0137945    0.00164423    0.00369147       10     
reco:pandoracalo:Calorimetry                          0.000190478   0.000818307    0.0039317    0.000487771   0.00105006       10     
reco:pandorapid:Chi2ParticleID                        5.9044e-05    0.000245089   0.00146858    0.000102177   0.000408768      10     
reco:cvnmap:CVNMapper                                  0.0854294     0.110823      0.157028      0.104588      0.0220947       10     
reco:cvneva:CVNEvaluator                               0.928239       1.44465       4.21246       1.06537      0.949617        10     
reco:energyrecnumu:EnergyReco                          0.0813218     0.109113      0.171799      0.100205      0.0265765       10     
reco:energyrecnue:EnergyReco                          0.00162178    0.00284514    0.00993521    0.00214477    0.00238851       10     
reco:energyrecnc:EnergyReco                            0.0090425     0.011318      0.0145568    0.00988691     0.0022645       10     
[art]:TriggerResults:TriggerResultInserter            1.5433e-05    2.68825e-05   8.5832e-05    2.0492e-05    1.99803e-05      10     
end_path:out1:RootOutput                               7.762e-06    1.06416e-05   2.3467e-05    9.1455e-06    4.40844e-06      10     
end_path:out1:RootOutput(write)                        0.342654      0.467607       0.61777      0.431937      0.114779        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14061.5 MB
  Peak resident set size usage (VmHWM): 2797.04 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
True Neutrino Energy: 9.82877
 Size of all hits: 16
hit initial : 33.8277 | hit correction : 1.32413
hit initial : 22.5958 | hit correction : 1.32411
hit initial : 90.5185 | hit correction : 1.32565
hit initial : 265.767 | hit correction : 1.32577
hit initial : 178.014 | hit correction : 1.32594
 *** Total Charge : 783.088
 Looking at view: 0 | Number of hits in view: 5
hit initial : 39.2717 | hit correction : 1.32412
hit initial : 20.7157 | hit correction : 1.32409
hit initial : 152.656 | hit correction : 1.32567
hit initial : 252.589 | hit correction : 1.32583
hit initial : 124.299 | hit correction : 1.32608
hit initial : 18.9113 | hit correction : 1.32635
 *** Total Charge : 806.604
 Looking at view: 1 | Number of hits in view: 6
hit initial : 19.2621 | hit correction : 1.32556
hit initial : 122.427 | hit correction : 1.32552
hit initial : 168.64 | hit correction : 1.32571
hit initial : 176.469 | hit correction : 1.32605
hit initial : 82.4974 | hit correction : 1.32629
 *** Total Charge : 754.802
 Looking at view: 2 | Number of hits in view: 5
Begin processing the 10th record. run: 5026 subRun: 0 event: 31880 at 14-Dec-2025 09:16:21 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  41
Added hit list length: 41 | Max was set to: -3.40282e+38
This pfp has size:  8
True Neutron Energy: 939.573
True Electron Energy: 42.1631
Completed MCParticle Loop
True Neutrino Energy: 45.9151
 Size of all hits: 41
hit initial : 23.2099 | hit correction : 1.37025
hit initial : 85.9934 | hit correction : 1.37024
hit initial : 90.3726 | hit correction : 1.37026
hit initial : 41.2434 | hit correction : 1.36849
hit initial : 29.3057 | hit correction : 1.36847
hit initial : 33.0467 | hit correction : 1.36542
hit initial : 73.3004 | hit correction : 1.3654
hit initial : 42.3404 | hit correction : 1.36547
hit initial : 70.5461 | hit correction : 1.36608
hit initial : 221.539 | hit correction : 1.36611
hit initial : 370.03 | hit correction : 1.36618
hit initial : 361.427 | hit correction : 1.36622
hit initial : 16.186 | hit correction : 1.36592
hit initial : 260.979 | hit correction : 1.3662
hit initial : 113.035 | hit correction : 1.36598
hit initial : 214.215 | hit correction : 1.36607
 *** Total Charge : 2797.07
 Looking at view: 0 | Number of hits in view: 16
hit initial : 24.6552 | hit correction : 1.37025
hit initial : 100.43 | hit correction : 1.37023
hit initial : 82.4069 | hit correction : 1.37024
hit initial : 25.2042 | hit correction : 1.36846
hit initial : 37.2497 | hit correction : 1.36845
hit initial : 15.5031 | hit correction : 1.36539
hit initial : 47.1281 | hit correction : 1.3654
hit initial : 19.1106 | hit correction : 1.36544
hit initial : 47.5852 | hit correction : 1.36602
hit initial : 197.754 | hit correction : 1.36606
hit initial : 392.598 | hit correction : 1.36614
hit initial : 517.333 | hit correction : 1.36621
hit initial : 393.125 | hit correction : 1.36616
hit initial : 101.07 | hit correction : 1.36608
 *** Total Charge : 2734.8
 Looking at view: 1 | Number of hits in view: 14
hit initial : 125.687 | hit correction : 1.37024
hit initial : 88.5467 | hit correction : 1.37023
hit initial : 65.9574 | hit correction : 1.36844
hit initial : 50.2868 | hit correction : 1.36539
hit initial : 37.9836 | hit correction : 1.36539
hit initial : 338.645 | hit correction : 1.36608
hit initial : 363.264 | hit correction : 1.36613
hit initial : 500.989 | hit correction : 1.3662
hit initial : 213.61 | hit correction : 1.36627
hit initial : 27.4188 | hit correction : 1.3659
hit initial : 188.548 | hit correction : 1.36597
 *** Total Charge : 2734.52
 Looking at view: 2 | Number of hits in view: 11
14-Dec-2025 09:16:26 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.100853      0.151115      0.317464      0.127815      0.0619076       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000461751   0.00878037     0.0824943    0.000526762    0.0245721       10     
end_path:ana:AnalyzeEvents         0.100147      0.142127      0.315675      0.127009      0.0585222       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11964.9 MB
  Peak resident set size usage (VmHWM): 1543.64 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:09:28 UTC       justIN version: 01.06.00