justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262269.1@dunegpschedd02.fnal.gov

Jobsub ID262269.1@dunegpschedd02.fnal.gov
Workflow ID11177
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 16:48:50
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 21:51:47
From worker nodeHostnamedunegli-7890610-0-fnpc9038.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 16:49:25
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205519Z_gen_003440_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (18133s)
CPU time4h (17995s = 99%)
Max RSS bytes2798379008 (2668 MiB)
Outputting started2025-12-13 21:51:38
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11177/1/001/analysisLowE_20251213T164926Z.root
Finished2025-12-13 21:51:47
Saved logsjustin-logs:262269.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

orkspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 21:50:51 UTC run: 5026 subRun: 0 event: 34400
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6742, 6597, 6682
minwire 0: 23
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2349, 2848
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.103246, 
Output 1: 0.259354, 0.0640125, 0.199098, 0.477535, 
Output 2: 0.0306134, 0.07293, 0.88742, 0.00903677, 
Output 3: 0.962523, 0.0328794, 0.00174456, 0.00285297, 
Output 4: 0.583136, 0.370402, 0.0381087, 0.00835338, 
Output 5: 0.197382, 0.416943, 0.276946, 0.108728, 
Output 6: 0.0950099, 0.197723, 0.128505, 0.578762, 

13-Dec-2025 21:50:54 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 21:50:54 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/86/66/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205519Z_gen_003440_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1491.86       1782.64       2973.8        1585.38       420.29         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000649051    0.0045852     0.0224182    0.000806779   0.00766984       10     
reco:pandora:StandardPandora                            1490.27       1780.64       2968.84       1583.74       419.341        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.0368005     0.0487843     0.0864656     0.0459968     0.0144842       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00141004    0.00623731     0.0283294    0.00472724    0.00759285       10     
reco:pandoracalo:Calorimetry                           0.0003437    0.000993363   0.00520776    0.000490272    0.0014196       10     
reco:pandorapid:Chi2ParticleID                        8.6525e-05    0.000332913    0.0023885    0.000107088   0.000685283      10     
reco:cvnmap:CVNMapper                                  0.0609225     0.0702545     0.099582      0.0664225     0.0115187       10     
reco:cvneva:CVNEvaluator                                1.05516       1.39068       4.08143       1.08132      0.897845        10     
reco:energyrecnumu:EnergyReco                          0.0798869     0.106542      0.159783      0.108306      0.0212491       10     
reco:energyrecnue:EnergyReco                          0.00134725    0.00157072    0.00252474    0.00143726    0.000339909      10     
reco:energyrecnc:EnergyReco                           0.00797254    0.00840384    0.00936698    0.00824788    0.000401342      10     
[art]:TriggerResults:TriggerResultInserter            1.4949e-05    2.32004e-05   8.3389e-05     1.612e-05    2.00987e-05      10     
end_path:out1:RootOutput                               4.883e-06    7.1692e-06    2.3295e-05     5.427e-06    5.38705e-06      10     
end_path:out1:RootOutput(write)                        0.336317      0.352508       0.45381      0.339029      0.0343502       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11904 MB
  Peak resident set size usage (VmHWM): 2798.38 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 191.68 | hit correction : 1.09031
hit initial : 173.246 | hit correction : 1.08999
 *** Total Charge : 5712.86
 Looking at view: 2 | Number of hits in view: 25
Begin processing the 9th record. run: 5026 subRun: 0 event: 34399 at 13-Dec-2025 21:51:32 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  25
Added hit list length: 25 | Max was set to: -3.40282e+38
This pfp has size:  10
True Electron Energy: 10.1528
Completed MCParticle Loop
True Neutrino Energy: 15.9187
 Size of all hits: 25
hit initial : 309.274 | hit correction : 1.42186
hit initial : 53.8895 | hit correction : 1.42185
hit initial : 112.776 | hit correction : 1.42184
hit initial : 43.5108 | hit correction : 1.42186
hit initial : 255.804 | hit correction : 1.422
hit initial : 64.3212 | hit correction : 1.42206
hit initial : 115.312 | hit correction : 1.4215
 *** Total Charge : 1357.72
 Looking at view: 0 | Number of hits in view: 7
hit initial : 35.7081 | hit correction : 1.42185
hit initial : 32.0832 | hit correction : 1.42195
hit initial : 105.933 | hit correction : 1.42184
hit initial : 63.866 | hit correction : 1.42185
hit initial : 115.057 | hit correction : 1.42199
hit initial : 185.579 | hit correction : 1.42203
hit initial : 206.557 | hit correction : 1.42189
hit initial : 147.263 | hit correction : 1.42152
hit initial : 33.398 | hit correction : 1.42133
 *** Total Charge : 1315.84
 Looking at view: 1 | Number of hits in view: 9
hit initial : 18.8889 | hit correction : 1.42186
hit initial : 106.198 | hit correction : 1.42182
hit initial : 103.24 | hit correction : 1.42185
hit initial : 114.05 | hit correction : 1.42203
hit initial : 140.92 | hit correction : 1.42208
hit initial : 113.639 | hit correction : 1.42194
hit initial : 281.914 | hit correction : 1.42169
hit initial : 30.8607 | hit correction : 1.42163
hit initial : 71.0954 | hit correction : 1.42133
 *** Total Charge : 1394.53
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 10th record. run: 5026 subRun: 0 event: 34400 at 13-Dec-2025 21:51:35 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  18
Added hit list length: 18 | Max was set to: -3.40282e+38
This pfp has size:  8
True Neutron Energy: 939.571
True Neutron Energy: 943.268
True Neutron Energy: 941.26
True Electron Energy: 10.1749
Completed MCParticle Loop
True Neutrino Energy: 22.7634
 Size of all hits: 18
hit initial : 112.88 | hit correction : 1.30749
hit initial : 230.585 | hit correction : 1.30736
hit initial : 247.745 | hit correction : 1.30729
hit initial : 209.723 | hit correction : 1.30727
hit initial : 140.548 | hit correction : 1.30732
hit initial : 68.858 | hit correction : 1.30737
hit initial : 18.3416 | hit correction : 1.30739
 *** Total Charge : 1344.83
 Looking at view: 0 | Number of hits in view: 7
hit initial : 13.6403 | hit correction : 1.30748
hit initial : 376.045 | hit correction : 1.30732
hit initial : 139.542 | hit correction : 1.30738
hit initial : 355.332 | hit correction : 1.3073
hit initial : 106.454 | hit correction : 1.30731
 *** Total Charge : 1295.58
 Looking at view: 1 | Number of hits in view: 5
hit initial : 98.2832 | hit correction : 1.30746
hit initial : 386.847 | hit correction : 1.30731
hit initial : 218.511 | hit correction : 1.30726
hit initial : 170.74 | hit correction : 1.30731
hit initial : 125.325 | hit correction : 1.30737
hit initial : 15.0697 | hit correction : 1.30744
 *** Total Charge : 1326.64
 Looking at view: 2 | Number of hits in view: 6
13-Dec-2025 21:51:38 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0842256     0.0897529     0.100609      0.0883956    0.00439862       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000360527   0.000456563   0.000847239   0.000417015   0.000132568      10     
end_path:ana:AnalyzeEvents         0.0837066     0.089154      0.0993294     0.0878593     0.0042127       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9814.44 MB
  Peak resident set size usage (VmHWM): 1540.03 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:12:13 UTC       justIN version: 01.06.00