justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268013.1@dunegpschedd01.fnal.gov

Jobsub ID268013.1@dunegpschedd01.fnal.gov
Workflow ID11169
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 11:52:35
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 22:05:26
From worker nodeHostnamedunegli-7890498-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 11:53:11
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T161831Z_gen_000313_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time10h (36720s)
CPU time9h (34636s = 94%)
Max RSS bytes2793635840 (2664 MiB)
Outputting started2025-12-13 22:05:11
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11169/1/001/analysisLowE_20251213T115314Z.root
Finished2025-12-13 22:05:26
Saved logsjustin-logs:268013.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 22:04:01 UTC run: 5026 subRun: 0 event: 3130
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 22:04:01 UTC run: 5026 subRun: 0 event: 3130
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6620, 6555, 6656
minwire 0: 28
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.160597, 
Output 1: 0.0799378, 0.11501, 0.180202, 0.62485, 
Output 2: 0.0463591, 0.166692, 0.76044, 0.0265095, 
Output 3: 0.969904, 0.0205713, 0.00327053, 0.00625394, 
Output 4: 0.501937, 0.430428, 0.0592573, 0.00837754, 
Output 5: 0.241498, 0.47631, 0.228884, 0.0533075, 
Output 6: 0.048985, 0.117414, 0.109977, 0.723624, 

13-Dec-2025 22:04:05 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 22:04:05 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/c2/5f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T161831Z_gen_000313_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2107.49       3604.14       5126.89       3567.36       1058.86        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000902881    0.0069908     0.0313461    0.00142543     0.0098383       10     
reco:pandora:StandardPandora                            2105.03       3600.8        5119.64       3564.53       1057.94        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000831646    0.0674199     0.125712      0.0725055     0.0419518       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000716981   0.00896445     0.0417808    0.00412438     0.0115666       10     
reco:pandoracalo:Calorimetry                          0.000369095   0.00243895     0.0121887    0.000970358    0.0035856       10     
reco:pandorapid:Chi2ParticleID                        0.000129574   0.000734417   0.00549935    0.000197177    0.0015899       10     
reco:cvnmap:CVNMapper                                  0.0614576     0.105264      0.144409      0.112012       0.02563        10     
reco:cvneva:CVNEvaluator                                1.87147       2.5712        6.10245       2.16477       1.2084         10     
reco:energyrecnumu:EnergyReco                          0.0896814     0.118882      0.189863      0.108148       0.02987        10     
reco:energyrecnue:EnergyReco                          0.00157127    0.00376283     0.0125617    0.00290155    0.00313427       10     
reco:energyrecnc:EnergyReco                           0.00921106     0.011222      0.0150661     0.0113024    0.00165927       10     
[art]:TriggerResults:TriggerResultInserter            1.9237e-05    4.16854e-05   0.000140715   2.96105e-05   3.44362e-05      10     
end_path:out1:RootOutput                               7.965e-06    1.5438e-05    4.2179e-05    1.23085e-05   9.62715e-06      10     
end_path:out1:RootOutput(write)                        0.364177      0.449454      0.699865      0.407808      0.111021        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21570.2 MB
  Peak resident set size usage (VmHWM): 2793.64 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 65.8175 | hit correction : 1.24643
hit initial : 254.153 | hit correction : 1.24638
hit initial : 521.051 | hit correction : 1.24636
 *** Total Charge : 2719.46
 Looking at view: 2 | Number of hits in view: 13
Begin processing the 9th record. run: 5026 subRun: 0 event: 3129 at 13-Dec-2025 22:05:02 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  25
Added hit list length: 25 | Max was set to: -3.40282e+38
This pfp has size:  18
This pfp has size:  6
True Electron Energy: 24.4767
Completed MCParticle Loop
True Neutrino Energy: 32.2803
 Size of all hits: 25
hit initial : 46.2956 | hit correction : 1.33612
hit initial : 52.7549 | hit correction : 1.33543
hit initial : 149.929 | hit correction : 1.33607
hit initial : 182.443 | hit correction : 1.33542
hit initial : 445.107 | hit correction : 1.33554
hit initial : 430.677 | hit correction : 1.33541
hit initial : 333.537 | hit correction : 1.33529
hit initial : 225.727 | hit correction : 1.33518
hit initial : 217.416 | hit correction : 1.33494
hit initial : 194.383 | hit correction : 1.33481
 *** Total Charge : 3042.31
 Looking at view: 0 | Number of hits in view: 10
hit initial : 118.638 | hit correction : 1.33593
hit initial : 594.461 | hit correction : 1.3354
hit initial : 408.11 | hit correction : 1.33574
hit initial : 580.587 | hit correction : 1.33526
hit initial : 408.89 | hit correction : 1.33503
hit initial : 31.3824 | hit correction : 1.33479
 *** Total Charge : 2860.47
 Looking at view: 1 | Number of hits in view: 6
hit initial : 124.732 | hit correction : 1.33608
hit initial : 47.962 | hit correction : 1.33546
hit initial : 258.153 | hit correction : 1.3359
hit initial : 154.367 | hit correction : 1.33541
hit initial : 158.334 | hit correction : 1.33533
hit initial : 651.496 | hit correction : 1.33536
hit initial : 124.6 | hit correction : 1.3353
hit initial : 245.846 | hit correction : 1.33521
hit initial : 374.57 | hit correction : 1.33493
 *** Total Charge : 2857.78
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 10th record. run: 5026 subRun: 0 event: 3130 at 13-Dec-2025 22:05:07 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  17
Added hit list length: 17 | Max was set to: -3.40282e+38
This pfp has size:  4
This pfp has size:  4
True Neutron Energy: 939.577
True Electron Energy: 11.911
Completed MCParticle Loop
True Neutrino Energy: 15.1963
 Size of all hits: 17
hit initial : 58.0058 | hit correction : 1.20199
hit initial : 224.38 | hit correction : 1.20195
hit initial : 281.305 | hit correction : 1.20195
hit initial : 259.916 | hit correction : 1.20219
hit initial : 209.822 | hit correction : 1.20235
hit initial : 78.5738 | hit correction : 1.20242
hit initial : 14.98 | hit correction : 1.20251
 *** Total Charge : 1354.77
 Looking at view: 0 | Number of hits in view: 7
hit initial : 128.607 | hit correction : 1.20199
hit initial : 265.888 | hit correction : 1.20197
hit initial : 425.835 | hit correction : 1.20212
hit initial : 303.019 | hit correction : 1.20229
hit initial : 58.6787 | hit correction : 1.20239
 *** Total Charge : 1420.95
 Looking at view: 1 | Number of hits in view: 5
hit initial : 94.461 | hit correction : 1.20188
hit initial : 567.422 | hit correction : 1.20197
hit initial : 334.45 | hit correction : 1.20224
hit initial : 178.069 | hit correction : 1.20242
hit initial : 36.1834 | hit correction : 1.20253
 *** Total Charge : 1455.27
 Looking at view: 2 | Number of hits in view: 5
13-Dec-2025 22:05:10 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          0.11707      0.152934      0.251494      0.143079      0.0373132       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000548464   0.000851602   0.00180174    0.000649745   0.000422772      10     
end_path:ana:AnalyzeEvents         0.116155      0.151811      0.249191       0.14168      0.0369658       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19474.1 MB
  Peak resident set size usage (VmHWM): 1530.01 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:07:52 UTC       justIN version: 01.06.00