justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257913.98@dunegpschedd02.fnal.gov

Jobsub ID257913.98@dunegpschedd02.fnal.gov
Workflow ID11027
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 11:27:40
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 12:00:23
From worker nodeHostnamedunegli-7832230-0-fnpc18004.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-06 00:43:14
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T213113Z_gen_004106_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time11h (40596s)
CPU time10h (38285s = 94%)
Max RSS bytes2997637120 (2858 MiB)
Outputting started2025-12-06 11:59:51
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11027/1/001/analysisLowE_20251206T004315Z.root
Finished2025-12-06 12:00:23
Saved logsjustin-logs:257913.98-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6692, 6752, 6591
minwire 0: 45
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1250, 1749
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.176746, 
Output 1: 0.123326, 0.117444, 0.148104, 0.611126, 
Output 2: 0.030223, 0.168389, 0.775131, 0.0262571, 
Output 3: 0.980533, 0.00755575, 0.0020558, 0.00985535, 
Output 4: 0.6211, 0.332228, 0.0406509, 0.00602087, 
Output 5: 0.217809, 0.407435, 0.292557, 0.0821995, 
Output 6: 0.0383071, 0.0826577, 0.0895725, 0.789463, 

06-Dec-2025 11:58:21 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 11:58:21 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/b6/51/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T213113Z_gen_004106_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3080.04       4027.01       4941.75       4093.13       465.198        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0022897     0.0116016     0.0213537     0.0104071    0.00519494       10     
reco:pandora:StandardPandora                            3076.05       3967.79       4873.84       4018.82       445.805        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00595799     0.0553815     0.244397      0.0208799     0.0690767       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00219992     0.010652      0.0397066    0.00877225     0.0103776       10     
reco:pandoracalo:Calorimetry                          0.000613288    0.0230861     0.086953     0.00301809     0.0320964       10     
reco:pandorapid:Chi2ParticleID                        0.000128322   0.00101863    0.00472193    0.000200784   0.00166195       10     
reco:cvnmap:CVNMapper                                  0.0600951     0.0835871     0.126676      0.0832516     0.0164397       10     
reco:cvneva:CVNEvaluator                                2.33934       3.49565       6.66274       3.12034       1.23385        10     
reco:energyrecnumu:EnergyReco                         0.00373396      13.3112       70.3692      0.0217835      26.5778        10     
reco:energyrecnue:EnergyReco                          0.00211778    0.00863511     0.0202517    0.00316499    0.00708335       10     
reco:energyrecnc:EnergyReco                            0.0143702      41.599        74.3323       63.3126       34.1881        10     
[art]:TriggerResults:TriggerResultInserter            2.6811e-05    5.74746e-05   0.000165312   4.72395e-05   3.78004e-05      10     
end_path:out1:RootOutput                               1.047e-05    1.90368e-05   6.5002e-05    1.36055e-05   1.55854e-05      10     
end_path:out1:RootOutput(write)                        0.475488      0.616035      0.690482      0.627421       0.05751        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21751 MB
  Peak resident set size usage (VmHWM): 2997.64 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 212.058 | hit correction : 1.13779
hit initial : 267.784 | hit correction : 1.13941
hit initial : 406.952 | hit correction : 1.13862
hit initial : 224.694 | hit correction : 1.13827
hit initial : 326.566 | hit correction : 1.13793
hit initial : 346.721 | hit correction : 1.13782
hit initial : 143.288 | hit correction : 1.13783
hit initial : 175.907 | hit correction : 1.13792
hit initial : 238.092 | hit correction : 1.13799
hit initial : 131.234 | hit correction : 1.13807
 *** Total Charge : 3239.78
 Looking at view: 2 | Number of hits in view: 13
Begin processing the 7th record. run: 5026 subRun: 0 event: 41057 at 06-Dec-2025 11:59:28 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 47.3848
Completed MCParticle Loop
True Neutrino Energy: 51.1133
 Size of all hits: 0
Begin processing the 8th record. run: 5026 subRun: 0 event: 41058 at 06-Dec-2025 11:59:33 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 6.52921
Completed MCParticle Loop
True Neutrino Energy: 14.317
 Size of all hits: 0
Begin processing the 9th record. run: 5026 subRun: 0 event: 41059 at 06-Dec-2025 11:59:39 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  33
Added hit list length: 33 | Max was set to: -3.40282e+38
True Electron Energy: 18.0238
Completed MCParticle Loop
True Neutrino Energy: 22.8682
 Size of all hits: 33
hit initial : 47.6984 | hit correction : 1.37911
hit initial : 50.605 | hit correction : 1.37914
hit initial : 47.6017 | hit correction : 1.37615
hit initial : 78.6774 | hit correction : 1.37525
hit initial : 107.807 | hit correction : 1.3761
hit initial : 37.6914 | hit correction : 1.37252
hit initial : 108.478 | hit correction : 1.37595
hit initial : 49.3361 | hit correction : 1.37529
hit initial : 283.604 | hit correction : 1.37509
hit initial : 198.255 | hit correction : 1.37452
hit initial : 185.402 | hit correction : 1.37401
hit initial : 200.431 | hit correction : 1.37365
hit initial : 220.967 | hit correction : 1.37316
hit initial : 148.885 | hit correction : 1.37272
 *** Total Charge : 2426.83
 Looking at view: 0 | Number of hits in view: 14
hit initial : 95.0831 | hit correction : 1.37259
hit initial : 184.782 | hit correction : 1.37284
hit initial : 392.09 | hit correction : 1.37489
hit initial : 125.019 | hit correction : 1.37606
hit initial : 64.5283 | hit correction : 1.37612
hit initial : 47.4059 | hit correction : 1.37531
hit initial : 367.62 | hit correction : 1.37465
hit initial : 61.2406 | hit correction : 1.37449
hit initial : 116.925 | hit correction : 1.37375
hit initial : 215.266 | hit correction : 1.37337
 *** Total Charge : 2295.09
 Looking at view: 1 | Number of hits in view: 10
hit initial : 57.1748 | hit correction : 1.37519
hit initial : 20.7066 | hit correction : 1.37674
hit initial : 325.588 | hit correction : 1.37526
hit initial : 214.006 | hit correction : 1.37605
hit initial : 163.374 | hit correction : 1.3746
hit initial : 144.9 | hit correction : 1.3742
hit initial : 128.158 | hit correction : 1.37256
hit initial : 112.715 | hit correction : 1.37354
hit initial : 476.727 | hit correction : 1.37338
 *** Total Charge : 2258.53
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 10th record. run: 5026 subRun: 0 event: 41060 at 06-Dec-2025 11:59:44 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.577
True Electron Energy: 4.93575
Completed MCParticle Loop
True Neutrino Energy: 8.2207
 Size of all hits: 0
06-Dec-2025 11:59:50 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          0.13103      0.155795      0.185251      0.159533      0.0155828       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000595425   0.000843843   0.00186437    0.000771221   0.000351663      10     
end_path:ana:AnalyzeEvents         0.130141      0.154654      0.184156      0.158531      0.0156556       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19472.4 MB
  Peak resident set size usage (VmHWM): 1530.44 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:10:31 UTC       justIN version: 01.06.00