justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262039.1@dunegpschedd02.fnal.gov

Jobsub ID262039.1@dunegpschedd02.fnal.gov
Workflow ID11169
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 07:30:21
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 15:29:22
From worker nodeHostnamedunegli-7890560-0-fnpc17112.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 07:31:32
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T133556Z_gen_000498_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time7h (28653s)
CPU time7h (27319s = 95%)
Max RSS bytes2796445696 (2666 MiB)
Outputting started2025-12-13 15:29:07
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11169/1/001/analysisLowE_20251213T073133Z.root
Finished2025-12-13 15:29:22
Saved logsjustin-logs:262039.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

rs/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoracontent/LArMonitoring/EventValidationBaseAlgorithm.cc line#: 64
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6801, 6714, 6668
minwire 0: 29
minwire 1: 1
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.109376, 
Output 1: 0.194649, 0.0906108, 0.248505, 0.466235, 
Output 2: 0.0462967, 0.10331, 0.831896, 0.0184971, 
Output 3: 0.943275, 0.0436668, 0.00516474, 0.00789349, 
Output 4: 0.555564, 0.400645, 0.037281, 0.00651031, 
Output 5: 0.245652, 0.451502, 0.227537, 0.0753092, 
Output 6: 0.133343, 0.184619, 0.114834, 0.567203, 

13-Dec-2025 15:27:35 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 15:27:35 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f3/e3/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250116T133556Z_gen_000498_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2213.36       2826.31       3280.86       2813.51       314.049        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00142945    0.00589195     0.0179217    0.00160277    0.00652795       10     
reco:pandora:StandardPandora                            2211.8        2823.35       3277.37       2810.82       313.199        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000443664    0.0752755     0.104413      0.0940297     0.0375796       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00127049     0.0159037     0.0873477    0.00426094     0.026496        10     
reco:pandoracalo:Calorimetry                          0.00025843    0.00178736    0.00980981    0.000811925   0.00272018       10     
reco:pandorapid:Chi2ParticleID                        8.3667e-05    0.000623463   0.00448791    0.000208933   0.00128982       10     
reco:cvnmap:CVNMapper                                  0.128657      0.195194      0.395598      0.178494      0.0718877       10     
reco:cvneva:CVNEvaluator                               0.935584       1.92062        5.158        1.52003       1.15709        10     
reco:energyrecnumu:EnergyReco                          0.0906919     0.141812      0.223235      0.138386      0.0364219       10     
reco:energyrecnue:EnergyReco                          0.00157631    0.00372001     0.0097665    0.00308128    0.00210536       10     
reco:energyrecnc:EnergyReco                            0.0083145     0.0157813     0.0396524     0.0139677    0.00883559       10     
[art]:TriggerResults:TriggerResultInserter            1.9208e-05    4.59311e-05   0.000146865   3.7969e-05    3.50352e-05      10     
end_path:out1:RootOutput                               7.442e-06    1.68579e-05   5.3641e-05    1.2775e-05    1.27182e-05      10     
end_path:out1:RootOutput(write)                        0.330466      0.584628      0.779322      0.599179      0.142638        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14064.8 MB
  Peak resident set size usage (VmHWM): 2796.45 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 19.8745 | hit correction : 1.42484
hit initial : 103.686 | hit correction : 1.42476
hit initial : 33.1661 | hit correction : 1.42473
hit initial : 128.199 | hit correction : 1.37928
hit initial : 207.675 | hit correction : 1.37934
hit initial : 41.3324 | hit correction : 1.37943
 *** Total Charge : 1421.95
 Looking at view: 2 | Number of hits in view: 10
Begin processing the 9th record. run: 4696 subRun: 0 event: 4979 at 13-Dec-2025 15:28:51 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  15
Added hit list length: 15 | Max was set to: -3.40282e+38
True Electron Energy: 41.5149
Completed MCParticle Loop
True Neutrino Energy: 46.0269
 Size of all hits: 15
hit initial : 73.2657 | hit correction : 1.2831
hit initial : 183.432 | hit correction : 1.28311
hit initial : 192.566 | hit correction : 1.28303
hit initial : 82.9256 | hit correction : 1.28224
hit initial : 59.6236 | hit correction : 1.2828
hit initial : 177.584 | hit correction : 1.28214
hit initial : 66.0163 | hit correction : 1.28201
 *** Total Charge : 1071.57
 Looking at view: 0 | Number of hits in view: 7
hit initial : 108.47 | hit correction : 1.28306
hit initial : 224.719 | hit correction : 1.2831
hit initial : 182.339 | hit correction : 1.28299
hit initial : 187.499 | hit correction : 1.28227
 *** Total Charge : 901.872
 Looking at view: 1 | Number of hits in view: 4
hit initial : 141.055 | hit correction : 1.28309
hit initial : 291.911 | hit correction : 1.28309
hit initial : 224.619 | hit correction : 1.28251
hit initial : 154.192 | hit correction : 1.28197
 *** Total Charge : 1041.28
 Looking at view: 2 | Number of hits in view: 4
Begin processing the 10th record. run: 4696 subRun: 0 event: 4980 at 13-Dec-2025 15:28:57 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  28
Added hit list length: 28 | Max was set to: -3.40282e+38
True Neutron Energy: 939.57
True Electron Energy: 33.5039
Completed MCParticle Loop
True Neutrino Energy: 39.0619
 Size of all hits: 28
hit initial : 336.65 | hit correction : 1.00748
hit initial : 186.402 | hit correction : 1.00746
hit initial : 26.9761 | hit correction : 1.00745
hit initial : 233.565 | hit correction : 1.00636
hit initial : 476.822 | hit correction : 1.00635
hit initial : 23.6666 | hit correction : 1.00744
hit initial : 75.6817 | hit correction : 1.00644
hit initial : 50.6714 | hit correction : 1.00587
hit initial : 25.4816 | hit correction : 1.01154
hit initial : 126.01 | hit correction : 1.00583
 *** Total Charge : 1572.54
 Looking at view: 0 | Number of hits in view: 10
hit initial : 187.682 | hit correction : 1.00751
hit initial : 289.47 | hit correction : 1.00748
hit initial : 135.864 | hit correction : 1.00743
hit initial : 270.427 | hit correction : 1.00635
hit initial : 454.939 | hit correction : 1.0064
hit initial : 23.5055 | hit correction : 1.00742
hit initial : 364.948 | hit correction : 1.0063
hit initial : 225.314 | hit correction : 1.0059
hit initial : 29.8752 | hit correction : 1.01154
hit initial : 133.51 | hit correction : 1.00581
 *** Total Charge : 2129.67
 Looking at view: 1 | Number of hits in view: 10
hit initial : 37.9033 | hit correction : 1.00831
hit initial : 635.769 | hit correction : 1.00747
hit initial : 450.516 | hit correction : 1.00636
hit initial : 599.164 | hit correction : 1.00639
hit initial : 33.3219 | hit correction : 1.0074
hit initial : 295.162 | hit correction : 1.00595
hit initial : 57.3112 | hit correction : 1.01155
hit initial : 200.098 | hit correction : 1.00578
 *** Total Charge : 2324.83
 Looking at view: 2 | Number of hits in view: 8
13-Dec-2025 15:29:04 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.132677      0.255568      0.771127      0.204988      0.176497        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000723671    0.0319209     0.311706     0.00083416     0.0932619       10     
end_path:ana:AnalyzeEvents          0.13155      0.223395       0.45873      0.204003      0.0881557       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11960.5 MB
  Peak resident set size usage (VmHWM): 1539.54 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:09:13 UTC       justIN version: 01.06.00