justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262355.0@dunegpschedd02.fnal.gov

Jobsub ID262355.0@dunegpschedd02.fnal.gov
Workflow ID11170
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 22:37:06
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-14 09:01:16
From worker nodeHostnamedunegli-8066212-0-fnpc17126.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-14 03:31:11
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T222540Z_gen_000292_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time5h (19785s)
CPU time5h (19351s = 97%)
Max RSS bytes2797539328 (2667 MiB)
Outputting started2025-12-14 09:00:57
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11170/1/001/analysisLowE_20251214T033113Z.root
Finished2025-12-14 09:01:10
Saved logsjustin-logs:262355.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ile:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoracontent/LArMonitoring/EventValidationBaseAlgorithm.cc line#: 64
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6863, 6740, 6734
minwire 0: 78
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.116047, 
Output 1: 0.368147, 0.0617388, 0.300291, 0.269823, 
Output 2: 0.0395886, 0.0588027, 0.891903, 0.00970597, 
Output 3: 0.952914, 0.0413168, 0.00266778, 0.00310153, 
Output 4: 0.524819, 0.418673, 0.044789, 0.0117193, 
Output 5: 0.276498, 0.444446, 0.193403, 0.0856533, 
Output 6: 0.188311, 0.297827, 0.130386, 0.383476, 

14-Dec-2025 08:59:18 UTC  Closed output file "matt_reco2.root"
14-Dec-2025 08:59:18 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/22/8d/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T222540Z_gen_000292_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1586.02       1944.3        2474.69       1874.84       250.912        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000818748   0.00570087     0.0242563    0.00115017    0.00863097       10     
reco:pandora:StandardPandora                            1583.77       1941.83       2472.13       1873.21       251.111        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000387449    0.0425569     0.0839863     0.0545007     0.0323982       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000398776   0.00492055     0.0268045    0.00237282    0.00750754       10     
reco:pandoracalo:Calorimetry                          0.000347284    0.0159064     0.079855     0.000866392    0.0295792       10     
reco:pandorapid:Chi2ParticleID                        7.3282e-05    0.000302366   0.00175187    0.000141186   0.000485878      10     
reco:cvnmap:CVNMapper                                  0.0946912     0.136955       0.3226       0.119029      0.0649152       10     
reco:cvneva:CVNEvaluator                               0.843218       1.61089       3.87338       1.46057      0.874129        10     
reco:energyrecnumu:EnergyReco                          0.0787143     0.118803      0.136545      0.127851      0.0189815       10     
reco:energyrecnue:EnergyReco                          0.00174216    0.00680646     0.014266     0.00603048    0.00487297       10     
reco:energyrecnc:EnergyReco                            0.0086823     0.0114387     0.0149184    0.00985894    0.00242503       10     
[art]:TriggerResults:TriggerResultInserter            1.9167e-05    3.68349e-05   0.000155607   2.2975e-05    3.98821e-05      10     
end_path:out1:RootOutput                               7.56e-06     1.35272e-05   4.3882e-05     9.192e-06    1.06079e-05      10     
end_path:out1:RootOutput(write)                        0.343323      0.520444       1.00089      0.483656      0.197148        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14061.5 MB
  Peak resident set size usage (VmHWM): 2797.54 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
This pfp has size:  60
Added hit list length: 60 | Max was set to: -3.40282e+38
This pfp has size:  12
This pfp has size:  11
True Electron Energy: 41.3231
Completed MCParticle Loop
True Neutrino Energy: 47.5745
 Size of all hits: 60
hit initial : 102.022 | hit correction : 1.11398
hit initial : 79.3954 | hit correction : 1.114
hit initial : 101.21 | hit correction : 1.09492
hit initial : 262.29 | hit correction : 1.0951
hit initial : 187.506 | hit correction : 1.09524
hit initial : 69.8295 | hit correction : 1.09528
hit initial : 92.9295 | hit correction : 1.09402
hit initial : 349.757 | hit correction : 1.09369
hit initial : 674.083 | hit correction : 1.0935
hit initial : 193.129 | hit correction : 1.09348
hit initial : 160.234 | hit correction : 1.09229
hit initial : 80.5244 | hit correction : 1.09211
hit initial : 262.387 | hit correction : 1.09071
hit initial : 420.824 | hit correction : 1.09052
hit initial : 180.673 | hit correction : 1.09029
hit initial : 76.9125 | hit correction : 1.08939
hit initial : 88.835 | hit correction : 1.08897
 *** Total Charge : 3700.06
 Looking at view: 0 | Number of hits in view: 17
hit initial : 41.0202 | hit correction : 1.11395
hit initial : 117.286 | hit correction : 1.11397
hit initial : 44.0184 | hit correction : 1.11396
hit initial : 262.381 | hit correction : 1.09486
hit initial : 173.533 | hit correction : 1.09522
hit initial : 153.241 | hit correction : 1.09524
hit initial : 111.352 | hit correction : 1.0941
hit initial : 199.464 | hit correction : 1.09386
hit initial : 229.532 | hit correction : 1.09369
hit initial : 370.375 | hit correction : 1.09348
hit initial : 551.491 | hit correction : 1.09336
hit initial : 439.78 | hit correction : 1.0929
hit initial : 330.323 | hit correction : 1.09224
hit initial : 365.752 | hit correction : 1.09168
hit initial : 429.807 | hit correction : 1.09089
hit initial : 419.745 | hit correction : 1.09005
hit initial : 272.689 | hit correction : 1.08927
hit initial : 33.3983 | hit correction : 1.08902
 *** Total Charge : 4970.1
 Looking at view: 1 | Number of hits in view: 18
hit initial : 29.7467 | hit correction : 1.10783
hit initial : 233.599 | hit correction : 1.09127
hit initial : 37.3005 | hit correction : 1.09679
hit initial : 29.714 | hit correction : 1.11451
hit initial : 16.7394 | hit correction : 1.11389
hit initial : 182.752 | hit correction : 1.11397
hit initial : 18.8461 | hit correction : 1.11395
hit initial : 137.81 | hit correction : 1.09508
hit initial : 485.852 | hit correction : 1.09521
hit initial : 302.99 | hit correction : 1.09435
hit initial : 211.329 | hit correction : 1.09393
hit initial : 202.23 | hit correction : 1.09377
hit initial : 177.109 | hit correction : 1.09359
hit initial : 505.416 | hit correction : 1.09349
hit initial : 333.005 | hit correction : 1.0933
hit initial : 177.034 | hit correction : 1.09302
hit initial : 160.117 | hit correction : 1.09277
hit initial : 187.543 | hit correction : 1.09251
hit initial : 165.29 | hit correction : 1.09223
hit initial : 177.699 | hit correction : 1.092
hit initial : 205.098 | hit correction : 1.09172
hit initial : 231.226 | hit correction : 1.09081
hit initial : 264.954 | hit correction : 1.09032
hit initial : 336.017 | hit correction : 1.08964
hit initial : 256.329 | hit correction : 1.08902
 *** Total Charge : 5540.77
 Looking at view: 2 | Number of hits in view: 25
Begin processing the 10th record. run: 5027 subRun: 0 event: 2920 at 14-Dec-2025 09:00:48 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Neutron Energy: 939.579
True Electron Energy: 7.57529
Completed MCParticle Loop
True Neutrino Energy: 10.8608
 Size of all hits: 0
14-Dec-2025 09:00:55 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0931891     0.278633       0.54657      0.271627      0.131675        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000531703   0.00100173    0.00184147    0.000849032    0.0004021       10     
end_path:ana:AnalyzeEvents         0.0924284     0.277373      0.545485      0.270038      0.131493        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11962.1 MB
  Peak resident set size usage (VmHWM): 1537.41 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:08:17 UTC       justIN version: 01.06.00