justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258262.0@dunegpschedd02.fnal.gov

Jobsub ID258262.0@dunegpschedd02.fnal.gov
Workflow ID11012
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 22:53:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 11:12:25
From worker nodeHostnamedunegli-7989276-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:16
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001304Z_gen_002003_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time9h (34014s)
CPU time8h (31119s = 91%)
Max RSS bytes3056185344 (2914 MiB)
Outputting started2025-12-06 11:05:11
Output files
Finished2025-12-06 11:12:25
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

T_INITIALIZED
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6761, 6717, 6790
minwire 0: 35
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 535, 1034
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.100486, 
Output 1: 0.104938, 0.0827976, 0.15247, 0.659795, 
Output 2: 0.0343097, 0.112589, 0.841843, 0.011258, 
Output 3: 0.976225, 0.0200547, 0.00132236, 0.00239748, 
Output 4: 0.59667, 0.347894, 0.0467612, 0.00867413, 
Output 5: 0.30187, 0.474362, 0.179491, 0.044277, 
Output 6: 0.0466188, 0.122885, 0.103376, 0.72712, 

06-Dec-2025 11:03:24 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 11:03:24 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7a/f2/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T001304Z_gen_002003_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2562.56       3363.46       5128.11       3010.16       776.614        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00526685     0.0377211     0.237602      0.0114368     0.0676296       10     
reco:pandora:StandardPandora                            2559.9        3357.85       5112.87       3006.16       773.15         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000658241    0.0428104     0.303399     0.00566102     0.0889238       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000549146   0.00988226     0.0656182    0.00283196     0.0188139       10     
reco:pandoracalo:Calorimetry                          0.00032345     0.0281338     0.0947628     0.0188326     0.0279898       10     
reco:pandorapid:Chi2ParticleID                        9.7714e-05    0.000804655   0.00560623    0.000168948   0.00161522       10     
reco:cvnmap:CVNMapper                                  0.0426856     0.0889744      0.25864      0.0721767     0.0606646       10     
reco:cvneva:CVNEvaluator                                2.02415       4.51756       13.3328       3.31271       3.39062        10     
reco:energyrecnumu:EnergyReco                          0.0173647     0.118554      0.212651      0.122147      0.0597232       10     
reco:energyrecnue:EnergyReco                          0.00273499     0.0136854     0.026345      0.0129265    0.00839238       10     
reco:energyrecnc:EnergyReco                            0.0104417     0.0417348     0.140523      0.0161759     0.0464919       10     
[art]:TriggerResults:TriggerResultInserter            2.9255e-05    5.22756e-05   0.000200508   3.25665e-05   5.00197e-05      10     
end_path:out1:RootOutput                              1.1892e-05    2.28951e-05   0.000103355    1.338e-05    2.68987e-05      10     
end_path:out1:RootOutput(write)                        0.418222      0.706813       1.05793      0.701204      0.214707        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21789.7 MB
  Peak resident set size usage (VmHWM): 3056.19 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 25.9263 | hit correction : 1.26535
hit initial : 23.6461 | hit correction : 1.26758
hit initial : 184.807 | hit correction : 1.26867
hit initial : 188.44 | hit correction : 1.26882
hit initial : 219.722 | hit correction : 1.26896
hit initial : 409.287 | hit correction : 1.26896
hit initial : 580.298 | hit correction : 1.26901
hit initial : 491.017 | hit correction : 1.26951
hit initial : 135.826 | hit correction : 1.26978
 *** Total Charge : 3921.79
 Looking at view: 2 | Number of hits in view: 16
Begin processing the 8th record. run: 5027 subRun: 0 event: 20028 at 06-Dec-2025 11:04:48 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 55.4532
Completed MCParticle Loop
True Neutrino Energy: 60.8697
 Size of all hits: 0
Begin processing the 9th record. run: 5027 subRun: 0 event: 20029 at 06-Dec-2025 11:04:55 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  40
Added hit list length: 40 | Max was set to: -3.40282e+38
True Electron Energy: 23.7457
Completed MCParticle Loop
True Neutrino Energy: 27.0448
 Size of all hits: 40
hit initial : 193.093 | hit correction : 1.1595
hit initial : 28.1766 | hit correction : 1.16032
hit initial : 569.192 | hit correction : 1.15942
hit initial : 182.538 | hit correction : 1.16007
hit initial : 108.752 | hit correction : 1.15919
hit initial : 665.093 | hit correction : 1.1592
hit initial : 957.893 | hit correction : 1.15924
 *** Total Charge : 3135.75
 Looking at view: 0 | Number of hits in view: 7
hit initial : 224.866 | hit correction : 1.15907
hit initial : 44.2382 | hit correction : 1.16015
hit initial : 175.98 | hit correction : 1.15901
hit initial : 100.134 | hit correction : 1.1601
hit initial : 130.004 | hit correction : 1.15906
hit initial : 61.9134 | hit correction : 1.15944
hit initial : 324.352 | hit correction : 1.15947
hit initial : 336.434 | hit correction : 1.1592
hit initial : 412.602 | hit correction : 1.15931
hit initial : 381.01 | hit correction : 1.15945
hit initial : 280.146 | hit correction : 1.15949
hit initial : 99.553 | hit correction : 1.15949
 *** Total Charge : 2980.97
 Looking at view: 1 | Number of hits in view: 12
hit initial : 60.5401 | hit correction : 1.15947
hit initial : 108.327 | hit correction : 1.15948
hit initial : 110.067 | hit correction : 1.15949
hit initial : 124.214 | hit correction : 1.15949
hit initial : 140.304 | hit correction : 1.15951
hit initial : 120.167 | hit correction : 1.15951
hit initial : 116.08 | hit correction : 1.15943
hit initial : 278.21 | hit correction : 1.1592
hit initial : 144.721 | hit correction : 1.1591
hit initial : 165.922 | hit correction : 1.159
hit initial : 124.42 | hit correction : 1.15898
hit initial : 180.589 | hit correction : 1.16016
hit initial : 124.286 | hit correction : 1.15902
hit initial : 266.987 | hit correction : 1.15978
hit initial : 134.938 | hit correction : 1.15907
hit initial : 222.248 | hit correction : 1.15934
hit initial : 25.1296 | hit correction : 1.15856
hit initial : 32.7149 | hit correction : 1.15806
hit initial : 16.9096 | hit correction : 1.15802
hit initial : 274.648 | hit correction : 1.15925
hit initial : 119.081 | hit correction : 1.15935
 *** Total Charge : 3351.1
 Looking at view: 2 | Number of hits in view: 21
Begin processing the 10th record. run: 5027 subRun: 0 event: 20030 at 06-Dec-2025 11:05:02 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.571
True Neutron Energy: 939.718
True Electron Energy: 39.4753
Completed MCParticle Loop
True Neutrino Energy: 48.467
 Size of all hits: 0
06-Dec-2025 11:05:10 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.117296      0.267499      0.661731      0.160574      0.175022        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000648202   0.00108399    0.00258794    0.00102492    0.000529717      10     
end_path:ana:AnalyzeEvents         0.116009      0.266049      0.658054      0.159281      0.174467        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.2 MB
  Peak resident set size usage (VmHWM): 1540.03 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 13:10:04 UTC       justIN version: 01.05.03