justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 249613.0@dunegpschedd02.fnal.gov

Jobsub ID249613.0@dunegpschedd02.fnal.gov
Workflow ID10411
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-22 10:41:28
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-22 13:47:16
From worker nodeHostnamedunegli-7693717-0-fnpc23018.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-22 11:35:36
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194331Z_gen_000990_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time2h (7890s)
CPU time2h (7812s = 99%)
Max RSS bytes3072221184 (2929 MiB)
Outputting started2025-11-22 13:47:07
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/10411/1/001/analysisLowE_20251122T113537Z.root
Finished2025-11-22 13:47:16
Saved logsjustin-logs:249613.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0005, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6712, 6676, 6638
minwire 0: 10
minwire 1: 1
minwire 2: 3
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1818, 2317
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.160374, 
Output 1: 0.0860275, 0.0484675, 0.111144, 0.754361, 
Output 2: 0.13221, 0.243946, 0.571912, 0.0519327, 
Output 3: 0.976871, 0.00737741, 0.00150745, 0.0142438, 
Output 4: 0.395591, 0.557903, 0.0418738, 0.00463254, 
Output 5: 0.584251, 0.361687, 0.0424992, 0.0115622, 
Output 6: 0.0172115, 0.0435197, 0.0621455, 0.877123, 

22-Nov-2025 13:46:40 UTC  Closed output file "matt_reco2.root"
22-Nov-2025 13:46:40 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ea/93/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194331Z_gen_000990_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              667.903       780.091       906.091       775.079       72.2477        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00139047     0.0103261     0.047554     0.00490464     0.0136151       10     
reco:pandora:StandardPandora                            664.711       778.854       905.101       774.022       72.5909        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000213749    0.0172856     0.118333     0.00203871     0.0352182       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000202537   0.00604799     0.0301031    0.000679599   0.00974834       10     
reco:pandoracalo:Calorimetry                          0.00016174    0.00135961    0.00565742    0.000229914   0.00189994       10     
reco:pandorapid:Chi2ParticleID                        3.8744e-05    0.000158036   0.00105294    6.1212e-05    0.000298568      10     
reco:cvnmap:CVNMapper                                  0.0190182     0.024469      0.0366354     0.0239275    0.00467222       10     
reco:cvneva:CVNEvaluator                               0.700871      0.893693       2.46452      0.717305      0.523745        10     
reco:energyrecnumu:EnergyReco                         0.00110924     0.0344364     0.0810334     0.0375468     0.0270848       10     
reco:energyrecnue:EnergyReco                          0.000780883   0.00253391    0.00508796    0.00107613    0.00198284       10     
reco:energyrecnc:EnergyReco                           0.00470396     0.0265363     0.0915012    0.00507872     0.0296313       10     
[art]:TriggerResults:TriggerResultInserter             8.466e-06    2.55528e-05   0.000135449   1.38615e-05   3.66855e-05      10     
end_path:out1:RootOutput                               4.479e-06    9.7487e-06    3.5157e-05     7.399e-06    8.55403e-06      10     
end_path:out1:RootOutput(write)                        0.193582      0.219214      0.397933      0.200198      0.0596693       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21780.3 MB
  Peak resident set size usage (VmHWM): 3072.22 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Completed MCParticle Loop
True Neutrino Energy: 21.0662
 Size of all hits: 24
hit initial : 47.8617 | hit correction : 1.2238
hit initial : 95.0006 | hit correction : 1.22377
hit initial : 26.9416 | hit correction : 1.22248
hit initial : 115.194 | hit correction : 1.22239
hit initial : 81.3332 | hit correction : 1.22238
hit initial : 61.1959 | hit correction : 1.2216
hit initial : 273.964 | hit correction : 1.22156
hit initial : 419.574 | hit correction : 1.2213
hit initial : 193.765 | hit correction : 1.22072
hit initial : 59.9332 | hit correction : 1.22054
hit initial : 114.201 | hit correction : 1.21957
hit initial : 159.908 | hit correction : 1.21955
 *** Total Charge : 2013.82
 Looking at view: 0 | Number of hits in view: 12
hit initial : 134.735 | hit correction : 1.21956
hit initial : 195.125 | hit correction : 1.21956
hit initial : 57.9543 | hit correction : 1.21957
hit initial : 49.0868 | hit correction : 1.22244
hit initial : 50.5628 | hit correction : 1.2224
hit initial : 41.9297 | hit correction : 1.22156
hit initial : 68.8632 | hit correction : 1.22152
hit initial : 118.143 | hit correction : 1.22144
hit initial : 476.372 | hit correction : 1.22141
hit initial : 81.4131 | hit correction : 1.22066
hit initial : 89.2146 | hit correction : 1.22064
hit initial : 131.46 | hit correction : 1.22053
 *** Total Charge : 1824.99
 Looking at view: 1 | Number of hits in view: 12
Begin processing the 7th record. run: 3799 subRun: 0 event: 9897 at 22-Nov-2025 13:47:00 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 44.8933
Completed MCParticle Loop
True Neutrino Energy: 48.2065
 Size of all hits: 0
Begin processing the 8th record. run: 3799 subRun: 0 event: 9898 at 22-Nov-2025 13:47:02 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 1.94785
Completed MCParticle Loop
True Neutrino Energy: 5.2315
 Size of all hits: 0
Begin processing the 9th record. run: 3799 subRun: 0 event: 9899 at 22-Nov-2025 13:47:03 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  14
Added hit list length: 14 | Max was set to: -3.40282e+38
True Electron Energy: 30.7601
Completed MCParticle Loop
True Neutrino Energy: 53.7817
 Size of all hits: 14
hit initial : 133.872 | hit correction : 1.08988
hit initial : 85.3987 | hit correction : 1.08917
hit initial : 311.15 | hit correction : 1.08938
hit initial : 73.7087 | hit correction : 1.08912
 *** Total Charge : 658.158
 Looking at view: 0 | Number of hits in view: 4
hit initial : 49.4263 | hit correction : 1.08994
hit initial : 235.219 | hit correction : 1.08974
hit initial : 57.0154 | hit correction : 1.08898
hit initial : 54.811 | hit correction : 1.08897
hit initial : 49.8171 | hit correction : 1.08953
hit initial : 238.828 | hit correction : 1.08948
 *** Total Charge : 746.45
 Looking at view: 1 | Number of hits in view: 6
hit initial : 111.215 | hit correction : 1.08995
hit initial : 111.834 | hit correction : 1.089
hit initial : 20.8219 | hit correction : 1.08891
hit initial : 403.862 | hit correction : 1.0895
 *** Total Charge : 705.688
 Looking at view: 2 | Number of hits in view: 4
Begin processing the 10th record. run: 3799 subRun: 0 event: 9900 at 22-Nov-2025 13:47:05 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.569
True Electron Energy: 7.03872
Completed MCParticle Loop
True Neutrino Energy: 10.3224
 Size of all hits: 0
22-Nov-2025 13:47:07 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0466621     0.0559489     0.0824478     0.053498      0.0104718       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000355429   0.000440209   0.000577374   0.000435458   6.22849e-05      10     
end_path:ana:AnalyzeEvents         0.0460369     0.0553383     0.0812718     0.0529482     0.0103201       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19470.5 MB
  Peak resident set size usage (VmHWM): 1533.94 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-09 03:28:12 UTC       justIN version: 01.06.00