justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261982.1@dunegpschedd02.fnal.gov

Jobsub ID261982.1@dunegpschedd02.fnal.gov
Workflow ID11171
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 06:18:18
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 16:48:55
From worker nodeHostnamedunegli-7890501-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 06:20:50
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T030917Z_gen_004683_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time10h (37646s)
CPU time9h (35445s = 94%)
Max RSS bytes2796134400 (2666 MiB)
Outputting started2025-12-13 16:48:16
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11171/1/001/analysisLowE_20251213T062054Z.root
Finished2025-12-13 16:48:55
Saved logsjustin-logs:261982.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 16:47:11 UTC run: 5027 subRun: 0 event: 46830
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6818, 6723, 6626
minwire 0: 20
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1066, 1565
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0859215, 
Output 1: 0.0250824, 0.131682, 0.128386, 0.71485, 
Output 2: 0.0543977, 0.274826, 0.644943, 0.0258334, 
Output 3: 0.956021, 0.0322399, 0.00395786, 0.0077816, 
Output 4: 0.167083, 0.755081, 0.0746847, 0.00315183, 
Output 5: 0.791512, 0.182778, 0.0209814, 0.00472861, 
Output 6: 0.00965396, 0.0375716, 0.0566258, 0.896149, 

13-Dec-2025 16:47:15 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 16:47:15 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ee/04/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T030917Z_gen_004683_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2015.89       3704.45       7764.46       2651.9        2093.8         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00110322     0.0128253     0.0843195    0.00187102     0.0251688       10     
reco:pandora:StandardPandora                            2013.41       3679.12       7690.18       2649.2        2063.01        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000468282    0.0872032     0.344577      0.0656874     0.0938709       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000465406    0.0151221     0.100702     0.00569335     0.0287497       10     
reco:pandoracalo:Calorimetry                          0.000238839   0.00335926     0.025811     0.000717992   0.00750817       10     
reco:pandorapid:Chi2ParticleID                         9.566e-05    0.000917229   0.00707115    0.000188064   0.00205811       10     
reco:cvnmap:CVNMapper                                  0.0690463     0.160414      0.346684      0.112122      0.0952434       10     
reco:cvneva:CVNEvaluator                                1.82823       3.64164       13.528        2.05277       3.48275        10     
reco:energyrecnumu:EnergyReco                          0.0702915      20.8475       72.2891       0.18146       31.602         10     
reco:energyrecnue:EnergyReco                          0.00164272    0.00645293     0.0210448    0.00267002    0.00604508       10     
reco:energyrecnc:EnergyReco                           0.00939633     0.0142536     0.0281256     0.0109604    0.00650541       10     
[art]:TriggerResults:TriggerResultInserter            2.4366e-05    6.08475e-05   0.00029214    2.76475e-05   7.95824e-05      10     
end_path:out1:RootOutput                               1.058e-05    2.69819e-05   0.000112081    1.353e-05    3.07682e-05      10     
end_path:out1:RootOutput(write)                        0.362709      0.540381       1.43365      0.397724      0.324225        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21579.6 MB
  Peak resident set size usage (VmHWM): 2796.13 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 265.825 | hit correction : 1.15052
hit initial : 197.688 | hit correction : 1.15011
 *** Total Charge : 2544.4
 Looking at view: 1 | Number of hits in view: 9
hit initial : 260.985 | hit correction : 1.15257
hit initial : 244.241 | hit correction : 1.15253
hit initial : 193.347 | hit correction : 1.15237
hit initial : 222.604 | hit correction : 1.15219
hit initial : 394.786 | hit correction : 1.15178
hit initial : 274.465 | hit correction : 1.15133
hit initial : 294.855 | hit correction : 1.15079
hit initial : 281.887 | hit correction : 1.15035
hit initial : 191.014 | hit correction : 1.14993
 *** Total Charge : 2715.53
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 10th record. run: 5027 subRun: 0 event: 46830 at 13-Dec-2025 16:48:12 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  49
Added hit list length: 49 | Max was set to: -3.40282e+38
This pfp has size:  10
This pfp has size:  6
True Neutron Energy: 939.568
True Electron Energy: 54.0709
Completed MCParticle Loop
True Neutrino Energy: 59.574
 Size of all hits: 49
hit initial : 70.7209 | hit correction : 1.03153
hit initial : 296.151 | hit correction : 1.03197
hit initial : 95.2559 | hit correction : 1.03205
hit initial : 110.385 | hit correction : 1.03317
hit initial : 187.677 | hit correction : 1.0331
hit initial : 141.899 | hit correction : 1.03308
hit initial : 59.2418 | hit correction : 1.03371
hit initial : 138.265 | hit correction : 1.04964
hit initial : 314.901 | hit correction : 1.04964
hit initial : 95.087 | hit correction : 1.0498
hit initial : 128.464 | hit correction : 1.06012
hit initial : 735.689 | hit correction : 1.06145
hit initial : 67.4946 | hit correction : 1.06136
hit initial : 441.624 | hit correction : 1.06179
hit initial : 56.0507 | hit correction : 1.06225
hit initial : 26.2682 | hit correction : 1.06283
hit initial : 97.3085 | hit correction : 1.06341
hit initial : 69.9883 | hit correction : 1.06397
 *** Total Charge : 3291.17
 Looking at view: 0 | Number of hits in view: 18
hit initial : 44.4979 | hit correction : 1.05903
hit initial : 197.146 | hit correction : 1.03176
hit initial : 308.099 | hit correction : 1.03194
hit initial : 78.7606 | hit correction : 1.03215
hit initial : 80.9527 | hit correction : 1.03315
hit initial : 212.305 | hit correction : 1.03313
hit initial : 153.221 | hit correction : 1.0331
hit initial : 77.9785 | hit correction : 1.0331
hit initial : 41.559 | hit correction : 1.03371
hit initial : 30.6957 | hit correction : 1.03371
hit initial : 198.657 | hit correction : 1.04957
hit initial : 287.511 | hit correction : 1.04968
hit initial : 93.3739 | hit correction : 1.04976
hit initial : 198.343 | hit correction : 1.06005
hit initial : 16.5615 | hit correction : 1.06137
hit initial : 1062.18 | hit correction : 1.06158
hit initial : 20.4679 | hit correction : 1.06227
hit initial : 38.6086 | hit correction : 1.06294
hit initial : 81.9552 | hit correction : 1.06379
 *** Total Charge : 3379.99
 Looking at view: 1 | Number of hits in view: 19
hit initial : 316.606 | hit correction : 1.03256
hit initial : 167.697 | hit correction : 1.03152
hit initial : 189.545 | hit correction : 1.03177
hit initial : 395.881 | hit correction : 1.03207
hit initial : 299.653 | hit correction : 1.03315
hit initial : 313.88 | hit correction : 1.03309
hit initial : 32.0948 | hit correction : 1.03311
hit initial : 96.8824 | hit correction : 1.03371
hit initial : 158.126 | hit correction : 1.04943
hit initial : 416.089 | hit correction : 1.04973
hit initial : 2297.25 | hit correction : 1.06143
hit initial : 71.4933 | hit correction : 1.0636
 *** Total Charge : 4988.33
 Looking at view: 2 | Number of hits in view: 12
13-Dec-2025 16:48:16 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0898414     0.133864      0.269991      0.120077      0.0479113       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000540318   0.000986022   0.00268414    0.000721856   0.000614142      10     
end_path:ana:AnalyzeEvents         0.0889972     0.132613      0.269089      0.119124      0.0479464       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19480.1 MB
  Peak resident set size usage (VmHWM): 1538.71 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:03:55 UTC       justIN version: 01.06.00