justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268105.1@dunegpschedd01.fnal.gov

Jobsub ID268105.1@dunegpschedd01.fnal.gov
Workflow ID11174
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 17:56:53
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-14 02:02:57
From worker nodeHostnamedunegli-8047129-0-fnpc19101.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 18:08:05
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T204320Z_gen_003132_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time7h (28478s)
CPU time7h (28128s = 98%)
Max RSS bytes2793361408 (2663 MiB)
Outputting started2025-12-14 02:02:44
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11174/1/001/analysisLowE_20251213T180806Z.root
Finished2025-12-14 02:02:57
Saved logsjustin-logs:268105.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  14-Dec-2025 02:01:53 UTC run: 5026 subRun: 0 event: 31320
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6838, 6792, 6685
minwire 0: 50
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 627, 1126
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.11743, 
Output 1: 0.0637705, 0.0739109, 0.0824614, 0.779857, 
Output 2: 0.0524905, 0.245575, 0.660927, 0.041007, 
Output 3: 0.97204, 0.0175433, 0.00280847, 0.00760804, 
Output 4: 0.672444, 0.290785, 0.0327467, 0.00402444, 
Output 5: 0.415679, 0.339946, 0.187319, 0.0570562, 
Output 6: 0.0274585, 0.0698985, 0.083751, 0.818892, 

14-Dec-2025 02:01:56 UTC  Closed output file "matt_reco2.root"
14-Dec-2025 02:01:56 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/bf/96/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T204320Z_gen_003132_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2449.84       2829.92       3234.27       2879.37       237.334        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00106083    0.00559782     0.0221032    0.00179266    0.00804718       10     
reco:pandora:StandardPandora                            2448.01       2827.72       3232.3        2876.39       237.384        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00276537     0.0692108     0.0906133     0.0746385     0.0239801       10     
reco:pandoraShower:LArPandoraModularShowerCreation     0.0020822    0.00860902     0.024332      0.0075422    0.00563043       10     
reco:pandoracalo:Calorimetry                          0.000407504   0.00110562    0.00440464    0.000665232   0.00112717       10     
reco:pandorapid:Chi2ParticleID                        0.000103081   0.000339757   0.00194293    0.000138771   0.000540216      10     
reco:cvnmap:CVNMapper                                  0.0721409     0.0880777     0.116248      0.0796877     0.0157026       10     
reco:cvneva:CVNEvaluator                                1.25604       1.59116       3.29148       1.34954      0.593342        10     
reco:energyrecnumu:EnergyReco                          0.0758299     0.103203      0.150151      0.100274      0.0204675       10     
reco:energyrecnue:EnergyReco                          0.00155097    0.00229459    0.00331589    0.00206945    0.000549147      10     
reco:energyrecnc:EnergyReco                            0.0077025    0.00892692     0.0129001    0.00839714    0.00148835       10     
[art]:TriggerResults:TriggerResultInserter             1.577e-05    3.43934e-05   9.9761e-05    2.6105e-05    2.32284e-05      10     
end_path:out1:RootOutput                               1.026e-05    1.77871e-05    3.268e-05    1.71305e-05   6.33033e-06      10     
end_path:out1:RootOutput(write)                        0.304077       0.32512      0.458162      0.309125      0.0446735       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21575.5 MB
  Peak resident set size usage (VmHWM): 2793.36 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 156.088 | hit correction : 1.2335
hit initial : 67.4451 | hit correction : 1.23082
hit initial : 50.4722 | hit correction : 1.23312
hit initial : 133.256 | hit correction : 1.2318
hit initial : 316.222 | hit correction : 1.2315
hit initial : 431.454 | hit correction : 1.23126
hit initial : 405.525 | hit correction : 1.23107
hit initial : 314.231 | hit correction : 1.23093
hit initial : 300.946 | hit correction : 1.23082
hit initial : 242.421 | hit correction : 1.23078
hit initial : 75.9558 | hit correction : 1.23078
 *** Total Charge : 4766.09
 Looking at view: 1 | Number of hits in view: 18
hit initial : 177.143 | hit correction : 1.23488
hit initial : 193.831 | hit correction : 1.23445
hit initial : 196.651 | hit correction : 1.23397
hit initial : 163.609 | hit correction : 1.23373
hit initial : 170.165 | hit correction : 1.23361
hit initial : 117.155 | hit correction : 1.23361
hit initial : 136.726 | hit correction : 1.23361
hit initial : 51.5613 | hit correction : 1.23088
hit initial : 110.588 | hit correction : 1.23362
hit initial : 115.477 | hit correction : 1.23356
hit initial : 129.785 | hit correction : 1.2335
hit initial : 292.713 | hit correction : 1.23082
hit initial : 32.9133 | hit correction : 1.23075
hit initial : 131.035 | hit correction : 1.23339
hit initial : 186.593 | hit correction : 1.23312
hit initial : 204.93 | hit correction : 1.23284
hit initial : 169.006 | hit correction : 1.23245
hit initial : 207.447 | hit correction : 1.23208
hit initial : 221.202 | hit correction : 1.23181
hit initial : 297.021 | hit correction : 1.23152
hit initial : 273.611 | hit correction : 1.23138
hit initial : 282.425 | hit correction : 1.2312
hit initial : 269.613 | hit correction : 1.23108
hit initial : 235.977 | hit correction : 1.23098
hit initial : 248.487 | hit correction : 1.23089
hit initial : 320.267 | hit correction : 1.23079
hit initial : 233.046 | hit correction : 1.23077
hit initial : 33.6225 | hit correction : 1.23079
 *** Total Charge : 6410.65
 Looking at view: 2 | Number of hits in view: 28
Begin processing the 10th record. run: 5026 subRun: 0 event: 31320 at 14-Dec-2025 02:02:41 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  21
Added hit list length: 21 | Max was set to: -3.40282e+38
This pfp has size:  12
This pfp has size:  7
True Neutron Energy: 939.574
True Electron Energy: 10.7726
Completed MCParticle Loop
True Neutrino Energy: 16.2751
 Size of all hits: 21
hit initial : 156.995 | hit correction : 1.19358
hit initial : 315.693 | hit correction : 1.19357
hit initial : 166.734 | hit correction : 1.19352
hit initial : 264.536 | hit correction : 1.19269
hit initial : 155.053 | hit correction : 1.19263
 *** Total Charge : 1263.62
 Looking at view: 0 | Number of hits in view: 5
hit initial : 88.9243 | hit correction : 1.19356
hit initial : 261.964 | hit correction : 1.19357
hit initial : 212.905 | hit correction : 1.19358
hit initial : 75.6157 | hit correction : 1.19354
hit initial : 88.619 | hit correction : 1.19295
hit initial : 220.906 | hit correction : 1.19275
hit initial : 193.175 | hit correction : 1.1925
hit initial : 46.4965 | hit correction : 1.19238
 *** Total Charge : 1418.18
 Looking at view: 1 | Number of hits in view: 8
hit initial : 127.613 | hit correction : 1.19345
hit initial : 161.582 | hit correction : 1.19357
hit initial : 155.228 | hit correction : 1.19367
hit initial : 152.066 | hit correction : 1.19356
hit initial : 187.679 | hit correction : 1.19315
hit initial : 226.654 | hit correction : 1.19267
hit initial : 159.162 | hit correction : 1.19234
hit initial : 20.7907 | hit correction : 1.19218
 *** Total Charge : 1420.76
 Looking at view: 2 | Number of hits in view: 8
14-Dec-2025 02:02:44 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0844526     0.0918696     0.111995      0.0885539    0.00844993       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000431484   0.00066883    0.00204027    0.000505815   0.000463893      10     
end_path:ana:AnalyzeEvents         0.0836092     0.0909533     0.109278      0.0878194    0.00797735       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19478 MB
  Peak resident set size usage (VmHWM): 1539.87 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:04:40 UTC       justIN version: 01.06.00