justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261257.1@dunegpschedd02.fnal.gov

Jobsub ID261257.1@dunegpschedd02.fnal.gov
Workflow ID11114
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 21:54:37
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-11 23:44:12
From worker nodeHostnamedunegli-7878631-0-fnpc23006.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-11 21:55:17
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T022935Z_gen_004033_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time1h (6524s)
CPU time1h (6481s = 99%)
Max RSS bytes2928730112 (2793 MiB)
Outputting started2025-12-11 23:44:01
Output files
Finished2025-12-11 23:44:12
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

1905, 
Output 1: 0.0610286, 0.0657612, 0.176238, 0.696973, 
Output 2: 0.0790813, 0.16579, 0.72526, 0.0298683, 
Output 3: 0.963639, 0.0289547, 0.00285373, 0.00455262, 
Output 4: 0.416696, 0.525389, 0.0521235, 0.00579152, 
Output 5: 0.435022, 0.432011, 0.1068, 0.0261675, 
Output 6: 0.0364982, 0.0884331, 0.0861165, 0.788952, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 40330 at 11-Dec-2025 23:32:43 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
Boundary wire vector sizes: 6800, 6797, 6688
minwire 0: 28
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0875122, 
Output 1: 0.145402, 0.0564663, 0.170034, 0.628097, 
Output 2: 0.0620149, 0.11327, 0.811302, 0.0134135, 
Output 3: 0.941935, 0.0527481, 0.0025328, 0.00278439, 
Output 4: 0.492438, 0.449208, 0.0495349, 0.00881861, 
Output 5: 0.277742, 0.542998, 0.145081, 0.0341788, 
Output 6: 0.0991032, 0.218564, 0.136508, 0.545825, 

11-Dec-2025 23:43:32 UTC  Closed output file "matt_reco2.root"
11-Dec-2025 23:43:32 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7e/4d/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T022935Z_gen_004033_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              638.192       643.993       654.11        643.362       4.52286        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00103761     0.0053615     0.0180169    0.00365288    0.00454135       10     
reco:pandora:StandardPandora                            635.828       642.847       653.084       642.352       4.70368        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00291626     0.0165347     0.0550003     0.0156298     0.0141777       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00056438    0.00286222    0.00944547    0.00190502    0.00250077       10     
reco:pandoracalo:Calorimetry                          0.000163072   0.00388278     0.0203739    0.000276274   0.00689887       10     
reco:pandorapid:Chi2ParticleID                        4.2742e-05    0.000191249   0.00075229    6.4563e-05    0.000240284      10     
reco:cvnmap:CVNMapper                                  0.0140414     0.024636      0.0318195     0.0255003    0.00495831       10     
reco:cvneva:CVNEvaluator                               0.686942      0.817703       1.94702      0.691752       0.37647        10     
reco:energyrecnumu:EnergyReco                         0.00241385     0.0292391     0.0849212    0.00532335     0.0322989       10     
reco:energyrecnue:EnergyReco                          0.00074155    0.00166597    0.00475209    0.000791214   0.00155899       10     
reco:energyrecnc:EnergyReco                           0.00462009     0.0370925     0.0644536     0.0528573     0.0267054       10     
[art]:TriggerResults:TriggerResultInserter             7.855e-06    1.60515e-05   5.5667e-05    1.16515e-05   1.34912e-05      10     
end_path:out1:RootOutput                               2.636e-06    8.2778e-06    2.3265e-05     5.28e-06     6.15908e-06      10     
end_path:out1:RootOutput(write)                        0.194182      0.205949      0.218098      0.201696     0.00823533       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21683.1 MB
  Peak resident set size usage (VmHWM): 2928.73 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 77.9408 | hit correction : 1.17879
hit initial : 405.098 | hit correction : 1.17917
hit initial : 396.148 | hit correction : 1.17961
 *** Total Charge : 1699.35
 Looking at view: 0 | Number of hits in view: 5
hit initial : 67.7177 | hit correction : 1.17869
hit initial : 174.607 | hit correction : 1.17878
hit initial : 196.519 | hit correction : 1.17895
hit initial : 64.7326 | hit correction : 1.18007
hit initial : 187.746 | hit correction : 1.17913
hit initial : 190.788 | hit correction : 1.17929
hit initial : 194.595 | hit correction : 1.17956
hit initial : 209.049 | hit correction : 1.17987
 *** Total Charge : 1516.28
 Looking at view: 1 | Number of hits in view: 8
hit initial : 36.5602 | hit correction : 1.1786
hit initial : 220.189 | hit correction : 1.17875
hit initial : 410.15 | hit correction : 1.17993
hit initial : 147.434 | hit correction : 1.18035
hit initial : 234.664 | hit correction : 1.17898
hit initial : 213.194 | hit correction : 1.17917
hit initial : 276.027 | hit correction : 1.17945
 *** Total Charge : 1814.23
 Looking at view: 2 | Number of hits in view: 7
Begin processing the 9th record. run: 5027 subRun: 0 event: 40329 at 11-Dec-2025 23:43:57 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  35
Added hit list length: 35 | Max was set to: -3.40282e+38
True Electron Energy: 27.3687
Completed MCParticle Loop
True Neutrino Energy: 31.1225
 Size of all hits: 35
hit initial : 32.9246 | hit correction : 1.25809
hit initial : 33.574 | hit correction : 1.25807
hit initial : 75.2151 | hit correction : 1.26092
hit initial : 54.0678 | hit correction : 1.26043
hit initial : 157.94 | hit correction : 1.26047
hit initial : 276.601 | hit correction : 1.26048
hit initial : 308.199 | hit correction : 1.26057
hit initial : 256.46 | hit correction : 1.26084
hit initial : 289.91 | hit correction : 1.2611
hit initial : 438.244 | hit correction : 1.26107
hit initial : 322.921 | hit correction : 1.26099
 *** Total Charge : 2831.7
 Looking at view: 0 | Number of hits in view: 11
hit initial : 127.203 | hit correction : 1.26084
hit initial : 580.375 | hit correction : 1.26086
hit initial : 95.5744 | hit correction : 1.26111
hit initial : 82.8776 | hit correction : 1.26051
hit initial : 238.083 | hit correction : 1.26109
hit initial : 75.3627 | hit correction : 1.25691
hit initial : 241.412 | hit correction : 1.25689
hit initial : 184.706 | hit correction : 1.25686
hit initial : 36.1579 | hit correction : 1.2609
hit initial : 226.526 | hit correction : 1.26053
 *** Total Charge : 2378.83
 Looking at view: 1 | Number of hits in view: 10
hit initial : 38.477 | hit correction : 1.26079
hit initial : 23.1079 | hit correction : 1.26029
hit initial : 82.7908 | hit correction : 1.26044
hit initial : 141.943 | hit correction : 1.26047
hit initial : 195.246 | hit correction : 1.26047
hit initial : 291.175 | hit correction : 1.26052
hit initial : 183.127 | hit correction : 1.2607
hit initial : 159.76 | hit correction : 1.26093
hit initial : 120.186 | hit correction : 1.26112
hit initial : 228.022 | hit correction : 1.26115
hit initial : 197.926 | hit correction : 1.26084
hit initial : 165.849 | hit correction : 1.26095
hit initial : 240.858 | hit correction : 1.26115
hit initial : 127.922 | hit correction : 1.2611
 *** Total Charge : 2769.25
 Looking at view: 2 | Number of hits in view: 14
Begin processing the 10th record. run: 5027 subRun: 0 event: 40330 at 11-Dec-2025 23:43:59 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.567
True Electron Energy: 10.9258
Completed MCParticle Loop
True Neutrino Energy: 14.6554
 Size of all hits: 0
11-Dec-2025 23:44:01 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0463186     0.0599029     0.0817969     0.0581941    0.00841233       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000294504   0.000382891   0.000562336   0.000358988   8.02764e-05      10     
end_path:ana:AnalyzeEvents         0.0457939     0.0594007     0.0808933     0.0576986    0.00830804       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19477.6 MB
  Peak resident set size usage (VmHWM): 1539.19 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 04:23:23 UTC       justIN version: 01.05.03