justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 259479.77@dunegpschedd02.fnal.gov

Jobsub ID259479.77@dunegpschedd02.fnal.gov
Workflow ID11111
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 10:51:44
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-10 09:20:57
From worker nodeHostnamedunegli-48793-0-cmswn4069.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-09 16:32:45
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232730Z_gen_001708_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time16h (60085s)
CPU time16h (58742s = 97%)
Max RSS bytes2601758720 (2481 MiB)
Outputting started2025-12-10 09:14:11
Output files
Finished2025-12-10 09:20:57
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

, 0.00724192, 
Output 5: 0.443993, 0.390553, 0.132095, 0.0333593, 
Output 6: 0.0289187, 0.0609786, 0.0730869, 0.837016, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 17080 at 10-Dec-2025 08:47:42 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  10-Dec-2025 09:13:40 UTC run: 5027 subRun: 0 event: 17080
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6788, 6774, 6717
minwire 0: 32
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1270, 1769
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.124688, 
Output 1: 0.324563, 0.114488, 0.252619, 0.30833, 
Output 2: 0.035901, 0.0620796, 0.894807, 0.00721206, 
Output 3: 0.96209, 0.0353476, 0.00105891, 0.00150358, 
Output 4: 0.46016, 0.476578, 0.0515118, 0.01175, 
Output 5: 0.212386, 0.414296, 0.244425, 0.128893, 
Output 6: 0.188669, 0.314806, 0.146608, 0.349917, 

10-Dec-2025 09:13:42 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:13:42 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/e5/bb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232730Z_gen_001708_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1556.09       5955.11       8687.3        6539.79       2007.93        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0119983     0.0235787     0.0506709     0.0220384     0.0108863       10     
reco:pandora:StandardPandora                            1555.03       5888.03       8579.72       6470.76       1985.46        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00679547     0.100134      0.616887      0.0238868     0.182957        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00191859     0.0379726      0.21199     0.00927181     0.063023        10     
reco:pandoracalo:Calorimetry                          0.000391173    0.0707583     0.205818      0.0159556     0.0843046       10     
reco:pandorapid:Chi2ParticleID                        9.0211e-05    0.00152412    0.00962833    0.000489981   0.00281187       10     
reco:cvnmap:CVNMapper                                  0.0271165      0.2606        1.13326      0.182582      0.301469        10     
reco:cvneva:CVNEvaluator                               0.695416       4.43122       29.267        1.82214       8.30923        10     
reco:energyrecnumu:EnergyReco                         0.00539896      40.0532       73.4483       60.3403       32.7151        10     
reco:energyrecnue:EnergyReco                          0.000797575    0.0135857     0.0415918    0.00760451     0.0132671       10     
reco:energyrecnc:EnergyReco                           0.00719228      21.1375       73.629       0.0296176      32.3333        10     
[art]:TriggerResults:TriggerResultInserter            1.5018e-05    0.00016915    0.000572518   0.000139545   0.000153581      10     
end_path:out1:RootOutput                               6.853e-06    6.40685e-05   0.000241318   3.7927e-05    6.63051e-05      10     
end_path:out1:RootOutput(write)                        0.223712      0.940145       1.63967      0.846121      0.545853        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21678.8 MB
  Peak resident set size usage (VmHWM): 2601.76 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
True Neutron Energy: 941.53
True Neutron Energy: 940.003
True Electron Energy: 27.2955
Completed MCParticle Loop
True Neutrino Energy: 46.7744
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 17080 at 10-Dec-2025 09:14:09 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  59
Added hit list length: 59 | Max was set to: -3.40282e+38
This pfp has size:  12
True Neutron Energy: 939.572
True Electron Energy: 50.6334
Completed MCParticle Loop
True Neutrino Energy: 56.0879
 Size of all hits: 59
hit initial : 27.6428 | hit correction : 1.02871
hit initial : 32.2504 | hit correction : 1.0288
hit initial : 35.8395 | hit correction : 1.03196
hit initial : 19.4199 | hit correction : 1.03255
hit initial : 69.6129 | hit correction : 1.03145
hit initial : 53.0888 | hit correction : 1.02999
hit initial : 176.849 | hit correction : 1.03145
hit initial : 123.407 | hit correction : 1.03013
hit initial : 71.1245 | hit correction : 1.03145
hit initial : 77.6302 | hit correction : 1.03019
hit initial : 81.4683 | hit correction : 1.02659
hit initial : 89.4623 | hit correction : 1.02661
hit initial : 42.7203 | hit correction : 1.03228
hit initial : 82.6705 | hit correction : 1.03226
hit initial : 31.515 | hit correction : 1.03223
hit initial : 35.8567 | hit correction : 1.02644
hit initial : 69.0143 | hit correction : 1.02649
hit initial : 114.891 | hit correction : 1.02681
hit initial : 256.129 | hit correction : 1.027
hit initial : 328.618 | hit correction : 1.02751
hit initial : 370.748 | hit correction : 1.02776
hit initial : 310.2 | hit correction : 1.02814
hit initial : 261.142 | hit correction : 1.02848
hit initial : 277.392 | hit correction : 1.02887
hit initial : 588.034 | hit correction : 1.02916
hit initial : 988.011 | hit correction : 1.02934
hit initial : 707.789 | hit correction : 1.02954
hit initial : 204.009 | hit correction : 1.02967
 *** Total Charge : 5686.67
 Looking at view: 0 | Number of hits in view: 28
hit initial : 64.1544 | hit correction : 1.02687
hit initial : 48.8038 | hit correction : 1.02767
hit initial : 109.099 | hit correction : 1.02905
hit initial : 556.581 | hit correction : 1.02952
hit initial : 97.3752 | hit correction : 1.02963
hit initial : 105.956 | hit correction : 1.02659
hit initial : 47.0574 | hit correction : 1.02646
hit initial : 25.0613 | hit correction : 1.02651
hit initial : 83.0065 | hit correction : 1.02648
hit initial : 100.358 | hit correction : 1.02683
hit initial : 714.79 | hit correction : 1.02772
hit initial : 168.867 | hit correction : 1.02795
hit initial : 92.8364 | hit correction : 1.0285
hit initial : 570.599 | hit correction : 1.02912
hit initial : 1170.06 | hit correction : 1.02936
 *** Total Charge : 4067.94
 Looking at view: 1 | Number of hits in view: 15
hit initial : 15.5186 | hit correction : 1.03012
hit initial : 622.184 | hit correction : 1.02954
hit initial : 25.3743 | hit correction : 1.02963
hit initial : 246.837 | hit correction : 1.02652
hit initial : 95.6406 | hit correction : 1.02658
hit initial : 99.1402 | hit correction : 1.02667
hit initial : 301.583 | hit correction : 1.02697
hit initial : 412.581 | hit correction : 1.02754
hit initial : 263.382 | hit correction : 1.02777
hit initial : 231.719 | hit correction : 1.02808
hit initial : 254.034 | hit correction : 1.02838
hit initial : 292.436 | hit correction : 1.02871
hit initial : 663.948 | hit correction : 1.02911
hit initial : 806.582 | hit correction : 1.02926
hit initial : 195.94 | hit correction : 1.02946
hit initial : 195.272 | hit correction : 1.02974
 *** Total Charge : 4856.94
 Looking at view: 2 | Number of hits in view: 16
10-Dec-2025 09:14:11 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0455075     0.0551148     0.0840901     0.0521874     0.0112532       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000353121   0.000513564   0.00103912    0.000425639   0.000206485      10     
end_path:ana:AnalyzeEvents         0.044944      0.0544174     0.0824466     0.0514463     0.0109938       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19471.3 MB
  Peak resident set size usage (VmHWM): 1524.17 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 23:27:03 UTC       justIN version: 01.05.03