justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258263.0@dunegpschedd02.fnal.gov

Jobsub ID258263.0@dunegpschedd02.fnal.gov
Workflow ID11026
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 22:55:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:34:40
From worker nodeHostnamedunegli-7989276-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:37:55
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T191539Z_gen_002044_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (31769s)
CPU time8h (30648s = 96%)
Max RSS bytes3033198592 (2892 MiB)
Outputting started2025-12-06 10:27:25
Output files
Finished2025-12-06 10:34:40
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r summary: 
Output 0: 0.14657, 
Output 1: 0.104208, 0.0893285, 0.088386, 0.718078, 
Output 2: 0.077326, 0.295926, 0.591236, 0.0355128, 
Output 3: 0.978682, 0.0114998, 0.00167166, 0.00814711, 
Output 4: 0.590517, 0.360827, 0.0430963, 0.00556023, 
Output 5: 0.427232, 0.467238, 0.0887034, 0.0168264, 
Output 6: 0.0368596, 0.0714973, 0.0860457, 0.805598, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 20440 at 06-Dec-2025 09:40:58 UTC
Boundary wire vector sizes: 6491, 6456, 6505
minwire 0: 45
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0840507, 
Output 1: 0.0578104, 0.0862788, 0.0854018, 0.770509, 
Output 2: 0.0441098, 0.180257, 0.741888, 0.0337449, 
Output 3: 0.970411, 0.01368, 0.0033526, 0.0125567, 
Output 4: 0.846853, 0.132909, 0.016364, 0.00387405, 
Output 5: 0.433771, 0.32483, 0.179505, 0.0618942, 
Output 6: 0.0327729, 0.0653153, 0.069217, 0.832695, 

06-Dec-2025 10:25:15 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:25:15 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/85/94/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T191539Z_gen_002044_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2648.46       3143.59       3575.93       3059.86       273.477        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00303585     0.012445      0.0484079    0.00872658     0.0123111       10     
reco:pandora:StandardPandora                            2644.18       3138.09       3570.74       3055.31       273.915        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.0010146     0.0432069     0.197318      0.0188056     0.0549155       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00237093     0.0112259     0.0510924    0.00408855     0.0147247       10     
reco:pandoracalo:Calorimetry                          0.000640888   0.00218128     0.0126216    0.00103679    0.00348845       10     
reco:pandorapid:Chi2ParticleID                        0.000178416   0.000755291   0.00523251    0.000258663   0.00149336       10     
reco:cvnmap:CVNMapper                                  0.0572513     0.0741991     0.154295      0.0628425     0.0276368       10     
reco:cvneva:CVNEvaluator                                3.19155       4.44644       8.64816       3.85906       1.56157        10     
reco:energyrecnumu:EnergyReco                         0.00303798     0.0536816     0.282649      0.0149345     0.0847348       10     
reco:energyrecnue:EnergyReco                          0.00246556    0.00348974     0.0056962    0.00305429    0.00102722       10     
reco:energyrecnc:EnergyReco                            0.0146925     0.118266      0.168573      0.135179      0.0523974       10     
[art]:TriggerResults:TriggerResultInserter            4.0647e-05    6.24467e-05   0.000196911   4.6242e-05    4.55672e-05      10     
end_path:out1:RootOutput                              1.3486e-05    3.0823e-05    6.2007e-05    1.6736e-05    1.96313e-05      10     
end_path:out1:RootOutput(write)                        0.633582      0.731469       0.92888      0.682979      0.108641        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21780.7 MB
  Peak resident set size usage (VmHWM): 3033.2 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 315.085 | hit correction : 1.01481
hit initial : 518.676 | hit correction : 1.01498
hit initial : 381.361 | hit correction : 1.01505
hit initial : 239.048 | hit correction : 1.01584
hit initial : 448.422 | hit correction : 1.01605
hit initial : 343.067 | hit correction : 1.01619
hit initial : 28.1656 | hit correction : 1.01599
hit initial : 88.9947 | hit correction : 1.01624
 *** Total Charge : 3257.6
 Looking at view: 1 | Number of hits in view: 13
hit initial : 21.3032 | hit correction : 1.01778
hit initial : 16.3835 | hit correction : 1.01787
hit initial : 195.671 | hit correction : 1.01401
hit initial : 253.146 | hit correction : 1.01427
hit initial : 38.4488 | hit correction : 1.01601
hit initial : 318.015 | hit correction : 1.01453
hit initial : 418.563 | hit correction : 1.01483
hit initial : 74.4916 | hit correction : 1.01499
hit initial : 616.089 | hit correction : 1.015
hit initial : 319.351 | hit correction : 1.01541
hit initial : 169.448 | hit correction : 1.01577
hit initial : 214.821 | hit correction : 1.01598
hit initial : 152.666 | hit correction : 1.01608
hit initial : 309.718 | hit correction : 1.01621
hit initial : 185.136 | hit correction : 1.01628
 *** Total Charge : 3353.6
 Looking at view: 2 | Number of hits in view: 15
Begin processing the 10th record. run: 5026 subRun: 0 event: 20440 at 06-Dec-2025 10:27:13 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  39
Added hit list length: 39 | Max was set to: -3.40282e+38
True Neutron Energy: 939.566
True Electron Energy: 24.3314
Completed MCParticle Loop
True Neutrino Energy: 29.7364
 Size of all hits: 39
hit initial : 111.08 | hit correction : 1.05643
hit initial : 29.3848 | hit correction : 1.0566
hit initial : 31.7117 | hit correction : 1.05663
hit initial : 134.86 | hit correction : 1.06085
hit initial : 128.731 | hit correction : 1.06012
hit initial : 497.98 | hit correction : 1.06017
hit initial : 248.888 | hit correction : 1.06105
hit initial : 181.209 | hit correction : 1.06106
hit initial : 221.754 | hit correction : 1.05981
hit initial : 39.67 | hit correction : 1.06103
hit initial : 279.686 | hit correction : 1.05949
hit initial : 299.444 | hit correction : 1.05938
hit initial : 236.155 | hit correction : 1.05933
hit initial : 218.114 | hit correction : 1.05929
hit initial : 185.242 | hit correction : 1.05928
hit initial : 99.9513 | hit correction : 1.05926
hit initial : 33.1297 | hit correction : 1.05927
 *** Total Charge : 3154.8
 Looking at view: 0 | Number of hits in view: 17
hit initial : 42.0153 | hit correction : 1.05927
hit initial : 69.494 | hit correction : 1.06082
hit initial : 395.538 | hit correction : 1.06083
hit initial : 83.4028 | hit correction : 1.06005
hit initial : 180.675 | hit correction : 1.06104
hit initial : 60.8922 | hit correction : 1.06107
hit initial : 345.049 | hit correction : 1.05989
hit initial : 338.17 | hit correction : 1.05957
hit initial : 310.556 | hit correction : 1.05938
hit initial : 275.703 | hit correction : 1.05933
hit initial : 249.846 | hit correction : 1.0593
hit initial : 175.691 | hit correction : 1.05928
 *** Total Charge : 2678.4
 Looking at view: 1 | Number of hits in view: 12
hit initial : 199.851 | hit correction : 1.06082
hit initial : 224.155 | hit correction : 1.06057
hit initial : 32.0005 | hit correction : 1.06004
hit initial : 203.621 | hit correction : 1.06024
hit initial : 157.285 | hit correction : 1.06099
hit initial : 343.831 | hit correction : 1.06111
hit initial : 589.204 | hit correction : 1.05988
hit initial : 647.828 | hit correction : 1.05939
hit initial : 649.205 | hit correction : 1.0593
hit initial : 60.9878 | hit correction : 1.05922
 *** Total Charge : 3294.36
 Looking at view: 2 | Number of hits in view: 10
06-Dec-2025 10:27:23 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.273472      0.507112       1.42821      0.398528      0.322359        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000629237    0.0220616     0.213577     0.000799357    0.0638387       10     
end_path:ana:AnalyzeEvents         0.272218      0.484697       1.21358       0.39754       0.26209        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19473.9 MB
  Peak resident set size usage (VmHWM): 1534.85 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 13:09:28 UTC       justIN version: 01.05.03