justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 266830.1@dunegpschedd01.fnal.gov

Jobsub ID266830.1@dunegpschedd01.fnal.gov
Workflow ID11093
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 14:54:14
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-11 22:08:05
From worker nodeHostnamedunegli-7872663-0-fnpc9034.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-11 14:54:47
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T004118Z_gen_002683_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time7h (25981s)
CPU time7h (25613s = 98%)
Max RSS bytes2975309824 (2837 MiB)
Outputting started2025-12-11 22:07:48
Output files
Finished2025-12-11 22:08:05
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.889345, 0.0102642, 
Output 3: 0.961675, 0.0327387, 0.00247756, 0.00310905, 
Output 4: 0.535118, 0.403531, 0.0508875, 0.010464, 
Output 5: 0.166041, 0.410872, 0.311432, 0.111655, 
Output 6: 0.106536, 0.206974, 0.127225, 0.559264, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 26830 at 11-Dec-2025 21:34:10 UTC
Boundary wire vector sizes: 6770, 6720, 6712
minwire 0: 49
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 792, 1291
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.126804, 
Output 1: 0.109474, 0.0693665, 0.127063, 0.694096, 
Output 2: 0.0531126, 0.252974, 0.660676, 0.0332372, 
Output 3: 0.966766, 0.0185065, 0.00333796, 0.0113899, 
Output 4: 0.432572, 0.516297, 0.0473293, 0.00380124, 
Output 5: 0.558245, 0.339541, 0.0824716, 0.0197426, 
Output 6: 0.0157734, 0.0426255, 0.0615324, 0.880069, 

11-Dec-2025 22:06:36 UTC  Closed output file "matt_reco2.root"
11-Dec-2025 22:06:36 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/dd/96/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T004118Z_gen_002683_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1711.13       2565.28       3353.05       2384.63       558.69         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00304931     0.0126509     0.0227951     0.0129419    0.00476669       10     
reco:pandora:StandardPandora                            1708.41       2562.1        3350.23       2382.22       557.898        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00155855     0.0516202     0.244614      0.0297195     0.0682006       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00141708    0.00608016     0.0216724    0.00278939    0.00647983       10     
reco:pandoracalo:Calorimetry                          0.000397084   0.00834539     0.0539553    0.000950861    0.0158771       10     
reco:pandorapid:Chi2ParticleID                        0.000129219   0.000520949   0.00222149    0.00014744    0.000728171      10     
reco:cvnmap:CVNMapper                                  0.0580651      0.08616      0.142807      0.0837189     0.0246803       10     
reco:cvneva:CVNEvaluator                                1.09803       2.1995        7.18941       1.84697       1.69955        10     
reco:energyrecnumu:EnergyReco                          0.0101378     0.108837       0.16932      0.144426      0.062451        10     
reco:energyrecnue:EnergyReco                          0.00145169    0.00563013     0.0143154    0.00263936    0.00519589       10     
reco:energyrecnc:EnergyReco                            0.0084061     0.0493247     0.159799      0.0142285     0.0577285       10     
[art]:TriggerResults:TriggerResultInserter            1.8254e-05    4.57917e-05   0.000160923   3.4724e-05    3.90075e-05      10     
end_path:out1:RootOutput                               7.154e-06    1.62606e-05   4.8855e-05    1.0178e-05    1.2266e-05       10     
end_path:out1:RootOutput(write)                        0.376181       0.64893      0.947909      0.667003      0.164663        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12077.4 MB
  Peak resident set size usage (VmHWM): 2975.31 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 526.048 | hit correction : 1.16886
hit initial : 192.412 | hit correction : 1.16938
hit initial : 241.829 | hit correction : 1.17009
hit initial : 233.314 | hit correction : 1.17065
hit initial : 376.797 | hit correction : 1.17146
hit initial : 323.838 | hit correction : 1.17254
hit initial : 222.9 | hit correction : 1.17326
hit initial : 219.346 | hit correction : 1.1738
hit initial : 215.198 | hit correction : 1.1743
hit initial : 234.615 | hit correction : 1.17478
hit initial : 233.313 | hit correction : 1.17529
hit initial : 98.5457 | hit correction : 1.17559
 *** Total Charge : 3902.29
 Looking at view: 2 | Number of hits in view: 14
Begin processing the 10th record. run: 5027 subRun: 0 event: 26830 at 11-Dec-2025 22:07:42 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  52
Added hit list length: 52 | Max was set to: -3.40282e+38
True Neutron Energy: 944.738
True Electron Energy: 33.0556
Completed MCParticle Loop
True Neutrino Energy: 38.4861
 Size of all hits: 52
hit initial : 34.5728 | hit correction : 1.46365
hit initial : 19.8107 | hit correction : 1.46364
hit initial : 26.7082 | hit correction : 1.47064
hit initial : 28.8298 | hit correction : 1.47067
hit initial : 23.4415 | hit correction : 1.47271
hit initial : 73.2722 | hit correction : 1.47192
hit initial : 22.2634 | hit correction : 1.47277
hit initial : 61.5411 | hit correction : 1.46905
hit initial : 71.0882 | hit correction : 1.46823
hit initial : 74.0553 | hit correction : 1.46725
hit initial : 199.178 | hit correction : 1.47186
hit initial : 271.654 | hit correction : 1.4716
hit initial : 263.967 | hit correction : 1.4712
hit initial : 83.0316 | hit correction : 1.47104
hit initial : 159.426 | hit correction : 1.47026
hit initial : 296.613 | hit correction : 1.46989
hit initial : 238.967 | hit correction : 1.46952
hit initial : 376.436 | hit correction : 1.46837
hit initial : 184.488 | hit correction : 1.46808
hit initial : 106.485 | hit correction : 1.46722
 *** Total Charge : 3844.67
 Looking at view: 0 | Number of hits in view: 20
hit initial : 40.0966 | hit correction : 1.46219
hit initial : 64.5853 | hit correction : 1.46218
hit initial : 66.8269 | hit correction : 1.46714
hit initial : 52.9366 | hit correction : 1.46902
hit initial : 20.7198 | hit correction : 1.46901
hit initial : 29.567 | hit correction : 1.47282
hit initial : 35.076 | hit correction : 1.47275
hit initial : 94.9551 | hit correction : 1.47192
hit initial : 160.701 | hit correction : 1.47189
hit initial : 164.499 | hit correction : 1.47175
hit initial : 172.612 | hit correction : 1.4714
hit initial : 188.992 | hit correction : 1.47111
hit initial : 170.32 | hit correction : 1.47072
hit initial : 189.732 | hit correction : 1.47016
hit initial : 222.386 | hit correction : 1.4698
hit initial : 195.491 | hit correction : 1.4695
hit initial : 177.236 | hit correction : 1.46902
hit initial : 247.844 | hit correction : 1.46838
hit initial : 273.933 | hit correction : 1.4679
hit initial : 187.054 | hit correction : 1.46734
 *** Total Charge : 4049.36
 Looking at view: 1 | Number of hits in view: 20
hit initial : 164.335 | hit correction : 1.46724
hit initial : 134.653 | hit correction : 1.47111
hit initial : 39.8126 | hit correction : 1.47301
hit initial : 57.0722 | hit correction : 1.47276
hit initial : 130.912 | hit correction : 1.4719
hit initial : 305.16 | hit correction : 1.47183
hit initial : 327.032 | hit correction : 1.47122
hit initial : 301.723 | hit correction : 1.47015
hit initial : 409.752 | hit correction : 1.46953
hit initial : 332.743 | hit correction : 1.46885
hit initial : 31.7788 | hit correction : 1.46819
hit initial : 402.598 | hit correction : 1.46793
 *** Total Charge : 3876.99
 Looking at view: 2 | Number of hits in view: 12
11-Dec-2025 22:07:47 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0879018     0.145998      0.234946      0.129834      0.0462569       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000447553   0.00301328     0.0247066    0.000559891   0.00723269       10     
end_path:ana:AnalyzeEvents         0.0872646     0.142507      0.233775      0.129228      0.0423531       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9811.32 MB
  Peak resident set size usage (VmHWM): 1534.86 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 16:19:22 UTC       justIN version: 01.06.00