justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258267.0@dunegpschedd02.fnal.gov

Jobsub ID258267.0@dunegpschedd02.fnal.gov
Workflow ID11022
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 23:09:14
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:12:16
From worker nodeHostnamedunegli-7989276-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:31
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T014635Z_gen_003226_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (30382s)
CPU time8h (29087s = 95%)
Max RSS bytes3064500224 (2922 MiB)
Outputting started2025-12-06 10:04:54
Output files
Finished2025-12-06 10:12:16
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

726, 0.0277583, 
Output 3: 0.943066, 0.0444964, 0.00520892, 0.0072288, 
Output 4: 0.487266, 0.448757, 0.0577225, 0.0062542, 
Output 5: 0.549236, 0.370629, 0.0677011, 0.0124338, 
Output 6: 0.016535, 0.0452921, 0.0601565, 0.878016, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 32260 at 06-Dec-2025 09:12:55 UTC
Boundary wire vector sizes: 6786, 6701, 6709
minwire 0: 20
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2162, 2661
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.100306, 
Output 1: 0.0585971, 0.0818685, 0.177561, 0.681974, 
Output 2: 0.0643888, 0.174513, 0.739649, 0.0214493, 
Output 3: 0.965091, 0.027348, 0.00264086, 0.00492044, 
Output 4: 0.241047, 0.680758, 0.0724363, 0.00575863, 
Output 5: 0.626821, 0.31567, 0.0462138, 0.011295, 
Output 6: 0.0312002, 0.0849441, 0.0845842, 0.799272, 

06-Dec-2025 10:03:26 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:03:26 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/a7/eb/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T014635Z_gen_003226_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                               2610         3006.56       3604.22       2828.65       363.758        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00287808    0.00961887     0.0285872     0.0063191    0.00793556       10     
reco:pandora:StandardPandora                            2605.79       3001.63        3593         2824.46       362.417        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00110208     0.0454679     0.205782      0.0303402     0.0580067       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00192674     0.0107526     0.0562427    0.00567316     0.0154991       10     
reco:pandoracalo:Calorimetry                          0.000556439   0.00625463     0.0428342    0.000911955    0.0127157       10     
reco:pandorapid:Chi2ParticleID                        0.000156595   0.00079045    0.00610217    0.000194236   0.00177101       10     
reco:cvnmap:CVNMapper                                  0.0498043     0.073034      0.155383      0.0567657     0.0331597       10     
reco:cvneva:CVNEvaluator                                3.08534       3.84481       8.81263       3.23429       1.67189        10     
reco:energyrecnumu:EnergyReco                         0.00419053     0.0709835     0.185162      0.0229427     0.0688205       10     
reco:energyrecnue:EnergyReco                          0.00222793    0.00505529     0.0160358    0.00263681    0.00496123       10     
reco:energyrecnc:EnergyReco                            0.0149224     0.107307      0.200628      0.124836      0.064247        10     
[art]:TriggerResults:TriggerResultInserter            2.8814e-05    5.64575e-05   0.000231937   3.80565e-05   5.88483e-05      10     
end_path:out1:RootOutput                              1.1962e-05    1.86951e-05   6.3099e-05    1.40015e-05   1.48631e-05      10     
end_path:out1:RootOutput(write)                        0.616436      0.752945       1.56712       0.65853      0.275795        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21786.7 MB
  Peak resident set size usage (VmHWM): 3064.5 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 90.8627 | hit correction : 1.18852
hit initial : 512.071 | hit correction : 1.18821
 *** Total Charge : 1529.6
 Looking at view: 2 | Number of hits in view: 6
Begin processing the 10th record. run: 5027 subRun: 0 event: 32260 at 06-Dec-2025 10:04:49 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  62
Added hit list length: 62 | Max was set to: -3.40282e+38
True Neutron Energy: 939.569
True Electron Energy: 48.8548
Completed MCParticle Loop
True Neutrino Energy: 64.3802
 Size of all hits: 62
hit initial : 29.3886 | hit correction : 1.38501
hit initial : 104.618 | hit correction : 1.3799
hit initial : 338.377 | hit correction : 1.37931
hit initial : 474.894 | hit correction : 1.37852
hit initial : 290.914 | hit correction : 1.37811
hit initial : 134.929 | hit correction : 1.37799
hit initial : 81.3112 | hit correction : 1.37645
hit initial : 80.8781 | hit correction : 1.37628
hit initial : 89.6551 | hit correction : 1.37621
hit initial : 33.989 | hit correction : 1.37617
hit initial : 180.626 | hit correction : 1.35827
hit initial : 288.159 | hit correction : 1.35821
hit initial : 92.6872 | hit correction : 1.35816
hit initial : 173.785 | hit correction : 1.35691
hit initial : 317.791 | hit correction : 1.35652
hit initial : 312.007 | hit correction : 1.35569
hit initial : 319.737 | hit correction : 1.35486
hit initial : 380.407 | hit correction : 1.35397
hit initial : 306.68 | hit correction : 1.35372
hit initial : 103.722 | hit correction : 1.3536
 *** Total Charge : 5642.98
 Looking at view: 0 | Number of hits in view: 20
hit initial : 76.9568 | hit correction : 1.37977
hit initial : 138.445 | hit correction : 1.37987
hit initial : 48.6751 | hit correction : 1.37997
hit initial : 477.344 | hit correction : 1.37836
hit initial : 250.139 | hit correction : 1.37831
hit initial : 215.83 | hit correction : 1.37824
hit initial : 67.5811 | hit correction : 1.37742
hit initial : 62.6788 | hit correction : 1.37628
hit initial : 107.013 | hit correction : 1.37626
hit initial : 71.7058 | hit correction : 1.37623
hit initial : 28.3778 | hit correction : 1.37622
hit initial : 100.378 | hit correction : 1.35821
hit initial : 256.048 | hit correction : 1.35823
hit initial : 108.578 | hit correction : 1.3582
hit initial : 219.942 | hit correction : 1.35671
hit initial : 94.6359 | hit correction : 1.35632
hit initial : 58.1363 | hit correction : 1.35626
hit initial : 39.5116 | hit correction : 1.35514
hit initial : 247.313 | hit correction : 1.35493
hit initial : 90.3825 | hit correction : 1.35492
hit initial : 142.172 | hit correction : 1.35375
hit initial : 345.467 | hit correction : 1.35375
hit initial : 127.7 | hit correction : 1.35372
 *** Total Charge : 4610.23
 Looking at view: 1 | Number of hits in view: 23
hit initial : 141.25 | hit correction : 1.38004
hit initial : 226.713 | hit correction : 1.37946
hit initial : 354.065 | hit correction : 1.37875
hit initial : 78.6803 | hit correction : 1.37851
hit initial : 153.251 | hit correction : 1.37809
hit initial : 298.201 | hit correction : 1.37801
hit initial : 508.918 | hit correction : 1.37751
hit initial : 235.573 | hit correction : 1.37626
hit initial : 72.9356 | hit correction : 1.37621
hit initial : 30.4379 | hit correction : 1.35836
hit initial : 352.666 | hit correction : 1.35824
hit initial : 187.522 | hit correction : 1.35813
hit initial : 44.1724 | hit correction : 1.3574
hit initial : 239.74 | hit correction : 1.35677
hit initial : 462.651 | hit correction : 1.35604
hit initial : 301.443 | hit correction : 1.35498
hit initial : 278.024 | hit correction : 1.35422
hit initial : 332.863 | hit correction : 1.35374
hit initial : 235.585 | hit correction : 1.35364
 *** Total Charge : 6194.13
 Looking at view: 2 | Number of hits in view: 19
06-Dec-2025 10:04:52 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.117619      0.135768      0.170551      0.134401      0.0137602       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000627122    0.001094     0.00280609    0.000768839   0.000709938      10     
end_path:ana:AnalyzeEvents         0.116727      0.134326      0.167417      0.133434      0.0130769       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19476.2 MB
  Peak resident set size usage (VmHWM): 1539.55 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:09:16 UTC       justIN version: 01.06.00