justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267090.0@dunegpschedd01.fnal.gov

Jobsub ID267090.0@dunegpschedd01.fnal.gov
Workflow ID11115
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 22:54:41
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-12 02:59:58
From worker nodeHostnamedunegli-8035322-0-fnpc19121.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-11 22:55:24
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205914Z_gen_003589_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time4h (14659s)
CPU time4h (14549s = 99%)
Max RSS bytes2932043776 (2796 MiB)
Outputting started2025-12-12 02:59:44
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11115/1/001/analysisLowE_20251211T225524Z.root
Finished2025-12-12 02:59:58
Saved logsjustin-logs:267090.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tput 0: 0.119315, 
Output 1: 0.0934355, 0.0959363, 0.121632, 0.688996, 
Output 2: 0.0334832, 0.202422, 0.714748, 0.0493465, 
Output 3: 0.961169, 0.0147657, 0.00651733, 0.0175478, 
Output 4: 0.846772, 0.135171, 0.0153606, 0.00269602, 
Output 5: 0.323485, 0.287354, 0.288716, 0.100446, 
Output 6: 0.0317328, 0.0665689, 0.0724083, 0.82929, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 35890 at 12-Dec-2025 02:33:34 UTC
Error: A CaloHitList is empty
Error: A CaloHitList is empty
Boundary wire vector sizes: 6539, 6582, 6605
minwire 0: 16
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.143185, 
Output 1: 0.0956404, 0.0788617, 0.094648, 0.73085, 
Output 2: 0.097748, 0.306341, 0.566714, 0.0291976, 
Output 3: 0.979165, 0.0138851, 0.00110977, 0.00584015, 
Output 4: 0.377785, 0.575806, 0.0431475, 0.00326165, 
Output 5: 0.440606, 0.497315, 0.0535354, 0.00854362, 
Output 6: 0.0193505, 0.0473496, 0.069692, 0.863608, 

12-Dec-2025 02:59:01 UTC  Closed output file "matt_reco2.root"
12-Dec-2025 02:59:01 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/66/93/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T205914Z_gen_003589_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1413.66       1451.83       1523.3        1442.8        28.3376        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00106757    0.00701183     0.0292235    0.00344011    0.00855367       10     
reco:pandora:StandardPandora                            1412.42       1450.43       1521.95       1441.59       28.2383        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000460864   0.00788347     0.0276699    0.00386474    0.00944133       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000813218   0.00358763     0.0115985    0.00296681    0.00288827       10     
reco:pandoracalo:Calorimetry                          0.000204162   0.00522347     0.0251419    0.000303054   0.00925074       10     
reco:pandorapid:Chi2ParticleID                        6.3301e-05    0.000191222   0.000860539   8.1091e-05    0.000234394      10     
reco:cvnmap:CVNMapper                                  0.0261066     0.036368      0.0645297     0.0342931    0.00992219       10     
reco:cvneva:CVNEvaluator                               0.812043      0.969012       2.1982       0.830626      0.410136        10     
reco:energyrecnumu:EnergyReco                         0.00335377     0.0357725     0.0833079     0.016201      0.0331848       10     
reco:energyrecnue:EnergyReco                          0.000862008   0.00266859     0.0068553    0.00101838    0.00262722       10     
reco:energyrecnc:EnergyReco                           0.00649715     0.0499308     0.0940554     0.0734456     0.0358393       10     
[art]:TriggerResults:TriggerResultInserter              1.2e-05     1.93711e-05   7.5121e-05    1.3475e-05    1.85941e-05      10     
end_path:out1:RootOutput                               7.25e-06     1.05792e-05   3.1751e-05     7.76e-06     7.16846e-06      10     
end_path:out1:RootOutput(write)                        0.270035      0.279337      0.297896      0.278457     0.00873758       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21694.2 MB
  Peak resident set size usage (VmHWM): 2932.04 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 217.408 | hit correction : 1.39807
hit initial : 237.734 | hit correction : 1.39748
hit initial : 210.645 | hit correction : 1.39682
hit initial : 227.58 | hit correction : 1.39624
hit initial : 193.698 | hit correction : 1.39538
 *** Total Charge : 2248.95
 Looking at view: 1 | Number of hits in view: 10
hit initial : 37.4999 | hit correction : 1.3953
hit initial : 239.337 | hit correction : 1.39879
hit initial : 49.7714 | hit correction : 1.3988
hit initial : 282.01 | hit correction : 1.39785
hit initial : 110.043 | hit correction : 1.39769
hit initial : 304.761 | hit correction : 1.3971
hit initial : 263.438 | hit correction : 1.39666
hit initial : 233.322 | hit correction : 1.39537
hit initial : 16.7919 | hit correction : 1.39516
 *** Total Charge : 2147.45
 Looking at view: 2 | Number of hits in view: 9
Begin processing the 7th record. run: 5026 subRun: 0 event: 35887 at 12-Dec-2025 02:59:32 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 2.05075
Completed MCParticle Loop
True Neutrino Energy: 5.33439
 Size of all hits: 0
Begin processing the 8th record. run: 5026 subRun: 0 event: 35888 at 12-Dec-2025 02:59:35 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  27
Added hit list length: 27 | Max was set to: -3.40282e+38
True Electron Energy: 18.048
Completed MCParticle Loop
True Neutrino Energy: 21.7855
 Size of all hits: 27
hit initial : 47.7575 | hit correction : 1.08302
hit initial : 61.7143 | hit correction : 1.083
hit initial : 20.6575 | hit correction : 1.08305
hit initial : 304.12 | hit correction : 1.08096
hit initial : 112.593 | hit correction : 1.0809
hit initial : 204.003 | hit correction : 1.08085
hit initial : 461 | hit correction : 1.08034
hit initial : 177.071 | hit correction : 1.0803
hit initial : 101.063 | hit correction : 1.08034
hit initial : 84.4329 | hit correction : 1.08034
 *** Total Charge : 1701.6
 Looking at view: 0 | Number of hits in view: 10
hit initial : 207.7 | hit correction : 1.0803
hit initial : 588.243 | hit correction : 1.08033
hit initial : 439.176 | hit correction : 1.08091
hit initial : 233.219 | hit correction : 1.08094
hit initial : 55.3494 | hit correction : 1.08091
hit initial : 601.492 | hit correction : 1.08055
 *** Total Charge : 2296.45
 Looking at view: 1 | Number of hits in view: 6
hit initial : 150.537 | hit correction : 1.08023
hit initial : 141.076 | hit correction : 1.08029
hit initial : 322.512 | hit correction : 1.08033
hit initial : 429.961 | hit correction : 1.08038
hit initial : 203.52 | hit correction : 1.08091
hit initial : 560.168 | hit correction : 1.08097
hit initial : 18.069 | hit correction : 1.08104
hit initial : 226.696 | hit correction : 1.08057
hit initial : 55.256 | hit correction : 1.08053
hit initial : 151.033 | hit correction : 1.08073
hit initial : 186.485 | hit correction : 1.08058
 *** Total Charge : 2642.4
 Looking at view: 2 | Number of hits in view: 11
Begin processing the 9th record. run: 5026 subRun: 0 event: 35889 at 12-Dec-2025 02:59:38 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 18.46
Completed MCParticle Loop
True Neutrino Energy: 21.7599
 Size of all hits: 0
Begin processing the 10th record. run: 5026 subRun: 0 event: 35890 at 12-Dec-2025 02:59:40 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Neutron Energy: 939.568
True Electron Energy: 3.97422
Completed MCParticle Loop
True Neutrino Energy: 9.35184
 Size of all hits: 0
12-Dec-2025 02:59:43 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0672129     0.0910012     0.120218      0.0931357     0.0153237       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000336703   0.00199272    0.00587223    0.000403485   0.00205957       10     
end_path:ana:AnalyzeEvents         0.0648681     0.0888475     0.113897      0.0907451     0.0143994       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19479.7 MB
  Peak resident set size usage (VmHWM): 1538.44 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 14:58:55 UTC       justIN version: 01.06.00