justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252260.16@dunegpschedd02.fnal.gov

Jobsub ID252260.16@dunegpschedd02.fnal.gov
Workflow ID10704
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-25 23:32:03
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-25 23:47:07
From worker nodeHostnamedunegli-7738788-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-25 23:33:46
Input filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T155041Z_gen_000783_supernova_g4_halfActiveVol_detsim_reco_20250724T172910Z_reco2.root
JobscriptExit code0
Real time12m (773s)
CPU time12m (756s = 97%)
Max RSS bytes857821184 (818 MiB)
Outputting started2025-11-25 23:46:39
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/smanthey/ana/solar_ana_marley_flash_radiological_decay0_dune10kt_1x2x6_lateralAPA/fnal/10704/1/001/fdhd_ana_2025-11-25T_233401Z_hist.root
Finished2025-11-25 23:47:07
Saved logsjustin-logs:252260.16-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

d OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
Invalid OpWaveform handle
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150
OpFlash PE 748.91 with purity 0.97 time -17.98 plane 0
 - Vertex (-0.05, -367.35, 290.17)
	*** 1st Sanity check: Ratio 0.12 <= 1.00
	*** 2nd Sanity check: #OpHits 82 >= 82
OpFlash PE 97.57 with purity 0.97 time -16.03 plane 0
 - Vertex (-0.05, -398.16, 177.94)
	*** 1st Sanity check: Ratio 0.17 <= 1.00
	*** 2nd Sanity check: #OpHits 29 >= 29
OpFlash PE 49.14 with purity 0.84 time -14.46 plane 0
 - Vertex (-0.04, -322.65, 198.06)
	*** 1st Sanity check: Ratio 0.14 <= 1.00
	*** 2nd Sanity check: #OpHits 16 >= 16
OpFlash PE 26.71 with purity 0.83 time -13.26 plane 0
 - Vertex (-0.03, -261.00, 216.82)
	*** 1st Sanity check: Ratio 0.26 <= 1.00
	*** 2nd Sanity check: #OpHits 11 >= 11
OpFlash PE 84.54 with purity 0.95 time -12.20 plane 0
 - Vertex (-0.05, -404.76, 195.52)
	*** 1st Sanity check: Ratio 0.75 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 3.70 with purity 0.57 time -8.28 plane 0
 - Vertex (-0.05, -395.78, 82.88)
	*** 1st Sanity check: Ratio 0.57 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2

# of OpHits (ophitspe): 41673 for times between -2261.79 us and 2227.78 us.
# of OpFlashes (solarflash): 3656 for times between -2261.64 us and 2227.78 us.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:07 UTC run: 5117 subRun: 0 event: 39150

# OpHits (ophitspe) in full geometry: 41673
# OpFlashes (solarflash) in full geometry: 3656
# Hits (hitfd) in each view: 151, 160, 293, 0
# Cluster from the hits: 95, 96, 251, 0
# Tracks (pmtracktc) in full geometry: 3
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:38 UTC run: 5117 subRun: 0 event: 39150

# ClusterReco: Looping over 162 matched clusters
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:38 UTC run: 5117 subRun: 0 event: 39150
*** Matched preselection cluster: 38 from 162
 - MainTrackID 33109
 - Gen 1 -> marley TPC 5
 - Purity 1.00 Hits 12
 - Charge 2689.24 ( MaxHit 509.43 )
 - #AdjCl 2 ( 2 signal ):
    - Cluster 36 at distance 5.86 with time 557.24 and charge 72.67 in TPC 5 and hits 1
    - Cluster 41 at distance 14.06 with time 602.22 and charge 87.87 in TPC 5 and hits 1
 - RecoCol  Time,Y,Z ( 566.01, -377.80, 252.94 )
 - RecoInd0 Time,Y,Z ( 566.17, -377.73, 347.71 )
 - RecoInd1 Time,Y,Z ( 565.95, -377.87, 322.67 )
*** Matched flash: 1814 from 3656
 - Purity 97.48 % Plane 0 #Hits 82 PE 748.91 MainOpHitPE 89.12
 - Time -17.98 (us) Fast 69.69 % Residual 394.45
 - Reco Time,Y,Z ( -17.98, -367.35, 290.17 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 92.97, -376.35, 249.90 )
 - Main X,Y,Z ( 92.97, -377.38, 255.09 )
 - EDep X,Y,Z ( 94.09, -377.52, 251.00 )
 - Reco X,Y,Z ( 93.77, -377.80, 252.94 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:46:38 UTC run: 5117 subRun: 0 event: 39150
-----------------------------------------------------------------------------------------

%MSG
25-Nov-2025 23:46:39 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/3a/fb/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T155041Z_gen_000783_supernova_g4_halfActiveVol_detsim_reco_20250724T172910Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 25-Nov-2025 23:46:39 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          0.2926        13.7892       34.6217      0.610953       14.0849        50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00157199    0.00681319     0.0204668    0.00283139    0.00580506       50     
end_path:solarnuana:SolarNuAna      0.28651       13.7822       34.6198      0.608452       14.0853        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 25-Nov-2025 23:46:39 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1491.37 MB
  Peak resident set size usage (VmHWM): 844.853 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 710.384297 Real = 723.880384

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1491.37 VmHWM = 844.853

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-08 12:13:39 UTC       justIN version: 01.06.00