justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257930.0@dunegpschedd01.fnal.gov

Jobsub ID257930.0@dunegpschedd01.fnal.gov
Workflow ID10703
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-25 23:45:52
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-25 23:55:46
From worker nodeHostnamedunegli-7735290-0-fnpc19119.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-25 23:47:14
Input filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T165319Z_gen_002561_supernova_g4_halfActiveVol_detsim_reco_20250724T181118Z_reco2.root
JobscriptExit code0
Real time8m (500s)
CPU time8m (486s = 97%)
Max RSS bytes852033536 (812 MiB)
Outputting started2025-11-25 23:55:34
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/smanthey/ana/solar_ana_marley_flash_radiological_decay0_dune10kt_1x2x6_lateralAPA/fnal/10703/1/001/fdhd_ana_2025-11-25T_234722Z_hist.root
Finished2025-11-25 23:55:46
Saved logsjustin-logs:257930.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
Invalid OpWaveform handle
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050
OpFlash PE 17.37 with purity 0.11 time -18.60 plane 0
 - Vertex (-0.05, -274.87, 1212.84)
	*** 1st Sanity check: Ratio 0.41 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 1927.25 with purity 0.98 time -18.06 plane 0
 - Vertex (0.05, 404.76, 499.90)
	*** 1st Sanity check: Ratio 0.77 <= 1.00
	*** 2nd Sanity check: #OpHits 55 >= 55
OpFlash PE 49.78 with purity 1.00 time -16.11 plane 0
 - Vertex (0.05, 366.38, 513.38)
	*** 1st Sanity check: Ratio 0.22 <= 1.00
	*** 2nd Sanity check: #OpHits 17 >= 17
OpFlash PE 12.49 with purity 1.00 time -12.87 plane 0
 - Vertex (0.05, 370.25, 421.91)
	*** 1st Sanity check: Ratio 0.35 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 51.73 with purity 0.82 time -11.72 plane 0
 - Vertex (0.05, 529.30, 400.23)
	*** 1st Sanity check: Ratio 0.70 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 49.78 with purity 0.79 time -10.29 plane 0
 - Vertex (0.05, 341.04, 502.96)
	*** 1st Sanity check: Ratio 0.43 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 2.26 with purity 1.00 time -9.01 plane 0
 - Vertex (0.05, 404.76, 499.90)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 49.13 with purity 0.30 time 0.46 plane 0
 - Vertex (0.05, 529.30, 316.31)
	*** 1st Sanity check: Ratio 0.90 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3

# of OpHits (ophitspe): 40521 for times between -2260.48 us and 2226.13 us.
# of OpFlashes (solarflash): 3778 for times between -2260.48 us and 2226.13 us.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:19 UTC run: 5117 subRun: 0 event: 128050

# OpHits (ophitspe) in full geometry: 40521
# OpFlashes (solarflash) in full geometry: 3778
# Hits (hitfd) in each view: 127, 131, 260, 0
# Cluster from the hits: 78, 82, 237, 0
# Tracks (pmtracktc) in full geometry: 1
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:34 UTC run: 5117 subRun: 0 event: 128050

# ClusterReco: Looping over 125 matched clusters
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:34 UTC run: 5117 subRun: 0 event: 128050
*** Matched preselection cluster: 1 from 125
 - MainTrackID 32987
 - Gen 1 -> marley TPC 11
 - Purity 1.00 Hits 2
 - Charge 1083.83 ( MaxHit 971.94 )
 - #AdjCl 2 ( 2 signal ):
    - Cluster 8 at distance 22.70 with time 117.73 and charge 158.31 in TPC 11 and hits 1
    - Cluster 9 at distance 31.79 with time 119.48 and charge 209.18 in TPC 11 and hits 1
 - RecoCol  Time,Y,Z ( 12.86, 408.17, 486.50 )
 - RecoInd0 Time,Y,Z ( 12.85, 408.19, 544.59 )
 - RecoInd1 Time,Y,Z ( 12.89, 408.14, 580.10 )
*** Matched flash: 1904 from 3778
 - Purity 98.36 % Plane 0 #Hits 55 PE 1927.25 MainOpHitPE 1492.18
 - Time -18.06 (us) Fast 86.98 % Residual 370.36
 - Reco Time,Y,Z ( -18.06, 404.76, 499.90 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 5.59, 406.55, 486.30 )
 - Main X,Y,Z ( 5.31, 409.47, 486.71 )
 - EDep X,Y,Z ( 5.21, 408.82, 486.50 )
 - Reco X,Y,Z ( 4.96, 408.17, 486.50 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:34 UTC run: 5117 subRun: 0 event: 128050
*** Matched preselection cluster: 15 from 125
 - MainTrackID 32987
 - Gen 1 -> marley TPC 7
 - Purity 1.00 Hits 1
 - Charge 64.02 ( MaxHit 64.02 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 267.80, -53.96, 451.89 )
 - RecoInd0 Time,Y,Z ( 259.86, 11.87, 580.10 )
 - RecoInd1 Time,Y,Z ( 267.96, -119.80, 298.77 )
*** Matched flash: 1633 from 3778
 - Purity 0.00 % Plane 0 #Hits 4 PE 137.50 MainOpHitPE 78.15
 - Time -317.61 (us) Fast 57.57 % Residual 162.34
 - Reco Time,Y,Z ( -317.61, -113.86, 476.25 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 5.59, 406.55, 486.30 )
 - Main X,Y,Z ( 5.31, 409.47, 486.71 )
 - EDep X,Y,Z ( 5.21, 408.82, 486.50 )
 - Reco X,Y,Z ( 94.00, -53.96, 451.89 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:55:34 UTC run: 5117 subRun: 0 event: 128050
-----------------------------------------------------------------------------------------

%MSG
25-Nov-2025 23:55:34 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/f5/22/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T165319Z_gen_002561_supernova_g4_halfActiveVol_detsim_reco_20250724T181118Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 25-Nov-2025 23:55:34 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.213055       9.02902       25.9821      0.711779       9.21297        50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00116139    0.00306891     0.0360347    0.00189709    0.00533709       50     
end_path:solarnuana:SolarNuAna     0.211303       9.02576       25.9804      0.709351       9.21183        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 25-Nov-2025 23:55:34 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1490.58 MB
  Peak resident set size usage (VmHWM): 847.135 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 467.317144 Real = 478.163038

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1490.58 VmHWM = 847.135

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-08 12:11:48 UTC       justIN version: 01.06.00