justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252288.1@dunegpschedd02.fnal.gov

Jobsub ID252288.1@dunegpschedd02.fnal.gov
Workflow ID10704
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-25 23:45:52
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-26 00:00:12
From worker nodeHostnamedunegli-7736252-0-fnpc9072.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-25 23:47:16
Input filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T165420Z_gen_000340_supernova_g4_halfActiveVol_detsim_reco_20250724T193051Z_reco2.root
JobscriptExit code0
Real time12m (756s)
CPU time12m (720s = 95%)
Max RSS bytes854130688 (814 MiB)
Outputting started2025-11-25 23:59:52
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/smanthey/ana/solar_ana_marley_flash_radiological_decay0_dune10kt_1x2x6_lateralAPA/fnal/10704/1/001/fdhd_ana_2025-11-25T_234741Z_hist.root
Finished2025-11-26 00:00:12
Saved logsjustin-logs:252288.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
Invalid OpWaveform handle
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000
OpFlash PE 149.62 with purity 1.00 time -17.89 plane 0
 - Vertex (-0.03, -0.49, 865.23)
	*** 1st Sanity check: Ratio 0.18 <= 1.00
	*** 2nd Sanity check: #OpHits 44 >= 44
OpFlash PE 3.65 with purity 1.00 time -17.58 plane 0
 - Vertex (0.05, 377.80, 1219.57)
	*** 1st Sanity check: Ratio 0.51 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 63.27 with purity 0.88 time -16.95 plane 0
 - Vertex (-0.05, -179.41, 899.67)
	*** 1st Sanity check: Ratio 0.24 <= 1.00
	*** 2nd Sanity check: #OpHits 20 >= 20
OpFlash PE 24.83 with purity 0.79 time -16.36 plane 0
 - Vertex (-0.00, 27.14, 846.05)
	*** 1st Sanity check: Ratio 0.13 <= 1.00
	*** 2nd Sanity check: #OpHits 12 >= 12
OpFlash PE 12.36 with purity 1.00 time -15.03 plane 0
 - Vertex (0.00, -10.38, 779.24)
	*** 1st Sanity check: Ratio 0.24 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 25.51 with purity 0.27 time -14.12 plane 0
 - Vertex (0.05, 155.68, 427.91)
	*** 1st Sanity check: Ratio 0.82 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 20.30 with purity 0.69 time -12.92 plane 0
 - Vertex (0.05, 155.68, 892.69)
	*** 1st Sanity check: Ratio 0.57 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 9.24 with purity 0.58 time -9.78 plane 0
 - Vertex (-0.01, 149.54, 922.62)
	*** 1st Sanity check: Ratio 0.58 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3

# of OpHits (ophitspe): 41037 for times between -2261.70 us and 2227.44 us.
# of OpFlashes (solarflash): 3756 for times between -2261.67 us and 2227.44 us.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:30 UTC run: 5100 subRun: 0 event: 17000

# OpHits (ophitspe) in full geometry: 41037
# OpFlashes (solarflash) in full geometry: 3756
# Hits (hitfd) in each view: 143, 135, 287, 0
# Cluster from the hits: 88, 82, 264, 0
# Tracks (pmtracktc) in full geometry: 1
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:51 UTC run: 5100 subRun: 0 event: 17000

# ClusterReco: Looping over 153 matched clusters
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:51 UTC run: 5100 subRun: 0 event: 17000
*** Matched preselection cluster: 81 from 153
 - MainTrackID 33299
 - Gen 1 -> marley TPC 15
 - Purity 1.00 Hits 7
 - Charge 1032.25 ( MaxHit 203.63 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 1368.27, 7.11, 880.94 )
 - RecoInd0 Time,Y,Z ( 1368.80, 7.19, 902.15 )
 - RecoInd1 Time,Y,Z ( 1368.56, 7.04, 791.27 )
*** Matched flash: 1868 from 3756
 - Purity 99.62 % Plane 0 #Hits 44 PE 149.62 MainOpHitPE 27.67
 - Time -17.89 (us) Fast 31.66 % Residual 204.72
 - Reco Time,Y,Z ( -17.89, -0.49, 865.23 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 222.67, 8.09, 879.93 )
 - Main X,Y,Z ( 221.88, 6.64, 881.53 )
 - EDep X,Y,Z ( 222.78, 7.72, 880.20 )
 - Reco X,Y,Z ( 222.56, 7.11, 880.94 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  25-Nov-2025 23:59:51 UTC run: 5100 subRun: 0 event: 17000
-----------------------------------------------------------------------------------------

%MSG
25-Nov-2025 23:59:52 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/c1/5c/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T165420Z_gen_000340_supernova_g4_halfActiveVol_detsim_reco_20250724T193051Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 25-Nov-2025 23:59:52 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.435669       13.0354       28.6557       10.6933       12.6205        50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)             0.0014146     0.0134336     0.0414006     0.0140422    0.00901947       50     
end_path:solarnuana:SolarNuAna     0.414291       13.0217       28.6372       10.666        12.6212        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 25-Nov-2025 23:59:52 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1493.35 MB
  Peak resident set size usage (VmHWM): 849.297 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 680.613791 Real = 695.002442

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1493.35 VmHWM = 849.297

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-08 12:13:33 UTC       justIN version: 01.06.00