justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267134.9@dunegpschedd02.fnal.gov

Jobsub ID267134.9@dunegpschedd02.fnal.gov
Workflow ID11627
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-08 19:09:19
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2026-01-08 19:33:28
From worker nodeHostnamedunegli-8121869-0-fnpc22024.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-08 19:27:32
Input filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T091345Z_gen_000229_supernova_g4_detsim_reco_20250724T120629Z_reco2.root
JobscriptExit code0
Real time5m (348s)
CPU time5m (341s = 97%)
Max RSS bytes905990144 (864 MiB)
Outputting started2026-01-08 19:33:21
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/smanthey/ana/solar_ana_marley_flash_radiological_decay0_dune10kt_1x2x6_centralAPA/fnal/11627/1/001/fdhd_ana_2026-01-08T_192738Z_hist.root
Finished2026-01-08 19:33:28
Saved logsjustin-logs:267134.9-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

33119; 
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:06 UTC run: 5082 subRun: 0 event: 11450

Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:08 UTC run: 5082 subRun: 0 event: 11450
OpFlash PE 63.79 with purity 0.37 time -32.18 plane 0
 - Vertex (0.01, 3.46, 938.83)
	*** 1st Sanity check: Ratio 0.51 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 1190.68 with purity 0.98 time -18.07 plane 0
 - Vertex (0.05, 155.68, 1245.87)
	*** 1st Sanity check: Ratio 0.60 <= 1.00
	*** 2nd Sanity check: #OpHits 32 >= 32
OpFlash PE 30.76 with purity 0.51 time -17.66 plane 0
 - Vertex (-0.03, -318.60, 1243.00)
	*** 1st Sanity check: Ratio 0.25 <= 1.00
	*** 2nd Sanity check: #OpHits 10 >= 10
OpFlash PE 51.04 with purity 0.62 time -17.01 plane 0
 - Vertex (-0.05, -280.22, 1197.07)
	*** 1st Sanity check: Ratio 0.41 <= 1.00
	*** 2nd Sanity check: #OpHits 13 >= 13
OpFlash PE 10.86 with purity 0.43 time -16.60 plane 0
 - Vertex (0.05, 280.22, 379.11)
	*** 1st Sanity check: Ratio 0.85 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 26.28 with purity 0.94 time -16.47 plane 0
 - Vertex (0.05, 197.12, 1205.47)
	*** 1st Sanity check: Ratio 0.38 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 30.02 with purity 0.70 time -16.03 plane 0
 - Vertex (0.05, 115.81, 1145.91)
	*** 1st Sanity check: Ratio 0.24 <= 1.00
	*** 2nd Sanity check: #OpHits 10 >= 10
OpFlash PE 6.38 with purity 0.32 time -15.91 plane 0
 - Vertex (-0.02, -97.44, 957.15)
	*** 1st Sanity check: Ratio 0.35 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 16.43 with purity 0.89 time -15.58 plane 0
 - Vertex (0.03, 82.06, 1246.02)
	*** 1st Sanity check: Ratio 0.27 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 10.45 with purity 0.83 time -15.13 plane 0
 - Vertex (0.01, 76.14, 1255.75)
	*** 1st Sanity check: Ratio 0.25 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 6.50 with purity 0.56 time -14.67 plane 0
 - Vertex (0.05, 310.46, 1046.64)
	*** 1st Sanity check: Ratio 0.44 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 9.39 with purity 0.44 time -14.29 plane 0
 - Vertex (0.01, -11.03, 1140.12)
	*** 1st Sanity check: Ratio 0.44 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 14.25 with purity 0.87 time -13.95 plane 0
 - Vertex (0.05, 155.68, 1280.77)
	*** 1st Sanity check: Ratio 0.55 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 3.98 with purity 1.00 time -13.39 plane 0
 - Vertex (0.05, 405.70, 1100.31)
	*** 1st Sanity check: Ratio 0.51 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 1.69 with purity 1.00 time -12.75 plane 0
 - Vertex (0.05, 280.22, 548.70)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 4.82 with purity 1.00 time -12.62 plane 0
 - Vertex (0.05, 114.88, 1229.04)
	*** 1st Sanity check: Ratio 0.66 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 16.66 with purity 0.65 time -10.65 plane 0
 - Vertex (0.05, 108.03, 1208.53)
	*** 1st Sanity check: Ratio 0.59 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 8.05 with purity 0.28 time -9.74 plane 0
 - Vertex (0.00, -69.12, 1017.40)
	*** 1st Sanity check: Ratio 0.28 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 4.54 with purity 0.53 time -8.40 plane 0
 - Vertex (0.05, 243.73, 1298.47)
	*** 1st Sanity check: Ratio 0.53 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 11.81 with purity 0.17 time -7.68 plane 0
 - Vertex (0.05, 31.14, 1076.28)
	*** 1st Sanity check: Ratio 0.84 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2

# of OpHits (ophitspe): 70752 for times between -2261.86 us and 2227.81 us.
# of OpFlashes (solarflash): 9634 for times between -2261.86 us and 2227.81 us.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:08 UTC run: 5082 subRun: 0 event: 11450

# OpHits (ophitspe) in full geometry: 70752
# OpFlashes (solarflash) in full geometry: 9634
# Hits (hitfd) in each view: 222, 225, 471, 0
# Cluster from the hits: 135, 135, 424, 0
# Tracks (pmtracktc) in full geometry: 1
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:20 UTC run: 5082 subRun: 0 event: 11450

# ClusterReco: Looping over 296 matched clusters
%MSG
Selecting flash with PE 1190.68 closer to predicted PE 936.86 than reference flash PE 9.05667 predicted PE 71.4453
The differences are 0.213173 and 6.88869
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:20 UTC run: 5082 subRun: 0 event: 11450
*** Matched preselection cluster: 7 from 296
 - MainTrackID 33077
 - Gen 1 -> marley TPC 22
 - Purity 1.00 Hits 5
 - Charge 1430.75 ( MaxHit 552.82 )
 - #AdjCl 1 ( 1 signal ):
    - Cluster 6 at distance 19.27 with time 36.07 and charge 183.35 in TPC 22 and hits 1
 - RecoCol  Time,Y,Z ( 52.78, 146.31, 1255.26 )
 - RecoInd0 Time,Y,Z ( 52.17, 146.42, 1260.88 )
 - RecoInd1 Time,Y,Z ( 53.05, 146.19, 1277.27 )
*** Matched flash: 4834 from 9634
 - Purity 97.63 % Plane 0 #Hits 32
 - MainOpHitPE 714.75 (PE);  TotalPE 1190.68 vs expected 936.86 (PE)
 - Time -18.07 (us) Fast 92.61 % Residual 453.12
 - Reco Time,Y,Z ( -18.07, 155.68, 1245.87 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -10.66, 147.76, 1256.18 )
 - Main X,Y,Z ( -12.43, 146.59, 1254.80 )
 - EDep X,Y,Z ( -11.48, 145.73, 1254.96 )
 - Reco X,Y,Z ( -11.38, 146.31, 1255.26 )
%MSG
Selecting flash with PE 8.23 closer to predicted PE 23.949 than reference flash PE 6.68 predicted PE 22.6085
The differences are 1.90996 and 2.38451
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:20 UTC run: 5082 subRun: 0 event: 11450
*** Matched preselection cluster: 8 from 296
 - MainTrackID 33080
 - Gen 1 -> marley TPC 23
 - Purity 1.00 Hits 2
 - Charge 310.75 ( MaxHit 235.62 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 55.85, 167.03, 1201.94 )
 - RecoInd0 Time,Y,Z ( 55.89, 167.00, 1277.27 )
 - RecoInd1 Time,Y,Z ( 55.87, 167.07, 1241.68 )
*** Matched flash: 452 from 9634
 - Purity 0.00 % Plane 0 #Hits 3
 - MainOpHitPE 4.00 (PE);  TotalPE 8.23 vs expected 23.95 (PE)
 - Time -2065.16 (us) Fast 60.75 % Residual 0.68
 - Reco Time,Y,Z ( -2065.16, 123.67, 1134.45 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -10.66, 147.76, 1256.18 )
 - Main X,Y,Z ( 13.68, 184.76, 1209.40 )
 - EDep X,Y,Z ( 12.05, 167.04, 1201.79 )
 - Reco X,Y,Z ( 340.56, 167.03, 1201.94 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:20 UTC run: 5082 subRun: 0 event: 11450
*** Matched preselection cluster: 12 from 296
 - MainTrackID 33077
 - Gen 1 -> marley TPC 22
 - Purity 1.00 Hits 1
 - Charge 49.64 ( MaxHit 49.64 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 81.19, 794.24, 1233.92 )
 - RecoInd0 Time,Y,Z ( -1000000.00, -1000000.00, -1000000.00 )
 - RecoInd1 Time,Y,Z ( 76.90, 794.24, 1012.09 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -10.66, 147.76, 1256.18 )
 - Main X,Y,Z ( -12.43, 146.59, 1254.80 )
 - EDep X,Y,Z ( -11.48, 145.73, 1254.96 )
 - Reco X,Y,Z ( -1000000.00, 794.24, 1233.92 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:20 UTC run: 5082 subRun: 0 event: 11450
-----------------------------------------------------------------------------------------

%MSG
08-Jan-2026 19:33:21 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/c3/9b/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T091345Z_gen_000229_supernova_g4_detsim_reco_20250724T120629Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 08-Jan-2026 19:33:21 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.131927       6.21983       15.3942      0.397909       6.5566         50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00111413     0.0033917     0.0190179    0.00154668    0.00468042       50     
end_path:solarnuana:SolarNuAna     0.127266       6.21626       15.3927      0.395138       6.55723        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 08-Jan-2026 19:33:21 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1554.1 MB
  Peak resident set size usage (VmHWM): 908.579 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 326.222522 Real = 331.628928

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1554.1 VmHWM = 908.579

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-06 14:26:49 UTC       justIN version: 01.06.00