justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267134.0@dunegpschedd02.fnal.gov

Jobsub ID267134.0@dunegpschedd02.fnal.gov
Workflow ID11627
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-08 19:09:19
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2026-01-08 19:33:38
From worker nodeHostnamedunegli-8121842-0-fnpc23051.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-08 19:27:15
Input filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T170419Z_gen_002309_supernova_g4_detsim_reco_20250724T133134Z_reco2.root
JobscriptExit code0
Real time6m (376s)
CPU time5m (355s = 94%)
Max RSS bytes904327168 (862 MiB)
Outputting started2026-01-08 19:33:32
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/smanthey/ana/solar_ana_marley_flash_radiological_decay0_dune10kt_1x2x6_centralAPA/fnal/11627/1/001/fdhd_ana_2026-01-08T_192721Z_hist.root
Finished2026-01-08 19:33:38
Saved logsjustin-logs:267134.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

enInAPA
# of particles 3	from gen 23 U238ChainGenInAPA
# of particles 0	from gen 24 Th232ChainGenInAPA
# of particles 42	from gen 25 U238Th232K40GenInLArAPAboards1x2x6
# of particles 0	from gen 26 Rn222ChainGenInPDS
# of particles 19159	from gen 27 CavernwallGammasAtLAr1x2x6lowBgAPA
# of particles 764	from gen 28 foamGammasAtLAr1x2x6lowBgAPA
# of particles 0	from gen 29 CavernwallNeutronsAtLAr1x2x6lowBgAPA
# of particles 0	from gen 30 CryostatNGammasAtLAr1x2x6lowBgAPA
# of particles 0	from gen 31 CavernNGammasAtLAr1x2x6lowBgAPA
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:19 UTC run: 5092 subRun: 0 event: 115450

Number of Signal Particles: 6
Neutrino Interaction: -1
Neutrino Energy: 29.54 MeV
Position (254.49, -364.58, 721.76) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.27	(254.49, -364.58, 721.76)	32592	32574
marley	-12		0.44	(254.49, -364.58, 721.76)	32591	32574
marley	22		1.60	(254.49, -364.58, 721.76)	32576	0
marley	1000020040	5.79	(254.49, -364.58, 721.76)	32575	0
marley	1000170360	0.65	(254.49, -364.58, 721.76)	32574	0
marley	11		13.55	(254.49, -364.58, 721.76)	32573	0
Signal Track IDs: 32573; 32574; 32575; 32576; 32591; 32592; 
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:19 UTC run: 5092 subRun: 0 event: 115450

Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:21 UTC run: 5092 subRun: 0 event: 115450
OpFlash PE 14.70 with purity 0.37 time -19.17 plane 0
 - Vertex (0.05, 404.76, 781.09)
	*** 1st Sanity check: Ratio 0.73 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 219.00 with purity 0.95 time -17.96 plane 0
 - Vertex (-0.05, -360.43, 759.08)
	*** 1st Sanity check: Ratio 0.07 <= 1.00
	*** 2nd Sanity check: #OpHits 50 >= 50
OpFlash PE 40.89 with purity 0.99 time -17.35 plane 0
 - Vertex (-0.05, -401.12, 743.47)
	*** 1st Sanity check: Ratio 0.29 <= 1.00
	*** 2nd Sanity check: #OpHits 10 >= 10
OpFlash PE 34.21 with purity 0.95 time -17.05 plane 0
 - Vertex (-0.04, -302.28, 701.67)
	*** 1st Sanity check: Ratio 0.12 <= 1.00
	*** 2nd Sanity check: #OpHits 15 >= 15
OpFlash PE 7.47 with purity 0.25 time -16.61 plane 0
 - Vertex (0.03, 107.83, 276.37)
	*** 1st Sanity check: Ratio 0.26 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 40.15 with purity 0.96 time -16.20 plane 0
 - Vertex (-0.05, -372.68, 766.79)
	*** 1st Sanity check: Ratio 0.12 <= 1.00
	*** 2nd Sanity check: #OpHits 15 >= 15
OpFlash PE 28.25 with purity 0.80 time -15.52 plane 0
 - Vertex (-0.03, -254.63, 747.73)
	*** 1st Sanity check: Ratio 0.11 <= 1.00
	*** 2nd Sanity check: #OpHits 13 >= 13
OpFlash PE 5.76 with purity 0.69 time -14.97 plane 0
 - Vertex (-0.05, -264.35, 788.42)
	*** 1st Sanity check: Ratio 0.36 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 17.52 with purity 0.40 time -14.56 plane 0
 - Vertex (-0.04, -143.09, 347.38)
	*** 1st Sanity check: Ratio 0.24 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 16.40 with purity 0.13 time -14.17 plane 0
 - Vertex (-0.03, -100.81, 271.20)
	*** 1st Sanity check: Ratio 0.18 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 2.06 with purity 1.00 time -14.14 plane 0
 - Vertex (0.05, 342.49, 843.89)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 38.02 with purity 0.46 time -13.62 plane 0
 - Vertex (-0.05, -429.52, 494.41)
	*** 1st Sanity check: Ratio 0.43 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 4.59 with purity 0.67 time -13.00 plane 0
 - Vertex (-0.05, -508.54, 489.11)
	*** 1st Sanity check: Ratio 0.67 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 2.62 with purity 1.00 time -12.48 plane 0
 - Vertex (-0.05, -217.95, 611.50)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 3.95 with purity 1.00 time -8.69 plane 0
 - Vertex (-0.05, -316.71, 607.70)
	*** 1st Sanity check: Ratio 0.53 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 1.94 with purity 1.00 time -4.62 plane 0
 - Vertex (-0.05, -529.30, 892.69)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1

# of OpHits (ophitspe): 71966 for times between -2262.42 us and 2227.33 us.
# of OpFlashes (solarflash): 9688 for times between -2262.42 us and 2227.33 us.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:21 UTC run: 5092 subRun: 0 event: 115450

# OpHits (ophitspe) in full geometry: 71966
# OpFlashes (solarflash) in full geometry: 9688
# Hits (hitfd) in each view: 165, 190, 431, 0
# Cluster from the hits: 116, 125, 393, 0
# Tracks (pmtracktc) in full geometry: 1
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:31 UTC run: 5092 subRun: 0 event: 115450

# ClusterReco: Looping over 275 matched clusters
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:31 UTC run: 5092 subRun: 0 event: 115450
*** Matched preselection cluster: 141 from 275
 - MainTrackID 32573
 - Gen 1 -> marley TPC 13
 - Purity 1.00 Hits 1
 - Charge 50.76 ( MaxHit 50.76 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 1504.24, -573.03, 731.78 )
 - RecoInd0 Time,Y,Z ( 1497.47, -90.48, 1013.80 )
 - RecoInd1 Time,Y,Z ( 1497.53, -1055.58, 1109.57 )
*** Matched flash: 3961 from 9688
 - Purity 0.00 % Plane 0 #Hits 3
 - MainOpHitPE 8.67 (PE);  TotalPE 12.53 vs expected 21.29 (PE)
 - Time -438.07 (us) Fast 77.17 % Residual 6.43
 - Reco Time,Y,Z ( -438.07, -591.57, 781.09 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 254.49, -364.58, 721.76 )
 - Main X,Y,Z ( 252.91, -367.47, 721.92 )
 - EDep X,Y,Z ( 254.43, -364.64, 721.79 )
 - Reco X,Y,Z ( 311.86, -573.03, 731.78 )
%MSG
Selecting flash with PE 65.15 closer to predicted PE 77.2529 than reference flash PE 16.5367 predicted PE 64.7105
The differences are 0.18577 and 2.91315
Selecting flash with PE 396.643 closer to predicted PE 404.852 than reference flash PE 65.15 predicted PE 77.2529
The differences are 0.0206941 and 0.18577
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:31 UTC run: 5092 subRun: 0 event: 115450
*** Matched preselection cluster: 143 from 275
 - MainTrackID 32573
 - Gen 1 -> marley TPC 13
 - Purity 1.00 Hits 4
 - Charge 1164.34 ( MaxHit 749.05 )
 - #AdjCl 1 ( 1 signal ):
    - Cluster 137 at distance 18.35 with time 1467.16 and charge 107.97 in TPC 13 and hits 1
 - RecoCol  Time,Y,Z ( 1559.24, -366.22, 722.10 )
 - RecoInd0 Time,Y,Z ( 1559.53, -366.22, 812.49 )
 - RecoInd1 Time,Y,Z ( 1559.44, -366.23, 793.83 )
*** Matched flash: 7013 from 9688
 - Purity 0.00 % Plane 0 #Hits 35
 - MainOpHitPE 81.18 (PE);  TotalPE 396.64 vs expected 404.85 (PE)
 - Time 988.87 (us) Fast 31.51 % Residual 14682.58
 - Reco Time,Y,Z ( 988.87, -341.27, 629.43 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 254.49, -364.58, 721.76 )
 - Main X,Y,Z ( 252.91, -367.47, 721.92 )
 - EDep X,Y,Z ( 254.43, -364.64, 721.79 )
 - Reco X,Y,Z ( 91.58, -366.22, 722.10 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  08-Jan-2026 19:33:31 UTC run: 5092 subRun: 0 event: 115450
-----------------------------------------------------------------------------------------

%MSG
08-Jan-2026 19:33:31 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/91/5f/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T170419Z_gen_002309_supernova_g4_detsim_reco_20250724T133134Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 08-Jan-2026 19:33:31 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          0.15216       6.52473       15.3158      0.486292       6.5471         50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00140763    0.00747023     0.0240782     0.002356     0.00687857       50     
end_path:solarnuana:SolarNuAna     0.150317       6.51706       15.3138       0.47542       6.54698        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 08-Jan-2026 19:33:31 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1550.55 MB
  Peak resident set size usage (VmHWM): 906.342 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 339.413105 Real = 358.985510

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1550.55 VmHWM = 906.342

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-06 14:27:48 UTC       justIN version: 01.06.00