Jobsub ID 241448.0@dunegpschedd02.fnal.gov

Jobsub ID	241448.0@dunegpschedd02.fnal.gov
Workflow ID	9933
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-11 19:04:39
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2025-11-11 20:47:44
From worker node	Hostname	dunegli-7570624-0-fnpc23033.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-11-11 20:42:47
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T103122Z_gen_000769_supernova_g4_detsim_reco_20250724T133740Z_reco2.root
Outputting started	2025-11-11 20:47:44
Output files
Finished	2025-11-11 21:34:38
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

99.37, 523.84, 849.28)	32945	32939
marley	2212		0.01	(396.10, 525.41, 849.21)	32944	32939
marley	2212		0.08	(390.79, 529.43, 840.43)	32943	32939
marley	1000060120	0.03	(383.94, 552.41, 845.38)	32942	32939
marley	1000070140	0.01	(385.10, 548.50, 848.21)	32941	32939
marley	1000180400	0.00	(353.87, 527.75, 860.71)	32940	32939
marley	2112		0.13	(317.85, 492.32, 860.76)	32939	0
marley	1000190390	0.10	(317.85, 492.32, 860.76)	32938	0
marley	11		35.06	(317.85, 492.32, 860.76)	32937	0
Signal Track IDs: 32937; 32938; 32939; 32940; 32941; 32942; 32943; 32944; 32945; 32947; 32948; 32949; 32950; 32951; 32964; 32965; 32966; 32967; 32968; [0m
%MSG
%MSG-w ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:28 UTC run: 5064 subRun: 0 event: 38449
[31m
Kinetic energy of signal particle is above threshold of 30.00 MeV. Skipping event.
[0m
%MSG
Begin processing the 50th record. run: 5064 subRun: 0 event: 38450 at 11-Nov-2025 20:47:28 UTC
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:28 UTC run: 5064 subRun: 0 event: 38450
[35m
###################################################
Event:			38450
Flag:			1038348561
Geometry:		dune10kt_v6_1x2x6 (HD)
Signal Label:		marley
PDS Frequency in [MHz]:	62.50
PDS Tick in [us]:	0.016
TPC Frequency in [MHz]:	2.00
TPC Tick in [us]:	0.50
TPC Map:		24 TPCs
TPC DriftLength in [cm]:360.37
TPC DriftTime in [us]:	2244.44
Fiducial Volume in [cm]:759.89, 1357.70, 1746.48
###################################################[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:29 UTC run: 5064 subRun: 0 event: 38450
[91m
There are a total of 33024 Particles in the event

# of particles 7	from gen 1 marley
# of particles 11005	from gen 2 Ar39GenInLAr
# of particles 1216	from gen 3 Kr85GenInLAr
# of particles 3	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 12	from gen 6 Rn222ChainRn222GenInLAr
# of particles 8	from gen 7 Rn222ChainPo218GenInLAr
# of particles 29	from gen 8 Rn222ChainPb214GenInLAr
# of particles 19	from gen 9 Rn222ChainBi214GenInLAr
# of particles 9	from gen 10 Rn222ChainPb210GenInLAr
# of particles 2	from gen 11 Rn220ChainPb212GenInLAr
# of particles 20	from gen 12 K40GenInCPA
# of particles 236	from gen 13 U238ChainGenInCPA
# of particles 308	from gen 14 Th232ChainGenInCPA
# of particles 0	from gen 15 K42From42ArGenInCPA
# of particles 3	from gen 16 Rn222ChainPo218GenInCPA
# of particles 19	from gen 17 Rn222ChainPb214GenInCPA
# of particles 48	from gen 18 Rn222ChainBi214GenInCPA
# of particles 40	from gen 19 Rn222ChainPb210GenInCPA
# of particles 22	from gen 20 Rn222ChainFromBi210GenInCPA
# of particles 44	from gen 21 Rn220ChainFromPb212GenInCPA
# of particles 9	from gen 22 Co60GenInAPA
# of particles 0	from gen 23 U238ChainGenInAPA
# of particles 0	from gen 24 Th232ChainGenInAPA
# of particles 42	from gen 25 U238Th232K40GenInLArAPAboards1x2x6
# of particles 0	from gen 26 Rn222ChainGenInPDS
# of particles 19159	from gen 27 CavernwallGammasAtLAr1x2x6lowBgAPA
# of particles 764	from gen 28 foamGammasAtLAr1x2x6lowBgAPA
# of particles 0	from gen 29 CavernwallNeutronsAtLAr1x2x6lowBgAPA
# of particles 0	from gen 30 CryostatNGammasAtLAr1x2x6lowBgAPA
# of particles 0	from gen 31 CavernNGammasAtLAr1x2x6lowBgAPA[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:29 UTC run: 5064 subRun: 0 event: 38450
[33m
Number of Signal Particles: 6
Neutrino Interaction: -1
Neutrino Energy: 6.50 MeV
Position (-217.03, -36.06, 223.19) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.93	(-217.03, -36.06, 223.19)	33076	33019
marley	-12		0.38	(-217.03, -36.06, 223.19)	33075	33019
marley	1000200400	0.00	(-217.03, -36.06, 223.19)	33074	33019
marley	22		2.42	(-217.03, -36.06, 223.19)	33021	0
marley	22		0.31	(-217.03, -36.06, 223.19)	33020	0
marley	1000190400	0.00	(-217.03, -36.06, 223.19)	33019	0
marley	11		2.26	(-217.03, -36.06, 223.19)	33018	0
Signal Track IDs: 33018; 33019; 33020; 33021; 33074; 33075; 33076; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:29 UTC run: 5064 subRun: 0 event: 38450
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:31 UTC run: 5064 subRun: 0 event: 38450
[34mOpFlash PE 41.52 with purity 0.51 time 0.00 plane 0
 - Vertex (-0.05, -342.49, 379.11)
	*** 1st Sanity check: Ratio 0.66 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 5.85 with purity 1.00 time 0.01 plane 0
 - Vertex (0.05, 365.69, 196.57)
	*** 1st Sanity check: Ratio 0.36 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 32.09 with purity 0.95 time 0.01 plane 0
 - Vertex (-0.01, -61.06, 293.31)
	*** 1st Sanity check: Ratio 0.19 <= 1.00
	*** 2nd Sanity check: #OpHits 13 >= 13
OpFlash PE 1.60 with purity 1.00 time 0.02 plane 0
 - Vertex (-0.05, -591.57, 146.72)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 2.18 with purity 1.00 time 0.03 plane 0
 - Vertex (0.05, 280.22, 427.91)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 3.26 with purity 0.50 time 0.05 plane 0
 - Vertex (0.05, 93.41, 195.52)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 7.18 with purity 0.73 time 0.07 plane 0
 - Vertex (0.02, 15.44, 175.60)
	*** 1st Sanity check: Ratio 0.48 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 3.92 with purity 1.00 time 0.09 plane 0
 - Vertex (0.05, 227.22, 244.14)
	*** 1st Sanity check: Ratio 0.57 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2

# of OpFlashes (solarflash) in full geometry: 8683
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:31 UTC run: 5064 subRun: 0 event: 38450
[36m
# OpHits (ophitspe) in full geometry: 70906
# OpFlashes (solarflash) in full geometry: 8683
# Hits (hitfd) in each view: 220, 234, 476, 0
# Cluster from the hits: 138, 146, 433, 0
# Tracks (pmtracktc) in full geometry: 1[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:43 UTC run: 5064 subRun: 0 event: 38450
[32m
# ClusterReco: Looping over 283 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:44 UTC run: 5064 subRun: 0 event: 38450
[32m*** Matched preselection cluster: 133
 - MainTrackID 33018
 - Gen 1 -> marley TPC 0
 - Purity 1.00 Hits 2
 - Charge 281.13 ( MaxHit 171.44 )
 - #AdjCl 1 ( 1 signal ):
    - Cluster 132 at distance 6.35 with time 1313.78 and charge 157.81 in TPC 0 and hits 1
 - RecoCol  Time,Y,Z ( 1334.08, -36.32, 223.04 )
 - RecoInd0 Time,Y,Z ( 1334.00, -36.28, 214.16 )
 - RecoInd1 Time,Y,Z ( 1334.11, -36.35, 115.32 )
*** Matched flash: 
 - Purity 0.00 Plane 0 #Hits 3 PE 62.51 MaxPE 58.51
 - Time -9.72 Fast 0.98 Residual 1145.53
 - Reco Time,Y,Z ( -9.72, 31.14, 195.52 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -217.03, -36.06, 223.19 )
 - Main X,Y,Z ( -217.33, -36.14, 223.02 )
 - EDep X,Y,Z ( -217.17, -36.28, 223.04 )
 - Reco X,Y,Z ( -215.76, -36.32, 223.04 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  11-Nov-2025 20:47:44 UTC run: 5064 subRun: 0 event: 38450
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
11-Nov-2025 20:47:44 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/d0/21/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250205T103122Z_gen_000769_supernova_g4_detsim_reco_20250724T133740Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 11-Nov-2025 20:47:44 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.130185       5.14381       15.5043      0.256676       6.58199        50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00117143    0.00958393     0.0358796    0.00896527    0.00859695       50     
end_path:solarnuana:SolarNuAna     0.128894       5.13407       15.5026      0.246744       6.58274        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 11-Nov-2025 20:47:44 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1550.22 MB
  Peak resident set size usage (VmHWM): 908.382 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 269.757757 Real = 281.130750

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1550.22 VmHWM = 908.382

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0