Jobsub ID 288544.41@dunegpschedd02.fnal.gov

Jobsub ID	288544.41@dunegpschedd02.fnal.gov
Workflow ID	12007
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-20 14:48:38
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2026-01-20 22:14:16
From worker node	Hostname	dunegli-50386-0-cmswn2355.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-20 14:57:16
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T083024Z_gen_004373_supernova_g4stage1_g4stage2_detsim_reco_20250627T053005Z_reco2.root
Jobscript	Exit code	0
	Real time	7h (26201s)
	CPU time	7h (26013s = 99%)
	Max RSS bytes	2571296768 (2452 MiB)
Outputting started	2026-01-20 22:13:58
Output files
Finished	2026-01-20 22:14:16
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 83844	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3566	from gen 27 foamGammasAtLAr1x8x14
# of particles 0	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 0	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 20:40:46 UTC run: 5058 subRun: 0 event: 43729
[33m
Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 17.92 MeV
Position (243.03, -559.15, 2134.80) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.57	(243.03, -559.15, 2134.80)	132532	132448
marley	-12		0.74	(243.03, -559.15, 2134.80)	132531	132448
marley	1000200400	0.00	(243.03, -559.15, 2134.80)	132530	132448
marley	22		0.03	(243.03, -559.15, 2134.80)	132451	0
marley	22		2.02	(243.03, -559.15, 2134.80)	132450	0
marley	22		2.46	(243.03, -559.15, 2134.80)	132449	0
marley	1000190400	0.01	(243.03, -559.15, 2134.80)	132448	0
marley	11		11.89	(243.03, -559.15, 2134.80)	132447	0
Signal Track IDs: 132447; 132448; 132449; 132450; 132451; 132530; 132531; 132532; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 20:40:46 UTC run: 5058 subRun: 0 event: 43729
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 20:44:13 UTC run: 5058 subRun: 0 event: 43729
[34mOpFlash PE 229.41 with purity 0.21 time -0.16 plane 3
 - Vertex (285.07, -220.00, 2188.38)
	*** 1st Sanity check: Ratio 0.68 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 110.44 with purity 0.19 time 0.26 plane 2
 - Vertex (236.09, -743.30, 1939.46)
	*** 1st Sanity check: Ratio 0.30 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 93.67 with purity 0.18 time 1.23 plane 2
 - Vertex (119.99, -743.30, 748.10)
	*** 1st Sanity check: Ratio 0.45 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 8.72 with purity 1.00 time 2.06 plane 0
 - Vertex (-327.50, -468.10, 2044.46)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 184.84 with purity 0.27 time 2.01 plane 3
 - Vertex (285.07, -220.00, 2188.38)
	*** 1st Sanity check: Ratio 0.73 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 11.58 with purity 0.23 time 2.64 plane 2
 - Vertex (215.24, -743.30, 1939.46)
	*** 1st Sanity check: Ratio 0.53 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 20.77 with purity 0.12 time 2.81 plane 1
 - Vertex (108.82, 743.30, 152.42)
	*** 1st Sanity check: Ratio 0.29 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4

# of OpHits (ophit10ppm): 52847 for times between -4254.36 us and 295147919198126080.00 us.
# of OpFlashes (solarflash): 18369 for times between -4254.36 us and 295147919198126080.00 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 20:44:17 UTC run: 5058 subRun: 0 event: 43729
[36m
# OpHits (ophit10ppm) in full geometry: 52847
# OpFlashes (solarflash) in full geometry: 18369
# Hits (gaushit) in each view: 6912, 6818, 6728, 0
# Cluster from the hits: 2905, 2892, 5739, 0
# Tracks (pmtracktc) in full geometry: 493[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:13:48 UTC run: 5058 subRun: 0 event: 43729
[32m
# ClusterReco: Looping over 5736 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:13:48 UTC run: 5058 subRun: 0 event: 43729
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
Begin processing the 10th record. run: 5058 subRun: 0 event: 43730 at 20-Jan-2026 22:13:49 UTC
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:13:54 UTC run: 5058 subRun: 0 event: 43730
[35m
###################################################
Event:			43730
Flag:			490676144
Geometry:		dunevd10kt_3view_30deg_v5_refactored_1x8x14ref (VD)
Signal Label:		marley
PDS Frequency in [MHz]:	62.50
PDS Tick in [us]:	0.016
TPC Frequency in [MHz]:	2.00
TPC Tick in [us]:	0.50
TPC Map:		112 TPCs
TPC DriftLength in [cm]:650.07
TPC DriftTime in [us]:	4048.69
Fiducial Volume in [cm]:850.32, 1507.84, 2306.12
###################################################[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:13:56 UTC run: 5058 subRun: 0 event: 43730
[91m
There are a total of 132883 Particles in the event

# of particles 10	from gen 1 marley
# of particles 31154	from gen 2 Ar39GenInLAr
# of particles 3553	from gen 3 Kr85GenInLAr
# of particles 4	from gen 4 Ar42GenInLAr
# of particles 1	from gen 5 K42From42ArGenInLAr
# of particles 29	from gen 6 Rn222ChainRn222GenInLAr
# of particles 28	from gen 7 Rn222ChainPo218GenInLAr
# of particles 32	from gen 8 Rn222ChainPb214GenInLAr
# of particles 31	from gen 9 Rn222ChainBi214GenInLAr
# of particles 34	from gen 10 Rn222ChainPb210GenInLAr
# of particles 17	from gen 11 Rn220ChainPb212GenInLAr
# of particles 210	from gen 12 K40GenInCathode
# of particles 2859	from gen 13 U238ChainGenInCathode
# of particles 3689	from gen 14 Th232ChainGenInCathode
# of particles 116	from gen 15 K40GenInAnode
# of particles 1420	from gen 16 U238ChainGenInAnode
# of particles 1630	from gen 17 Th232ChainGenInAnode
# of particles 0	from gen 18 Rn222ChainGenInPDS
# of particles 4	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 4	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 18	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 46	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 50	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 36	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 36	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 24	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:13:56 UTC run: 5058 subRun: 0 event: 43730
[33m
Number of Signal Particles: 9
Neutrino Interaction: -1
Neutrino Energy: 47.69 MeV
Position (78.04, -374.82, 1072.01) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.37	(78.04, -374.82, 1072.01)	132947	132850
marley	-12		0.94	(78.04, -374.82, 1072.01)	132946	132850
marley	1000200400	0.00	(78.04, -374.82, 1072.01)	132945	132850
marley	22		0.03	(78.04, -374.82, 1072.01)	132855	0
marley	22		0.77	(78.04, -374.82, 1072.01)	132854	0
marley	22		1.16	(78.04, -374.82, 1072.01)	132853	0
marley	22		0.33	(78.04, -374.82, 1072.01)	132852	0
marley	22		2.09	(78.04, -374.82, 1072.01)	132851	0
marley	1000190400	0.00	(78.04, -374.82, 1072.01)	132850	0
marley	11		41.80	(78.04, -374.82, 1072.01)	132849	0
Signal Track IDs: 132849; 132850; 132851; 132852; 132853; 132854; 132855; 132945; 132946; 132947; [0m
%MSG
%MSG-w ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:13:56 UTC run: 5058 subRun: 0 event: 43730
[31m
Kinetic energy of signal particle is above threshold of 30.00 MeV. Skipping event.
[0m
%MSG
20-Jan-2026 22:13:57 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/df/f6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T083024Z_gen_004373_supernova_g4stage1_g4stage2_detsim_reco_20250627T053005Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 22:13:57 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          2.57534       2611.32       5584.97        2512         2612.23        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)             0.0011342     0.0131011     0.0254064     0.0112253    0.00922421       10     
end_path:solarnuana:SolarNuAna      2.56937       2611.31       5584.94       2511.98       2612.23        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 22:13:57 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3200.23 MB
  Peak resident set size usage (VmHWM): 2571.3 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 25991.782213 Real = 26177.510150

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3200.23 VmHWM = 2571.3

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===