Jobsub ID 296306.1@dunegpschedd02.fnal.gov

Jobsub ID	296306.1@dunegpschedd02.fnal.gov
Workflow ID	12412
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-28 13:32:43
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2026-01-28 20:25:57
From worker node	Hostname	dunegli-8465065-0-fnpc18001.fnal.gov
	cpuinfo	AMD EPYC 7551 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-28 13:33:56
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T081137Z_gen_003190_supernova_g4stage1_g4stage2_detsim_reco_20250618T102247Z_reco2.root
Jobscript	Exit code	0
	Real time	6h (24702s)
	CPU time	6h (22491s = 91%)
	Max RSS bytes	2584285184 (2464 MiB)
Outputting started	2026-01-28 20:25:39
Output files
Finished	2026-01-28 20:25:57
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

07
TPC DriftTime in [us]:	4048.69
Fiducial Volume in [cm]:850.32, 1507.84, 2306.12
###################################################[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:15:47 UTC run: 4933 subRun: 0 event: 31900
[91m
There are a total of 133178 Particles in the event

# of particles 11	from gen 1 marley
# of particles 31129	from gen 2 Ar39GenInLAr
# of particles 3708	from gen 3 Kr85GenInLAr
# of particles 1	from gen 4 Ar42GenInLAr
# of particles 3	from gen 5 K42From42ArGenInLAr
# of particles 31	from gen 6 Rn222ChainRn222GenInLAr
# of particles 28	from gen 7 Rn222ChainPo218GenInLAr
# of particles 92	from gen 8 Rn222ChainPb214GenInLAr
# of particles 36	from gen 9 Rn222ChainBi214GenInLAr
# of particles 32	from gen 10 Rn222ChainPb210GenInLAr
# of particles 23	from gen 11 Rn220ChainPb212GenInLAr
# of particles 231	from gen 12 K40GenInCathode
# of particles 3155	from gen 13 U238ChainGenInCathode
# of particles 3498	from gen 14 Th232ChainGenInCathode
# of particles 118	from gen 15 K40GenInAnode
# of particles 1314	from gen 16 U238ChainGenInAnode
# of particles 1659	from gen 17 Th232ChainGenInAnode
# of particles 1	from gen 18 Rn222ChainGenInPDS
# of particles 2	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 2	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 19	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 73	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 54	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 41	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 50	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 19	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:15:47 UTC run: 4933 subRun: 0 event: 31900
[33m
Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 26.51 MeV
Position (312.67, 66.30, 2102.13) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	22		1.46	(312.67, 66.30, 2102.13)	133215	133151
marley	1000180400	0.00	(312.67, 66.30, 2102.13)	133214	133151
marley	12		0.04	(312.67, 66.30, 2102.13)	133213	133151
marley	11		0.00	(312.67, 66.30, 2102.13)	133211	133151
marley	11		0.00	(312.67, 66.30, 2102.13)	133210	133151
marley	11		0.00	(312.67, 66.30, 2102.13)	133209	133151
marley	22		0.03	(312.67, 66.30, 2102.13)	133154	0
marley	22		1.61	(312.67, 66.30, 2102.13)	133153	0
marley	22		0.65	(312.67, 66.30, 2102.13)	133152	0
marley	1000190400	0.02	(312.67, 66.30, 2102.13)	133151	0
marley	11		22.69	(312.67, 66.30, 2102.13)	133150	0
Signal Track IDs: 133150; 133151; 133152; 133153; 133154; 133209; 133210; 133211; 133213; 133214; 133215; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:15:47 UTC run: 4933 subRun: 0 event: 31900
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:17:47 UTC run: 4933 subRun: 0 event: 31900
[34mOpFlash PE 633.77 with purity 0.79 time 0.18 plane 3
 - Vertex (247.36, 220.00, 2188.38)
	*** 1st Sanity check: Ratio 0.19 <= 1.00
	*** 2nd Sanity check: #OpHits 20 >= 20
OpFlash PE 43.40 with purity 0.44 time 0.70 plane 0
 - Vertex (-327.50, -295.21, 1832.24)
	*** 1st Sanity check: Ratio 0.24 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 54.76 with purity 0.07 time 0.75 plane 2
 - Vertex (85.26, -743.30, 1695.45)
	*** 1st Sanity check: Ratio 0.39 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 87.56 with purity 0.19 time 1.04 plane 0
 - Vertex (-327.50, 209.42, 745.19)
	*** 1st Sanity check: Ratio 0.33 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 14.28 with purity 0.78 time 1.45 plane 1
 - Vertex (210.07, 743.30, 1683.16)
	*** 1st Sanity check: Ratio 0.57 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 12.40 with purity 0.32 time 1.19 plane 2
 - Vertex (171.29, -743.30, 450.26)
	*** 1st Sanity check: Ratio 0.52 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 59.96 with purity 0.27 time 1.86 plane 1
 - Vertex (227.62, 743.30, 1235.29)
	*** 1st Sanity check: Ratio 0.22 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 225.81 with purity 0.18 time 2.90 plane 2
 - Vertex (195.64, -743.30, 988.65)
	*** 1st Sanity check: Ratio 0.30 <= 1.00
	*** 2nd Sanity check: #OpHits 9 >= 9
OpFlash PE 28.69 with purity 1.00 time 3.05 plane 3
 - Vertex (144.58, 152.02, 2188.38)
	*** 1st Sanity check: Ratio 0.28 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 20.61 with purity 0.54 time 2.93 plane 1
 - Vertex (210.07, 743.30, 1668.74)
	*** 1st Sanity check: Ratio 0.39 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 28.38 with purity 0.82 time 4.57 plane 3
 - Vertex (222.50, 151.29, 2188.38)
	*** 1st Sanity check: Ratio 0.33 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4

# of OpHits (ophit10ppm): 53051 for times between -4254.66 us and 295147919198126080.00 us.
# of OpFlashes (solarflash): 18212 for times between -4254.66 us and 295147919198126080.00 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:17:52 UTC run: 4933 subRun: 0 event: 31900
[36m
# OpHits (ophit10ppm) in full geometry: 53051
# OpFlashes (solarflash) in full geometry: 18212
# Hits (gaushit) in each view: 6670, 6652, 6736, 0
# Cluster from the hits: 2799, 2880, 5704, 0
# Tracks (pmtracktc) in full geometry: 454[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 20:25:35 UTC run: 4933 subRun: 0 event: 31900
[32m
# ClusterReco: Looping over 5700 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 20:25:35 UTC run: 4933 subRun: 0 event: 31900
[32m*** Matched preselection cluster: 391 from 5700
 - MainTrackID 133153
 - Gen 1 -> marley TPC 108
 - Purity 1.00 Hits 1
 - Charge 68.49 ( MaxHit 68.49 )
 - #AdjCl 4 ( 1 signal ):
    - Cluster 96 at distance 47.68 with time 131.17 and charge 41.48 in TPC 106 and hits 1
    - Cluster 100 at distance 45.05 with time 132.95 and charge 53.81 in TPC 108 and hits 1
    - Cluster 279 at distance 57.12 with time 249.82 and charge 30.48 in TPC 106 and hits 1
    - Cluster 973 at distance 99.47 with time 759.61 and charge 32.06 in TPC 110 and hits 1
 - RecoCol  Time,Y,Z ( 333.64, 74.67, 2061.54 )
 - RecoInd0 Time,Y,Z ( 333.49, 74.76, 2016.92 )
 - RecoInd1 Time,Y,Z ( 333.64, 74.57, 2016.92 )
*** Matched flash: 4732 from 18212
 - Purity 0.00 % Plane 0 #Hits 9
 - MainOpHitPE 36.55 (PE);  TotalPE 108.87 vs expected 40.61 (PE)
 - Time -2067.80 (us) Fast 34.49 % Residual 18.35
 - Reco Time,Y,Z ( -2067.80, 38.00, 1973.46 )
 - Found valid waveform: true
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 312.67, 66.30, 2102.13 )
 - Main X,Y,Z ( 268.71, 73.05, 2065.98 )
 - EDep X,Y,Z ( 271.47, 74.85, 2061.47 )
 - Reco X,Y,Z ( -60.55, 74.67, 2061.54 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 20:25:35 UTC run: 4933 subRun: 0 event: 31900
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
28-Jan-2026 20:25:36 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/46/71/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250127T081137Z_gen_003190_supernova_g4stage1_g4stage2_detsim_reco_20250618T102247Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 28-Jan-2026 20:25:36 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          1.57648       2440.07       5567.19       2128.88       2460.34        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00189494      6.66284       66.2204      0.0205252      19.8526        10     
end_path:solarnuana:SolarNuAna      1.57432       2433.4        5567.19       2096.19       2466.81        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 28-Jan-2026 20:25:36 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3197.54 MB
  Peak resident set size usage (VmHWM): 2584.29 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 22467.600777 Real = 24667.256645

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3197.54 VmHWM = 2584.29

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===