Jobsub ID 241929.1@dunegpschedd02.fnal.gov

Jobsub ID	241929.1@dunegpschedd02.fnal.gov
Workflow ID	9965
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-12 04:18:43
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-11-13 07:59:33
From worker node	Hostname	dunegli-7728647-0-fnpc23038.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-11-13 03:50:04
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T043927Z_gen_003309_supernova_g4stage1_g4stage2_detsim_reco_20250619T171703Z_reco2.root
Jobscript	Exit code	0
	Real time	4h (14953s)
	CPU time	4h (14825s = 99%)
	Max RSS bytes	1909526528 (1821 MiB)
Outputting started	2025-11-13 07:59:18
Output files
Finished	2025-11-13 07:59:33
List job events (HTCondor job logs unavailable)
Jobscript log (last 10,000 characters)

:solarnuana@BeginModule  13-Nov-2025 07:29:53 UTC run: 5019 subRun: 0 event: 33090
[33m
Number of Signal Particles: 5
Neutrino Interaction: -1
Neutrino Energy: 29.29 MeV
Position (149.62, -614.16, 396.84) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	22		2.22	(249.59, -822.77, 144.93)	133113	132980
marley	1000010020	0.00	(249.59, -822.77, 144.93)	133112	132980
marley	2212		0.00	(248.46, -818.35, 87.20)	133080	132980
marley	2212		0.00	(292.78, -829.76, 151.08)	133076	132980
marley	1000070140	0.00	(291.84, -837.91, 155.44)	133072	132980
marley	1000070140	0.00	(289.10, -840.23, 154.59)	133071	132980
marley	1000070140	0.00	(286.11, -838.95, 152.63)	133069	132980
marley	1000070140	0.00	(286.42, -838.38, 153.07)	133068	132980
marley	1000070140	0.00	(284.95, -841.35, 153.28)	133066	132980
marley	1000070140	0.00	(284.40, -840.73, 152.97)	133065	132980
marley	1000070140	0.00	(284.27, -841.18, 152.75)	133064	132980
marley	1000070140	0.00	(284.29, -841.14, 152.76)	133063	132980
marley	1000070140	0.00	(284.67, -841.85, 152.95)	133062	132980
marley	1000070140	0.00	(287.05, -842.53, 152.55)	133061	132980
marley	1000070140	0.00	(288.40, -842.35, 153.06)	133060	132980
marley	1000070140	0.00	(288.74, -842.08, 153.50)	133059	132980
marley	1000070140	0.00	(288.75, -840.73, 152.22)	133058	132980
marley	1000070140	0.00	(288.44, -840.89, 152.12)	133057	132980
marley	1000070140	0.00	(288.58, -840.74, 152.00)	133056	132980
marley	1000070140	0.00	(288.55, -840.94, 151.90)	133055	132980
marley	1000260560	0.00	(288.37, -840.60, 151.97)	133053	132980
marley	1000070140	0.00	(288.98, -840.52, 152.61)	133051	132980
marley	1000260560	0.00	(288.92, -840.43, 152.61)	133050	132980
marley	1000260560	0.00	(289.40, -841.15, 152.08)	133049	132980
marley	1000070140	0.00	(289.18, -841.07, 152.05)	133048	132980
marley	1000070140	0.00	(289.18, -840.97, 152.80)	133047	132980
marley	1000070140	0.00	(291.24, -841.59, 152.82)	133046	132980
marley	1000070140	0.00	(290.93, -842.72, 152.51)	133045	132980
marley	1000260560	0.00	(290.35, -840.62, 151.69)	133044	132980
marley	1000080160	0.00	(288.62, -841.64, 155.90)	133043	132980
marley	1000070140	0.00	(288.18, -841.40, 156.06)	133042	132980
marley	1000070140	0.00	(288.61, -839.85, 156.31)	133041	132980
marley	1000080160	0.00	(288.59, -839.49, 155.86)	133040	132980
marley	1000070140	0.00	(288.97, -837.45, 155.28)	133039	132980
marley	1000080160	0.00	(288.68, -837.57, 153.05)	133038	132980
marley	1000070140	0.00	(288.64, -837.56, 153.09)	133037	132980
marley	1000070140	0.00	(288.50, -837.56, 153.29)	133036	132980
marley	1000240530	0.00	(287.99, -836.52, 153.73)	133035	132980
marley	1000260540	0.00	(288.63, -834.16, 156.18)	133034	132980
marley	1000060120	0.00	(286.46, -824.80, 146.57)	133033	132980
marley	2212		0.01	(273.84, -819.50, 168.67)	133032	132980
marley	2212		0.05	(270.71, -816.83, 169.65)	133031	132980
marley	1000060120	0.02	(257.20, -818.00, 163.86)	133030	132980
marley	2212		0.04	(274.55, -791.33, 197.63)	133029	132980
marley	1000180400	0.01	(259.26, -717.59, 262.28)	133028	132980
marley	1000180400	0.01	(242.37, -736.63, 270.56)	133027	132980
marley	1000180360	0.01	(264.86, -599.06, 319.61)	133026	132980
marley	22		0.00	(144.93, -616.91, 390.92)	132984	132979
marley	22		0.99	(144.93, -616.91, 390.92)	132983	132979
marley	22		2.52	(144.93, -616.91, 390.92)	132982	132979
marley	1000180400	0.11	(144.93, -616.91, 390.92)	132981	132979
marley	2112		0.16	(144.93, -616.91, 390.92)	132980	132979
marley	2112		3.78	(149.62, -614.16, 396.84)	132979	0
marley	1000190390	0.03	(149.62, -614.16, 396.84)	132978	0
marley	11		16.17	(149.62, -614.16, 396.84)	132977	0
Signal Track IDs: 132977; 132978; 132979; 132980; 132981; 132982; 132983; 132984; 133026; 133027; 133028; 133029; 133030; 133031; 133032; 133033; 133034; 133035; 133036; 133037; 133038; 133039; 133040; 133041; 133042; 133043; 133044; 133045; 133046; 133047; 133048; 133049; 133050; 133051; 133053; 133055; 133056; 133057; 133058; 133059; 133060; 133061; 133062; 133063; 133064; 133065; 133066; 133068; 133069; 133071; 133072; 133076; 133080; 133112; 133113; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:29:53 UTC run: 5019 subRun: 0 event: 33090
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:30:19 UTC run: 5019 subRun: 0 event: 33090
[34mOpFlash PE 839.70 with purity 0.71 time 0.00 plane 2
 - Vertex (148.61, -743.30, 450.26)
	*** 1st Sanity check: Ratio 0.17 <= 1.00
	*** 2nd Sanity check: #OpHits 26 >= 26
OpFlash PE 67.56 with purity 0.05 time 0.00 plane 0
 - Vertex (-327.50, -261.17, 1810.38)
	*** 1st Sanity check: Ratio 0.23 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 29.91 with purity 0.43 time 0.02 plane 4
 - Vertex (188.74, 132.00, -96.50)
	*** 1st Sanity check: Ratio 0.22 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 262.39 with purity 0.24 time 0.01 plane 0
 - Vertex (-327.50, -426.55, 431.37)
	*** 1st Sanity check: Ratio 0.09 <= 1.00
	*** 2nd Sanity check: #OpHits 28 >= 28
OpFlash PE 19.76 with purity 0.18 time 0.04 plane 2
 - Vertex (225.72, -743.30, 1860.30)
	*** 1st Sanity check: Ratio 0.47 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3

# of OpFlashes (solarflash) in full geometry: 10715
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:30:21 UTC run: 5019 subRun: 0 event: 33090
[36m
# OpHits (ophit10ppm) in full geometry: 52536
# OpFlashes (solarflash) in full geometry: 10715
# Hits (gaushit) in each view: 6727, 6751, 6718, 0
# Cluster from the hits: 2802, 2864, 5682, 0
# Tracks (pmtracktc) in full geometry: 459[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:59:17 UTC run: 5019 subRun: 0 event: 33090
[32m
# ClusterReco: Looping over 5663 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:59:17 UTC run: 5019 subRun: 0 event: 33090
[32m*** Matched preselection cluster: 1373
 - MainTrackID 132977
 - Gen 1 -> marley TPC 16
 - Purity 1.00 Hits 9
 - Charge 1748.56 ( MaxHit 818.73 )
 - #AdjCl 5 ( 2 signal ):
    - Cluster 834 at distance 79.47 with time 692.77 and charge 125.82 in TPC 16 and hits 2
    - Cluster 937 at distance 69.07 with time 775.01 and charge 41.58 in TPC 16 and hits 1
    - Cluster 1064 at distance 45.18 with time 860.15 and charge 126.64 in TPC 16 and hits 1
    - Cluster 1223 at distance 19.20 with time 982.47 and charge 99.44 in TPC 16 and hits 2
    - Cluster 1544 at distance 27.32 with time 1198.93 and charge 72.83 in TPC 18 and hits 2
 - RecoCol  Time,Y,Z ( 1078.32, -616.42, 397.02 )
 - RecoInd0 Time,Y,Z ( 1081.81, -616.05, 373.80 )
 - RecoInd1 Time,Y,Z ( 1078.21, -616.79, 374.93 )
*** Matched flash: 
 - Purity 0.00 Plane 0 #Hits 19 PE 1522.75 MaxPE 451.04
 - Time -11.93 Fast 0.94 Residual 36492.52
 - Reco Time,Y,Z ( -11.93, -542.10, 338.26 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 149.62, -614.16, 396.84 )
 - Main X,Y,Z ( 152.05, -617.73, 396.01 )
 - EDep X,Y,Z ( 152.00, -617.55, 396.29 )
 - Reco X,Y,Z ( 149.98, -616.42, 397.02 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:59:17 UTC run: 5019 subRun: 0 event: 33090
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
13-Nov-2025 07:59:17 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ab/02/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T043927Z_gen_003309_supernova_g4stage1_g4stage2_detsim_reco_20250619T171703Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 13-Nov-2025 07:59:17 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.944911       1489.6        3925.78       883.568       1587.61        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000688139   0.00711726     0.0180837     0.0053913    0.00670266       10     
end_path:solarnuana:SolarNuAna     0.926834       1489.59       3925.77       883.561       1587.61        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 13-Nov-2025 07:59:17 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2534.1 MB
  Peak resident set size usage (VmHWM): 1909.53 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 14807.078292 Real = 14927.792012

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2534.1 VmHWM = 1909.53

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0