Jobsub ID 288544.28@dunegpschedd02.fnal.gov

Jobsub ID	288544.28@dunegpschedd02.fnal.gov
Workflow ID	12007
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-20 14:48:38
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2026-01-20 17:36:20
From worker node	Hostname	dunegli-50386-0-cmswn2355.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-20 14:56:59
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T202650Z_gen_004750_supernova_g4stage1_g4stage2_detsim_reco_20250701T211759Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (9541s)
	CPU time	2h (9334s = 97%)
	Max RSS bytes	2555260928 (2436 MiB)
Outputting started	2026-01-20 17:36:01
Output files
Finished	2026-01-20 17:36:20
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

58 plane 0
 - Vertex (-327.50, -496.66, 841.20)
	*** 1st Sanity check: Ratio 0.10 <= 1.00
	*** 2nd Sanity check: #OpHits 26 >= 26
OpFlash PE 287.64 with purity 0.00 time 0.43 plane 4
 - Vertex (241.73, -220.00, -96.50)
	*** 1st Sanity check: Ratio 0.26 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 48.91 with purity 0.29 time 1.12 plane 0
 - Vertex (-327.50, 193.75, 1194.06)
	*** 1st Sanity check: Ratio 0.19 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 25.58 with purity 0.18 time 1.46 plane 2
 - Vertex (225.76, -743.30, 1560.19)
	*** 1st Sanity check: Ratio 0.52 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 56.28 with purity 0.04 time 2.20 plane 4
 - Vertex (113.67, 220.00, -96.50)
	*** 1st Sanity check: Ratio 0.25 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 107.84 with purity 0.69 time 1.98 plane 0
 - Vertex (-327.50, -477.92, 743.07)
	*** 1st Sanity check: Ratio 0.23 <= 1.00
	*** 2nd Sanity check: #OpHits 12 >= 12
OpFlash PE 19.01 with purity 1.00 time 3.75 plane 2
 - Vertex (181.34, -743.30, 931.84)
	*** 1st Sanity check: Ratio 0.62 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 20.89 with purity 0.50 time 4.09 plane 0
 - Vertex (-327.50, -334.65, 499.75)
	*** 1st Sanity check: Ratio 0.56 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 21.72 with purity 0.60 time 5.08 plane 2
 - Vertex (184.92, -743.30, 946.06)
	*** 1st Sanity check: Ratio 0.43 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 4.82 with purity 1.00 time 6.52 plane 0
 - Vertex (-327.50, -632.80, 1079.94)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1

# of OpHits (ophit10ppm): 52521 for times between -4254.14 us and 295147919198126080.00 us.
# of OpFlashes (solarflash): 18116 for times between -4254.14 us and 295147919198126080.00 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 16:25:36 UTC run: 4899 subRun: 0 event: 47499
[36m
# OpHits (ophit10ppm) in full geometry: 52521
# OpFlashes (solarflash) in full geometry: 18116
# Hits (gaushit) in each view: 6734, 6656, 6698, 0
# Cluster from the hits: 2804, 2823, 5683, 0
# Tracks (pmtracktc) in full geometry: 490[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:34:36 UTC run: 4899 subRun: 0 event: 47499
[32m
# ClusterReco: Looping over 5673 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:34:36 UTC run: 4899 subRun: 0 event: 47499
[32m*** Matched preselection cluster: 2021 from 5673
 - MainTrackID 132445
 - Gen 1 -> marley TPC 40
 - Purity 1.00 Hits 9
 - Charge 1638.46 ( MaxHit 295.41 )
 - #AdjCl 3 ( 1 signal ):
    - Cluster 1557 at distance 55.63 with time 1217.34 and charge 55.70 in TPC 40 and hits 1
    - Cluster 2109 at distance 15.16 with time 1621.67 and charge 318.10 in TPC 40 and hits 2
    - Cluster 2439 at distance 81.79 with time 1834.10 and charge 36.84 in TPC 40 and hits 1
 - RecoCol  Time,Y,Z ( 1558.18, -656.48, 857.71 )
 - RecoInd0 Time,Y,Z ( 1558.43, -656.45, 816.19 )
 - RecoInd1 Time,Y,Z ( 1557.81, -656.52, 863.14 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 73.51, -655.05, 856.45 )
 - Main X,Y,Z ( 77.07, -657.54, 858.09 )
 - EDep X,Y,Z ( 73.64, -655.27, 856.85 )
 - Reco X,Y,Z ( -1000000.00, -656.48, 857.71 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:34:36 UTC run: 4899 subRun: 0 event: 47499
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
Begin processing the 10th record. run: 4899 subRun: 0 event: 47500 at 20-Jan-2026 17:34:37 UTC
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:35:56 UTC run: 4899 subRun: 0 event: 47500
[35m
###################################################
Event:			47500
Flag:			594887612
Geometry:		dunevd10kt_3view_30deg_v5_refactored_1x8x14ref (VD)
Signal Label:		marley
PDS Frequency in [MHz]:	62.50
PDS Tick in [us]:	0.016
TPC Frequency in [MHz]:	2.00
TPC Tick in [us]:	0.50
TPC Map:		112 TPCs
TPC DriftLength in [cm]:650.07
TPC DriftTime in [us]:	4048.69
Fiducial Volume in [cm]:850.32, 1507.84, 2306.12
###################################################[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:35:59 UTC run: 4899 subRun: 0 event: 47500
[91m
There are a total of 133337 Particles in the event

# of particles 13	from gen 1 marley
# of particles 31178	from gen 2 Ar39GenInLAr
# of particles 3605	from gen 3 Kr85GenInLAr
# of particles 0	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 41	from gen 6 Rn222ChainRn222GenInLAr
# of particles 26	from gen 7 Rn222ChainPo218GenInLAr
# of particles 59	from gen 8 Rn222ChainPb214GenInLAr
# of particles 32	from gen 9 Rn222ChainBi214GenInLAr
# of particles 24	from gen 10 Rn222ChainPb210GenInLAr
# of particles 23	from gen 11 Rn220ChainPb212GenInLAr
# of particles 210	from gen 12 K40GenInCathode
# of particles 3111	from gen 13 U238ChainGenInCathode
# of particles 3739	from gen 14 Th232ChainGenInCathode
# of particles 89	from gen 15 K40GenInAnode
# of particles 1324	from gen 16 U238ChainGenInAnode
# of particles 1779	from gen 17 Th232ChainGenInAnode
# of particles 2	from gen 18 Rn222ChainGenInPDS
# of particles 1	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 1	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 22	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 50	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 51	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 44	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 48	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 17	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:35:59 UTC run: 4899 subRun: 0 event: 47500
[33m
Number of Signal Particles: 8
Neutrino Interaction: -1
Neutrino Energy: 43.67 MeV
Position (156.86, 549.01, 226.57) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	22		1.46	(156.86, 549.01, 226.57)	133368	133309
marley	1000180400	0.00	(156.86, 549.01, 226.57)	133367	133309
marley	12		0.04	(156.86, 549.01, 226.57)	133366	133309
marley	11		0.00	(156.86, 549.01, 226.57)	133364	133309
marley	11		0.00	(156.86, 549.01, 226.57)	133363	133309
marley	11		0.00	(156.86, 549.01, 226.57)	133362	133309
marley	11		0.00	(156.86, 549.01, 226.57)	133361	133309
marley	22		0.03	(156.86, 549.01, 226.57)	133313	0
marley	22		1.61	(156.86, 549.01, 226.57)	133312	0
marley	22		0.65	(156.86, 549.01, 226.57)	133311	0
marley	22		2.09	(156.86, 549.01, 226.57)	133310	0
marley	1000190400	0.03	(156.86, 549.01, 226.57)	133309	0
marley	11		37.75	(156.86, 549.01, 226.57)	133308	0
Signal Track IDs: 133308; 133309; 133310; 133311; 133312; 133313; 133361; 133362; 133363; 133364; 133366; 133367; 133368; [0m
%MSG
%MSG-w ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:35:59 UTC run: 4899 subRun: 0 event: 47500
[31m
Kinetic energy of signal particle is above threshold of 30.00 MeV. Skipping event.
[0m
%MSG
20-Jan-2026 17:35:59 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/5a/c4/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250125T202650Z_gen_004750_supernova_g4stage1_g4stage2_detsim_reco_20250701T211759Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 17:35:59 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          1.40597       932.439       4930.83       2.80362       1863.82        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000606724   0.00958606     0.0215959     0.0084412    0.00803407       10     
end_path:solarnuana:SolarNuAna      1.3924        932.429       4930.81       2.80115       1863.82        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 17:35:59 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3186.37 MB
  Peak resident set size usage (VmHWM): 2555.26 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 9317.130039 Real = 9514.562750

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3186.37 VmHWM = 2555.26

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===