Jobsub ID 288544.38@dunegpschedd02.fnal.gov

Jobsub ID	288544.38@dunegpschedd02.fnal.gov
Workflow ID	12007
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-20 14:48:38
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2026-01-20 19:20:25
From worker node	Hostname	dunegli-50386-0-cmswn2355.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-20 14:57:36
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T004602Z_gen_000824_supernova_g4stage1_g4stage2_detsim_reco_20250620T211226Z_reco2.root
Jobscript	Exit code	0
	Real time	4h (15750s)
	CPU time	4h (15639s = 99%)
	Max RSS bytes	2562965504 (2444 MiB)
Outputting started	2026-01-20 19:20:06
Output files
Finished	2026-01-20 19:20:25
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

y in [MHz]:	2.00
TPC Tick in [us]:	0.50
TPC Map:		112 TPCs
TPC DriftLength in [cm]:650.07
TPC DriftTime in [us]:	4048.69
Fiducial Volume in [cm]:850.32, 1507.84, 2306.12
###################################################[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:53:47 UTC run: 5040 subRun: 0 event: 8240
[91m
There are a total of 132753 Particles in the event

# of particles 10	from gen 1 marley
# of particles 31270	from gen 2 Ar39GenInLAr
# of particles 3549	from gen 3 Kr85GenInLAr
# of particles 3	from gen 4 Ar42GenInLAr
# of particles 1	from gen 5 K42From42ArGenInLAr
# of particles 36	from gen 6 Rn222ChainRn222GenInLAr
# of particles 32	from gen 7 Rn222ChainPo218GenInLAr
# of particles 65	from gen 8 Rn222ChainPb214GenInLAr
# of particles 20	from gen 9 Rn222ChainBi214GenInLAr
# of particles 19	from gen 10 Rn222ChainPb210GenInLAr
# of particles 31	from gen 11 Rn220ChainPb212GenInLAr
# of particles 209	from gen 12 K40GenInCathode
# of particles 3092	from gen 13 U238ChainGenInCathode
# of particles 3404	from gen 14 Th232ChainGenInCathode
# of particles 103	from gen 15 K40GenInAnode
# of particles 1219	from gen 16 U238ChainGenInAnode
# of particles 1618	from gen 17 Th232ChainGenInAnode
# of particles 0	from gen 18 Rn222ChainGenInPDS
# of particles 3	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 2	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 24	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 34	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 49	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 44	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 48	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 20	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:53:47 UTC run: 5040 subRun: 0 event: 8240
[33m
Number of Signal Particles: 9
Neutrino Interaction: -1
Neutrino Energy: 14.82 MeV
Position (265.11, -583.19, 532.08) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.69	(265.11, -583.19, 532.08)	132819	132725
marley	-12		0.62	(265.11, -583.19, 532.08)	132818	132725
marley	1000200400	0.00	(265.11, -583.19, 532.08)	132817	132725
marley	22		0.03	(265.11, -583.19, 532.08)	132730	0
marley	22		0.77	(265.11, -583.19, 532.08)	132729	0
marley	22		0.84	(265.11, -583.19, 532.08)	132728	0
marley	22		0.65	(265.11, -583.19, 532.08)	132727	0
marley	22		2.09	(265.11, -583.19, 532.08)	132726	0
marley	1000190400	0.00	(265.11, -583.19, 532.08)	132725	0
marley	11		8.93	(265.11, -583.19, 532.08)	132724	0
Signal Track IDs: 132724; 132725; 132726; 132727; 132728; 132729; 132730; 132817; 132818; 132819; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:53:47 UTC run: 5040 subRun: 0 event: 8240
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:57:27 UTC run: 5040 subRun: 0 event: 8240
[34mOpFlash PE 10.68 with purity 0.75 time -3.18 plane 2
 - Vertex (285.07, -743.30, 152.42)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 301.81 with purity 0.05 time 0.11 plane 0
 - Vertex (-327.50, -377.40, 111.42)
	*** 1st Sanity check: Ratio 0.44 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 323.28 with purity 0.39 time 0.08 plane 2
 - Vertex (216.01, -743.30, 505.23)
	*** 1st Sanity check: Ratio 0.16 <= 1.00
	*** 2nd Sanity check: #OpHits 13 >= 13
OpFlash PE 13.90 with purity 0.37 time 0.78 plane 0
 - Vertex (-327.50, 430.22, 579.38)
	*** 1st Sanity check: Ratio 0.63 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 15.76 with purity 0.12 time 0.82 plane 0
 - Vertex (-327.50, -22.24, 1998.52)
	*** 1st Sanity check: Ratio 0.65 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 47.05 with purity 0.14 time 2.86 plane 0
 - Vertex (-327.50, -522.04, 396.72)
	*** 1st Sanity check: Ratio 0.19 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 51.24 with purity 0.16 time 3.27 plane 0
 - Vertex (-327.50, -458.47, 761.25)
	*** 1st Sanity check: Ratio 0.20 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 21.99 with purity 0.53 time 3.19 plane 2
 - Vertex (148.84, -743.30, 515.71)
	*** 1st Sanity check: Ratio 0.48 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3

# of OpHits (ophit10ppm): 52644 for times between -4254.57 us and 295147919198126080.00 us.
# of OpFlashes (solarflash): 18248 for times between -4254.57 us and 295147919198126080.00 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 17:57:30 UTC run: 5040 subRun: 0 event: 8240
[36m
# OpHits (ophit10ppm) in full geometry: 52644
# OpFlashes (solarflash) in full geometry: 18248
# Hits (gaushit) in each view: 6596, 6533, 6532, 0
# Cluster from the hits: 2734, 2767, 5591, 0
# Tracks (pmtracktc) in full geometry: 474[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 19:20:04 UTC run: 5040 subRun: 0 event: 8240
[32m
# ClusterReco: Looping over 5572 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 19:20:04 UTC run: 5040 subRun: 0 event: 8240
[32m*** Matched preselection cluster: 424 from 5572
 - MainTrackID 132724
 - Gen 1 -> marley TPC 24
 - Purity 1.00 Hits 3
 - Charge 951.79 ( MaxHit 581.24 )
 - #AdjCl 12 ( 5 signal ):
    - Cluster 300 at distance 17.19 with time 288.88 and charge 77.51 in TPC 24 and hits 1
    - Cluster 362 at distance 11.57 with time 329.15 and charge 129.17 in TPC 24 and hits 1
    - Cluster 370 at distance 98.32 with time 338.16 and charge 58.10 in TPC 32 and hits 1
    - Cluster 374 at distance 20.10 with time 341.64 and charge 192.19 in TPC 24 and hits 1
    - Cluster 398 at distance 88.64 with time 355.18 and charge 31.18 in TPC 24 and hits 1
    - Cluster 530 at distance 75.68 with time 447.71 and charge 32.93 in TPC 26 and hits 1
    - Cluster 531 at distance 14.66 with time 447.72 and charge 35.66 in TPC 24 and hits 1
    - Cluster 617 at distance 84.96 with time 514.81 and charge 30.63 in TPC 34 and hits 1
    - Cluster 862 at distance 69.26 with time 674.44 and charge 39.29 in TPC 24 and hits 1
    - Cluster 949 at distance 74.83 with time 754.78 and charge 25.59 in TPC 26 and hits 1
    - Cluster 1081 at distance 81.09 with time 848.35 and charge 44.63 in TPC 28 and hits 1
    - Cluster 1170 at distance 95.61 with time 921.16 and charge 221.97 in TPC 24 and hits 1
 - RecoCol  Time,Y,Z ( 368.83, -581.80, 532.47 )
 - RecoInd0 Time,Y,Z ( 370.13, -581.86, 522.72 )
 - RecoInd1 Time,Y,Z ( 368.99, -581.74, 522.72 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 265.11, -583.19, 532.08 )
 - Main X,Y,Z ( 266.29, -580.75, 532.58 )
 - EDep X,Y,Z ( 266.27, -580.86, 532.51 )
 - Reco X,Y,Z ( -1000000.00, -581.80, 532.47 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 19:20:04 UTC run: 5040 subRun: 0 event: 8240
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
20-Jan-2026 19:20:05 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/55/86/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250204T004602Z_gen_000824_supernova_g4stage1_g4stage2_detsim_reco_20250620T211226Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 19:20:05 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          2.38405       1564.38       5730.29       4.12535       2395.29        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00150957     0.0127881     0.0305756    0.00832407     0.0112709       10     
end_path:solarnuana:SolarNuAna      2.37067       1564.37       5730.28       4.09951       2395.3         10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 19:20:05 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3188.08 MB
  Peak resident set size usage (VmHWM): 2562.97 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 15604.664618 Real = 15714.246268

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3188.08 VmHWM = 2562.97

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===