Jobsub ID 296325.0@dunegpschedd02.fnal.gov

Jobsub ID	296325.0@dunegpschedd02.fnal.gov
Workflow ID	12415
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-28 15:04:48
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2026-01-29 00:30:43
From worker node	Hostname	dunegli-8465065-0-fnpc18001.fnal.gov
	cpuinfo	AMD EPYC 7551 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-28 15:06:42
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T224159Z_gen_002270_supernova_g4stage1_g4stage2_detsim_reco_20250701T162653Z_reco2.root
Jobscript	Exit code	0
	Real time	9h (33826s)
	CPU time	9h (32756s = 96%)
	Max RSS bytes	2584170496 (2464 MiB)
Outputting started	2026-01-29 00:30:29
Output files
Finished	2026-01-29 00:30:43
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

z]:	2.00
TPC Tick in [us]:	0.50
TPC Map:		112 TPCs
TPC DriftLength in [cm]:650.07
TPC DriftTime in [us]:	4048.69
Fiducial Volume in [cm]:850.32, 1507.84, 2306.12
###################################################[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 23:23:32 UTC run: 4924 subRun: 0 event: 22700
[91m
There are a total of 132922 Particles in the event

# of particles 8	from gen 1 marley
# of particles 31266	from gen 2 Ar39GenInLAr
# of particles 3642	from gen 3 Kr85GenInLAr
# of particles 1	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 30	from gen 6 Rn222ChainRn222GenInLAr
# of particles 31	from gen 7 Rn222ChainPo218GenInLAr
# of particles 33	from gen 8 Rn222ChainPb214GenInLAr
# of particles 31	from gen 9 Rn222ChainBi214GenInLAr
# of particles 16	from gen 10 Rn222ChainPb210GenInLAr
# of particles 20	from gen 11 Rn220ChainPb212GenInLAr
# of particles 203	from gen 12 K40GenInCathode
# of particles 3082	from gen 13 U238ChainGenInCathode
# of particles 3510	from gen 14 Th232ChainGenInCathode
# of particles 80	from gen 15 K40GenInAnode
# of particles 1379	from gen 16 U238ChainGenInAnode
# of particles 1550	from gen 17 Th232ChainGenInAnode
# of particles 0	from gen 18 Rn222ChainGenInPDS
# of particles 2	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 2	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 14	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 38	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 43	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 34	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 49	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 10	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 23:23:32 UTC run: 4924 subRun: 0 event: 22700
[33m
Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 21.55 MeV
Position (-211.82, -59.70, 910.83) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.73	(-211.82, -59.70, 910.83)	132945	132907
marley	-12		0.58	(-211.82, -59.70, 910.83)	132944	132907
marley	1000200400	0.00	(-211.82, -59.70, 910.83)	132943	132907
marley	22		0.03	(-211.82, -59.70, 910.83)	132910	0
marley	22		2.72	(-211.82, -59.70, 910.83)	132909	0
marley	22		1.64	(-211.82, -59.70, 910.83)	132908	0
marley	1000190400	0.01	(-211.82, -59.70, 910.83)	132907	0
marley	11		15.65	(-211.82, -59.70, 910.83)	132906	0
Signal Track IDs: 132906; 132907; 132908; 132909; 132910; 132943; 132944; 132945; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 23:23:32 UTC run: 4924 subRun: 0 event: 22700
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 23:25:48 UTC run: 4924 subRun: 0 event: 22700
[34mOpFlash PE 919.56 with purity 0.23 time 0.14 plane 0
 - Vertex (-327.50, -50.09, 911.08)
	*** 1st Sanity check: Ratio 0.08 <= 1.00
	*** 2nd Sanity check: #OpHits 36 >= 36
OpFlash PE 21.68 with purity 0.15 time 0.44 plane 2
 - Vertex (101.71, -743.30, 1806.98)
	*** 1st Sanity check: Ratio 0.44 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 71.40 with purity 0.05 time 2.12 plane 0
 - Vertex (-327.50, -60.94, 827.04)
	*** 1st Sanity check: Ratio 0.21 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 79.36 with purity 0.10 time 3.05 plane 0
 - Vertex (-327.50, 379.25, 625.85)
	*** 1st Sanity check: Ratio 0.36 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 6.34 with purity 0.50 time 3.40 plane 0
 - Vertex (-327.50, 464.10, 1529.62)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 13.11 with purity 0.43 time 4.87 plane 0
 - Vertex (-327.50, 71.26, 796.81)
	*** 1st Sanity check: Ratio 0.54 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2

# of OpHits (ophit10ppm): 52429 for times between -4254.60 us and 295147919198126080.00 us.
# of OpFlashes (solarflash): 18204 for times between -4254.60 us and 295147919198126080.00 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 23:25:52 UTC run: 4924 subRun: 0 event: 22700
[36m
# OpHits (ophit10ppm) in full geometry: 52429
# OpFlashes (solarflash) in full geometry: 18204
# Hits (gaushit) in each view: 6828, 6824, 6744, 0
# Cluster from the hits: 2854, 2909, 5732, 0
# Tracks (pmtracktc) in full geometry: 456[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Jan-2026 00:30:27 UTC run: 4924 subRun: 0 event: 22700
[32m
# ClusterReco: Looping over 5728 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Jan-2026 00:30:27 UTC run: 4924 subRun: 0 event: 22700
[32m*** Matched preselection cluster: 4462 from 5728
 - MainTrackID 132906
 - Gen 1 -> marley TPC 51
 - Purity 1.00 Hits 12
 - Charge 1614.92 ( MaxHit 244.07 )
 - #AdjCl 14 ( 1 signal ):
    - Cluster 3905 at distance 93.49 with time 2944.57 and charge 34.43 in TPC 53 and hits 1
    - Cluster 4018 at distance 78.26 with time 3032.67 and charge 53.40 in TPC 55 and hits 2
    - Cluster 4078 at distance 79.03 with time 3076.34 and charge 40.86 in TPC 43 and hits 1
    - Cluster 4218 at distance 67.62 with time 3161.42 and charge 35.97 in TPC 43 and hits 1
    - Cluster 4234 at distance 67.16 with time 3174.63 and charge 30.76 in TPC 55 and hits 1
    - Cluster 4284 at distance 65.87 with time 3213.62 and charge 89.05 in TPC 43 and hits 1
    - Cluster 4307 at distance 71.25 with time 3228.91 and charge 106.98 in TPC 43 and hits 2
    - Cluster 4438 at distance 71.33 with time 3328.17 and charge 21.88 in TPC 51 and hits 1
    - Cluster 4482 at distance 78.43 with time 3354.67 and charge 25.35 in TPC 55 and hits 1
    - Cluster 4550 at distance 55.40 with time 3406.67 and charge 40.95 in TPC 53 and hits 1
    - Cluster 4564 at distance 78.39 with time 3415.22 and charge 30.10 in TPC 53 and hits 1
    - Cluster 4610 at distance 94.90 with time 3447.56 and charge 70.35 in TPC 41 and hits 1
    - Cluster 5035 at distance 65.11 with time 3721.77 and charge 271.55 in TPC 51 and hits 2
    - Cluster 5096 at distance 93.61 with time 3758.71 and charge 36.80 in TPC 47 and hits 1
 - RecoCol  Time,Y,Z ( 3346.38, -57.67, 913.78 )
 - RecoInd0 Time,Y,Z ( 3346.38, -57.69, 971.48 )
 - RecoInd1 Time,Y,Z ( 3346.20, -57.64, 954.95 )
*** Matched flash: 9058 from 18204
 - Purity 22.97 % Plane 0 #Hits 36
 - MainOpHitPE 75.03 (PE);  TotalPE 919.56 vs expected 707.20 (PE)
 - Time 0.14 (us) Fast 26.02 % Residual 235.77
 - Reco Time,Y,Z ( 0.14, -50.09, 911.08 )
 - Found valid waveform: true
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -211.82, -59.70, 910.83 )
 - Main X,Y,Z ( -213.27, -55.92, 915.87 )
 - EDep X,Y,Z ( -212.11, -57.83, 913.76 )
 - Reco X,Y,Z ( -212.25, -57.67, 913.78 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Jan-2026 00:30:27 UTC run: 4924 subRun: 0 event: 22700
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
29-Jan-2026 00:30:28 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/34/69/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250126T224159Z_gen_002270_supernova_g4stage1_g4stage2_detsim_reco_20250701T162653Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 29-Jan-2026 00:30:28 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          1.83808       3372.81       5118.38       4032.71       1732.96        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00191259     0.0312757     0.198908      0.0108411     0.0569179       10     
end_path:solarnuana:SolarNuAna      1.83328       3372.78       5118.18       4032.69       1732.94        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 29-Jan-2026 00:30:28 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3195.73 MB
  Peak resident set size usage (VmHWM): 2584.17 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 32730.768689 Real = 33787.351516

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3195.73 VmHWM = 2584.17

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===