Jobsub ID 241885.11@dunegpschedd02.fnal.gov

Jobsub ID	241885.11@dunegpschedd02.fnal.gov
Workflow ID	9965
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-12 03:52:43
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-11-13 07:47:10
From worker node	Hostname	dunegli-7728493-0-fnpc23013.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-11-13 03:41:00
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T094818Z_gen_003403_supernova_g4stage1_g4stage2_detsim_reco_20250624T151634Z_reco2.root
Jobscript	Exit code	0
	Real time	4h (14757s)
	CPU time	4h (14569s = 98%)
	Max RSS bytes	1913249792 (1824 MiB)
Outputting started	2025-11-13 07:46:58
Output files
Finished	2025-11-13 07:47:10
List job events (HTCondor job logs unavailable)
Jobscript log (last 10,000 characters)

222.28 )
 - EDep X,Y,Z ( 230.44, -541.83, 221.91 )
 - Reco X,Y,Z ( 239.31, -541.95, 221.76 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:20:07 UTC run: 5004 subRun: 0 event: 34029
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
Begin processing the 10th record. run: 5004 subRun: 0 event: 34030 at 13-Nov-2025 07:20:07 UTC
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:20:09 UTC run: 5004 subRun: 0 event: 34030
[35m
###################################################
Event:			34030
Flag:			1733856792
Geometry:		dunevd10kt_3view_30deg_v5_refactored_1x8x14ref (VD)
Signal Label:		marley
PDS Frequency in [MHz]:	62.50
PDS Tick in [us]:	0.016
TPC Frequency in [MHz]:	2.00
TPC Tick in [us]:	0.50
TPC Map:		112 TPCs
TPC DriftLength in [cm]:650.07
TPC DriftTime in [us]:	4048.69
Fiducial Volume in [cm]:850.32, 1507.84, 2306.12
###################################################[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:20:10 UTC run: 5004 subRun: 0 event: 34030
[91m
There are a total of 132516 Particles in the event

# of particles 12	from gen 1 marley
# of particles 31041	from gen 2 Ar39GenInLAr
# of particles 3631	from gen 3 Kr85GenInLAr
# of particles 2	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 41	from gen 6 Rn222ChainRn222GenInLAr
# of particles 27	from gen 7 Rn222ChainPo218GenInLAr
# of particles 45	from gen 8 Rn222ChainPb214GenInLAr
# of particles 35	from gen 9 Rn222ChainBi214GenInLAr
# of particles 13	from gen 10 Rn222ChainPb210GenInLAr
# of particles 33	from gen 11 Rn220ChainPb212GenInLAr
# of particles 211	from gen 12 K40GenInCathode
# of particles 2928	from gen 13 U238ChainGenInCathode
# of particles 3446	from gen 14 Th232ChainGenInCathode
# of particles 97	from gen 15 K40GenInAnode
# of particles 1421	from gen 16 U238ChainGenInAnode
# of particles 1468	from gen 17 Th232ChainGenInAnode
# of particles 1	from gen 18 Rn222ChainGenInPDS
# of particles 3	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 0	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 10	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 50	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 27	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 36	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 40	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 50	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:20:10 UTC run: 5004 subRun: 0 event: 34030
[33m
Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 20.00 MeV
Position (27.12, 144.66, 772.27) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	22		1.46	(27.12, 144.66, 772.27)	132523	132456
marley	1000180400	0.00	(27.12, 144.66, 772.27)	132522	132456
marley	12		0.04	(27.12, 144.66, 772.27)	132521	132456
marley	11		0.00	(27.12, 144.66, 772.27)	132519	132456
marley	11		0.00	(27.12, 144.66, 772.27)	132518	132456
marley	11		0.00	(27.12, 144.66, 772.27)	132517	132456
marley	11		0.00	(27.12, 144.66, 772.27)	132516	132456
marley	22		0.03	(27.12, 144.66, 772.27)	132459	0
marley	22		0.77	(27.12, 144.66, 772.27)	132458	0
marley	22		3.71	(27.12, 144.66, 772.27)	132457	0
marley	1000190400	0.01	(27.12, 144.66, 772.27)	132456	0
marley	11		13.98	(27.12, 144.66, 772.27)	132455	0
Signal Track IDs: 132455; 132456; 132457; 132458; 132459; 132516; 132517; 132518; 132519; 132521; 132522; 132523; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:20:10 UTC run: 5004 subRun: 0 event: 34030
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:20:29 UTC run: 5004 subRun: 0 event: 34030
[34mOpFlash PE 114.22 with purity 0.09 time -0.05 plane 2
 - Vertex (285.07, -743.30, 1641.62)
	*** 1st Sanity check: Ratio 0.40 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 430.41 with purity 0.19 time 0.01 plane 0
 - Vertex (-327.50, 143.69, 744.40)
	*** 1st Sanity check: Ratio 0.07 <= 1.00
	*** 2nd Sanity check: #OpHits 39 >= 39
OpFlash PE 20.77 with purity 0.31 time 0.01 plane 0
 - Vertex (-327.50, 211.48, 1249.32)
	*** 1st Sanity check: Ratio 0.37 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3

# of OpFlashes (solarflash) in full geometry: 10677
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:20:31 UTC run: 5004 subRun: 0 event: 34030
[36m
# OpHits (ophit10ppm) in full geometry: 52395
# OpFlashes (solarflash) in full geometry: 10677
# Hits (gaushit) in each view: 6853, 6705, 6758, 0
# Cluster from the hits: 2862, 2857, 5770, 0
# Tracks (pmtracktc) in full geometry: 477[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:46:56 UTC run: 5004 subRun: 0 event: 34030
[32m
# ClusterReco: Looping over 5762 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:46:56 UTC run: 5004 subRun: 0 event: 34030
[32m*** Matched preselection cluster: 2440
 - MainTrackID 132455
 - Gen 1 -> marley TPC 44
 - Purity 1.00 Hits 4
 - Charge 1128.59 ( MaxHit 696.43 )
 - #AdjCl 11 ( 3 signal ):
    - Cluster 1861 at distance 87.36 with time 1445.96 and charge 98.35 in TPC 44 and hits 2
    - Cluster 1887 at distance 92.53 with time 1464.17 and charge 33.21 in TPC 36 and hits 1
    - Cluster 2291 at distance 56.74 with time 1756.72 and charge 50.73 in TPC 44 and hits 1
    - Cluster 2379 at distance 37.41 with time 1811.11 and charge 265.94 in TPC 44 and hits 3
    - Cluster 2449 at distance 27.88 with time 1868.01 and charge 359.78 in TPC 44 and hits 3
    - Cluster 2532 at distance 36.17 with time 1939.74 and charge 187.40 in TPC 44 and hits 2
    - Cluster 2624 at distance 51.82 with time 2008.53 and charge 64.03 in TPC 36 and hits 1
    - Cluster 2780 at distance 81.60 with time 2129.97 and charge 33.67 in TPC 38 and hits 1
    - Cluster 3087 at distance 86.81 with time 2328.37 and charge 21.69 in TPC 46 and hits 1
    - Cluster 3154 at distance 88.05 with time 2382.20 and charge 100.75 in TPC 44 and hits 2
    - Cluster 3211 at distance 95.38 with time 2419.57 and charge 39.97 in TPC 32 and hits 1
 - RecoCol  Time,Y,Z ( 1859.63, 142.90, 772.15 )
 - RecoInd0 Time,Y,Z ( 1859.47, 142.89, 821.56 )
 - RecoInd1 Time,Y,Z ( 1859.42, 142.91, 781.58 )
*** Matched flash: 
 - Purity 0.19 Plane 0 #Hits 39 PE 430.41 MaxPE 29.77
 - Time 0.01 Fast 0.71 Residual 184.63
 - Reco Time,Y,Z ( 0.01, 143.69, 744.40 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 27.12, 144.66, 772.27 )
 - Main X,Y,Z ( 26.40, 141.92, 772.47 )
 - EDep X,Y,Z ( 26.31, 142.39, 771.99 )
 - Reco X,Y,Z ( 26.45, 142.90, 772.15 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:46:56 UTC run: 5004 subRun: 0 event: 34030
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
13-Nov-2025 07:46:56 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f5/93/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250131T094818Z_gen_003403_supernova_g4stage1_g4stage2_detsim_reco_20250624T151634Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 13-Nov-2025 07:46:56 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.668613       1470.09       3853.55       840.286       1580.01        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000715388     7.22286       72.168      0.00576803      21.6484        10     
end_path:solarnuana:SolarNuAna     0.667188       1462.87       3853.55       804.378       1586.52        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 13-Nov-2025 07:46:56 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2531.88 MB
  Peak resident set size usage (VmHWM): 1913.25 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 14554.142350 Real = 14733.247040

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2531.88 VmHWM = 1913.25

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0