Jobsub ID 288544.35@dunegpschedd02.fnal.gov

Jobsub ID	288544.35@dunegpschedd02.fnal.gov
Workflow ID	12007
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-20 14:48:38
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2026-01-20 22:03:44
From worker node	Hostname	dunegli-50386-0-cmswn2355.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-20 14:57:12
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T145847Z_gen_000051_supernova_g4stage1_g4stage2_detsim_reco_20250702T143527Z_reco2.root
Jobscript	Exit code	0
	Real time	7h (25574s)
	CPU time	7h (25398s = 99%)
	Max RSS bytes	2571743232 (2452 MiB)
Outputting started	2026-01-20 22:03:27
Output files
Finished	2026-01-20 22:03:44
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

rom gen 9 Rn222ChainBi214GenInLAr
# of particles 24	from gen 10 Rn222ChainPb210GenInLAr
# of particles 14	from gen 11 Rn220ChainPb212GenInLAr
# of particles 228	from gen 12 K40GenInCathode
# of particles 2900	from gen 13 U238ChainGenInCathode
# of particles 3482	from gen 14 Th232ChainGenInCathode
# of particles 120	from gen 15 K40GenInAnode
# of particles 1377	from gen 16 U238ChainGenInAnode
# of particles 1646	from gen 17 Th232ChainGenInAnode
# of particles 5	from gen 18 Rn222ChainGenInPDS
# of particles 2	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 0	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 24	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 75	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 34	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 41	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 58	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 18	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 20:46:03 UTC run: 4799 subRun: 0 event: 510
[33m
Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 14.10 MeV
Position (289.12, -208.39, 1517.27) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.65	(289.12, -208.39, 1517.27)	132902	132869
marley	-12		0.66	(289.12, -208.39, 1517.27)	132901	132869
marley	1000200400	0.00	(289.12, -208.39, 1517.27)	132900	132869
marley	22		0.03	(289.12, -208.39, 1517.27)	132872	0
marley	22		1.61	(289.12, -208.39, 1517.27)	132871	0
marley	22		1.09	(289.12, -208.39, 1517.27)	132870	0
marley	1000190400	0.00	(289.12, -208.39, 1517.27)	132869	0
marley	11		9.86	(289.12, -208.39, 1517.27)	132868	0
Signal Track IDs: 132868; 132869; 132870; 132871; 132872; 132900; 132901; 132902; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 20:46:03 UTC run: 4799 subRun: 0 event: 510
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 20:49:23 UTC run: 4799 subRun: 0 event: 510
[34mOpFlash PE 7.51 with purity 0.50 time 0.44 plane 3
 - Vertex (60.07, -220.00, 2188.38)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 13.35 with purity 0.50 time 1.16 plane 0
 - Vertex (-327.50, -168.82, 1935.70)
	*** 1st Sanity check: Ratio 0.50 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 16.04 with purity 0.63 time 1.42 plane 2
 - Vertex (96.75, -743.30, 1568.78)
	*** 1st Sanity check: Ratio 0.42 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3

# of OpHits (ophit10ppm): 53188 for times between -4254.46 us and 295147919198126080.00 us.
# of OpFlashes (solarflash): 18313 for times between -4254.28 us and 295147919198126080.00 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 20:49:27 UTC run: 4799 subRun: 0 event: 510
[36m
# OpHits (ophit10ppm) in full geometry: 53188
# OpFlashes (solarflash) in full geometry: 18313
# Hits (gaushit) in each view: 6710, 6600, 6758, 0
# Cluster from the hits: 2778, 2810, 5737, 0
# Tracks (pmtracktc) in full geometry: 477[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:03:25 UTC run: 4799 subRun: 0 event: 510
[32m
# ClusterReco: Looping over 5733 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:03:25 UTC run: 4799 subRun: 0 event: 510
[32m*** Matched preselection cluster: 216 from 5733
 - MainTrackID 132871
 - Gen 1 -> marley TPC 82
 - Purity 1.00 Hits 1
 - Charge 176.41 ( MaxHit 176.41 )
 - #AdjCl 3 ( 0 signal ):
    - Cluster 94 at distance 47.08 with time 133.71 and charge 60.47 in TPC 78 and hits 2
    - Cluster 392 at distance 40.90 with time 342.99 and charge 85.25 in TPC 86 and hits 1
    - Cluster 669 at distance 78.70 with time 549.80 and charge 23.19 in TPC 86 and hits 1
 - RecoCol  Time,Y,Z ( 213.86, 384.32, 1516.89 )
 - RecoInd0 Time,Y,Z ( 213.16, 522.74, 1718.08 )
 - RecoInd1 Time,Y,Z ( 213.11, 245.90, 1733.47 )
*** Matched flash: 6743 from 18313
 - Purity 0.00 % Plane 0 #Hits 6
 - MainOpHitPE 32.86 (PE);  TotalPE 96.32 vs expected 17.91 (PE)
 - Time -1096.25 (us) Fast 48.21 % Residual 1167.70
 - Reco Time,Y,Z ( -1096.25, 386.71, 1592.28 )
 - Found valid waveform: true
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 289.12, -208.39, 1517.27 )
 - Main X,Y,Z ( 272.84, -198.76, 1513.53 )
 - EDep X,Y,Z ( 290.75, -208.24, 1517.03 )
 - Reco X,Y,Z ( 114.68, 384.32, 1516.89 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:03:25 UTC run: 4799 subRun: 0 event: 510
[32m*** Matched preselection cluster: 235 from 5733
 - MainTrackID 132868
 - Gen 1 -> marley TPC 82
 - Purity 1.00 Hits 3
 - Charge 999.99 ( MaxHit 447.65 )
 - #AdjCl 7 ( 2 signal ):
    - Cluster 55 at distance 96.58 with time 107.39 and charge 32.09 in TPC 82 and hits 1
    - Cluster 85 at distance 75.53 with time 126.83 and charge 77.02 in TPC 82 and hits 1
    - Cluster 137 at distance 22.98 with time 165.98 and charge 81.84 in TPC 82 and hits 1
    - Cluster 266 at distance 3.80 with time 253.95 and charge 50.64 in TPC 82 and hits 1
    - Cluster 431 at distance 77.07 with time 365.30 and charge 28.12 in TPC 84 and hits 1
    - Cluster 667 at distance 73.94 with time 548.41 and charge 32.57 in TPC 78 and hits 1
    - Cluster 701 at distance 83.98 with time 568.88 and charge 64.67 in TPC 82 and hits 1
 - RecoCol  Time,Y,Z ( 230.81, -208.02, 1517.62 )
 - RecoInd0 Time,Y,Z ( 228.34, -208.04, 1569.16 )
 - RecoInd1 Time,Y,Z ( 231.46, -208.00, 1540.69 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 289.12, -208.39, 1517.27 )
 - Main X,Y,Z ( 287.67, -207.18, 1517.54 )
 - EDep X,Y,Z ( 287.62, -207.63, 1517.79 )
 - Reco X,Y,Z ( -1000000.00, -208.02, 1517.62 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:03:25 UTC run: 4799 subRun: 0 event: 510
[32m*** Matched preselection cluster: 296 from 5733
 - MainTrackID 132868
 - Gen 1 -> marley TPC 82
 - Purity 1.00 Hits 1
 - Charge 86.56 ( MaxHit 86.56 )
 - #AdjCl 1 ( 0 signal ):
    - Cluster 808 at distance 99.66 with time 664.91 and charge 30.89 in TPC 82 and hits 1
 - RecoCol  Time,Y,Z ( 271.76, -451.09, 1517.90 )
 - RecoInd0 Time,Y,Z ( 273.26, -694.16, 382.30 )
 - RecoInd1 Time,Y,Z ( 271.66, -208.03, 1540.86 )
*** Matched flash: 5052 from 18313
 - Purity 0.00 % Plane 0 #Hits 7
 - MainOpHitPE 63.82 (PE);  TotalPE 204.85 vs expected 33.19 (PE)
 - Time -1894.38 (us) Fast 41.65 % Residual 317.64
 - Reco Time,Y,Z ( -1894.38, -508.64, 1493.06 )
 - Found valid waveform: true
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 289.12, -208.39, 1517.27 )
 - Main X,Y,Z ( 287.67, -207.18, 1517.54 )
 - EDep X,Y,Z ( 287.62, -207.63, 1517.79 )
 - Reco X,Y,Z ( -22.77, -451.09, 1517.90 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:03:25 UTC run: 4799 subRun: 0 event: 510
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
20-Jan-2026 22:03:26 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/5c/5a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T145847Z_gen_000051_supernova_g4stage1_g4stage2_detsim_reco_20250702T143527Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 22:03:26 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          1.55818       2550.1        5630.72       2323.59       2560.43        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000655625   0.00909653     0.0158007    0.00870398     0.0056408       10     
end_path:solarnuana:SolarNuAna      1.55735       2550.09       5630.71       2323.58       2560.42        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 22:03:26 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3196.04 MB
  Peak resident set size usage (VmHWM): 2571.74 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 25380.267525 Real = 25553.278648

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3196.04 VmHWM = 2571.74

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===