Jobsub ID 301988.1@dunegpschedd01.fnal.gov

Jobsub ID	301988.1@dunegpschedd01.fnal.gov
Workflow ID	12420
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-28 15:54:50
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2026-01-28 20:13:40
From worker node	Hostname	dunegli-8464114-0-fnpc22004.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-28 15:56:19
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T132217Z_gen_001344_supernova_g4stage1_g4stage2_detsim_reco_20250626T131143Z_reco2.root
Jobscript	Exit code	0
	Real time	4h (15433s)
	CPU time	4h (15321s = 99%)
	Max RSS bytes	2573950976 (2454 MiB)
Outputting started	2026-01-28 20:13:32
Output files
Finished	2026-01-28 20:13:40
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

-591.89, 747.53)	132553	132526
marley	1000180360	0.00	(-310.73, -594.05, 752.95)	132552	132526
marley	1000180360	0.00	(-316.15, -596.19, 754.51)	132551	132526
marley	1000180400	0.00	(-329.42, -568.70, 772.26)	132550	132526
marley	1000180360	0.00	(-318.79, -580.36, 815.90)	132549	132526
marley	1000180400	0.00	(-309.44, -584.95, 807.94)	132548	132526
marley	1000180360	0.00	(-308.65, -589.67, 808.64)	132547	132526
marley	1000180400	0.00	(-322.66, -582.66, 815.44)	132546	132526
marley	1000180360	0.00	(-327.76, -585.40, 826.95)	132545	132526
marley	1000180360	0.00	(-330.93, -574.62, 833.36)	132544	132526
marley	2212		0.04	(-327.97, -585.51, 831.46)	132543	132526
marley	1000060120	0.01	(-328.16, -585.60, 831.44)	132542	132526
marley	1000080160	0.01	(-328.05, -585.59, 831.54)	132541	132526
marley	1000180360	0.00	(-230.73, -266.22, 531.47)	132540	132526
marley	1000180400	0.00	(-228.00, -267.17, 529.42)	132539	132526
marley	1000180400	0.00	(-226.92, -301.52, 559.74)	132538	132526
marley	1000180400	0.00	(-228.46, -301.84, 558.55)	132537	132526
marley	1000180400	0.01	(-228.73, -302.65, 558.50)	132536	132526
marley	1000180400	0.01	(-228.88, -302.07, 558.63)	132535	132526
marley	1000180360	0.01	(-245.86, -318.65, 552.92)	132534	132526
marley	1000180400	0.01	(-243.07, -309.23, 556.93)	132533	132526
marley	1000180400	0.00	(-259.54, -356.61, 557.88)	132532	132526
marley	1000180400	0.00	(-276.06, -358.94, 561.55)	132531	132526
marley	1000180400	0.01	(-277.91, -360.78, 560.67)	132530	132526
marley	1000180360	0.00	(-297.85, -300.00, 573.32)	132529	132526
marley	1000180400	0.01	(-260.74, -242.99, 565.97)	132528	132526
marley	1000180400	0.01	(-292.69, -290.49, 528.43)	132527	132526
marley	2112		0.15	(-302.31, -283.83, 525.55)	132526	0
marley	1000190390	0.01	(-302.31, -283.83, 525.55)	132525	0
marley	11		5.54	(-302.31, -283.83, 525.55)	132524	0
Signal Track IDs: 132524; 132525; 132526; 132527; 132528; 132529; 132530; 132531; 132532; 132533; 132534; 132535; 132536; 132537; 132538; 132539; 132540; 132541; 132542; 132543; 132544; 132545; 132546; 132547; 132548; 132549; 132550; 132551; 132552; 132553; 132554; 132555; 132556; 132557; 132558; 132559; 132575; 132576; 132577; 132578; 132579; 132580; 132581; 132582; 132583; 132584; 132585; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:43:44 UTC run: 4966 subRun: 0 event: 13440
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:44:28 UTC run: 4966 subRun: 0 event: 13440
[34mOpFlash PE 23.41 with purity 0.32 time -0.47 plane 2
 - Vertex (132.89, -743.30, 152.42)
	*** 1st Sanity check: Ratio 0.35 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 981.96 with purity 0.68 time 0.02 plane 0
 - Vertex (-327.50, -297.40, 484.26)
	*** 1st Sanity check: Ratio 0.28 <= 1.00
	*** 2nd Sanity check: #OpHits 10 >= 10
OpFlash PE 38.67 with purity 0.21 time 0.60 plane 0
 - Vertex (-327.50, 134.79, 818.96)
	*** 1st Sanity check: Ratio 0.19 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 26.31 with purity 1.00 time 0.63 plane 1
 - Vertex (135.07, 743.30, 152.42)
	*** 1st Sanity check: Ratio 0.39 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 27.21 with purity 0.19 time 4.95 plane 0
 - Vertex (-327.50, 285.10, 993.96)
	*** 1st Sanity check: Ratio 0.38 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 4159.38 with purity 0.07 time 8.27 plane 0
 - Vertex (-327.50, -468.10, 1448.78)
	*** 1st Sanity check: Ratio 0.47 <= 1.00
	*** 2nd Sanity check: #OpHits 16 >= 16
OpFlash PE 1134.16 with purity 0.04 time 582.82 plane 0
 - Vertex (-327.50, -468.10, 334.95)
	*** 1st Sanity check: Ratio 0.30 <= 1.00
	*** 2nd Sanity check: #OpHits 20 >= 20
OpFlash PE 1174.22 with purity 0.46 time 592.04 plane 0
 - Vertex (-327.50, -535.20, 856.08)
	*** 1st Sanity check: Ratio 0.13 <= 1.00
	*** 2nd Sanity check: #OpHits 26 >= 26
OpFlash PE 18.59 with purity 0.08 time 592.47 plane 2
 - Vertex (79.99, -743.30, 690.36)
	*** 1st Sanity check: Ratio 0.50 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 6.50 with purity 0.33 time 592.38 plane 1
 - Vertex (285.07, 743.30, 152.42)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 45.12 with purity 0.26 time 593.81 plane 0
 - Vertex (-327.50, 260.56, 498.86)
	*** 1st Sanity check: Ratio 0.22 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 38.10 with purity 0.15 time 593.92 plane 0
 - Vertex (-327.50, 389.05, 816.42)
	*** 1st Sanity check: Ratio 0.29 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 14.18 with purity 0.22 time 600.80 plane 0
 - Vertex (-327.50, -377.18, 831.60)
	*** 1st Sanity check: Ratio 0.67 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 21.50 with purity 0.14 time 602.28 plane 0
 - Vertex (-327.50, -378.50, 600.50)
	*** 1st Sanity check: Ratio 0.27 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4

# of OpHits (ophit10ppm): 53627 for times between -4254.44 us and 295147919198126080.00 us.
# of OpFlashes (solarflash): 18286 for times between -4254.44 us and 295147919198126080.00 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:44:30 UTC run: 4966 subRun: 0 event: 13440
[36m
# OpHits (ophit10ppm) in full geometry: 53627
# OpFlashes (solarflash) in full geometry: 18286
# Hits (gaushit) in each view: 6553, 6617, 6545, 0
# Cluster from the hits: 2734, 2803, 5585, 0
# Tracks (pmtracktc) in full geometry: 457[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 20:13:32 UTC run: 4966 subRun: 0 event: 13440
[32m
# ClusterReco: Looping over 5578 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 20:13:32 UTC run: 4966 subRun: 0 event: 13440
[32m*** Matched preselection cluster: 5132 from 5578
 - MainTrackID 106577
 - Gen 1 -> marley TPC 26
 - Purity 0.91 Hits 5
 - Charge 584.07 ( MaxHit 306.95 )
 - #AdjCl 7 ( 0 signal ):
    - Cluster 4543 at distance 98.38 with time 3452.33 and charge 279.62 in TPC 30 and hits 2
    - Cluster 4704 at distance 69.41 with time 3576.63 and charge 41.26 in TPC 30 and hits 2
    - Cluster 4824 at distance 94.60 with time 3675.22 and charge 35.01 in TPC 24 and hits 1
    - Cluster 4853 at distance 61.65 with time 3698.45 and charge 29.56 in TPC 30 and hits 1
    - Cluster 5007 at distance 92.73 with time 3808.14 and charge 33.49 in TPC 28 and hits 1
    - Cluster 5059 at distance 14.95 with time 3854.70 and charge 46.28 in TPC 30 and hits 1
    - Cluster 5307 at distance 68.99 with time 4031.76 and charge 40.49 in TPC 30 and hits 1
 - RecoCol  Time,Y,Z ( 3905.45, 357.26, 525.21 )
 - RecoInd0 Time,Y,Z ( 3905.16, 779.20, 1243.64 )
 - RecoInd1 Time,Y,Z ( 3905.00, -64.68, 1282.72 )
*** Matched flash: 10323 from 18286
 - Purity 0.00 % Plane 0 #Hits 18
 - MainOpHitPE 38.50 (PE);  TotalPE 249.38 vs expected 266.54 (PE)
 - Time 557.25 (us) Fast 60.50 % Residual 144.17
 - Reco Time,Y,Z ( 557.25, 402.87, 541.78 )
 - Found valid waveform: true
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 0.00, 0.00, 0.00 )
 - Main X,Y,Z ( 0.00, 0.00, 0.00 )
 - EDep X,Y,Z ( 0.00, 0.00, 0.00 )
 - Reco X,Y,Z ( -212.56, 357.26, 525.21 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 20:13:32 UTC run: 4966 subRun: 0 event: 13440
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
28-Jan-2026 20:13:32 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/d9/9f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T132217Z_gen_001344_supernova_g4stage1_g4stage2_detsim_reco_20250626T131143Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 28-Jan-2026 20:13:32 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.558317       1538.47       2682.15       1871.96       1052.63        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000544014   0.00966806     0.0218783    0.00764333    0.00768814       10     
end_path:solarnuana:SolarNuAna     0.557652       1538.46       2682.13       1871.95       1052.64        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 28-Jan-2026 20:13:32 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3193.2 MB
  Peak resident set size usage (VmHWM): 2573.95 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 15309.374014 Real = 15409.624503

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3193.2 VmHWM = 2573.95

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===