Jobsub ID 274810.1@dunegpschedd01.fnal.gov

Jobsub ID	274810.1@dunegpschedd01.fnal.gov
Workflow ID	11705
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-11 17:05:02
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2026-01-12 13:23:15
From worker node	Hostname	dunegli-8304644-0-fnpc9010.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-12 13:00:30
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T174602Z_gen_003284_supernova_g4_detsim_reco_20250724T110930Z_reco2.root
Jobscript	Exit code	0
	Real time	22m (1350s)
	CPU time	22m (1329s = 98%)
	Max RSS bytes	908337152 (866 MiB)
Outputting started	2026-01-12 13:23:00
Output files
Finished	2026-01-12 13:23:15
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

318.35, 1097.48)	33051	33008
marley	1000180360	0.00	(-335.56, 335.66, 1070.93)	33050	33008
marley	1000180360	0.00	(-337.78, 335.45, 1071.95)	33049	33008
marley	1000180360	0.00	(-338.54, 365.28, 1087.89)	33048	33008
marley	1000180400	0.00	(-349.99, 372.38, 1089.47)	33046	33008
marley	1000180360	0.00	(-347.10, 374.90, 1088.91)	33045	33008
marley	1000180400	0.00	(-342.32, 374.23, 1088.80)	33044	33008
marley	1000180400	0.00	(-358.30, 364.32, 1068.63)	33043	33008
marley	1000180400	0.00	(-368.13, 351.74, 1071.58)	33042	33008
marley	1000180360	0.00	(-375.41, 338.92, 1065.75)	33041	33008
marley	1000180360	0.00	(-366.20, 333.05, 1063.37)	33040	33008
marley	1000180360	0.00	(-374.44, 338.62, 1085.87)	33039	33008
marley	1000240520	0.00	(-378.94, 353.02, 1086.68)	33038	33008
marley	1000180360	0.00	(-367.60, 338.66, 1083.94)	33036	33008
marley	2212		0.00	(-363.57, 346.20, 1083.18)	33035	33008
marley	2212		0.03	(-363.40, 346.32, 1082.91)	33034	33008
marley	1000180400	0.00	(-312.26, 312.69, 1070.10)	33033	33008
marley	1000180360	0.00	(-330.08, 476.50, 1212.17)	33032	33008
marley	1000280600	0.00	(-379.05, 448.47, 1201.15)	33031	33008
marley	1000260560	0.00	(-379.34, 449.00, 1200.90)	33030	33008
marley	1000240520	0.00	(-378.78, 447.88, 1201.95)	33029	33008
marley	1000240500	0.00	(-207.00, 699.05, 1372.52)	33028	33008
marley	1000260560	0.00	(-207.30, 698.59, 1372.18)	33027	33008
marley	1000180360	0.00	(-66.80, 545.97, 1069.39)	33026	33008
marley	1000180400	0.01	(-40.24, 619.01, 974.84)	33025	33008
marley	1000180400	0.00	(-152.47, 437.12, 1073.62)	33024	33008
marley	1000180400	0.01	(-151.60, 437.00, 1069.78)	33023	33008
marley	1000180400	0.00	(-151.62, 437.24, 1069.95)	33022	33008
marley	1000180400	0.00	(-152.24, 438.10, 1068.45)	33021	33008
marley	1000180400	0.00	(-153.14, 439.81, 1070.07)	33020	33008
marley	1000180400	0.00	(-150.17, 442.43, 1068.28)	33019	33008
marley	1000180360	0.01	(-143.33, 439.41, 1071.33)	33018	33008
marley	1000180400	0.00	(-146.48, 423.43, 1062.66)	33017	33008
marley	1000180400	0.01	(-150.88, 429.49, 1048.57)	33016	33008
marley	1000180400	0.00	(-139.65, 421.43, 1065.92)	33015	33008
marley	1000180400	0.01	(-126.67, 423.61, 1069.00)	33014	33008
marley	1000180360	0.00	(-128.50, 417.91, 1067.30)	33013	33008
marley	1000180400	0.00	(-141.45, 373.99, 1062.33)	33012	33008
marley	1000180400	0.00	(-131.36, 367.09, 1063.90)	33011	33008
marley	1000180400	0.00	(-126.38, 354.07, 1067.04)	33010	33008
marley	1000180400	0.01	(-129.97, 347.21, 1067.80)	33009	33008
marley	2112		0.14	(-99.78, 348.58, 1050.05)	33008	0
marley	1000190390	0.02	(-99.78, 348.58, 1050.05)	33007	0
marley	11		6.41	(-99.78, 348.58, 1050.05)	33006	0
Signal Track IDs: 33006; 33007; 33008; 33009; 33010; 33011; 33012; 33013; 33014; 33015; 33016; 33017; 33018; 33019; 33020; 33021; 33022; 33023; 33024; 33025; 33026; 33027; 33028; 33029; 33030; 33031; 33032; 33033; 33034; 33035; 33036; 33038; 33039; 33040; 33041; 33042; 33043; 33044; 33045; 33046; 33048; 33049; 33050; 33051; 33052; 33053; 33054; 33055; 33056; 33058; 33061; 33062; 33063; 33086; 33087; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  12-Jan-2026 13:22:04 UTC run: 5092 subRun: 0 event: 164200
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  12-Jan-2026 13:22:25 UTC run: 5092 subRun: 0 event: 164200
[34mOpFlash PE 103.57 with purity 0.94 time -18.02 plane 0
 - Vertex (0.05, 341.65, 1063.14)
	*** 1st Sanity check: Ratio 0.11 <= 1.00
	*** 2nd Sanity check: #OpHits 24 >= 24
OpFlash PE 46.77 with purity 1.00 time -17.60 plane 0
 - Vertex (0.05, 365.90, 1111.49)
	*** 1st Sanity check: Ratio 0.26 <= 1.00
	*** 2nd Sanity check: #OpHits 13 >= 13
OpFlash PE 38.00 with purity 0.64 time -16.70 plane 0
 - Vertex (0.05, 311.70, 826.20)
	*** 1st Sanity check: Ratio 0.30 <= 1.00
	*** 2nd Sanity check: #OpHits 11 >= 11
OpFlash PE 40.44 with purity 0.29 time -15.86 plane 0
 - Vertex (0.05, 546.85, 632.32)
	*** 1st Sanity check: Ratio 0.21 <= 1.00
	*** 2nd Sanity check: #OpHits 12 >= 12
OpFlash PE 12.76 with purity 0.77 time -15.28 plane 0
 - Vertex (0.05, 305.06, 990.97)
	*** 1st Sanity check: Ratio 0.24 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 5.91 with purity 0.34 time -14.89 plane 0
 - Vertex (0.05, 528.67, 696.76)
	*** 1st Sanity check: Ratio 0.35 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 3.92 with purity 0.48 time -13.85 plane 0
 - Vertex (0.05, 410.48, 843.15)
	*** 1st Sanity check: Ratio 0.52 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 10.21 with purity 0.33 time -13.58 plane 0
 - Vertex (0.05, 241.37, 670.82)
	*** 1st Sanity check: Ratio 0.30 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 10.32 with purity 0.49 time -13.20 plane 0
 - Vertex (0.05, 433.72, 851.41)
	*** 1st Sanity check: Ratio 0.37 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 3.67 with purity 0.56 time -12.56 plane 0
 - Vertex (0.05, 370.11, 835.15)
	*** 1st Sanity check: Ratio 0.56 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 5.84 with purity 0.28 time -12.22 plane 0
 - Vertex (-0.02, -105.03, 1068.27)
	*** 1st Sanity check: Ratio 0.37 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 2.16 with purity 1.00 time -11.82 plane 0
 - Vertex (0.05, 342.49, 1013.48)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 4.06 with purity 1.00 time -10.83 plane 0
 - Vertex (0.05, 369.48, 1099.60)
	*** 1st Sanity check: Ratio 0.52 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2

# of OpHits (ophitspe): 71650 for times between -2261.79 us and 2227.81 us.
# of OpFlashes (solarflash): 9615 for times between -2261.79 us and 2227.81 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  12-Jan-2026 13:22:25 UTC run: 5092 subRun: 0 event: 164200
[36m
# OpHits (ophitspe) in full geometry: 71650
# OpFlashes (solarflash) in full geometry: 9615
# Hits (hitfd) in each view: 193, 179, 435, 0
# Cluster from the hits: 122, 110, 399, 0
# Tracks (pmtracktc) in full geometry: 1[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  12-Jan-2026 13:23:00 UTC run: 5092 subRun: 0 event: 164200
[32m
# ClusterReco: Looping over 229 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  12-Jan-2026 13:23:00 UTC run: 5092 subRun: 0 event: 164200
[32m*** Matched preselection cluster: 48 from 229
 - MainTrackID 33006
 - Gen 1 -> marley TPC 18
 - Purity 1.00 Hits 4
 - Charge 599.02 ( MaxHit 194.43 )
 - #AdjCl 1 ( 1 signal ):
    - Cluster 44 at distance 11.91 with time 566.52 and charge 116.74 in TPC 18 and hits 1
 - RecoCol  Time,Y,Z ( 602.42, 348.57, 1051.18 )
 - RecoInd0 Time,Y,Z ( 602.34, 348.39, 1044.88 )
 - RecoInd1 Time,Y,Z ( 602.40, 348.75, 1044.88 )
*** Matched flash: 4889 from 9615
 - Purity 93.85 % Plane 0 #Hits 24
 - MainOpHitPE 11.11 (PE);  TotalPE 103.57 vs expected 216.89 (PE)
 - Time -18.02 (us) Fast 36.82 % Residual 28.59
 - Reco Time,Y,Z ( -18.02, 341.65, 1063.14 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -99.78, 348.58, 1050.05 )
 - Main X,Y,Z ( -99.87, 348.62, 1051.76 )
 - EDep X,Y,Z ( -99.87, 348.61, 1051.73 )
 - Reco X,Y,Z ( -99.62, 348.57, 1051.18 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  12-Jan-2026 13:23:00 UTC run: 5092 subRun: 0 event: 164200
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
12-Jan-2026 13:23:00 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/dd/31/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T174602Z_gen_003284_supernova_g4_detsim_reco_20250724T110930Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 12-Jan-2026 13:23:00 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.401489       24.9682       66.0578       1.25066       27.6432        50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00226667     0.0120969     0.0538045     0.0119645     0.0116183       50     
end_path:solarnuana:SolarNuAna     0.392403       24.9558       66.0432       1.23657       27.6454        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 12-Jan-2026 13:23:00 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1552.33 MB
  Peak resident set size usage (VmHWM): 906.916 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 1286.521825 Real = 1302.883512

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1552.33 VmHWM = 906.916

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===