Jobsub ID 241909.3@dunegpschedd02.fnal.gov

Jobsub ID	241909.3@dunegpschedd02.fnal.gov
Workflow ID	9965
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-12 04:06:43
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2025-11-13 08:04:33
From worker node	Hostname	dunegli-7572316-0-fnpc23039.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-11-13 03:44:48
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T132217Z_gen_001344_supernova_g4stage1_g4stage2_detsim_reco_20250626T131143Z_reco2.root
Jobscript	Exit code	0
	Real time	4h (15567s)
	CPU time	4h (15425s = 99%)
	Max RSS bytes	1911889920 (1823 MiB)
Outputting started	2025-11-13 08:04:16
Output files
Finished	2025-11-13 08:04:33
List job events (HTCondor job logs unavailable)
Jobscript log (last 10,000 characters)

(-268.23, -539.69, 839.60)	132583	132580
marley	-12		0.44	(-268.23, -539.69, 839.60)	132582	132580
marley	1000190419	0.00	(-268.23, -539.69, 839.60)	132581	132580
marley	1000180410	0.00	(-268.23, -539.69, 839.60)	132580	132526
marley	22		3.28	(-268.23, -539.69, 839.60)	132579	132526
marley	22		4.10	(-268.23, -539.69, 839.60)	132578	132526
marley	22		1.49	(-268.23, -539.69, 839.60)	132577	132526
marley	22		3.62	(-268.23, -539.69, 839.60)	132576	132526
marley	22		1.49	(-268.23, -539.69, 839.60)	132575	132526
marley	2212		0.00	(-326.27, -623.09, 744.76)	132559	132526
marley	2212		0.00	(-326.27, -623.12, 744.74)	132558	132526
marley	2212		0.00	(-326.16, -623.43, 744.97)	132557	132526
marley	1000180360	0.00	(-323.66, -625.91, 754.80)	132556	132526
marley	1000180360	0.00	(-327.73, -612.43, 752.61)	132555	132526
marley	1000180360	0.00	(-315.34, -594.99, 747.48)	132554	132526
marley	1000180360	0.00	(-317.00, -591.89, 747.53)	132553	132526
marley	1000180360	0.00	(-310.73, -594.05, 752.95)	132552	132526
marley	1000180360	0.00	(-316.15, -596.19, 754.51)	132551	132526
marley	1000180400	0.00	(-329.42, -568.70, 772.26)	132550	132526
marley	1000180360	0.00	(-318.79, -580.36, 815.90)	132549	132526
marley	1000180400	0.00	(-309.44, -584.95, 807.94)	132548	132526
marley	1000180360	0.00	(-308.65, -589.67, 808.64)	132547	132526
marley	1000180400	0.00	(-322.66, -582.66, 815.44)	132546	132526
marley	1000180360	0.00	(-327.76, -585.40, 826.95)	132545	132526
marley	1000180360	0.00	(-330.93, -574.62, 833.36)	132544	132526
marley	2212		0.04	(-327.97, -585.51, 831.46)	132543	132526
marley	1000060120	0.01	(-328.16, -585.60, 831.44)	132542	132526
marley	1000080160	0.01	(-328.05, -585.59, 831.54)	132541	132526
marley	1000180360	0.00	(-230.73, -266.22, 531.47)	132540	132526
marley	1000180400	0.00	(-228.00, -267.17, 529.42)	132539	132526
marley	1000180400	0.00	(-226.92, -301.52, 559.74)	132538	132526
marley	1000180400	0.00	(-228.46, -301.84, 558.55)	132537	132526
marley	1000180400	0.01	(-228.73, -302.65, 558.50)	132536	132526
marley	1000180400	0.01	(-228.88, -302.07, 558.63)	132535	132526
marley	1000180360	0.01	(-245.86, -318.65, 552.92)	132534	132526
marley	1000180400	0.01	(-243.07, -309.23, 556.93)	132533	132526
marley	1000180400	0.00	(-259.54, -356.61, 557.88)	132532	132526
marley	1000180400	0.00	(-276.06, -358.94, 561.55)	132531	132526
marley	1000180400	0.01	(-277.91, -360.78, 560.67)	132530	132526
marley	1000180360	0.00	(-297.85, -300.00, 573.32)	132529	132526
marley	1000180400	0.01	(-260.74, -242.99, 565.97)	132528	132526
marley	1000180400	0.01	(-292.69, -290.49, 528.43)	132527	132526
marley	2112		0.15	(-302.31, -283.83, 525.55)	132526	0
marley	1000190390	0.01	(-302.31, -283.83, 525.55)	132525	0
marley	11		5.54	(-302.31, -283.83, 525.55)	132524	0
Signal Track IDs: 132524; 132525; 132526; 132527; 132528; 132529; 132530; 132531; 132532; 132533; 132534; 132535; 132536; 132537; 132538; 132539; 132540; 132541; 132542; 132543; 132544; 132545; 132546; 132547; 132548; 132549; 132550; 132551; 132552; 132553; 132554; 132555; 132556; 132557; 132558; 132559; 132575; 132576; 132577; 132578; 132579; 132580; 132581; 132582; 132583; 132584; 132585; [0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:39:13 UTC run: 4966 subRun: 0 event: 13440
[33m
Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:39:36 UTC run: 4966 subRun: 0 event: 13440
[34mOpFlash PE 148.09 with purity 0.18 time -0.01 plane 1
 - Vertex (210.07, 743.30, 450.26)
	*** 1st Sanity check: Ratio 0.42 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 981.96 with purity 0.68 time 0.00 plane 0
 - Vertex (-327.50, -297.40, 484.26)
	*** 1st Sanity check: Ratio 0.28 <= 1.00
	*** 2nd Sanity check: #OpHits 10 >= 10
OpFlash PE 957.56 with purity 0.01 time 0.04 plane 1
 - Vertex (285.07, 743.30, 450.26)
	*** 1st Sanity check: Ratio 0.25 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 103.79 with purity 0.05 time 0.05 plane 0
 - Vertex (-327.50, 293.99, 1322.58)
	*** 1st Sanity check: Ratio 0.31 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 4194.35 with purity 0.07 time 0.13 plane 0
 - Vertex (-327.50, -468.10, 1448.78)
	*** 1st Sanity check: Ratio 0.46 <= 1.00
	*** 2nd Sanity check: #OpHits 20 >= 20
OpFlash PE 1322.78 with purity 0.04 time 9.33 plane 0
 - Vertex (-327.50, -468.10, 334.95)
	*** 1st Sanity check: Ratio 0.26 <= 1.00
	*** 2nd Sanity check: #OpHits 30 >= 30
OpFlash PE 36.36 with purity 0.33 time 9.45 plane 0
 - Vertex (-327.50, 350.52, 1033.24)
	*** 1st Sanity check: Ratio 0.35 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 1227.04 with purity 0.47 time 9.47 plane 0
 - Vertex (-327.50, -535.20, 856.08)
	*** 1st Sanity check: Ratio 0.12 <= 1.00
	*** 2nd Sanity check: #OpHits 34 >= 34
OpFlash PE 6.50 with purity 0.33 time 9.48 plane 1
 - Vertex (285.07, 743.30, 152.42)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 83.17 with purity 0.21 time 9.52 plane 0
 - Vertex (-327.50, 383.56, 853.60)
	*** 1st Sanity check: Ratio 0.13 <= 1.00
	*** 2nd Sanity check: #OpHits 13 >= 13

# of OpFlashes (solarflash) in full geometry: 10648
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 07:39:38 UTC run: 4966 subRun: 0 event: 13440
[36m
# OpHits (ophit10ppm) in full geometry: 53627
# OpFlashes (solarflash) in full geometry: 10648
# Hits (gaushit) in each view: 6553, 6617, 6545, 0
# Cluster from the hits: 2734, 2803, 5585, 0
# Tracks (pmtracktc) in full geometry: 457[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 08:04:15 UTC run: 4966 subRun: 0 event: 13440
[32m
# ClusterReco: Looping over 5578 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 08:04:15 UTC run: 4966 subRun: 0 event: 13440
[32m*** Matched preselection cluster: 5132
 - MainTrackID 106577
 - Gen 1 -> marley TPC 26
 - Purity 0.91 Hits 5
 - Charge 584.07 ( MaxHit 306.95 )
 - #AdjCl 7 ( 0 signal ):
    - Cluster 4543 at distance 98.38 with time 3452.33 and charge 279.62 in TPC 30 and hits 2
    - Cluster 4704 at distance 69.41 with time 3576.63 and charge 41.26 in TPC 30 and hits 2
    - Cluster 4824 at distance 94.60 with time 3675.22 and charge 35.01 in TPC 24 and hits 1
    - Cluster 4853 at distance 61.65 with time 3698.45 and charge 29.56 in TPC 30 and hits 1
    - Cluster 5007 at distance 92.73 with time 3808.14 and charge 33.49 in TPC 28 and hits 1
    - Cluster 5059 at distance 14.95 with time 3854.70 and charge 46.28 in TPC 30 and hits 1
    - Cluster 5307 at distance 68.99 with time 4031.76 and charge 40.49 in TPC 30 and hits 1
 - RecoCol  Time,Y,Z ( 3905.45, 357.26, 525.21 )
 - RecoInd0 Time,Y,Z ( 3905.16, 779.20, 1243.64 )
 - RecoInd1 Time,Y,Z ( 3905.00, -64.68, 1282.72 )
*** Matched flash: 
 - Purity 0.00 Plane 0 #Hits 20 PE 263.27 MaxPE 38.50
 - Time 8.92 Fast 0.64 Residual 290.02
 - Reco Time,Y,Z ( 8.92, 402.87, 541.78 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 0.00, 0.00, 0.00 )
 - Main X,Y,Z ( 0.00, 0.00, 0.00 )
 - EDep X,Y,Z ( 0.00, 0.00, 0.00 )
 - Reco X,Y,Z ( -300.60, 357.26, 525.21 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 08:04:15 UTC run: 4966 subRun: 0 event: 13440
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
13-Nov-2025 08:04:15 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/d9/9f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T132217Z_gen_001344_supernova_g4stage1_g4stage2_detsim_reco_20250626T131143Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 13-Nov-2025 08:04:15 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.914459       1551.57       2679.16        1812         1090.26        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000686784   0.00980276     0.0230834    0.00776901     0.0091459       10     
end_path:solarnuana:SolarNuAna     0.898423       1551.56       2679.16       1811.98       1090.26        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 13-Nov-2025 08:04:15 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2536.42 MB
  Peak resident set size usage (VmHWM): 1911.89 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 15409.380871 Real = 15547.885131

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2536.42 VmHWM = 1911.89

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0