Jobsub ID 255024.2@dunegpschedd02.fnal.gov

Jobsub ID	255024.2@dunegpschedd02.fnal.gov
Workflow ID	10891
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-29 04:38:43
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-11-29 04:48:51
From worker node	Hostname	dunegli-7929758-0-fnpc9053.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-29 04:39:50
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T201832Z_gen_004846_supernova_g4_halfActiveVol_detsim_reco_20250724T160921Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (531s)
	CPU time	8m (519s = 97%)
	Max RSS bytes	850956288 (811 MiB)
Outputting started	2025-11-29 04:48:41
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/smanthey/ana/solar_ana_marley_flash_radiological_decay0_dune10kt_1x2x6_lateralAPA/fnal/10891/1/001/fdhd_ana_2025-11-29T_044000Z_hist.root
Finished	2025-11-29 04:48:51
Saved logs	justin-logs:255024.2-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nt: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[34mOpFlash PE 1415.69 with purity 0.99 time -18.03 plane 0
 - Vertex (-0.05, -280.22, 1197.07)
	*** 1st Sanity check: Ratio 0.62 <= 1.00
	*** 2nd Sanity check: #OpHits 59 >= 59
OpFlash PE 6.16 with purity 0.34 time -17.58 plane 0
 - Vertex (0.05, 317.72, 763.91)
	*** 1st Sanity check: Ratio 0.34 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 54.14 with purity 0.98 time -16.42 plane 0
 - Vertex (-0.05, -242.19, 1204.93)
	*** 1st Sanity check: Ratio 0.18 <= 1.00
	*** 2nd Sanity check: #OpHits 16 >= 16
OpFlash PE 34.97 with purity 0.91 time -15.08 plane 0
 - Vertex (-0.04, -207.22, 1199.09)
	*** 1st Sanity check: Ratio 0.14 <= 1.00
	*** 2nd Sanity check: #OpHits 13 >= 13
OpFlash PE 14.24 with purity 0.12 time -14.16 plane 0
 - Vertex (0.05, 217.95, 892.69)
	*** 1st Sanity check: Ratio 0.76 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 6.65 with purity 1.00 time -13.13 plane 0
 - Vertex (-0.05, -358.26, 1265.69)
	*** 1st Sanity check: Ratio 0.37 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 6.61 with purity 0.67 time -12.01 plane 0
 - Vertex (-0.02, -69.68, 1250.73)
	*** 1st Sanity check: Ratio 0.41 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 2.04 with purity 1.00 time -11.06 plane 0
 - Vertex (-0.05, -342.49, 1013.48)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1
OpFlash PE 8.67 with purity 0.79 time -9.09 plane 0
 - Vertex (-0.05, -263.90, 1088.09)
	*** 1st Sanity check: Ratio 0.60 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 161.19 with purity 0.14 time -6.04 plane 0
 - Vertex (0.05, 155.68, 1197.07)
	*** 1st Sanity check: Ratio 0.82 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 100.56 with purity 0.71 time 0.50 plane 0
 - Vertex (-0.05, -217.95, 1197.07)
	*** 1st Sanity check: Ratio 0.94 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3

# of OpHits (ophitspe): 40034 for times between -2259.41 us and 2227.26 us.
# of OpFlashes (solarflash): 3692 for times between -2259.41 us and 2227.18 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:26 UTC run: 5100 subRun: 0 event: 242300
[36m
# OpHits (ophitspe) in full geometry: 40034
# OpFlashes (solarflash) in full geometry: 3692
# Hits (hitfd) in each view: 139, 162, 300, 0
# Cluster from the hits: 87, 100, 267, 0
# Tracks (pmtracktc) in full geometry: 2[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:41 UTC run: 5100 subRun: 0 event: 242300
[32m
# ClusterReco: Looping over 154 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:41 UTC run: 5100 subRun: 0 event: 242300
[32m*** Matched preselection cluster: 12 from 154
 - MainTrackID 32901
 - Gen 1 -> marley TPC 21
 - Purity 1.00 Hits 1
 - Charge 49.91 ( MaxHit 49.91 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 116.74, 1165.07, 1211.41 )
 - RecoInd0 Time,Y,Z ( 118.24, 1165.07, 115.32 )
 - RecoInd1 Time,Y,Z ( -1000000.00, -1000000.00, -1000000.00 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 23.68, -237.34, 1213.81 )
 - Main X,Y,Z ( 34.06, -207.42, 1220.34 )
 - EDep X,Y,Z ( 21.77, -237.91, 1211.34 )
 - Reco X,Y,Z ( -1000000.00, 1165.07, 1211.41 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:41 UTC run: 5100 subRun: 0 event: 242300
[32m*** Matched preselection cluster: 16 from 154
 - MainTrackID 32898
 - Gen 1 -> marley TPC 21
 - Purity 1.00 Hits 6
 - Charge 1174.41 ( MaxHit 272.88 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 131.85, -238.23, 1212.70 )
 - RecoInd0 Time,Y,Z ( 131.79, -238.17, 1272.57 )
 - RecoInd1 Time,Y,Z ( 131.95, -238.29, 1277.27 )
*** Matched flash: 1818 from 3692
 - Purity 98.68 % Plane 0 #Hits 59 PE 1415.69 MainOpHitPE 877.43
 - Time -18.03 (us) Fast 85.69 % Residual 85672.98
 - Reco Time,Y,Z ( -18.03, -280.22, 1197.07 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 23.68, -237.34, 1213.81 )
 - Main X,Y,Z ( 24.63, -238.89, 1211.42 )
 - EDep X,Y,Z ( 24.09, -238.11, 1212.89 )
 - Reco X,Y,Z ( 24.07, -238.23, 1212.70 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:48:41 UTC run: 5100 subRun: 0 event: 242300
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
29-Nov-2025 04:48:41 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/35/2b/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250209T201832Z_gen_004846_supernova_g4_halfActiveVol_detsim_reco_20250724T160921Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 29-Nov-2025 04:48:41 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.262677       9.23566       25.5031      0.701196       9.38343        50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00132494    0.00651424     0.0222569    0.00213774    0.00705603       50     
end_path:solarnuana:SolarNuAna     0.260473       9.22895       25.5011      0.683021       9.38395        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 29-Nov-2025 04:48:41 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1489.84 MB
  Peak resident set size usage (VmHWM): 847.057 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 481.861889 Real = 488.384055

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1489.84 VmHWM = 847.057

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===