Jobsub ID 249793.2@dunegpschedd02.fnal.gov

Jobsub ID	249793.2@dunegpschedd02.fnal.gov
Workflow ID	10517
Stage ID	1
User name	smanthey@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-22 20:25:50
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-11-22 21:10:47
From worker node	Hostname	dunegli-7851131-0-fnpc23042.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-22 20:32:36
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T062123Z_gen_001879_supernova_g4_halfActiveVol_detsim_reco_20250725T052213Z_reco2.root
Jobscript	Exit code	0
	Real time	2m (176s)
	CPU time	2m (169s = 96%)
	Max RSS bytes	850509824 (811 MiB)
Outputting started	2025-11-22 20:35:33
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/smanthey/ana/solar_ana_marley_flash_radiological_decay0_dune10kt_1x2x6_lateralAPA/fnal/10517/1/001/fdhd_ana_2025-11-22T_203241Z_hist.root
Finished	2025-11-22 21:10:47
Saved logs	justin-logs:249793.2-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

cerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[31mInvalid OpWaveform handle[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[34mOpFlash PE 15.89 with purity 0.25 time -24.07 plane 0
 - Vertex (-0.05, -93.41, 427.91)
	*** 1st Sanity check: Ratio 0.76 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 88.26 with purity 1.00 time -17.95 plane 0
 - Vertex (-0.05, -464.24, 241.53)
	*** 1st Sanity check: Ratio 0.12 <= 1.00
	*** 2nd Sanity check: #OpHits 29 >= 29
OpFlash PE 13.91 with purity 1.00 time -16.73 plane 0
 - Vertex (-0.05, -458.38, 266.07)
	*** 1st Sanity check: Ratio 0.30 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 9.42 with purity 0.59 time -16.54 plane 0
 - Vertex (-0.01, -151.05, 670.04)
	*** 1st Sanity check: Ratio 0.22 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 10.10 with purity 1.00 time -15.10 plane 0
 - Vertex (-0.05, -479.42, 424.64)
	*** 1st Sanity check: Ratio 0.28 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 1.89 with purity 1.00 time -14.00 plane 0
 - Vertex (0.05, 217.95, 611.50)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1

# of OpHits (ophitspe): 40976 for times between -2260.88 us and 2227.57 us.
# of OpFlashes (solarflash): 3863 for times between -2260.88 us and 2227.47 us.
[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:25 UTC run: 5102 subRun: 0 event: 93950
[36m
# OpHits (ophitspe) in full geometry: 40976
# OpFlashes (solarflash) in full geometry: 3863
# Hits (hitfd) in each view: 155, 172, 295, 0
# Cluster from the hits: 86, 95, 259, 0
# Tracks (pmtracktc) in full geometry: 2[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:32 UTC run: 5102 subRun: 0 event: 93950
[32m
# ClusterReco: Looping over 156 matched clusters[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:32 UTC run: 5102 subRun: 0 event: 93950
[32m*** Matched preselection cluster: 120 from 156
 - MainTrackID 33194
 - Gen 1 -> marley TPC 1
 - Purity 1.00 Hits 14
 - Charge 2954.53 ( MaxHit 710.54 )
 - #AdjCl 4 ( 1 signal ):
    - Cluster 113 at distance 89.30 with time 1877.74 and charge 40.97 in TPC 3 and hits 1
    - Cluster 114 at distance 16.53 with time 1879.43 and charge 50.57 in TPC 1 and hits 1
    - Cluster 117 at distance 44.20 with time 1920.59 and charge 148.68 in TPC 5 and hits 1
    - Cluster 125 at distance 39.56 with time 2008.67 and charge 60.10 in TPC 5 and hits 1
 - RecoCol  Time,Y,Z ( 1978.31, -578.16, 199.57 )
 - RecoInd0 Time,Y,Z ( 1978.62, -578.18, 207.26 )
 - RecoInd1 Time,Y,Z ( 1978.00, -578.14, 107.81 )
*** Matched flash: 2666 from 3863
 - Purity 0.00 % Plane 0 #Hits 7 PE 23.75 MainOpHitPE 7.42
 - Time 858.19 (us) Fast 35.47 % Residual 9.00
 - Reco Time,Y,Z ( 858.19, -529.30, 171.75 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 316.74, -574.73, 200.29 )
 - Main X,Y,Z ( 322.74, -580.62, 196.78 )
 - EDep X,Y,Z ( 321.56, -579.41, 200.03 )
 - Reco X,Y,Z ( 179.85, -578.16, 199.57 )[0m
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  22-Nov-2025 20:35:32 UTC run: 5102 subRun: 0 event: 93950
[32m-----------------------------------------------------------------------------------------
[0m
%MSG
22-Nov-2025 20:35:32 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/91/b2/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20250210T062123Z_gen_001879_supernova_g4_halfActiveVol_detsim_reco_20250725T052213Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 22-Nov-2025 20:35:32 UTC  ModuleEndJob
[35mFinished running the SolarNuAna module[0m
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.118505       2.94288       9.23291      0.210641       3.56648        50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000907113   0.00225863     0.0199067    0.00136404    0.00371455       50     
end_path:solarnuana:SolarNuAna     0.116955       2.94052       9.23097      0.209216       3.56714        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 22-Nov-2025 20:35:32 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1491.03 MB
  Peak resident set size usage (VmHWM): 845.308 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 156.973629 Real = 161.309796

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1491.03 VmHWM = 845.308

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0