justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 255008.1@dunegpschedd02.fnal.gov

Jobsub ID255008.1@dunegpschedd02.fnal.gov
Workflow ID10885
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-29 04:32:42
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-11-29 04:41:46
From worker nodeHostnamedunegli-7929757-0-fnpc23101.fnal.gov
cpuinfoAMD EPYC 9354 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-29 04:33:19
Input filesfardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T115009Z_gen_001764_supernova_g4_detsim_reco_20250723T213213Z_reco2.root
JobscriptExit code0
Real time8m (499s)
CPU time8m (493s = 98%)
Max RSS bytes922411008 (879 MiB)
Outputting started2025-11-29 04:41:38
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/smanthey/ana/solar_ana_marley_flash_radiological_decay0_dune10kt_1x2x6_centralAPA/fnal/10885/1/001/fdhd_ana_2025-11-29T_043324Z_hist.root
Finished2025-11-29 04:41:46
Saved logsjustin-logs:255008.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200
Invalid OpWaveform handle
%MSG
%MSG-e ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200
Invalid OpWaveform handle
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200
OpFlash PE 35.03 with purity 0.08 time -25.20 plane 0
 - Vertex (0.05, 31.14, 660.30)
	*** 1st Sanity check: Ratio 0.48 <= 1.00
	*** 2nd Sanity check: #OpHits 8 >= 8
OpFlash PE 4.56 with purity 0.36 time -19.29 plane 0
 - Vertex (0.05, 569.17, 707.98)
	*** 1st Sanity check: Ratio 0.64 <= 1.00
	*** 2nd Sanity check: #OpHits 2 >= 2
OpFlash PE 7.33 with purity 0.10 time -17.88 plane 0
 - Vertex (-0.05, -388.87, 252.93)
	*** 1st Sanity check: Ratio 0.32 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 1312.28 with purity 0.99 time -18.04 plane 0
 - Vertex (0.05, 565.63, 426.58)
	*** 1st Sanity check: Ratio 0.21 <= 1.00
	*** 2nd Sanity check: #OpHits 56 >= 56
OpFlash PE 29.46 with purity 0.34 time -16.66 plane 0
 - Vertex (-0.01, -91.19, 575.47)
	*** 1st Sanity check: Ratio 0.44 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 119.10 with purity 0.95 time -16.57 plane 0
 - Vertex (0.05, 436.78, 512.27)
	*** 1st Sanity check: Ratio 0.13 <= 1.00
	*** 2nd Sanity check: #OpHits 29 >= 29
OpFlash PE 8.58 with purity 0.19 time -15.74 plane 0
 - Vertex (-0.05, -287.11, 121.87)
	*** 1st Sanity check: Ratio 0.58 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 46.35 with purity 1.00 time -15.27 plane 0
 - Vertex (0.05, 580.94, 458.97)
	*** 1st Sanity check: Ratio 0.20 <= 1.00
	*** 2nd Sanity check: #OpHits 17 >= 17
OpFlash PE 23.28 with purity 0.91 time -14.52 plane 0
 - Vertex (0.05, 418.07, 472.85)
	*** 1st Sanity check: Ratio 0.14 <= 1.00
	*** 2nd Sanity check: #OpHits 11 >= 11
OpFlash PE 23.38 with purity 0.89 time -13.20 plane 0
 - Vertex (0.05, 504.17, 468.67)
	*** 1st Sanity check: Ratio 0.15 <= 1.00
	*** 2nd Sanity check: #OpHits 10 >= 10
OpFlash PE 14.11 with purity 0.87 time -11.34 plane 0
 - Vertex (0.05, 525.52, 404.95)
	*** 1st Sanity check: Ratio 0.30 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 10.68 with purity 1.00 time -8.90 plane 0
 - Vertex (0.05, 529.30, 499.90)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1

# of OpHits (ophitspe): 71441 for times between -2261.57 us and 2228.00 us.
# of OpFlashes (solarflash): 5196 for times between -2261.27 us and 2227.96 us.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:18 UTC run: 5097 subRun: 0 event: 88200

# OpHits (ophitspe) in full geometry: 71441
# OpFlashes (solarflash) in full geometry: 5196
# Hits (hitfd) in each view: 206, 200, 428, 0
# Cluster from the hits: 126, 120, 386, 0
# Tracks (pmtracktc) in full geometry: 2
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:38 UTC run: 5097 subRun: 0 event: 88200

# ClusterReco: Looping over 237 matched clusters
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:38 UTC run: 5097 subRun: 0 event: 88200
*** Matched preselection cluster: 11 from 237
 - MainTrackID 32750
 - Gen 1 -> marley TPC 6
 - Purity 1.00 Hits 1
 - Charge 101.61 ( MaxHit 101.61 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 149.36, 1189.81, 418.84 )
 - RecoInd0 Time,Y,Z ( 149.36, 557.43, 425.63 )
 - RecoInd1 Time,Y,Z ( 156.66, 1822.20, 1344.83 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -41.33, 566.67, 449.64 )
 - Main X,Y,Z ( -40.33, 564.52, 445.60 )
 - EDep X,Y,Z ( -41.34, 566.43, 449.29 )
 - Reco X,Y,Z ( -1000000.00, 1189.81, 418.84 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:38 UTC run: 5097 subRun: 0 event: 88200
*** Matched preselection cluster: 14 from 237
 - MainTrackID 32752
 - Gen 1 -> marley TPC 6
 - Purity 1.00 Hits 1
 - Charge 41.41 ( MaxHit 41.41 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 173.26, 1676.39, 453.81 )
 - RecoInd0 Time,Y,Z ( 171.58, 1676.39, 1310.07 )
 - RecoInd1 Time,Y,Z ( -1000000.00, -1000000.00, -1000000.00 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -41.33, 566.67, 449.64 )
 - Main X,Y,Z ( -38.37, 579.56, 455.90 )
 - EDep X,Y,Z ( -39.67, 566.25, 449.26 )
 - Reco X,Y,Z ( -1000000.00, 1676.39, 453.81 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:38 UTC run: 5097 subRun: 0 event: 88200
*** Matched preselection cluster: 16 from 237
 - MainTrackID 32750
 - Gen 1 -> marley TPC 6
 - Purity 1.00 Hits 11
 - Charge 2488.48 ( MaxHit 539.78 )
 - #AdjCl 1 ( 1 signal ):
    - Cluster 15 at distance 29.74 with time 197.83 and charge 397.36 in TPC 6 and hits 1
 - RecoCol  Time,Y,Z ( 236.07, 565.35, 447.60 )
 - RecoInd0 Time,Y,Z ( 236.33, 565.32, 442.84 )
 - RecoInd1 Time,Y,Z ( 236.21, 565.38, 347.71 )
*** Matched flash: 2623 from 5196
 - Purity 99.31 % Plane 0 #Hits 56 PE 1312.28 MainOpHitPE 277.36
 - Time -18.04 (us) Fast 87.82 % Residual 224.64
 - Reco Time,Y,Z ( -18.04, 565.63, 426.58 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -41.33, 566.67, 449.64 )
 - Main X,Y,Z ( -40.33, 564.52, 445.60 )
 - EDep X,Y,Z ( -41.34, 566.43, 449.29 )
 - Reco X,Y,Z ( -40.80, 565.35, 447.60 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:38 UTC run: 5097 subRun: 0 event: 88200
*** Matched preselection cluster: 19 from 237
 - MainTrackID 32753
 - Gen 1 -> marley TPC 10
 - Purity 1.00 Hits 1
 - Charge 44.62 ( MaxHit 44.62 )
 - #AdjCl 0 ( 0 signal ):
 - RecoCol  Time,Y,Z ( 252.70, 228.01, 468.25 )
 - RecoInd0 Time,Y,Z ( 246.66, -731.87, 1077.63 )
 - RecoInd1 Time,Y,Z ( 246.39, 1187.90, 1044.88 )
*** Matched flash: 2094 from 5196
 - Purity 0.00 % Plane 0 #Hits 32 PE 179.83 MainOpHitPE 47.78
 - Time -473.93 (us) Fast 33.29 % Residual 866.89
 - Reco Time,Y,Z ( -473.93, 298.23, 507.58 )
 - Found valid waveform: false
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -41.33, 566.67, 449.64 )
 - Main X,Y,Z ( -57.45, 565.70, 474.18 )
 - EDep X,Y,Z ( -43.58, 575.15, 468.23 )
 - Reco X,Y,Z ( -116.67, 228.01, 468.25 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  29-Nov-2025 04:41:38 UTC run: 5097 subRun: 0 event: 88200
-----------------------------------------------------------------------------------------

%MSG
29-Nov-2025 04:41:38 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/05/7d/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T115009Z_gen_001764_supernova_g4_detsim_reco_20250723T213213Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 29-Nov-2025 04:41:38 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.130009       9.25295       25.046       0.594099       10.0248        50     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00113717    0.00164975    0.00226093     0.0016249    0.000260933      50     
end_path:solarnuana:SolarNuAna     0.127623       9.25113       25.044       0.592274       10.0248        50     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 29-Nov-2025 04:41:38 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1550.41 MB
  Peak resident set size usage (VmHWM): 906.514 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 50 passed = 50 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         50         50          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 479.616949 Real = 482.997534

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1550.41 VmHWM = 906.514

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-08 02:45:13 UTC       justIN version: 01.06.00