justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 278547.1@dunegpschedd01.fnal.gov

Jobsub ID278547.1@dunegpschedd01.fnal.gov
Workflow ID11760
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-15 01:41:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2026-01-15 04:33:49
From worker nodeHostnamedunegli-8325406-0-fnpc22034.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-15 01:42:24
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T111129Z_gen_004914_supernova_g4stage1_g4stage2_detsim_reco_20250731T224120Z_reco2.root
JobscriptExit code0
Real time2h (10275s)
CPU time2h (9810s = 95%)
Max RSS bytes2569613312 (2450 MiB)
Outputting started2026-01-15 04:33:39
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/smanthey/ana/solar_ana_marley_flash_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_nominal/fnal/11760/1/001/fdhd_ana_2026-01-15T_014253Z_hist.root
Finished2026-01-15 04:33:49
Saved logsjustin-logs:278547.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2795; 132796; 132797; 132798; 132799; 132800; 132801; 132802; 132803; 132804; 132805; 132806; 132807; 132808; 132809; 132810; 132811; 132812; 132813; 132814; 132815; 132816; 132817; 132818; 132819; 132820; 132821; 132822; 132823; 132824; 132825; 132826; 132827; 132828; 132829; 132830; 132831; 132832; 132833; 132835; 132840; 132864; 132865; 132866; 132867; 132868; 132869; 132870; 132871; 132872; 132873; 132874; 
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  15-Jan-2026 04:03:09 UTC run: 5021 subRun: 0 event: 49140

Kinetic energy of signal particle is below threshold of 30.00 MeV. Selecting event.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  15-Jan-2026 04:03:57 UTC run: 5021 subRun: 0 event: 49140
OpFlash PE 134.37 with purity 0.01 time -2.12 plane 0
 - Vertex (-327.50, 138.51, 1334.97)
	*** 1st Sanity check: Ratio 0.27 <= 1.00
	*** 2nd Sanity check: #OpHits 10 >= 10
OpFlash PE 99.95 with purity 0.21 time 0.66 plane 0
 - Vertex (-327.50, 16.70, 1051.52)
	*** 1st Sanity check: Ratio 0.15 <= 1.00
	*** 2nd Sanity check: #OpHits 11 >= 11
OpFlash PE 55.84 with purity 0.43 time 1.90 plane 1
 - Vertex (106.35, 743.30, 748.10)
	*** 1st Sanity check: Ratio 0.50 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 106.21 with purity 0.09 time 1.64 plane 1
 - Vertex (285.07, 743.30, 1939.46)
	*** 1st Sanity check: Ratio 0.38 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 17.53 with purity 0.12 time 2.95 plane 2
 - Vertex (142.06, -743.30, 861.16)
	*** 1st Sanity check: Ratio 0.38 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 59.59 with purity 0.23 time 2.56 plane 0
 - Vertex (-327.50, -36.15, 1108.32)
	*** 1st Sanity check: Ratio 0.21 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 124.82 with purity 0.27 time 1047.94 plane 0
 - Vertex (-327.50, -68.00, 1804.23)
	*** 1st Sanity check: Ratio 0.15 <= 1.00
	*** 2nd Sanity check: #OpHits 14 >= 14
OpFlash PE 68.55 with purity 0.65 time 1048.42 plane 1
 - Vertex (218.96, 743.30, 1607.33)
	*** 1st Sanity check: Ratio 0.25 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 36.62 with purity 0.46 time 1049.78 plane 1
 - Vertex (181.55, 743.30, 1484.30)
	*** 1st Sanity check: Ratio 0.34 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 51.63 with purity 0.06 time 1052.31 plane 1
 - Vertex (210.07, 743.30, 1343.78)
	*** 1st Sanity check: Ratio 0.49 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 5.45 with purity 1.00 time 1052.95 plane 0
 - Vertex (-327.50, -132.70, 1746.62)
	*** 1st Sanity check: Ratio 1.00 <= 1.00
	*** 2nd Sanity check: #OpHits 1 >= 1

# of OpHits (ophit10ppm): 52608 for times between -4253.86 us and 295147919198126080.00 us.
# of OpFlashes (solarflash): 18212 for times between -4253.86 us and 295147919198126080.00 us.

%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  15-Jan-2026 04:03:59 UTC run: 5021 subRun: 0 event: 49140

# OpHits (ophit10ppm) in full geometry: 52608
# OpFlashes (solarflash) in full geometry: 18212
# Hits (gaushit) in each view: 6864, 6856, 6734, 0
# Cluster from the hits: 2871, 2918, 5716, 0
# Tracks (pmtracktc) in full geometry: 476
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  15-Jan-2026 04:33:39 UTC run: 5021 subRun: 0 event: 49140

# ClusterReco: Looping over 5710 matched clusters
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  15-Jan-2026 04:33:39 UTC run: 5021 subRun: 0 event: 49140
*** Matched preselection cluster: 412 from 5710
 - MainTrackID 132729
 - Gen 1 -> marley TPC 69
 - Purity 1.00 Hits 4
 - Charge 755.87 ( MaxHit 362.06 )
 - #AdjCl 6 ( 3 signal ):
    - Cluster 177 at distance 62.26 with time 197.86 and charge 50.28 in TPC 69 and hits 1
    - Cluster 387 at distance 12.35 with time 363.89 and charge 149.34 in TPC 69 and hits 1
    - Cluster 393 at distance 99.95 with time 372.27 and charge 20.69 in TPC 59 and hits 1
    - Cluster 415 at distance 19.34 with time 388.70 and charge 148.33 in TPC 69 and hits 2
    - Cluster 470 at distance 33.76 with time 423.49 and charge 55.88 in TPC 69 and hits 1
    - Cluster 533 at distance 23.06 with time 456.06 and charge 349.07 in TPC 69 and hits 2
 - RecoCol  Time,Y,Z ( 385.59, 296.17, 1248.75 )
 - RecoInd0 Time,Y,Z ( 385.65, 296.12, 1270.32 )
 - RecoInd1 Time,Y,Z ( 385.63, 296.23, 1256.72 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 262.93, 295.82, 1247.80 )
 - Main X,Y,Z ( 262.97, 296.42, 1248.95 )
 - EDep X,Y,Z ( 263.21, 296.15, 1248.78 )
 - Reco X,Y,Z ( -1000000.00, 296.17, 1248.75 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  15-Jan-2026 04:33:39 UTC run: 5021 subRun: 0 event: 49140
*** Matched preselection cluster: 1655 from 5710
 - MainTrackID 132867
 - Gen 1 -> marley TPC 86
 - Purity 1.00 Hits 3
 - Charge 338.24 ( MaxHit 219.78 )
 - #AdjCl 7 ( 2 signal ):
    - Cluster 1189 at distance 59.06 with time 951.54 and charge 28.38 in TPC 80 and hits 1
    - Cluster 1222 at distance 88.05 with time 976.46 and charge 28.46 in TPC 84 and hits 1
    - Cluster 1268 at distance 65.68 with time 1018.46 and charge 120.60 in TPC 86 and hits 2
    - Cluster 1342 at distance 59.37 with time 1067.23 and charge 110.24 in TPC 86 and hits 1
    - Cluster 1514 at distance 68.11 with time 1201.10 and charge 26.66 in TPC 82 and hits 1
    - Cluster 1757 at distance 14.62 with time 1393.49 and charge 80.80 in TPC 86 and hits 1
    - Cluster 2122 at distance 74.94 with time 1673.39 and charge 173.93 in TPC 86 and hits 1
 - RecoCol  Time,Y,Z ( 1315.08, 417.95, 1615.44 )
 - RecoInd0 Time,Y,Z ( 1315.02, 418.20, 1569.16 )
 - RecoInd1 Time,Y,Z ( 1315.33, 417.70, 1569.16 )
*** Matched flash: 9011 from 18212
 - Purity 0.00 % Plane 0 #Hits 3
 - MainOpHitPE 10.02 (PE);  TotalPE 24.67 vs expected 25.42 (PE)
 - Time -40.89 (us) Fast 44.68 % Residual 4.85
 - Reco Time,Y,Z ( -40.89, 464.10, 1529.62 )
 - Found valid waveform: true
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 262.93, 295.82, 1247.80 )
 - Main X,Y,Z ( 267.13, 287.71, 1239.45 )
 - EDep X,Y,Z ( 282.19, 417.68, 1615.63 )
 - Reco X,Y,Z ( 107.32, 417.95, 1615.44 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  15-Jan-2026 04:33:39 UTC run: 5021 subRun: 0 event: 49140
*** Matched preselection cluster: 1704 from 5710
 - MainTrackID 132864
 - Gen 1 -> marley TPC 86
 - Purity 1.00 Hits 5
 - Charge 474.40 ( MaxHit 159.98 )
 - #AdjCl 2 ( 0 signal ):
    - Cluster 1328 at distance 98.49 with time 1055.01 and charge 51.61 in TPC 84 and hits 2
    - Cluster 2107 at distance 99.57 with time 1663.15 and charge 78.01 in TPC 86 and hits 1
 - RecoCol  Time,Y,Z ( 1347.88, 246.89, 1604.21 )
 - RecoInd0 Time,Y,Z ( 1354.13, 394.33, 1569.16 )
 - RecoInd1 Time,Y,Z ( 1353.20, 99.45, 797.83 )
*** Matched flash: 6267 from 18212
 - Purity 0.00 % Plane 0 #Hits 3
 - MainOpHitPE 8.12 (PE);  TotalPE 20.67 vs expected 112.88 (PE)
 - Time -1309.10 (us) Fast 44.14 % Residual 1.19
 - Reco Time,Y,Z ( -1309.10, 269.52, 1634.54 )
 - Found valid waveform: true
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 262.93, 295.82, 1247.80 )
 - Main X,Y,Z ( 267.13, 287.71, 1239.45 )
 - EDep X,Y,Z ( 277.92, 401.78, 1604.79 )
 - Reco X,Y,Z ( -101.58, 246.89, 1604.21 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  15-Jan-2026 04:33:39 UTC run: 5021 subRun: 0 event: 49140
-----------------------------------------------------------------------------------------

%MSG
15-Jan-2026 04:33:39 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/d2/8c/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T111129Z_gen_004914_supernova_g4stage1_g4stage2_detsim_reco_20250731T224120Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 15-Jan-2026 04:33:39 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          0.66594       1020.85       2504.67       901.609       1038.37        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000530915   0.00959873     0.0190017     0.012174     0.00710021       10     
end_path:solarnuana:SolarNuAna     0.646236       1020.84       2504.67       901.602       1038.37        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 15-Jan-2026 04:33:39 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3191.96 MB
  Peak resident set size usage (VmHWM): 2569.61 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 9797.431319 Real = 10237.395692

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3191.96 VmHWM = 2569.61

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-06 09:00:45 UTC       justIN version: 01.06.00