justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 296364.1@dunegpschedd02.fnal.gov

Jobsub ID296364.1@dunegpschedd02.fnal.gov
Workflow ID12414
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-28 16:20:51
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2026-01-28 19:52:50
From worker nodeHostnamedunegli-8463070-0-fnpc23031.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2026-01-28 16:22:39
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T234613Z_gen_001088_supernova_g4stage1_g4stage2_detsim_reco_20250701T085923Z_reco2.root
JobscriptExit code0
Real time3h (12604s)
CPU time3h (12497s = 99%)
Max RSS bytes2579488768 (2459 MiB)
Outputting started2026-01-28 19:52:43
Output files
Finished2026-01-28 19:52:50
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

, 176.64 )
 - Reco X,Y,Z ( 225.55, -772.41, 172.96 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:52:39 UTC run: 5028 subRun: 0 event: 10879
*** Matched preselection cluster: 4135 from 5611
 - MainTrackID 132906
 - Gen 1 -> marley TPC 10
 - Purity 1.00 Hits 3
 - Charge 1126.21 ( MaxHit 585.10 )
 - #AdjCl 12 ( 3 signal ):
    - Cluster 3484 at distance 81.61 with time 2666.78 and charge 102.79 in TPC 10 and hits 1
    - Cluster 3539 at distance 74.85 with time 2705.53 and charge 50.78 in TPC 8 and hits 1
    - Cluster 3636 at distance 99.36 with time 2763.95 and charge 20.74 in TPC 8 and hits 1
    - Cluster 3834 at distance 76.97 with time 2898.72 and charge 56.12 in TPC 2 and hits 1
    - Cluster 3947 at distance 79.80 with time 2995.46 and charge 51.73 in TPC 10 and hits 1
    - Cluster 3974 at distance 59.20 with time 3020.22 and charge 117.02 in TPC 10 and hits 1
    - Cluster 4017 at distance 14.02 with time 3058.26 and charge 273.94 in TPC 10 and hits 3
    - Cluster 4111 at distance 4.48 with time 3129.48 and charge 299.08 in TPC 10 and hits 5
    - Cluster 4136 at distance 88.91 with time 3144.58 and charge 26.65 in TPC 6 and hits 1
    - Cluster 4358 at distance 29.71 with time 3291.37 and charge 108.72 in TPC 10 and hits 2
    - Cluster 4437 at distance 75.39 with time 3345.15 and charge 49.55 in TPC 10 and hits 1
    - Cluster 4611 at distance 76.26 with time 3489.75 and charge 36.67 in TPC 4 and hits 1
 - RecoCol  Time,Y,Z ( 3144.29, -261.93, 176.46 )
 - RecoInd0 Time,Y,Z ( 3144.77, -261.75, 223.88 )
 - RecoInd1 Time,Y,Z ( 3149.08, -262.11, 223.88 )
*** Matched flash: 14961 from 18322
 - Purity 0.00 % Plane 0 #Hits 6
 - MainOpHitPE 6.11 (PE);  TotalPE 31.65 vs expected 33.59 (PE)
 - Time 2701.73 (us) Fast 22.48 % Residual 3.02
 - Reco Time,Y,Z ( 2701.73, -230.89, 236.66 )
 - Found valid waveform: true
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -183.29, -262.38, 176.76 )
 - Main X,Y,Z ( -177.13, -261.35, 175.83 )
 - EDep X,Y,Z ( -179.81, -261.66, 176.64 )
 - Reco X,Y,Z ( 253.98, -261.93, 176.46 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:52:39 UTC run: 5028 subRun: 0 event: 10879
*** Matched preselection cluster: 4161 from 5611
 - MainTrackID 132906
 - Gen 1 -> marley TPC 10
 - Purity 1.00 Hits 1
 - Charge 354.33 ( MaxHit 354.33 )
 - #AdjCl 8 ( 0 signal ):
    - Cluster 3725 at distance 83.24 with time 2821.29 and charge 30.80 in TPC 12 and hits 1
    - Cluster 3789 at distance 87.43 with time 2863.11 and charge 32.96 in TPC 4 and hits 1
    - Cluster 4031 at distance 51.64 with time 3066.22 and charge 41.23 in TPC 4 and hits 1
    - Cluster 4187 at distance 92.07 with time 3174.82 and charge 52.38 in TPC 4 and hits 2
    - Cluster 4359 at distance 92.36 with time 3291.77 and charge 35.06 in TPC 14 and hits 1
    - Cluster 4556 at distance 74.85 with time 3443.89 and charge 35.16 in TPC 6 and hits 1
    - Cluster 4664 at distance 63.56 with time 3527.56 and charge 37.99 in TPC 14 and hits 1
    - Cluster 4693 at distance 97.98 with time 3544.52 and charge 38.88 in TPC 4 and hits 2
 - RecoCol  Time,Y,Z ( 3158.58, -16.46, 176.71 )
 - RecoInd0 Time,Y,Z ( 3155.43, 56.69, 1097.47 )
 - RecoInd1 Time,Y,Z ( 3157.84, -89.62, 522.72 )
*** Matched flash: 9852 from 18322
 - Purity 0.00 % Plane 0 #Hits 8
 - MainOpHitPE 7.81 (PE);  TotalPE 53.26 vs expected 110.38 (PE)
 - Time 325.46 (us) Fast 16.54 % Residual 18.84
 - Reco Time,Y,Z ( 325.46, 6.68, 219.77 )
 - Found valid waveform: true
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -183.29, -262.38, 176.76 )
 - Main X,Y,Z ( -177.13, -261.35, 175.83 )
 - EDep X,Y,Z ( -179.81, -261.66, 176.64 )
 - Reco X,Y,Z ( -129.86, -16.46, 176.71 )
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:52:39 UTC run: 5028 subRun: 0 event: 10879
-----------------------------------------------------------------------------------------

%MSG
Begin processing the 10th record. run: 5028 subRun: 0 event: 10880 at 28-Jan-2026 19:52:40 UTC
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:52:42 UTC run: 5028 subRun: 0 event: 10880

###################################################
Event:			10880
Flag:			1267303458
Geometry:		dunevd10kt_3view_30deg_v5_refactored_1x8x14ref (VD)
Signal Label:		marley
PDS Frequency in [MHz]:	62.50
PDS Tick in [us]:	0.016
TPC Frequency in [MHz]:	2.00
TPC Tick in [us]:	0.50
TPC Map:		112 TPCs
TPC DriftLength in [cm]:650.07
TPC DriftTime in [us]:	4048.69
Fiducial Volume in [cm]:850.32, 1507.84, 2306.12
###################################################
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:52:42 UTC run: 5028 subRun: 0 event: 10880

There are a total of 132846 Particles in the event

# of particles 6	from gen 1 marley
# of particles 31156	from gen 2 Ar39GenInLAr
# of particles 3555	from gen 3 Kr85GenInLAr
# of particles 5	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 31	from gen 6 Rn222ChainRn222GenInLAr
# of particles 40	from gen 7 Rn222ChainPo218GenInLAr
# of particles 67	from gen 8 Rn222ChainPb214GenInLAr
# of particles 41	from gen 9 Rn222ChainBi214GenInLAr
# of particles 6	from gen 10 Rn222ChainPb210GenInLAr
# of particles 27	from gen 11 Rn220ChainPb212GenInLAr
# of particles 195	from gen 12 K40GenInCathode
# of particles 3099	from gen 13 U238ChainGenInCathode
# of particles 3530	from gen 14 Th232ChainGenInCathode
# of particles 118	from gen 15 K40GenInAnode
# of particles 1396	from gen 16 U238ChainGenInAnode
# of particles 1486	from gen 17 Th232ChainGenInAnode
# of particles 0	from gen 18 Rn222ChainGenInPDS
# of particles 1	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 2	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 20	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 59	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 46	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 35	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 58	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 19	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:52:42 UTC run: 5028 subRun: 0 event: 10880

Number of Signal Particles: 6
Neutrino Interaction: -1
Neutrino Energy: 31.41 MeV
Position (145.30, 114.86, 1128.38) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.25	(145.30, 114.86, 1128.38)	132841	132824
marley	-12		0.46	(145.30, 114.86, 1128.38)	132840	132824
marley	22		1.16	(145.30, 114.86, 1128.38)	132826	0
marley	1000020040	6.04	(145.30, 114.86, 1128.38)	132825	0
marley	1000170360	0.85	(145.30, 114.86, 1128.38)	132824	0
marley	11		15.42	(145.30, 114.86, 1128.38)	132823	0
Signal Track IDs: 132823; 132824; 132825; 132826; 132840; 132841; 
%MSG
%MSG-w ProducerUtils:  SolarNuAna:solarnuana@BeginModule  28-Jan-2026 19:52:42 UTC run: 5028 subRun: 0 event: 10880

Kinetic energy of signal particle is above threshold of 30.00 MeV. Skipping event.

%MSG
28-Jan-2026 19:52:42 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f2/e4/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T234613Z_gen_001088_supernova_g4stage1_g4stage2_detsim_reco_20250701T085923Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 28-Jan-2026 19:52:42 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          0.55951       1256.57       1944.17       1852.96       845.747        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00154976     0.0100268     0.0226038    0.00881822    0.00795225       10     
end_path:solarnuana:SolarNuAna     0.556779       1256.56       1944.15       1852.95       845.742        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 28-Jan-2026 19:52:42 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3200.97 MB
  Peak resident set size usage (VmHWM): 2579.49 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 12484.676818 Real = 12588.608523

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3200.97 VmHWM = 2579.49

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-04 04:27:02 UTC       justIN version: 01.06.00