justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242009.29@dunegpschedd02.fnal.gov

Jobsub ID242009.29@dunegpschedd02.fnal.gov
Workflow ID9957
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-12 06:57:22
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-13 08:13:58
From worker nodeHostnamedunegli-7572317-0-fnpc23039.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-13 05:38:43
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193255Z_gen_000185_supernova_g4stage1_g4stage2_detsim_reco_20250730T141350Z_reco2.root
JobscriptExit code0
Real time2h (9300s)
CPU time2h (9189s = 98%)
Max RSS bytes1906028544 (1817 MiB)
Outputting started2025-11-13 08:13:44
Output files
Finished2025-11-13 08:13:58
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

33068	133067
marley	1000180410	0.00	(315.15, -479.92, 1608.51)	133067	132963
marley	22		1.05	(315.15, -479.92, 1608.51)	133066	132963
marley	22		3.41	(315.15, -479.92, 1608.51)	133065	132963
marley	22		3.57	(315.15, -479.92, 1608.51)	133064	132963
marley	22		1.49	(315.15, -479.92, 1608.51)	133063	132963
marley	22		3.56	(315.15, -479.92, 1608.51)	133062	132963
marley	22		2.80	(315.15, -479.92, 1608.51)	133061	132963
marley	22		3.07	(315.15, -479.92, 1608.51)	133060	132963
marley	22		1.73	(315.15, -479.92, 1608.51)	133059	132963
marley	2212		0.00	(451.92, -281.94, 1292.24)	133047	132963
marley	2212		0.00	(461.68, -275.17, 1308.99)	133046	132963
marley	2212		0.00	(493.83, -263.63, 1316.47)	133042	132963
marley	2212		0.00	(501.79, -261.18, 1311.89)	133041	132963
marley	1000070140	0.00	(506.16, -257.88, 1312.84)	133040	132963
marley	1000240530	0.00	(506.12, -258.02, 1312.80)	133039	132963
marley	1000070140	0.00	(505.95, -259.78, 1313.41)	133038	132963
marley	1000070140	0.00	(506.29, -259.70, 1313.39)	133037	132963
marley	1000080160	0.00	(506.51, -261.20, 1314.63)	133036	132963
marley	1000280600	0.00	(507.85, -262.75, 1313.88)	133035	132963
marley	1000070140	0.00	(507.91, -262.80, 1313.67)	133034	132963
marley	1000280580	0.00	(507.93, -262.00, 1312.58)	133032	132963
marley	1000070140	0.00	(509.34, -261.05, 1312.08)	133031	132963
marley	1000240530	0.00	(510.48, -261.81, 1311.24)	133030	132963
marley	1000070140	0.00	(510.52, -261.76, 1311.18)	133029	132963
marley	1000070140	0.00	(511.25, -262.10, 1310.58)	133028	132963
marley	1000070140	0.00	(512.31, -261.04, 1311.81)	133027	132963
marley	1000080160	0.00	(511.57, -261.74, 1313.18)	133026	132963
marley	1000070140	0.01	(511.47, -261.84, 1313.13)	133025	132963
marley	1000070140	0.01	(512.10, -261.72, 1313.25)	133024	132963
marley	1000070140	0.00	(510.35, -261.18, 1314.14)	133023	132963
marley	1000260560	0.00	(509.36, -260.06, 1313.23)	133022	132963
marley	1000240520	0.00	(509.35, -260.08, 1313.23)	133021	132963
marley	1000070140	0.01	(509.29, -260.25, 1313.22)	133020	132963
marley	1000070140	0.02	(509.07, -260.07, 1313.49)	133019	132963
marley	1000260560	0.00	(509.01, -260.09, 1313.14)	133018	132963
marley	1000070140	0.01	(509.03, -259.98, 1312.78)	133017	132963
marley	1000070140	0.02	(509.06, -259.70, 1312.76)	133016	132963
marley	1000070140	0.00	(509.00, -259.72, 1313.16)	133015	132963
marley	1000080160	0.01	(508.95, -259.74, 1313.32)	133014	132963
marley	22		0.85	(509.55, -257.54, 1315.10)	132966	132843
marley	22		3.52	(509.55, -257.54, 1315.10)	132965	132843
marley	1000260560	0.09	(509.55, -257.54, 1315.10)	132964	132843
marley	2112		0.11	(509.55, -257.54, 1315.10)	132963	132843
marley	1000060120	0.58	(479.66, -263.87, 1317.83)	132962	132843
marley	22		2.82	(356.73, -123.46, 1140.28)	132848	0
marley	22		0.78	(356.73, -123.46, 1140.28)	132847	0
marley	22		0.35	(356.73, -123.46, 1140.28)	132846	0
marley	22		1.22	(356.73, -123.46, 1140.28)	132845	0
marley	22		2.64	(356.73, -123.46, 1140.28)	132844	0
marley	2112		5.15	(356.73, -123.46, 1140.28)	132843	0
marley	1000190390	0.22	(356.73, -123.46, 1140.28)	132842	0
marley	11		20.03	(356.73, -123.46, 1140.28)	132841	0
Signal Track IDs: 132841; 132842; 132843; 132844; 132845; 132846; 132847; 132848; 132962; 132963; 132964; 132965; 132966; 133014; 133015; 133016; 133017; 133018; 133019; 133020; 133021; 133022; 133023; 133024; 133025; 133026; 133027; 133028; 133029; 133030; 133031; 133032; 133034; 133035; 133036; 133037; 133038; 133039; 133040; 133041; 133042; 133046; 133047; 133059; 133060; 133061; 133062; 133063; 133064; 133065; 133066; 133067; 133068; 133069; 133070; 133071; 133072; 
%MSG
%MSG-w ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 08:13:40 UTC run: 3799 subRun: 0 event: 1849

Kinetic energy of signal particle is above threshold of 30.00 MeV. Skipping event.

%MSG
Begin processing the 10th record. run: 3799 subRun: 0 event: 1850 at 13-Nov-2025 08:13:40 UTC
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 08:13:43 UTC run: 3799 subRun: 0 event: 1850

###################################################
Event:			1850
Flag:			453793153
Geometry:		dunevd10kt_3view_30deg_v5_refactored_1x8x14ref (VD)
Signal Label:		marley
PDS Frequency in [MHz]:	62.50
PDS Tick in [us]:	0.016
TPC Frequency in [MHz]:	2.00
TPC Tick in [us]:	0.50
TPC Map:		112 TPCs
TPC DriftLength in [cm]:650.07
TPC DriftTime in [us]:	4048.69
Fiducial Volume in [cm]:850.32, 1507.84, 2306.12
###################################################
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 08:13:44 UTC run: 3799 subRun: 0 event: 1850

There are a total of 132477 Particles in the event

# of particles 8	from gen 1 marley
# of particles 30894	from gen 2 Ar39GenInLAr
# of particles 3427	from gen 3 Kr85GenInLAr
# of particles 3	from gen 4 Ar42GenInLAr
# of particles 1	from gen 5 K42From42ArGenInLAr
# of particles 33	from gen 6 Rn222ChainRn222GenInLAr
# of particles 19	from gen 7 Rn222ChainPo218GenInLAr
# of particles 61	from gen 8 Rn222ChainPb214GenInLAr
# of particles 49	from gen 9 Rn222ChainBi214GenInLAr
# of particles 15	from gen 10 Rn222ChainPb210GenInLAr
# of particles 25	from gen 11 Rn220ChainPb212GenInLAr
# of particles 234	from gen 12 K40GenInCathode
# of particles 3039	from gen 13 U238ChainGenInCathode
# of particles 3510	from gen 14 Th232ChainGenInCathode
# of particles 120	from gen 15 K40GenInAnode
# of particles 1332	from gen 16 U238ChainGenInAnode
# of particles 1567	from gen 17 Th232ChainGenInAnode
# of particles 0	from gen 18 Rn222ChainGenInPDS
# of particles 1	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 0	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 29	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 65	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 74	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 39	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 65	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 19	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 08:13:44 UTC run: 3799 subRun: 0 event: 1850

Number of Signal Particles: 8
Neutrino Interaction: -1
Neutrino Energy: 56.05 MeV
Position (-28.05, -506.61, 1123.72) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.76	(-28.05, -506.61, 1123.72)	132555	132453
marley	-12		0.55	(-28.05, -506.61, 1123.72)	132554	132453
marley	22		0.03	(-28.05, -506.61, 1123.72)	132457	0
marley	22		0.77	(-28.05, -506.61, 1123.72)	132456	0
marley	22		1.49	(-28.05, -506.61, 1123.72)	132455	0
marley	22		2.09	(-28.05, -506.61, 1123.72)	132454	0
marley	1000190400	0.09	(-28.05, -506.61, 1123.72)	132453	0
marley	11		50.07	(-28.05, -506.61, 1123.72)	132452	0
Signal Track IDs: 132452; 132453; 132454; 132455; 132456; 132457; 132554; 132555; 
%MSG
%MSG-w ProducerUtils:  SolarNuAna:solarnuana@BeginModule  13-Nov-2025 08:13:44 UTC run: 3799 subRun: 0 event: 1850

Kinetic energy of signal particle is above threshold of 30.00 MeV. Skipping event.

%MSG
13-Nov-2025 08:13:44 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/f1/50/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193255Z_gen_000185_supernova_g4stage1_g4stage2_detsim_reco_20250730T141350Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 13-Nov-2025 08:13:44 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          0.49335       924.934       3074.99       0.98528       1195.42        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000510709    0.011902      0.0566266    0.00629978     0.0165168       10     
end_path:solarnuana:SolarNuAna     0.492541       924.922       3074.97      0.950466       1195.42        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 13-Nov-2025 08:13:44 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2530.98 MB
  Peak resident set size usage (VmHWM): 1906.03 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 9178.439563 Real = 9282.692858

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2530.98 VmHWM = 1906.03

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-10 06:20:09 UTC       justIN version: 01.06.00