justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 288544.27@dunegpschedd02.fnal.gov

Jobsub ID288544.27@dunegpschedd02.fnal.gov
Workflow ID12007
Stage ID1
User namesmanthey@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-20 14:48:38
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2026-01-20 22:55:55
From worker nodeHostnamedunegli-50386-0-cmswn2355.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2026-01-20 14:57:20
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T194913Z_gen_004445_supernova_g4stage1_g4stage2_detsim_reco_20250621T225249Z_reco2.root
JobscriptExit code0
Real time7h (28697s)
CPU time7h (28516s = 99%)
Max RSS bytes2564218880 (2445 MiB)
Outputting started2026-01-20 22:55:37
Output files
Finished2026-01-20 22:55:55
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

(-26.29, 362.21, 197.74)	132731	132658
marley	1000180400	0.03	(-0.33, 341.20, 196.60)	132730	132658
marley	1000180400	0.03	(-2.51, 339.99, 196.89)	132729	132658
marley	1000180400	0.05	(-3.73, 352.29, 191.64)	132728	132658
marley	1000180400	0.05	(-3.95, 337.03, 203.34)	132727	132658
marley	1000180400	0.04	(-4.70, 339.68, 200.18)	132726	132658
marley	1000180400	0.06	(-8.40, 333.93, 200.47)	132725	132658
marley	1000180400	0.05	(17.88, 362.69, 205.39)	132724	132658
marley	1000180400	0.01	(13.96, 353.80, 217.55)	132723	132658
marley	1000180400	0.04	(-4.86, 356.97, 251.42)	132722	132658
marley	1000180400	0.00	(-6.07, 357.64, 250.69)	132721	132658
marley	1000180400	0.07	(-7.94, 359.23, 250.58)	132720	132658
marley	1000180400	0.03	(-5.38, 353.04, 247.48)	132719	132658
marley	1000180400	0.08	(-31.01, 335.99, 230.37)	132718	132658
marley	1000180400	0.05	(-8.03, 352.93, 230.92)	132717	132658
marley	1000180400	0.01	(-6.78, 351.33, 231.54)	132716	132658
marley	1000180400	0.08	(-6.64, 351.31, 231.54)	132715	132658
marley	1000180400	0.00	(-10.38, 347.12, 230.17)	132714	132658
marley	1000180400	0.01	(-11.21, 346.04, 229.74)	132713	132658
marley	1000180400	0.01	(-12.60, 340.83, 225.12)	132712	132658
marley	1000180400	0.02	(-10.31, 334.64, 212.42)	132711	132658
marley	1000180400	0.01	(-7.77, 335.73, 210.08)	132710	132658
marley	1000180400	0.05	(88.91, 334.26, 188.81)	132709	132658
marley	1000180400	0.03	(94.01, 329.90, 208.19)	132708	132658
marley	1000180400	0.01	(93.91, 329.80, 208.29)	132707	132658
marley	1000180400	0.01	(63.25, 302.10, 214.24)	132706	132658
marley	1000180400	0.00	(62.84, 301.94, 214.21)	132705	132658
marley	1000180400	0.02	(59.47, 300.56, 213.25)	132704	132658
marley	1000180400	0.00	(49.92, 304.65, 206.30)	132703	132658
marley	1000180400	0.12	(38.36, 310.27, 198.94)	132702	132658
marley	1000180400	0.03	(48.64, 304.14, 211.88)	132701	132658
marley	1000180400	0.14	(51.05, 304.31, 212.06)	132700	132658
marley	1000180400	0.11	(15.28, 296.30, 197.76)	132699	132658
marley	1000180400	0.00	(31.21, 283.60, 188.93)	132698	132658
marley	1000180400	0.07	(32.47, 283.20, 188.19)	132697	132658
marley	1000180400	0.06	(33.14, 293.31, 178.48)	132696	132658
marley	1000180400	0.05	(11.64, 297.58, 169.54)	132695	132658
marley	1000180400	0.19	(-9.53, 275.55, 173.82)	132694	132658
marley	22		0.00	(-8.11, 276.44, 173.78)	132661	132654
marley	22		1.46	(-8.11, 276.44, 173.78)	132660	132654
marley	1000180400	0.16	(-8.11, 276.44, 173.78)	132659	132654
marley	2112		1.96	(-8.11, 276.44, 173.78)	132658	132654
marley	1000180400	0.06	(-5.82, 271.23, 174.62)	132657	132654
marley	1000180400	0.02	(-3.18, 257.31, 164.27)	132656	132654
marley	1000180400	0.02	(-2.16, 253.81, 163.84)	132655	132654
marley	2112		3.68	(-1.33, 249.47, 165.42)	132654	0
marley	1000190390	0.24	(-1.33, 249.47, 165.42)	132653	0
marley	11		49.26	(-1.33, 249.47, 165.42)	132652	0
Signal Track IDs: 132652; 132653; 132654; 132655; 132656; 132657; 132658; 132659; 132660; 132661; 132694; 132695; 132696; 132697; 132698; 132699; 132700; 132701; 132702; 132703; 132704; 132705; 132706; 132707; 132708; 132709; 132710; 132711; 132712; 132713; 132714; 132715; 132716; 132717; 132718; 132719; 132720; 132721; 132722; 132723; 132724; 132725; 132726; 132727; 132728; 132729; 132730; 132731; 132732; 132733; 132734; 132735; 132736; 132737; 132738; 132739; 132740; 132741; 132742; 132743; 132744; 132745; 132746; 132747; 132748; 132749; 132750; 132751; 132752; 132753; 132754; 132755; 132756; 132757; 132758; 132759; 132760; 132761; 132762; 132763; 132764; 132765; 132766; 132767; 132768; 132769; 132770; 132772; 132773; 132774; 132775; 132819; 132820; 
%MSG
%MSG-w ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:55:28 UTC run: 4969 subRun: 0 event: 44449

Kinetic energy of signal particle is above threshold of 30.00 MeV. Skipping event.

%MSG
Begin processing the 10th record. run: 4969 subRun: 0 event: 44450 at 20-Jan-2026 22:55:29 UTC
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:55:34 UTC run: 4969 subRun: 0 event: 44450

###################################################
Event:			44450
Flag:			2050521354
Geometry:		dunevd10kt_3view_30deg_v5_refactored_1x8x14ref (VD)
Signal Label:		marley
PDS Frequency in [MHz]:	62.50
PDS Tick in [us]:	0.016
TPC Frequency in [MHz]:	2.00
TPC Tick in [us]:	0.50
TPC Map:		112 TPCs
TPC DriftLength in [cm]:650.07
TPC DriftTime in [us]:	4048.69
Fiducial Volume in [cm]:850.32, 1507.84, 2306.12
###################################################
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:55:36 UTC run: 4969 subRun: 0 event: 44450

There are a total of 132921 Particles in the event

# of particles 8	from gen 1 marley
# of particles 31089	from gen 2 Ar39GenInLAr
# of particles 3632	from gen 3 Kr85GenInLAr
# of particles 3	from gen 4 Ar42GenInLAr
# of particles 2	from gen 5 K42From42ArGenInLAr
# of particles 32	from gen 6 Rn222ChainRn222GenInLAr
# of particles 25	from gen 7 Rn222ChainPo218GenInLAr
# of particles 57	from gen 8 Rn222ChainPb214GenInLAr
# of particles 26	from gen 9 Rn222ChainBi214GenInLAr
# of particles 16	from gen 10 Rn222ChainPb210GenInLAr
# of particles 34	from gen 11 Rn220ChainPb212GenInLAr
# of particles 249	from gen 12 K40GenInCathode
# of particles 3049	from gen 13 U238ChainGenInCathode
# of particles 3748	from gen 14 Th232ChainGenInCathode
# of particles 90	from gen 15 K40GenInAnode
# of particles 1335	from gen 16 U238ChainGenInAnode
# of particles 1439	from gen 17 Th232ChainGenInAnode
# of particles 5	from gen 18 Rn222ChainGenInPDS
# of particles 3	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 1	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 25	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 52	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 66	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 39	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 27	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 21	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14
%MSG
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:55:36 UTC run: 4969 subRun: 0 event: 44450

Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 54.93 MeV
Position (309.77, -167.14, 1170.63) cm
Gen.	PdgCode		Energy		EndPosition		TrackID	MotherID
--------------------------------------------------------------------------------
marley	11		0.59	(309.77, -167.14, 1170.63)	132921	132895
marley	-12		0.72	(309.77, -167.14, 1170.63)	132920	132895
marley	1000200400	0.00	(309.77, -167.14, 1170.63)	132919	132895
marley	22		0.03	(309.77, -167.14, 1170.63)	132898	0
marley	22		1.61	(309.77, -167.14, 1170.63)	132897	0
marley	22		0.65	(309.77, -167.14, 1170.63)	132896	0
marley	1000190400	0.08	(309.77, -167.14, 1170.63)	132895	0
marley	11		51.05	(309.77, -167.14, 1170.63)	132894	0
Signal Track IDs: 132894; 132895; 132896; 132897; 132898; 132919; 132920; 132921; 
%MSG
%MSG-w ProducerUtils:  SolarNuAna:solarnuana@BeginModule  20-Jan-2026 22:55:36 UTC run: 4969 subRun: 0 event: 44450

Kinetic energy of signal particle is above threshold of 30.00 MeV. Skipping event.

%MSG
20-Jan-2026 22:55:36 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/a7/13/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T194913Z_gen_004445_supernova_g4stage1_g4stage2_detsim_reco_20250621T225249Z_reco2.root"
%MSG-i ProducerUtils:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 22:55:36 UTC  ModuleEndJob
Finished running the SolarNuAna module
%MSG

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          2.3095        2862.36       5111.5        4436.15       2346.02        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00136551     0.0121384     0.030369     0.00940213    0.00933282       10     
end_path:solarnuana:SolarNuAna      2.30404       2862.35       5111.48       4436.15       2346.02        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 20-Jan-2026 22:55:36 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3187.19 MB
  Peak resident set size usage (VmHWM): 2564.22 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 28497.482633 Real = 28677.232489

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3187.19 VmHWM = 2564.22

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-04 04:28:47 UTC       justIN version: 01.06.00