Jobsub ID 293531.2@dunegpschedd02.fnal.gov

Jobsub ID	293531.2@dunegpschedd02.fnal.gov
Workflow ID	12230
Stage ID	1
User name	muditk@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	6291456000 (6000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-22 11:32:57
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2026-01-22 11:50:42
From worker node	Hostname	dunegli-8232568-0-fnpc22009.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	7864320000 (7500 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-22 11:34:05
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T180016Z_gen_003545_supernova_g4_detsim_reco_20250724T103854Z_reco2.root
Jobscript	Exit code	0
	Real time	16m (987s)
	CPU time	16m (972s = 98%)
	Max RSS bytes	4283572224 (4085 MiB)
Outputting started	2026-01-22 11:50:32
Output files
Finished	2026-01-22 11:50:42
Saved logs	justin-logs:293531.2-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

imer: WireCell::Gen::Digitizer : 1.9599993 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Digitizer : 1.9499993 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.919999 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.909999 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Digitizer : 1.8799994 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.8599992 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.8599991 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.8199992 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.7299993 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.7199993 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.6799992 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.6799989 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.6199989 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.609999 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.589999 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 1.5699989 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.559999 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.519999 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.499999 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.499999 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.4699991 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.4299991 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 1.419999 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.3899992 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.3599992 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 1.2899991 sec
[11:47:27.292] I [ timer  ] Timer: WireCell::Gen::Retagger : 1.1999992 sec
[11:47:27.292] I [ timer  ] Timer: wcls::FrameSaver : 1.1699992 sec
[11:47:27.292] I [ timer  ] Timer: wcls::SimDepoSource : 1.1499993 sec
[11:47:27.292] I [ timer  ] Timer: Total node execution : 662.2796442508698 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 5th record. run: 5092 subRun: 0 event: 177205 at 22-Jan-2026 11:47:35 UTC
SimDepoSource got 141904 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 141904 depos spanning: [-2245.92, 8.99662e+22]us
Retagger: tagging trace set: daq with 30720 traces, 0 summary
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 70.489914 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 69.89989 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 67.559906 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 65.04992 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 58.129887 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 57.00993 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 56.619896 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 56.34989 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 52.289978 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 50.589905 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 50.149876 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 48.959976 sec
[11:50:23.675] I [ timer  ] Timer: wcls::SimChannelSink : 7.7900834 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Drifter : 3.4599972 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 2.9899976 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.7699988 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.7499986 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.669999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.669999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.669999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.649999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.629999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.6099987 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.589999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.5899987 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.5799987 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.529999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.4899988 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.469999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.4199991 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.399999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.389999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.359999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.349999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.3299992 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.299999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.219999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.219999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.219999 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.0199986 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.0099986 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.9799986 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.9599986 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8999988 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 1.8999985 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8899987 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8699987 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8499987 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8299987 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 1.7799987 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.7499988 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.7299988 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.7099988 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::Retagger : 1.5999988 sec
[11:50:23.675] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 1.5999988 sec
[11:50:23.675] I [ timer  ] Timer: wcls::SimDepoSource : 1.429999 sec
[11:50:23.675] I [ timer  ] Timer: wcls::FrameSaver : 1.429999 sec
[11:50:23.675] I [ timer  ] Timer: Total node execution : 809.458994269371 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
22-Jan-2026 11:50:32 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T180016Z_gen_003545_supernova_g4_detsim_reco_20250724T103854Z_reco2_detsim_2026-01-22T_113408Z.root"
22-Jan-2026 11:50:32 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/02/58/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250207T180016Z_gen_003545_supernova_g4_detsim_reco_20250724T103854Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      171.568       191.149        209.3        190.197       15.7894         5     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00126557     0.0039157    0.00537336    0.00425544    0.00138923        5     
simulate:tpcrawdecoder:WireCellToolkit          165.207       184.575       202.906       183.79        15.6344         5     
simulate:rns:RandomNumberSaver                5.1968e-05    0.000140023   0.000337969   5.5875e-05    0.00011455        5     
[art]:TriggerResults:TriggerResultInserter    2.1301e-05    2.83256e-05   4.8331e-05    2.3335e-05    1.01891e-05       5     
end_path:out1:RootOutput                       4.749e-06     9.624e-06    2.3905e-05     6.873e-06    7.23672e-06       5     
end_path:out1:RootOutput(write)                 6.35696       6.56907       7.17927       6.40274      0.309906         5     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5901.83 MB
  Peak resident set size usage (VmHWM): 4283.57 MB
====================================================================================================
ProvidedPedestalAdditionService::dtor: Deleting random engine with seed 376695977
ExponentialChannelNoiseService::dtor: Deleting random engine with seed 820915101
StuckBitAdcDistortionService:dtor: Deleting random engine with seed 650999119
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0