Jobsub ID 293531.4@dunegpschedd02.fnal.gov

Jobsub ID	293531.4@dunegpschedd02.fnal.gov
Workflow ID	12230
Stage ID	1
User name	muditk@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	6291456000 (6000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-22 11:32:57
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2026-01-22 11:53:06
From worker node	Hostname	dunegli-8241032-0-fnpc22013.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	7864320000 (7500 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-22 11:34:15
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T120419Z_gen_002038_supernova_g4_detsim_reco_20250724T135820Z_reco2.root
Jobscript	Exit code	0
	Real time	18m (1123s)
	CPU time	18m (1108s = 98%)
	Max RSS bytes	4292182016 (4093 MiB)
Outputting started	2026-01-22 11:52:59
Output files
Finished	2026-01-22 11:53:06
Saved logs	justin-logs:293531.4-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

WireCell::Gen::IncoherentAddNoise : 2.129999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.129999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.099999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.0899992 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.049999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.039999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.0299993 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.969999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.9499991 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.909999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.8999991 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8399988 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 1.8099986 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.7599988 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.7399988 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.599999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.529999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.509999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.509999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.499999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.4799991 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 1.439999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.4199991 sec
[11:49:11.470] I [ timer  ] Timer: wcls::FrameSaver : 1.419999 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.4099991 sec
[11:49:11.470] I [ timer  ] Timer: wcls::SimDepoSource : 1.3499991 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Retagger : 1.3399991 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 1.3199991 sec
[11:49:11.470] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.3099992 sec
[11:49:11.470] I [ timer  ] Timer: Total node execution : 748.3294578790665 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 5th record. run: 5097 subRun: 0 event: 101855 at 22-Jan-2026 11:49:20 UTC
SimDepoSource got 139818 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 139818 depos spanning: [-2245.29, 2244.77]us
Retagger: tagging trace set: daq with 30720 traces, 0 summary
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 76.74993 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 75.10996 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 74.59989 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 74.56987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 73.149895 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 73.12986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 65.9899 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 65.73991 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 64.85989 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 61.159893 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 59.369926 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 55.58996 sec
[11:52:49.965] I [ timer  ] Timer: wcls::SimChannelSink : 8.750105 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Drifter : 3.8499968 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 3.569997 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 3.1299987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.9899988 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.9599986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8799987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8699987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8699987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8599987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8099988 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.7999988 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.7599986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.7399988 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.7099986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.6999989 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.669999 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.6699986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.6299987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.6099987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.569999 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.5499988 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.5199988 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.4999988 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.4299989 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.419999 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.389999 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.2499986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 2.1999984 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.1999984 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.0199986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.0199986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.9399986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.9399986 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8999987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8899987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8699987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8599987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Retagger : 1.7699987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.7599988 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 1.7399987 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.6899989 sec
[11:52:49.965] I [ timer  ] Timer: wcls::FrameSaver : 1.6899987 sec
[11:52:49.965] I [ timer  ] Timer: wcls::SimDepoSource : 1.6599988 sec
[11:52:49.965] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 1.6299988 sec
[11:52:49.965] I [ timer  ] Timer: Total node execution : 935.2589312791824 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
22-Jan-2026 11:52:59 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T120419Z_gen_002038_supernova_g4_detsim_reco_20250724T135820Z_reco2_detsim_2026-01-22T_113418Z.root"
22-Jan-2026 11:52:59 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/64/c7/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250208T120419Z_gen_002038_supernova_g4_detsim_reco_20250724T135820Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      211.113       218.068       230.447       217.782       7.03158         5     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000932936   0.00734019     0.0171421    0.00610179    0.00532282        5     
simulate:tpcrawdecoder:WireCellToolkit          204.135       211.041       223.131       210.789       6.91848         5     
simulate:rns:RandomNumberSaver                5.5145e-05    0.000143674   0.000312271   7.0343e-05    0.000104343       5     
[art]:TriggerResults:TriggerResultInserter    2.2222e-05    4.53176e-05   0.000109256   2.4797e-05    3.32827e-05       5     
end_path:out1:RootOutput                       6.622e-06    9.3494e-06    1.9887e-05     6.743e-06    5.26937e-06       5     
end_path:out1:RootOutput(write)                 6.86153       7.01898       7.31359       6.97454       0.15354         5     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5914.39 MB
  Peak resident set size usage (VmHWM): 4292.18 MB
====================================================================================================
ProvidedPedestalAdditionService::dtor: Deleting random engine with seed 629287847
ExponentialChannelNoiseService::dtor: Deleting random engine with seed 99631793
StuckBitAdcDistortionService:dtor: Deleting random engine with seed 500575429
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0