Jobsub ID 293531.0@dunegpschedd02.fnal.gov

Jobsub ID	293531.0@dunegpschedd02.fnal.gov
Workflow ID	12230
Stage ID	1
User name	muditk@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	6291456000 (6000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-22 11:32:57
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2026-01-22 11:52:19
From worker node	Hostname	dunegli-8241032-0-fnpc22013.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	7864320000 (7500 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2026-01-22 11:34:35
Input files	fardet-hd:prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2.root
Jobscript	Exit code	0
	Real time	17m (1056s)
	CPU time	17m (1043s = 98%)
	Max RSS bytes	4296912896 (4097 MiB)
Outputting started	2026-01-22 11:52:12
Output files
Finished	2026-01-22 11:52:19
Saved logs	justin-logs:293531.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

: WireCell::Gen::Digitizer : 2.1299992 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.1299992 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.109999 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.109999 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.0699992 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.059999 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.039999 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.9499991 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.9299991 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.9099991 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.8399992 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.7199988 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 1.7099987 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.6599989 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.6499989 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.6199988 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.569999 sec
[11:48:30.217] I [ timer  ] Timer: wcls::FrameSaver : 1.5599989 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.549999 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.509999 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.4799991 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.4299991 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.4199991 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.369999 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Retagger : 1.3499991 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 1.3399991 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.3199992 sec
[11:48:30.217] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 1.2999991 sec
[11:48:30.217] I [ timer  ] Timer: wcls::SimDepoSource : 1.2299992 sec
[11:48:30.217] I [ timer  ] Timer: Total node execution : 690.639589548111 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 5th record. run: 5082 subRun: 0 event: 227905 at 22-Jan-2026 11:48:38 UTC
SimDepoSource got 141617 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 141617 depos spanning: [-2245.93, 8.61693e+22]us
Retagger: tagging trace set: daq with 30720 traces, 0 summary
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 72.46993 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 71.60991 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 69.87989 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 67.9499 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 66.02991 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 65.7899 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 65.11991 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 62.989906 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 56.13989 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 55.369892 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 52.3799 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 51.69997 sec
[11:52:02.954] I [ timer  ] Timer: wcls::SimChannelSink : 8.200092 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Drifter : 3.9499967 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 3.8999968 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 3.0299985 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 3.0199986 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.9199986 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8899987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8699987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8599987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8599987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.8599987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.7799988 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.7699988 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.7599986 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.6899986 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.6799989 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.6699986 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.629999 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Digitizer : 2.629999 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.6299987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.5799987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.5599988 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.5399988 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.4399989 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.409999 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.309999 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 2.279999 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.1199985 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.1099985 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 2.0899985 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 2.0199986 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.9699986 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.9699985 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.9299986 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8899987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8899987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8399987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.8299987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.7999988 sec
[11:52:02.954] I [ timer  ] Timer: wcls::FrameSaver : 1.7699987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Reframer : 1.7699987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::Retagger : 1.7599987 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 1.6599988 sec
[11:52:02.954] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 1.5899988 sec
[11:52:02.954] I [ timer  ] Timer: wcls::SimDepoSource : 1.5199989 sec
[11:52:02.954] I [ timer  ] Timer: Total node execution : 871.6789408922195 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
22-Jan-2026 11:52:12 UTC  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2_detsim_2026-01-22T_113438Z.root"
22-Jan-2026 11:52:12 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/8c/9e/prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_centralAPA_20250206T145355Z_gen_004559_supernova_g4_detsim_reco_20250724T031052Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      194.911       205.071       212.242       206.529       6.24389         5     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000831634   0.00450125    0.00647125    0.00508593    0.00205613        5     
simulate:tpcrawdecoder:WireCellToolkit          188.169       198.236       205.348       199.433       6.17472         5     
simulate:rns:RandomNumberSaver                5.9913e-05     0.000171     0.000401169   6.1356e-05    0.00014127        5     
[art]:TriggerResults:TriggerResultInserter    2.2473e-05    2.98166e-05   4.8452e-05     2.625e-05    9.43873e-06       5     
end_path:out1:RootOutput                       4.338e-06    7.2358e-06    1.7263e-05     4.899e-06    5.01916e-06       5     
end_path:out1:RootOutput(write)                 6.64038       6.83003       7.09429       6.78907      0.154442         5     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5921.03 MB
  Peak resident set size usage (VmHWM): 4296.91 MB
====================================================================================================
ProvidedPedestalAdditionService::dtor: Deleting random engine with seed 263079513
ExponentialChannelNoiseService::dtor: Deleting random engine with seed 507255280
StuckBitAdcDistortionService:dtor: Deleting random engine with seed 464194836
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0