Jobsub ID 223384.136@dunegpschedd02.fnal.gov

Jobsub ID	223384.136@dunegpschedd02.fnal.gov
Workflow ID	8497
Stage ID	1
User name	epennacc@fnal.gov
HTCondor Group	group_dune.prod_mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-03 10:16:06
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2025-10-03 10:28:19
From worker node	Hostname	b9p07p7557.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	aborted
Started	2025-10-03 10:18:58
Input files	monte-carlo-008497-005299
Outputting started	2025-10-03 10:26:07
Output files	fardet-hd:prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299_g4_detsim.root
Finished	2025-10-03 10:28:19
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

::Gen::DepoTransform : 0.15 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.14 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.14 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.14 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.03 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.03 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: wcls::SimChannelSink : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.02 sec
[12:25:30.453] I [ timer  ] Timer: wcls::FrameSaver : 0.01 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::Drifter : 0 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 0 sec
[12:25:30.453] I [ timer  ] Timer: wcls::SimDepoSource : 0 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 0 sec
[12:25:30.453] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[12:25:30.453] I [ timer  ] Timer: Total node execution : 8.340000065043569 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 10th record. run: 8497 subRun: 0 event: 52990 at 03-Oct-2025 12:25:41 CEST
SimDepoSource got 2698 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 2698 depos spanning: [1, 1.00579]us
Retagger: tagging trace set: daq with 30720 traces, 0 summary
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 1.15 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.32 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.31 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.3 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.3 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.29 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.29 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.29 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.29 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.28 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.27 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.27 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.26 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.26 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.26 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.26 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.26 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.26 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.26 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.23 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.22 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.2 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.2 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.2 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.19 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.19 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.16 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.15 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.14 sec
[12:25:51.992] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.13 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.12 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.12 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.1 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 0.1 sec
[12:25:51.993] I [ timer  ] Timer: wcls::SimChannelSink : 0.09 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.07 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.06 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.06 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.05 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.05 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.04 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.04 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.04 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.04 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.03 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.03 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.03 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.03 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[12:25:51.993] I [ timer  ] Timer: wcls::FrameSaver : 0.02 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::Drifter : 0.02 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 0.02 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0.01 sec
[12:25:51.993] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 0.01 sec
[12:25:51.993] I [ timer  ] Timer: wcls::SimDepoSource : 0.01 sec
[12:25:51.993] I [ timer  ] Timer: Total node execution : 9.709999926388264 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
03-Oct-2025 12:26:03 CEST  Closed output file "prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299_g4_detsim.root"
03-Oct-2025 12:26:03 CEST  Closed input file "prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299_g4.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      15.677        18.2862       21.4168       18.2316       1.56614        10     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00057713    0.000869817   0.00149902    0.000735814   0.000261475      10     
simulate:tpcrawdecoder:WireCellToolkit          9.15895       11.5861       14.3828       11.6421       1.38288        10     
simulate:opdigi:OpDetDigitizerDUNE             0.0212153     0.0584707     0.0921845     0.0583988     0.0241634       10     
simulate:rns:RandomNumberSaver                 2.058e-05    6.77748e-05   0.00037439     3.551e-05    0.000102887      10     
[art]:TriggerResults:TriggerResultInserter     8.71e-06     1.78559e-05   5.4049e-05     1.48e-05     1.26149e-05      10     
end_path:out1:RootOutput                       2.94e-06     1.0513e-05     3.305e-05     6.675e-06    1.01872e-05      10     
end_path:out1:RootOutput(write)                 6.13359       6.64032       7.11268       6.67265      0.347014        10     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4738.59 MB
  Peak resident set size usage (VmHWM): 3039.4 MB
====================================================================================================
Art has completed and will exit with status 0.
written output file: prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299.root 99193
written output file: prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299_g4.root 1676029
written output file: prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299_g4_detsim.root 1798442730
written output file: prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299_g4_detsim.root.ext.json 1453
written output file: prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299_g4_detsim.root.json 1759
written output file: prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299_g4_detsim.root.temp.json 1759
written output file: prod_p_1-500MeV_dune10kt_1x2x6_20251003T101902Z_gen_005299_g4_detsim_hist.root 519