justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 208680.21@dunegpschedd02.fnal.gov

Jobsub ID208680.21@dunegpschedd02.fnal.gov
Workflow ID8191
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-09-28 22:55:38
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-09-29 00:37:16
From worker nodeHostnamewn65
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-09-28 22:57:41
Input filesmonte-carlo-008191-009104
Outputting started2025-09-29 00:20:36
Output files
Finished2025-09-29 00:37:16
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.89 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.87 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.87 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.85 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.84 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.83 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 0.81 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.81 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 0.8 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.79 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.79 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.76 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.74 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.7 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.7 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.69 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.68 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.67 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.66 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.65 sec
[21:14:02.183] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.65 sec
[21:14:02.184] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.65 sec
[21:14:02.184] I [ timer  ] Timer: wcls::FrameSaver : 0.64 sec
[21:14:02.184] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.62 sec
[21:14:02.184] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0.56 sec
[21:14:02.184] I [ timer  ] Timer: wcls::SimDepoSource : 0.2 sec
[21:14:02.184] I [ timer  ] Timer: Total node execution : 287.35000009834766 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 10th record. run: 8191 subRun: 0 event: 91040 at 28-Sep-2025 21:14:32 -03
SimDepoSource got 137424 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 137424 depos spanning: [-2245.97, 2245.58]us
Retagger: tagging trace set: daq with 30720 traces, 0 summary
[21:20:00.876] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 27.12 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 23.77 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 23.76 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 22.59 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 22.13 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 21.2 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 19.82 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 18.88 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 18.49 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 17.23 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 16.18 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 13.37 sec
[21:20:00.877] I [ timer  ] Timer: wcls::SimChannelSink : 3.25 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Drifter : 1.09 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.05 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1.03 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 1.02 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 1 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.99 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.98 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.97 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.97 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.97 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.96 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.95 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.93 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.92 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.91 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.9 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.9 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.89 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.88 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.88 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.88 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.88 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.88 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 0.87 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.84 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.83 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.82 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.81 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.77 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.76 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.75 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.72 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 0.7 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.68 sec
[21:20:00.877] I [ timer  ] Timer: wcls::FrameSaver : 0.65 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.64 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.64 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0.62 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.62 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.61 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.6 sec
[21:20:00.877] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.54 sec
[21:20:00.877] I [ timer  ] Timer: wcls::SimDepoSource : 0.26 sec
[21:20:00.877] I [ timer  ] Timer: Total node execution : 284.34999990463257 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
28-Sep-2025 21:20:29 -03  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T225748Z_gen_009104_supernova_g4_detsim.root"
28-Sep-2025 21:20:29 -03  Closed input file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T225748Z_gen_009104_supernova_g4.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      349.551       363.951       387.46        357.37        12.2275        10     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00103584    0.00558866     0.0089094    0.00565679    0.00192554       10     
simulate:tpcrawdecoder:WireCellToolkit          317.848       336.705       356.312       333.658       12.133         10     
simulate:opdigi:OpDetDigitizerDUNE              3.77847       4.03342       4.23595        4.065       0.121774        10     
simulate:rns:RandomNumberSaver                2.8341e-05    6.08913e-05   0.000334593   3.0936e-05    9.12486e-05      10     
[art]:TriggerResults:TriggerResultInserter     1.525e-05    2.06738e-05   5.6683e-05    1.6566e-05    1.20467e-05      10     
end_path:out1:RootOutput                       3.24e-06     5.8341e-06     2.269e-05     4.035e-06    5.63051e-06      10     
end_path:out1:RootOutput(write)                 17.2399       23.2067       27.7991       23.6772       3.74517        10     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5176.63 MB
  Peak resident set size usage (VmHWM): 3511.98 MB
====================================================================================================
Art has completed and will exit with status 0.
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T225748Z_gen_009104.root 21976045
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T225748Z_gen_009104_supernova_g4.root 180450590
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T225748Z_gen_009104_supernova_g4_detsim.root 2352966248
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T225748Z_gen_009104_supernova_g4_detsim.root.ext.json 1561
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T225748Z_gen_009104_supernova_g4_detsim.root.json 1827
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T225748Z_gen_009104_supernova_g4_detsim.root.temp.json 1827
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T225748Z_gen_009104_supernova_g4_detsim_hist.root 519
justIN time: 2025-11-03 18:44:28 UTC       justIN version: 01.05.01