justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 213707.124@dunegpschedd01.fnal.gov

Jobsub ID213707.124@dunegpschedd01.fnal.gov
Workflow ID8146
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-09-27 13:06:49
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce02
Last heartbeat2025-09-27 15:18:24
From worker nodeHostnamedune004-8295208.0-lcg2642.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-09-27 13:08:50
Input filesmonte-carlo-008146-008914
JobscriptExit code0
Real time1h (3856s)
CPU time1h (3794s = 98%)
Max RSS bytes3487875072 (3326 MiB)
Outputting started2025-09-27 14:14:02
Output files
Finished2025-09-27 15:18:24
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.75 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.75 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.74 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 0.69 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.68 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.68 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.67 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.67 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.66 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.66 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.65 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.65 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.62 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.62 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.61 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.61 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.59 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.58 sec
[14:07:36.488] I [ timer  ] Timer: wcls::FrameSaver : 0.58 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.56 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.56 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0.55 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.51 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 0.49 sec
[14:07:36.488] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.48 sec
[14:07:36.488] I [ timer  ] Timer: wcls::SimDepoSource : 0.17 sec
[14:07:36.488] I [ timer  ] Timer: Total node execution : 256.5100025087595 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 10th record. run: 8146 subRun: 0 event: 89140 at 27-Sep-2025 14:07:54 UTC
SimDepoSource got 138678 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 138678 depos spanning: [-2245.57, 2643.2]us
Retagger: tagging trace set: daq with 30720 traces, 0 summary
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 23.84 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 21.3 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 21.27 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 18.6 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 17.81 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 17.74 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 17.48 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 17.29 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 17.18 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 17 sec
[14:12:43.434] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 16.3 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::DepoTransform : 15.61 sec
[14:12:43.435] I [ timer  ] Timer: wcls::SimChannelSink : 2.63 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Drifter : 1.1 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.9 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.89 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.86 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.85 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.85 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.84 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.84 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.83 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.83 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.83 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.82 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.82 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.81 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.78 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.77 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.77 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.76 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.76 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.75 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.75 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.73 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::DepoBagger : 0.71 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.71 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Digitizer : 0.7 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.69 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.69 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::IncoherentAddNoise : 0.69 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.69 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.68 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.67 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.62 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0.62 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.62 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.59 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.59 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.58 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.55 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.54 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Reframer : 0.46 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.44 sec
[14:12:43.435] I [ timer  ] Timer: wcls::FrameSaver : 0.43 sec
[14:12:43.435] I [ timer  ] Timer: WireCell::Gen::DepoSetFanout : 0.41 sec
[14:12:43.435] I [ timer  ] Timer: wcls::SimDepoSource : 0.14 sec
[14:12:43.435] I [ timer  ] Timer: Total node execution : 255.0099990516901 sec
wclsFrameSaver: saving raw::RawDigits tagged "daq"
27-Sep-2025 14:13:02 UTC  Closed output file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T130856Z_gen_008914_supernova_g4_detsim.root"
27-Sep-2025 14:13:02 UTC  Closed input file "prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T130856Z_gen_008914_supernova_g4.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      264.769       300.992       318.634       306.34        15.401         10     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00177395    0.00558004    0.00805109    0.00581459     0.0018316       10     
simulate:tpcrawdecoder:WireCellToolkit          250.642       287.84        307.447       293.124       15.8008        10     
simulate:opdigi:OpDetDigitizerDUNE              2.82037       3.30907       3.6114        3.39019      0.259612        10     
simulate:rns:RandomNumberSaver                4.7141e-05    0.000100551   0.000517316   5.2597e-05    0.000139068      10     
[art]:TriggerResults:TriggerResultInserter     1.524e-05    2.56288e-05   7.4372e-05    2.07355e-05   1.67004e-05      10     
end_path:out1:RootOutput                       5.91e-06     1.08482e-05   4.0911e-05    7.7305e-06    1.00614e-05      10     
end_path:out1:RootOutput(write)                 7.29796       9.83734       11.1682       10.5031       1.18892        10     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5302.21 MB
  Peak resident set size usage (VmHWM): 3487.88 MB
====================================================================================================
Art has completed and will exit with status 0.
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T130856Z_gen_008914.root 21973625
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T130856Z_gen_008914_supernova_g4.root 179952412
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T130856Z_gen_008914_supernova_g4_detsim.root 2350859171
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T130856Z_gen_008914_supernova_g4_detsim.root.ext.json 1561
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T130856Z_gen_008914_supernova_g4_detsim.root.json 1827
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T130856Z_gen_008914_supernova_g4_detsim.root.temp.json 1827
written output file: prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T130856Z_gen_008914_supernova_g4_detsim_hist.root 519
justIN time: 2025-11-03 18:40:25 UTC       justIN version: 01.05.01