justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 299566.61@dunegpschedd01.fnal.gov

Jobsub ID299566.61@dunegpschedd01.fnal.gov
Workflow ID12285
Stage ID1
User namenivedpm@fnal.gov
RequestedProcessors4
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-23 00:37:39
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2026-01-23 02:13:49
From worker nodeHostnamewn-la-22.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors4
RSS bytes7864320000 (7500 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-23 01:01:19
Input filesmonte-carlo-012285-000030
JobscriptExit code0
Real time1h (4160s)
CPU time1h (4196s = 100%)
Max RSS bytes9430437888 (8993 MiB)
Outputting started2026-01-23 02:10:46
Output files
Finished2026-01-23 02:13:49
Saved logsjustin-logs:299566.61-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

iener"
FrameSaver: q=2.52944e+06 n=405696 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 41172 traces, 41172 summary
Retagger: tagging trace set: gauss with 38496 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 38496 traces tagged "gauss"
FrameSaver: q=2.97356e+06 n=544225 tag=gauss
wclsFrameSaver: saving 41172 traces tagged "wiener"
FrameSaver: q=3.22973e+06 n=491792 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 39499 traces, 39499 summary
Retagger: tagging trace set: gauss with 36885 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 36885 traces tagged "gauss"
FrameSaver: q=2.48116e+06 n=561833 tag=gauss
wclsFrameSaver: saving 39499 traces tagged "wiener"
FrameSaver: q=2.61475e+06 n=503077 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 31754 traces, 31754 summary
Retagger: tagging trace set: gauss with 31436 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 31436 traces tagged "gauss"
FrameSaver: q=2.24046e+06 n=463490 tag=gauss
wclsFrameSaver: saving 31754 traces tagged "wiener"
FrameSaver: q=2.41173e+06 n=427368 tag=wiener

Error in processing reco
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  hitpdune
  pandora
  pandora2Shower
  pandora2Track
  pandora2cali
  pandora2calipid
  pandora2pid
  pandoraShower
  pandoraTrack
  pandoraWriter
  pandoracalipid
  pandorapid
  reco3d
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/gdml/protodune_v7.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 36176 nodes/ 5435 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
23-Jan-2026 02:37:19 CET  Initiating request to open input file "prod_mpvmpr_protodunesp_1_299566_61_1769130126_gen_g4_stage1_detsim.root"
23-Jan-2026 02:37:20 CET  Opened input file "prod_mpvmpr_protodunesp_1_299566_61_1769130126_gen_g4_stage1_detsim.root"
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 45044 traces, 45044 summary
Retagger: tagging trace set: gauss with 40855 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 40855 traces tagged "gauss"
FrameSaver: q=3.06936e+06 n=628935 tag=gauss
wclsFrameSaver: saving 45044 traces tagged "wiener"
FrameSaver: q=3.26311e+06 n=561113 tag=wiener
23-Jan-2026 02:40:53 CET  Opened output file with pattern "prod_mpvmpr_protodunesp_1_299566_61_1769130126_gen_g4_stage1_detsim_reco.root"
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 57803 traces, 57803 summary
Retagger: tagging trace set: gauss with 48160 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 48160 traces tagged "gauss"
FrameSaver: q=3.95979e+06 n=722225 tag=gauss
wclsFrameSaver: saving 57803 traces tagged "wiener"
FrameSaver: q=4.30417e+06 n=635986 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 36889 traces, 36889 summary
Retagger: tagging trace set: gauss with 35210 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 35210 traces tagged "gauss"
FrameSaver: q=2.52864e+06 n=515187 tag=gauss
wclsFrameSaver: saving 36889 traces tagged "wiener"
FrameSaver: q=2.65301e+06 n=460783 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 36465 traces, 36465 summary
Retagger: tagging trace set: gauss with 35084 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 35084 traces tagged "gauss"
FrameSaver: q=2.90367e+06 n=544148 tag=gauss
wclsFrameSaver: saving 36465 traces tagged "wiener"
FrameSaver: q=3.14089e+06 n=498342 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 38673 traces, 38673 summary
Retagger: tagging trace set: gauss with 36608 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 36608 traces tagged "gauss"
FrameSaver: q=3.43292e+06 n=541921 tag=gauss
wclsFrameSaver: saving 38673 traces tagged "wiener"
FrameSaver: q=3.67463e+06 n=490105 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 28776 traces, 28776 summary
Retagger: tagging trace set: gauss with 28778 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 28778 traces tagged "gauss"
FrameSaver: q=1.54592e+06 n=395255 tag=gauss
wclsFrameSaver: saving 28776 traces tagged "wiener"
FrameSaver: q=1.63872e+06 n=356480 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 31461 traces, 31461 summary
Retagger: tagging trace set: gauss with 31635 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 31635 traces tagged "gauss"
FrameSaver: q=1.69587e+06 n=439130 tag=gauss
wclsFrameSaver: saving 31461 traces tagged "wiener"
FrameSaver: q=1.79343e+06 n=395005 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 36551 traces, 36551 summary
Retagger: tagging trace set: gauss with 33505 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 33505 traces tagged "gauss"
FrameSaver: q=2.19315e+06 n=446475 tag=gauss
wclsFrameSaver: saving 36551 traces tagged "wiener"
FrameSaver: q=2.38983e+06 n=395307 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 36497 traces, 36497 summary
Retagger: tagging trace set: gauss with 33958 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 33958 traces tagged "gauss"
FrameSaver: q=2.34621e+06 n=493559 tag=gauss
wclsFrameSaver: saving 36497 traces tagged "wiener"
FrameSaver: q=2.5562e+06 n=448551 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 43478 traces, 43478 summary
Retagger: tagging trace set: gauss with 36942 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 36942 traces tagged "gauss"
FrameSaver: q=2.27466e+06 n=473644 tag=gauss
wclsFrameSaver: saving 43478 traces tagged "wiener"
FrameSaver: q=2.52944e+06 n=405696 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 41172 traces, 41172 summary
Retagger: tagging trace set: gauss with 38496 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 38496 traces tagged "gauss"
FrameSaver: q=2.97356e+06 n=544225 tag=gauss
wclsFrameSaver: saving 41172 traces tagged "wiener"
FrameSaver: q=3.22973e+06 n=491792 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 39499 traces, 39499 summary
Retagger: tagging trace set: gauss with 36885 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 36885 traces tagged "gauss"
FrameSaver: q=2.48116e+06 n=561833 tag=gauss
wclsFrameSaver: saving 39499 traces tagged "wiener"
FrameSaver: q=2.61475e+06 n=503077 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 31754 traces, 31754 summary
Retagger: tagging trace set: gauss with 31436 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 31436 traces tagged "gauss"
FrameSaver: q=2.24046e+06 n=463490 tag=gauss
wclsFrameSaver: saving 31754 traces tagged "wiener"
FrameSaver: q=2.41173e+06 n=427368 tag=wiener

ls: cannot access *larcvpdsp*.root: No such file or directory
Output files:
 larcv files: 
formed
{
  "name": "",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_16_00d00",
    "core.data_tier": "root-tuple-virtual",
    "core.runs": null,
    "core.runs_subruns": null,
    "dune.config_file": "run_supera_pdunesp_mpvmpr_d.fcl",
    "dune.campaign": "sp-protodune-spine-reco",
    "core.start_time": 1769130123.0,
    "core.end_time": 1769134234.0
  }
}
justIN time: 2026-02-04 06:03:56 UTC       justIN version: 01.06.00