justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301705.1@dunegpschedd01.fnal.gov

Jobsub ID301705.1@dunegpschedd01.fnal.gov
Workflow ID12436
Stage ID1
User namenivedpm@fnal.gov
RequestedProcessors4
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-27 16:43:37
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-01-27 18:03:28
From worker nodeHostnamewn5916290.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors4
RSS bytes7864320000 (7500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-27 16:45:04
Input filesmonte-carlo-012436-001601
JobscriptExit code0
Real time1h (4613s)
CPU time1h (4564s = 98%)
Max RSS bytes5150404608 (4911 MiB)
Outputting started2026-01-27 18:01:59
Output files
Finished2026-01-27 18:03:28
Saved logsjustin-logs:301705.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tagged "wiener"
FrameSaver: q=3.7229e+06 n=576660 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 34149 traces, 34149 summary
Retagger: tagging trace set: gauss with 34441 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 34441 traces tagged "gauss"
FrameSaver: q=3.27588e+06 n=545660 tag=gauss
wclsFrameSaver: saving 34149 traces tagged "wiener"
FrameSaver: q=3.49121e+06 n=505634 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 34756 traces, 34756 summary
Retagger: tagging trace set: gauss with 35235 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 35235 traces tagged "gauss"
FrameSaver: q=2.7285e+06 n=571343 tag=gauss
wclsFrameSaver: saving 34756 traces tagged "wiener"
FrameSaver: q=2.85229e+06 n=530709 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 44970 traces, 44970 summary
Retagger: tagging trace set: gauss with 40853 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 40853 traces tagged "gauss"
FrameSaver: q=3.52493e+06 n=657963 tag=gauss
wclsFrameSaver: saving 44970 traces tagged "wiener"
FrameSaver: q=3.74868e+06 n=594481 tag=wiener

Error in processing reco
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  hitpdune
  pandora
  pandora2Shower
  pandora2Track
  pandora2cali
  pandora2calipid
  pandora2pid
  pandoraShower
  pandoraTrack
  pandoraWriter
  pandoracalipid
  pandorapid
  reco3d
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/gdml/protodune_v7.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 36176 nodes/ 5435 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
27-Jan-2026 17:27:17 GMT  Initiating request to open input file "prod_mpvmpr_protodunesp_1_301705_1_1769532320_gen_g4_stage1_detsim.root"
27-Jan-2026 17:27:18 GMT  Opened input file "prod_mpvmpr_protodunesp_1_301705_1_1769532320_gen_g4_stage1_detsim.root"
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 38769 traces, 38769 summary
Retagger: tagging trace set: gauss with 35098 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 35098 traces tagged "gauss"
FrameSaver: q=2.65765e+06 n=476449 tag=gauss
wclsFrameSaver: saving 38769 traces tagged "wiener"
FrameSaver: q=2.87496e+06 n=420148 tag=wiener
27-Jan-2026 17:29:30 GMT  Opened output file with pattern "prod_mpvmpr_protodunesp_1_301705_1_1769532320_gen_g4_stage1_detsim_reco.root"
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 44256 traces, 44256 summary
Retagger: tagging trace set: gauss with 38188 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 38188 traces tagged "gauss"
FrameSaver: q=2.58156e+06 n=531936 tag=gauss
wclsFrameSaver: saving 44256 traces tagged "wiener"
FrameSaver: q=2.81584e+06 n=463226 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 47069 traces, 47069 summary
Retagger: tagging trace set: gauss with 39742 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 39742 traces tagged "gauss"
FrameSaver: q=3.00168e+06 n=566490 tag=gauss
wclsFrameSaver: saving 47069 traces tagged "wiener"
FrameSaver: q=3.25494e+06 n=490563 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 43429 traces, 43429 summary
Retagger: tagging trace set: gauss with 37733 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 37733 traces tagged "gauss"
FrameSaver: q=2.47704e+06 n=514592 tag=gauss
wclsFrameSaver: saving 43429 traces tagged "wiener"
FrameSaver: q=2.64397e+06 n=439749 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 45531 traces, 45531 summary
Retagger: tagging trace set: gauss with 42665 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 42665 traces tagged "gauss"
FrameSaver: q=4.19416e+06 n=690972 tag=gauss
wclsFrameSaver: saving 45531 traces tagged "wiener"
FrameSaver: q=4.54198e+06 n=635218 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 41606 traces, 41606 summary
Retagger: tagging trace set: gauss with 36458 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 36458 traces tagged "gauss"
FrameSaver: q=2.58266e+06 n=510719 tag=gauss
wclsFrameSaver: saving 41606 traces tagged "wiener"
FrameSaver: q=2.7648e+06 n=439405 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 27308 traces, 27308 summary
Retagger: tagging trace set: gauss with 27572 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 27572 traces tagged "gauss"
FrameSaver: q=1.16086e+06 n=353074 tag=gauss
wclsFrameSaver: saving 27308 traces tagged "wiener"
FrameSaver: q=1.2451e+06 n=313810 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 43589 traces, 43589 summary
Retagger: tagging trace set: gauss with 40575 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 40575 traces tagged "gauss"
FrameSaver: q=3.44085e+06 n=609993 tag=gauss
wclsFrameSaver: saving 43589 traces tagged "wiener"
FrameSaver: q=3.69246e+06 n=546121 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 38947 traces, 38947 summary
Retagger: tagging trace set: gauss with 36344 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 36344 traces tagged "gauss"
FrameSaver: q=2.6369e+06 n=524447 tag=gauss
wclsFrameSaver: saving 38947 traces tagged "wiener"
FrameSaver: q=2.87358e+06 n=478958 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 46641 traces, 46641 summary
Retagger: tagging trace set: gauss with 41789 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 41789 traces tagged "gauss"
FrameSaver: q=3.47827e+06 n=647082 tag=gauss
wclsFrameSaver: saving 46641 traces tagged "wiener"
FrameSaver: q=3.7229e+06 n=576660 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 34149 traces, 34149 summary
Retagger: tagging trace set: gauss with 34441 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 34441 traces tagged "gauss"
FrameSaver: q=3.27588e+06 n=545660 tag=gauss
wclsFrameSaver: saving 34149 traces tagged "wiener"
FrameSaver: q=3.49121e+06 n=505634 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 34756 traces, 34756 summary
Retagger: tagging trace set: gauss with 35235 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 35235 traces tagged "gauss"
FrameSaver: q=2.7285e+06 n=571343 tag=gauss
wclsFrameSaver: saving 34756 traces tagged "wiener"
FrameSaver: q=2.85229e+06 n=530709 tag=wiener
RawFrameSource: got 15360 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 44970 traces, 44970 summary
Retagger: tagging trace set: gauss with 40853 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 40853 traces tagged "gauss"
FrameSaver: q=3.52493e+06 n=657963 tag=gauss
wclsFrameSaver: saving 44970 traces tagged "wiener"
FrameSaver: q=3.74868e+06 n=594481 tag=wiener

ls: cannot access *larcvpdsp*.root: No such file or directory
Output files:
 larcv files: 
formed
{
  "name": "",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_16_00d00",
    "core.data_tier": "root-tuple-virtual",
    "core.runs": null,
    "core.runs_subruns": null,
    "dune.config_file": "run_supera_pdunesp_mpvmpr_d.fcl",
    "dune.campaign": "sp-protodune-spine-reco",
    "core.start_time": 1769532318.0,
    "core.end_time": 1769536913.0
  }
}
justIN time: 2026-02-04 10:08:52 UTC       justIN version: 01.06.00