justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 273596.0@dunegpschedd02.fnal.gov

Jobsub ID273596.0@dunegpschedd02.fnal.gov
Workflow ID11917
Stage ID1
User namenivedpm@fnal.gov
RequestedProcessors4
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-16 09:05:07
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2026-01-16 09:49:48
From worker nodeHostnamewn-f01-15-06-02.cr.cnaf.infn.it
cpuinfoIntel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors4
RSS bytes7864320000 (7500 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-16 09:05:38
Input filesmonte-carlo-011917-000554
JobscriptExit code0
Real time43m (2619s)
CPU time42m (2533s = 96%)
Max RSS bytes1854148608 (1768 MiB)
Outputting started2026-01-16 09:49:18
Output files
Finished2026-01-16 09:49:48
Saved logsjustin-logs:273596.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gauss"
FrameSaver: q=5.33104e+06 n=421173 tag=gauss
wclsFrameSaver: saving 34787 traces tagged "wiener"
FrameSaver: q=5.72257e+06 n=379874 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 19981 traces, 19981 summary
Retagger: tagging trace set: gauss with 15011 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 15011 traces tagged "gauss"
FrameSaver: q=3.0646e+06 n=241077 tag=gauss
wclsFrameSaver: saving 19981 traces tagged "wiener"
FrameSaver: q=3.29252e+06 n=220004 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 17340 traces, 17340 summary
Retagger: tagging trace set: gauss with 13857 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 13857 traces tagged "gauss"
FrameSaver: q=2.57552e+06 n=241507 tag=gauss
wclsFrameSaver: saving 17340 traces tagged "wiener"
FrameSaver: q=2.7378e+06 n=227822 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 33866 traces, 33866 summary
Retagger: tagging trace set: gauss with 24013 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 24013 traces tagged "gauss"
FrameSaver: q=6.23743e+06 n=424784 tag=gauss
wclsFrameSaver: saving 33866 traces tagged "wiener"
FrameSaver: q=6.68874e+06 n=384066 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 22414 traces, 22414 summary
Retagger: tagging trace set: gauss with 17588 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 17588 traces tagged "gauss"
FrameSaver: q=3.40187e+06 n=292860 tag=gauss
wclsFrameSaver: saving 22414 traces tagged "wiener"
FrameSaver: q=3.59551e+06 n=269336 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 20236 traces, 20236 summary
Retagger: tagging trace set: gauss with 16274 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 16274 traces tagged "gauss"
FrameSaver: q=2.94512e+06 n=259921 tag=gauss
wclsFrameSaver: saving 20236 traces tagged "wiener"
FrameSaver: q=3.162e+06 n=244969 tag=wiener

Error in processing reco
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  emtrkmichelid
  nhitsfilter
  trigfilter
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/gdml/protodunevd_v5_ggd.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16484 nodes/ 2469 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:spsaver"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
16-Jan-2026 10:30:27 CET  Initiating request to open input file "prod_muon_400MeV_protodunevd_1_273596_0_1768554385_gen_g4_stage1_g4_stage2_detsim.root"
16-Jan-2026 10:30:28 CET  Opened input file "prod_muon_400MeV_protodunevd_1_273596_0_1768554385_gen_g4_stage1_g4_stage2_detsim.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 18609 traces, 18609 summary
Retagger: tagging trace set: gauss with 15066 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 15066 traces tagged "gauss"
FrameSaver: q=3.15947e+06 n=267336 tag=gauss
wclsFrameSaver: saving 18609 traces tagged "wiener"
FrameSaver: q=3.34743e+06 n=251673 tag=wiener
16-Jan-2026 10:32:03 CET  Opened output file with pattern "prod_muon_400MeV_protodunevd_1_273596_0_1768554385_gen_g4_stage1_g4_stage2_detsim_reco.root"
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 15776 traces, 15776 summary
Retagger: tagging trace set: gauss with 12130 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 12130 traces tagged "gauss"
FrameSaver: q=2.16996e+06 n=200177 tag=gauss
wclsFrameSaver: saving 15776 traces tagged "wiener"
FrameSaver: q=2.35417e+06 n=186245 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 23941 traces, 23941 summary
Retagger: tagging trace set: gauss with 18329 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 18329 traces tagged "gauss"
FrameSaver: q=3.77323e+06 n=287820 tag=gauss
wclsFrameSaver: saving 23941 traces tagged "wiener"
FrameSaver: q=4.14116e+06 n=259753 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 28213 traces, 28213 summary
Retagger: tagging trace set: gauss with 19530 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 19530 traces tagged "gauss"
FrameSaver: q=6.61878e+06 n=354155 tag=gauss
wclsFrameSaver: saving 28213 traces tagged "wiener"
FrameSaver: q=7.24031e+06 n=322304 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 20902 traces, 20902 summary
Retagger: tagging trace set: gauss with 17120 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 17120 traces tagged "gauss"
FrameSaver: q=3.58217e+06 n=280187 tag=gauss
wclsFrameSaver: saving 20902 traces tagged "wiener"
FrameSaver: q=3.78479e+06 n=260240 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 34787 traces, 34787 summary
Retagger: tagging trace set: gauss with 25333 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 25333 traces tagged "gauss"
FrameSaver: q=5.33104e+06 n=421173 tag=gauss
wclsFrameSaver: saving 34787 traces tagged "wiener"
FrameSaver: q=5.72257e+06 n=379874 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 19981 traces, 19981 summary
Retagger: tagging trace set: gauss with 15011 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 15011 traces tagged "gauss"
FrameSaver: q=3.0646e+06 n=241077 tag=gauss
wclsFrameSaver: saving 19981 traces tagged "wiener"
FrameSaver: q=3.29252e+06 n=220004 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 17340 traces, 17340 summary
Retagger: tagging trace set: gauss with 13857 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 13857 traces tagged "gauss"
FrameSaver: q=2.57552e+06 n=241507 tag=gauss
wclsFrameSaver: saving 17340 traces tagged "wiener"
FrameSaver: q=2.7378e+06 n=227822 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 33866 traces, 33866 summary
Retagger: tagging trace set: gauss with 24013 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 24013 traces tagged "gauss"
FrameSaver: q=6.23743e+06 n=424784 tag=gauss
wclsFrameSaver: saving 33866 traces tagged "wiener"
FrameSaver: q=6.68874e+06 n=384066 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 22414 traces, 22414 summary
Retagger: tagging trace set: gauss with 17588 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 17588 traces tagged "gauss"
FrameSaver: q=3.40187e+06 n=292860 tag=gauss
wclsFrameSaver: saving 22414 traces tagged "wiener"
FrameSaver: q=3.59551e+06 n=269336 tag=wiener
RawFrameSource: got 12288 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 20236 traces, 20236 summary
Retagger: tagging trace set: gauss with 16274 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 16274 traces tagged "gauss"
FrameSaver: q=2.94512e+06 n=259921 tag=gauss
wclsFrameSaver: saving 20236 traces tagged "wiener"
FrameSaver: q=3.162e+06 n=244969 tag=wiener

ls: cannot access *larcvpdvd*.root: No such file or directory
Output files:
 larcv files: 
formed
{
  "name": "",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_16_00d00",
    "core.data_tier": "root-tuple-virtual",
    "core.runs": null,
    "core.runs_subruns": null,
    "dune.config_file": "run_supera_pdunevd_mpvmpr_doublets.fcl",
    "dune.campaign": "vd-protodune-spine-reco",
    "core.start_time": 1768554377.0,
    "core.end_time": 1768556951.0
  }
}
justIN time: 2026-02-04 06:23:48 UTC       justIN version: 01.06.00