justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 296290.1@dunegpschedd02.fnal.gov

Jobsub ID296290.1@dunegpschedd02.fnal.gov
Workflow ID12474
Stage ID1
User namenivedpm@fnal.gov
RequestedProcessors4
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2026-01-28 11:48:37
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2026-01-28 11:50:21
From worker nodeHostnamedunegli-8464114-0-fnpc22004.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors4
RSS bytes7864320000 (7500 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2026-01-28 11:50:01
Input filesmonte-carlo-012474-000001
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2026-01-28 11:50:21
Saved logsjustin-logs:296290.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

id.org/products/dune/dunepdlegacy/v1_01_04/slf7.x86_64.e26.prof
DUNEEXAMPLES_INC=/cvmfs/dune.opensciencegrid.org/products/dune/duneexamples/v10_16_00d00/include
DUNEANAOBJ_VERSION=v03_11_00
DUNE_PARDATA_VERSION=v0_02_00
DUNEANAOBJ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneanaobj/v03_11_00
DUNEDAQDATAFORMATS_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunedaqdataformats/v4_4_5
SETUP_DUNERECO=dunereco v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
PROTODUNEANA_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_16_00d00
DUNEDATAPREP_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v10_16_00d00/include
DUNESW_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunesw/v10_16_00d00/slf7.x86_64.e26.prof
SETUP_DUNEUTIL=duneutil v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNESW_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunesw/v10_16_00d00
DUNE_PARDATA_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dune_pardata/v01_84_00
DUNETRIGGER_VERSION=v10_16_00d00
DUNESIM_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v10_16_00d00
DUNEOPDET_VERSION=v10_16_00d00
SETUP_DUNE_PARDATA=dune_pardata v01_84_00 -f NULL -z /cvmfs/dune.opensciencegrid.org/products/dune
DUNERECO_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_16_00d00/slf7.x86_64.e26.prof
DUNEEXAMPLES_VERSION=v10_16_00d00
DUNERECO_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_16_00d00/include
DUNEUTIL_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_16_00d00
DUNEPDLEGACY_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_01_04
DUNECORE_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00
DUNECALIB_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunecalib/v10_16_00d00/slf7.x86_64.e26.prof
DUNEANA_VERSION=v10_16_00d00
DUNETRIGGER_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunetrigger/v10_16_00d00
DUNEPROTOTYPES_VERSION=v10_16_00d00
DUNEOPDET_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneopdet/v10_16_00d00
DUNEPROTOTYPES_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v10_16_00d00
SETUP_DUNEDATAPREP=dunedataprep v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNETRIGGER_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunetrigger/v10_16_00d00/slf7.x86_64.e26.prof
DUNEOPDET_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneopdet/v10_16_00d00/slf7.x86_64.e26.prof
PROTODUNEANA_VERSION=v10_16_00d00
DUNEEXAMPLES_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneexamples/v10_16_00d00/slf7.x86_64.e26.prof
SETUP_DUNEPROTOTYPES=duneprototypes v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNEDATAPREP_VERSION=v10_16_00d00
DUNECORE_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/include
DUNEPDLEGACY_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_01_04/include
SETUP_DUNEANA=duneana v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNEPROTOTYPES_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v10_16_00d00/slf7.x86_64.e26.prof/lib
DUNEDETDATAFORMATS_VERSION=v4_4_5
DUNEUTIL_VERSION=v10_16_00d00
DUNESW_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunesw/v10_16_00d00/slf7.x86_64.e26.prof/lib
SETUP_PROTODUNEANA=protoduneana v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
SETUP_DUNESIM=dunesim v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
PROTODUNEANA_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_16_00d00/slf7.x86_64.e26.prof
DUNESIM_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v10_16_00d00/slf7.x86_64.e26.prof/lib
SETUP_DUNEOPDET=duneopdet v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNESIM_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v10_16_00d00/slf7.x86_64.e26.prof
SETUP_DUNEPDLEGACY=dunepdlegacy v1_01_04 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNECALIB_VERSION=v10_16_00d00
SETUP_DUNEDAQDATAFORMATS=dunedaqdataformats v4_4_5 -f NULL -z /cvmfs/dune.opensciencegrid.org/products/dune
DUNEEXAMPLES_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneexamples/v10_16_00d00
DUNECORE_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/slf7.x86_64.e26.prof/lib
DUNEPDLEGACY_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_01_04/slf7.x86_64.e26.prof/lib
DUNEANA_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneana/v10_16_00d00/slf7.x86_64.e26.prof/lib
SETUP_DUNETRIGGER=dunetrigger v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
SETUP_DUNECORE=dunecore v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
SETUP_DUNEANAOBJ=duneanaobj v03_11_00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNETRIGGER_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunetrigger/v10_16_00d00/slf7.x86_64.e26.prof/lib
DUNESIM_VERSION=v10_16_00d00
DUNECALIB_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunecalib/v10_16_00d00/include
DUNEOPDET_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneopdet/v10_16_00d00/slf7.x86_64.e26.prof/lib
DUNEUTIL_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_16_00d00/slf7.x86_64.e26.prof
DUNEANAOBJ_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneanaobj/v03_11_00/slf7.x86_64.e26.prof/lib
DUNEPROTOTYPES_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v10_16_00d00/slf7.x86_64.e26.prof
PROTODUNEANA_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_16_00d00/slf7.x86_64.e26.prof/lib
DUNEPROTOTYPES_INC=/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v10_16_00d00/include
DUNEEXAMPLES_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneexamples/v10_16_00d00/slf7.x86_64.e26.prof/lib
DUNESW_TAR=/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/99144ae8ad88d5f6cd320eace484d0ff71ca6dc1
SETUP_DUNECALIB=dunecalib v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNERECO_VERSION=v10_16_00d00
DUNEDETDATAFORMATS_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunedetdataformats/v4_4_5/include
SETUP_DUNESW=dunesw v10_16_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
PROTODUNEANA_INC=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_16_00d00/include
DUNEDATAPREP_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v10_16_00d00/slf7.x86_64.e26.prof/lib
DUNERECO_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_16_00d00
Will use justin-get-file
input file: monte-carlo-012474-000001
jobsub_id: 296290.1@dunegpschedd02.fnal.gov
Getting run subrun from monte-carlo-012474-000001
Traceback (most recent call last):
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/f322c8172ea6a33d03aca9d395dc03dc5dcf3c88/beam_job_utils.py", line 369, in <module>
    routines[args.routine](args)
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/f322c8172ea6a33d03aca9d395dc03dc5dcf3c88/beam_job_utils.py", line 46, in get_run_subrun
    md = mc.get_file(did=args.i, with_metadata=True)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1251, in get_file
    namespace, name = parse_name(did, None)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 11, in parse_name
    assert not not default_namespace, "Null default namespace"
AssertionError: Null default namespace
error in get_run_subrun

nevents: 1
input flag: -i 000001 --parent monte-carlo-012474-000001
usage: beam_job_utils.py [-h] [-i I] [--yaml YAML] [-o O] [--json JSON]
                         [--run RUN] [--subrun SUBRUN]
                         [--exclude EXCLUDE [EXCLUDE ...]]
                         [--overrides OVERRIDES [OVERRIDES ...]]
                         [--parent PARENT] [--event EVENT] [--nevents NEVENTS]
                         [--past_fcls PAST_FCLS [PAST_FCLS ...]]
                         [--past_apps PAST_APPS [PAST_APPS ...]]
                         [--past_vers PAST_VERS [PAST_VERS ...]]
                         {get_run_subrun,make_metadata,run_job,get_artroot_nevents}
beam_job_utils.py: error: argument --run: expected one argument
ls: cannot access *larcvpdvd*.root: No such file or directory
Output files:
 larcv files: 
grep: /cvmfs/fifeuser4.opensciencegrid.org/sw/dune/f322c8172ea6a33d03aca9d395dc03dc5dcf3c88/: Is a directory
usage: meta_maker.py [-h] --file FILE [--get_events] [--start_time START_TIME]
                     [--end_time END_TIME] [--file_format FILE_FORMAT]
                     [--app_family APP_FAMILY] [--app_name APP_NAME]
                     [--app_version APP_VERSION] [--data_tier DATA_TIER]
                     [--fcl FCL] [--campaign CAMPAIGN] [--runs RUNS]
                     [--subruns SUBRUNS]
                     [--past_fcls PAST_FCLS [PAST_FCLS ...]]
                     [--past_apps PAST_APPS [PAST_APPS ...]] [--parent PARENT]
                     [--parent_as_json] --json JSON
meta_maker.py: error: argument --fcl: expected one argument
Error in larcv metadata
justIN time: 2026-02-04 19:35:54 UTC       justIN version: 01.06.00