justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 226222.118@dunegpschedd02.fnal.gov

Jobsub ID226222.118@dunegpschedd02.fnal.gov
Workflow ID8696
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes6291456000 (6000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-08 16:32:44
SiteUK_Glasgow
EntryCLAS12_T3_UK_ScotGrid_GLA_ce04_scitok
Last heartbeat2025-10-08 18:29:26
From worker nodeHostnamewn-d20-003.beowulf.cluster
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes7864320000 (7500 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-10-08 18:25:18
Input filesmonte-carlo-008696-001742
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-10-08 18:29:26
Saved logsjustin-logs:226222.118-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

/products/dune/dunedetdataformats/v4_4_5/include
SETUP_DUNESW=dunesw v10_05_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
PROTODUNEANA_INC=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_05_00d00/include
DUNEDATAPREP_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v10_05_00d00/slf7.x86_64.e26.prof/lib
DUNERECO_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_05_00d00
Installing yaml
Collecting pyyaml
  Downloading pyyaml-6.0.3-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (750 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/cachecontrol/filewrapper.py", line 62, in read
    data = self.__fp.read(amt)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/http/client.py", line 463, in read
    n = self.readinto(b)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/http/client.py", line 507, in readinto
    n = self.fp.readinto(b)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/ssl.py", line 1242, in recv_into
    return self.read(nbytes, buffer)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/ssl.py", line 1100, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/cli/base_command.py", line 173, in _main
    status = self.run(options, args)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/cli/req_command.py", line 203, in wrapper
    return func(self, options, args)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/commands/install.py", line 315, in run
    requirement_set = resolver.resolve(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/resolvelib/resolvers.py", line 341, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/resolution/resolvelib/found_candidates.py", line 140, in __bool__
    return any(self)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/resolution/resolvelib/found_candidates.py", line 128, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/resolution/resolvelib/found_candidates.py", line 32, in _iter_built
    candidate = func()
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/resolution/resolvelib/factory.py", line 204, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/resolution/resolvelib/candidates.py", line 295, in __init__
    super().__init__(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
    dist = self._prepare_distribution()
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/resolution/resolvelib/candidates.py", line 305, in _prepare_distribution
    return self._factory.preparer.prepare_linked_requirement(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/operations/prepare.py", line 550, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/operations/prepare.py", line 239, in unpack_url
    file = get_http_url(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/operations/prepare.py", line 102, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/network/download.py", line 145, in __call__
    for chunk in chunks:
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/cli/progress_bars.py", line 144, in iter
    for x in it:
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_internal/network/utils.py", line 63, in response_chunks
    for chunk in response.raw.stream(
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/pip/_vendor/urllib3/response.py", line 443, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
WARNING: You are using pip version 21.2.4; however, version 25.2 is available.
You should consider upgrading via the '/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python3 -m pip install --upgrade pip' command.
Done
Will use justin-get-file
input file: monte-carlo-008696-001742
jobsub_id: 226222.118@dunegpschedd02.fnal.gov
nevents: 10
input flag: 
Traceback (most recent call last):
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/8903d6be21c5250084ac1854692983819112b027/beam_job_utils.py", line 369, in <module>
    routines[args.routine](args)
  File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/8903d6be21c5250084ac1854692983819112b027/beam_job_utils.py", line 250, in run_job
    import yaml
ModuleNotFoundError: No module named 'yaml'
Error running. Exiting with 1
justIN time: 2025-11-04 12:31:27 UTC       justIN version: 01.05.01