Jobsub ID 206865.1@dunegpschedd02.fnal.gov
| Jobsub ID | 206865.1@dunegpschedd02.fnal.gov |
| Workflow ID | 8046 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 6815744000 (6500 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-09-25 18:38:34 |
| Site | US_Wisconsin |
| Entry | HCCHTPC_US_Wisconsin_osg01_rhel7 |
| Last heartbeat | 2025-09-26 06:48:34 |
| From worker node | Hostname | e2470 |
| cpuinfo | AMD EPYC 7452 32-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 7864320000 (7500 MiB) |
| Wall seconds limit | 82800 (23 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-09-25 18:39:41 |
| Input files | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-09-26 06:48:34 |
| Saved logs | justin-logs:206865.1-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ologicals_protodunehd_1_205934_6_1758725473_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/02/9f/prod_cosmics_radiologicals_protodunehd_1_205934_66_1758725451_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/af/d7/prod_cosmics_radiologicals_protodunehd_1_205934_67_1758725437_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/d2/af/prod_cosmics_radiologicals_protodunehd_1_205934_70_1758725454_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/75/19/prod_cosmics_radiologicals_protodunehd_1_205934_71_1758725441_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/de/cb/prod_cosmics_radiologicals_protodunehd_1_205934_73_1758725457_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/9a/66/prod_cosmics_radiologicals_protodunehd_1_205934_75_1758725449_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/96/6d/prod_cosmics_radiologicals_protodunehd_1_205934_76_1758725470_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/d0/af/prod_cosmics_radiologicals_protodunehd_1_205934_78_1758725441_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/1a/b0/prod_cosmics_radiologicals_protodunehd_1_205934_79_1758725449_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/df/a9/prod_cosmics_radiologicals_protodunehd_1_205934_80_1758725471_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/02/4a/prod_cosmics_radiologicals_protodunehd_1_205934_81_1758725462_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/68/07/prod_cosmics_radiologicals_protodunehd_1_205934_8_1758725473_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/23/05/prod_cosmics_radiologicals_protodunehd_1_205934_82_1758725471_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b8/5e/prod_cosmics_radiologicals_protodunehd_1_205934_83_1758725435_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/9a/3b/prod_cosmics_radiologicals_protodunehd_1_205934_84_1758725451_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/63/e8/prod_cosmics_radiologicals_protodunehd_1_205934_87_1758725462_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/dc/b1/prod_cosmics_radiologicals_protodunehd_1_205934_88_1758725454_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/af/1b/prod_cosmics_radiologicals_protodunehd_1_205934_90_1758725473_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/d9/d5/prod_cosmics_radiologicals_protodunehd_1_205934_91_1758725465_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/55/95/prod_cosmics_radiologicals_protodunehd_1_205934_9_1758725435_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/f8/60/prod_cosmics_radiologicals_protodunehd_1_205934_95_1758725476_reco_box_sce.root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/04/34/prod_cosmics_radiologicals_protodunehd_1_205934_96_1758725444_reco_box_sce.root
Output file: prod_cosmics_radiologicals_protodunehd_1_205934_0_1758725473_reco_box_sce_20250925T183944Z_ana.root
Running singlehit
Ran successfully
Forming metadata
Traceback (most recent call last):
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
socket.timeout: _ssl.c:1112: The handshake operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/util/retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/packages/six.py", line 770, in reraise
raise value
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 385, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/urllib3/connectionpool.py", line 336, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='metacat.fnal.gov', port=9443): Read timed out. (read timeout=600)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b1187e46da36fee81ba3ffb85c909ed3a28f1403/ntuple_prod_utils.py", line 257, in <module>
commands[args.command](args)
File "/cvmfs/fifeuser2.opensciencegrid.org/sw/dune/b1187e46da36fee81ba3ffb85c909ed3a28f1403/ntuple_prod_utils.py", line 97, in get_metadata
input_file = mc.get_file(did=did, with_provenance=False, with_metadata=True)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1259, in get_file
return self.get_json(url)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 206, in get_json
return self.unpack_json_data(self.send_request("get", uri_suffix, headers=headers, stream=True))
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 154, in send_request
self.LastResponse = response = self.retry_request(method, url, headers=headers, **args)
File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 131, in retry_request
response = requests.get(url, timeout=self.Timeout, **args)
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/cvmfs/fermilab.opensciencegrid.org/products/common/prd/python_future_six_request/v1_3_1/Linux64bit-3-10-2-17-python3-9/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='metacat.fnal.gov', port=9443): Read timed out. (read timeout=600)
Error in ntuple metdata production
cat: md.log: No such file or directory 