http2 connection is not renewed after server closes?

[twisteroidambassador]

Using AsyncClient with http2=True, after sending many requests, all subsequent requests fail immediately with an exception like this: httpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:19999, additional_data:None>

It feels like the server has closed the connection after serving a fixed number of request (20000?), but httpx keeps trying to use the same connection, without attempting to create a new one.

Here's the full traceback:

Traceback (most recent call last):
  File "~\venv\lib\site-packages\h2\connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~\venv\lib\site-packages\httpcore\_async\http2.py", line 105, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "~\venv\lib\site-packages\httpcore\_async\http2.py", line 202, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "~\venv\lib\site-packages\h2\connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "~\venv\lib\site-packages\h2\connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~\venv\lib\site-packages\httpx\_transports\default.py", line 60, in map_httpcore_exceptions
    yield
  File "~\venv\lib\site-packages\httpx\_transports\default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "~\venv\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in handle_async_request
    raise exc
  File "~\venv\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "~\venv\lib\site-packages\httpcore\_async\connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "~\venv\lib\site-packages\httpcore\_async\http2.py", line 138, in handle_async_request
    raise RemoteProtocolError(self._connection_error_event)
httpcore.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:19999, additional_data:None>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~\decryptor.py", line 138, in do_decrypt_work
    resp = await client.get(test_v3_url)
  File "~\venv\lib\site-packages\httpx\_client.py", line 1729, in get
    return await self.request(
  File "~\venv\lib\site-packages\httpx\_client.py", line 1506, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "~\venv\lib\site-packages\httpx\_client.py", line 1593, in send
    response = await self._send_handling_auth(
  File "~\venv\lib\site-packages\httpx\_client.py", line 1621, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "~\venv\lib\site-packages\httpx\_client.py", line 1658, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "~\venv\lib\site-packages\httpx\_client.py", line 1695, in _send_single_request
    response = await transport.handle_async_request(request)
  File "~\venv\lib\site-packages\httpx\_transports\default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "~\venv\lib\site-packages\httpx\_transports\default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:19999, additional_data:None>

Package versions:
h2-4.1.0
httpcore-0.14.7
httpx-0.22.0

This may be related? encode/httpcore#45

[adriangb]

I've run into this as well. I'm just disabling http2 for now, which is not great. I'll try to look into causes

[set92]

2023 and still here boys! :D

PrefectHQ/prefect#7442

I haven't seen a workaround, so I suppose I'll disable http2 for now, which sucks but seems to be the only way to fix it.

[akdienes]

I am also hitting this.

[python273]

Same error

code: https://github.com/python273/eimi/blob/5fc4a1744191a5954955d77786e94bcaf8d6d5de/main.py
tracebacks: https://gist.githubusercontent.com/python273/651744e3e7140a3ed012661922ab075c/raw/c0bac785225b6ec475f6e8c6f0c913f60c534a1b/httpx_error.log

[zanieb]

I have a reproduction at https://github.com/madkinsz/httpcore-h2-closed and a draft fix at encode/httpcore#679

[clairem-sl]

I've been having the same issue. How to work around this?

OS: Windows 10 Pro 64-bit
Python: CPython 3.11
h2==4.1.0
httpcore==0.17.2
httpx==0.24.1

Edit: I've been looking around httpx documentation, there doesn't seem a way to trigger httpx.AsynClient to just invalidate its connection pool and rebuild a new pool. If there's a way to do that, I can have the first asyncio Task that sees ConnectionTerminated to trigger that invalidation+rebuilding process...

[python273]

I think there was a fix in httpcore 0.17.1

[joemirizio]

I upgraded to the latest versions of httpx and httpcore and still get a ConnectionTerminated. My workaround has also been disabling http2.

Python: CPython 3.11.2
h2==4.1.0
httpcore==0.17.2
httpx==0.24.1

Traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
                         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http2.py", line 119, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http2.py", line 216, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/usr/local/lib/python3.11/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/usr/local/lib/python3.11/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 261, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http2.py", line 152, in handle_async_request
    raise RemoteProtocolError(self._connection_error_event)
httpcore.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:19999, additional_data:None>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspaces/mycode/loader.py", line 69, in process_table_package
    package = await vendor.get_table_package(table_id)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/vendor.py", line 309, in get_table_package
    table, columns = await asyncio.gather(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 684, in _wrap_awaitable
    return (yield from awaitable.__await__())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiostream/core.py", line 33, in wait_stream
    async for item in streamer:
  File "/usr/local/lib/python3.11/site-packages/aiostream/stream/aggregate.py", line 71, in list
    async for item in streamer:
  File "/workspaces/mycode/vendor.py", line 260, in get_column_objects
    column_json["_deprecated"] = await self.is_object_deprecated(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/vendor.py", line 290, in is_object_deprecated
    flags = await self.get_flag_objects(oid, otype)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/vendor.py", line 282, in get_flag_objects
    flags = await stream.list(self.get_flags(oid, otype))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiostream/core.py", line 33, in wait_stream
    async for item in streamer:
  File "/usr/local/lib/python3.11/site-packages/aiostream/stream/aggregate.py", line 71, in list
    async for item in streamer:
  File "/workspaces/mycode/vendor.py", line 72, in _api_get_paginated
    resp = await self._api_get(
           ^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/client.py", line 57, in _api_get
    return await self._api_request("GET", endpoint, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/client.py", line 52, in _api_request
    response = await self.client.request(method, f"{self.uri}/{endpoint}", **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:19999, additional_data:None>

[zanieb]

The previous fix just made it so you can catch this error and retry the request yourself without it failing again.

encode/httpcore#683 is needed to automatically retry the request.

[joemirizio]

Gotcha - thanks for the clarification!

[injust]

The previous fix just made it so you can catch this error and retry the request yourself without it failing again.

encode/httpcore#683 is needed to automatically retry the request.

@zanieb I see that encode/httpcore#683 was superseded by encode/httpcore#733, so I thought that these failures should be gracefully retried now.

But I'm still seeing this ~daily. I've worked around it by retrying the httpx requests with tenacity.

h2==4.2.0
httpcore==1.0.7
httpx==0.28.1

Traceback

Traceback (most recent call last):

  File "~/foo/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "~/foo/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
                 │    │     │                    └ <Request [b'POST']>
                 │    │     └ <function AsyncConnectionPool.handle_async_request at 0x111e68720>
                 │    └ <AsyncConnectionPool [Requests: 0 active, 0 queued | Connections: 0 active, 0 idle]>
                 └ <httpx.AsyncHTTPTransport object at 0x111c8e120>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
                     │          └ <function AsyncHTTPConnection.handle_async_request at 0x111e4f880>
                     └ <AsyncHTTPConnection ['https://example.com:443', HTTP/2, CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
                 │    │           │                    └ <Request [b'POST']>
                 │    │           └ <function AsyncHTTP2Connection.handle_async_request at 0x111e6a5c0>
                 │    └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
                 └ <AsyncHTTPConnection ['https://example.com:443', HTTP/2, CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 183, in handle_async_request
    raise exc
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 146, in handle_async_request
    status, headers = await self._receive_response(
                            │    └ <function AsyncHTTP2Connection._receive_response at 0x111e6a980>
                            └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 290, in _receive_response
    event = await self._receive_stream_event(request, stream_id)
                  │    │                     │        └ 647
                  │    │                     └ <Request [b'POST']>
                  │    └ <function AsyncHTTP2Connection._receive_stream_event at 0x111e6aac0>
                  └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 329, in _receive_stream_event
    await self._receive_events(request, stream_id)
          │    │               │        └ 647
          │    │               └ <Request [b'POST']>
          │    └ <function AsyncHTTP2Connection._receive_events at 0x111e6ab60>
          └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
  File "~/foo/.venv/lib/python3.13/site-packages/httpcore/_async/http2.py", line 348, in _receive_events
    raise RemoteProtocolError(self._connection_terminated)
          │                   │    └ <ConnectionTerminated error_code:0, last_stream_id:647, additional_data:None>
          │                   └ <AsyncHTTP2Connection ['https://example.com:443', CLOSED, Request Count: 324]>
          └ <class 'httpcore.RemoteProtocolError'>

httpcore.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:647, additional_data:None>

[zanieb]

It might be worth opening a new discussion with a reproduction. It'll get lost here.

[injust]

#3549

[khalezin]

We had the same problem. We have following configuration
proxy(nginx) > uvicorn
We fixed the issue by increasing this value on the side of the:

keepalive_requests 1000;

http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests

[mhxion]

Can confirm. This issue still occurs with HTTPX v0.28.1 with HTTP/2 enabled. Increasing keepalive_requests like @khalezin suggested does fix it for now.