[Remark] gunicorn_conf can lead to huge number of workers

[euri10]

I used your config file for gunicorn on a project, so not sure this issue applies here as I'm using asyncpg, but I found it interesting enough to post.

web_concurrency can reach too high levels, in my case using defaults it was trying to setup 96 workers and I couldn't start my app, because of an asyncpg.exceptions.TooManyConnectionsError

Indeed, the default max_size in asyncpg.pool.create_pool is 10 so should you get more than 10 workers, db will complain this way:

[2019-02-26 12:52:14 +0000] [135] [ERROR] Exception in 'lifespan' protocol
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/uvicorn/lifespan/on.py", line 40, in main
    await app_instance(self.receive, self.send)
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 478, in asgi
    await self.startup()
  File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 461, in startup
    await handler()
  File "/usr/src/app/app/app.py", line 27, in startup
    await database.connect()
  File "/usr/local/lib/python3.7/site-packages/databases/core.py", line 59, in connect
    await self._backend.connect()
  File "/usr/local/lib/python3.7/site-packages/databases/backends/postgres.py", line 55, in connect
    self._pool = await asyncpg.create_pool(str(self._database_url), **kwargs)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/pool.py", line 400, in _async__init__
    await self._initialize()
  File "/usr/local/lib/python3.7/site-packages/asyncpg/pool.py", line 417, in _initialize
    await first_ch.connect()
  File "/usr/local/lib/python3.7/site-packages/asyncpg/pool.py", line 125, in connect
    self._con = await self._pool._get_new_connection()
  File "/usr/local/lib/python3.7/site-packages/asyncpg/pool.py", line 463, in _get_new_connection
    **self._connect_kwargs)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connection.py", line 1688, in connect
    max_cacheable_statement_size=max_cacheable_statement_size)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connect_utils.py", line 543, in _connect
    connection_class=connection_class)
  File "/usr/local/lib/python3.7/site-packages/asyncpg/connect_utils.py", line 519, in _connect_addr
    await asyncio.wait_for(connected, loop=loop, timeout=timeout)
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
asyncpg.exceptions.TooManyConnectionsError: sorry, too many clients already

I added the following, so that the max number of workers matches the max_size of asyncpg create_pool()

diff --git a/services/backend/gunicorn_conf.py b/services/backend/gunicorn_conf.py
index c031db5..e181880 100644
--- a/services/backend/gunicorn_conf.py
+++ b/services/backend/gunicorn_conf.py
@@ -2,6 +2,7 @@ import json
 import multiprocessing
 import os
 
+max_workers = os.getenv("MAX_WORKERS", 10)
 workers_per_core_str = os.getenv("WORKERS_PER_CORE", "2")
 web_concurrency_str = os.getenv("WEB_CONCURRENCY", None)
 host = os.getenv("HOST", "0.0.0.0")
@@ -20,7 +21,7 @@ if web_concurrency_str:
     web_concurrency = int(web_concurrency_str)
     assert web_concurrency > 0
 else:
-    web_concurrency = int(default_web_concurrency)
+    web_concurrency = min(max_workers, int(default_web_concurrency))
 
 # Gunicorn config variables
 loglevel = use_loglevel

[tiangolo]

Sorry for the delay @euri10, and thanks for the very interesting insight. I think it's worth adding that variable (MAX_WORKERS), tests, and docs to the base images. And then use those defaults here.