[Self-Signed] Can't proxy https site with self-signed cert

[uuund3rfl0w]

Process Threadless-2:1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/core/acceptor/threadless.py", line 175, in run
    self.run_once()
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/core/acceptor/threadless.py", line 165, in run_once
    self.loop.run_until_complete(self.wait_for_tasks(tasks))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/core/acceptor/threadless.py", line 98, in wait_for_tasks
    teardown = await asyncio.wait_for(tasks[work_id], DEFAULT_TIMEOUT)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 483, in wait_for
    return fut.result()
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/core/acceptor/threadless.py", line 89, in handle_events
    return self.works[fileno].handle_events(readables, writables)
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/http/handler.py", line 152, in handle_events
    teardown = self.handle_readables(readables)
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/http/handler.py", line 298, in handle_readables
    upgraded_sock = plugin.on_request_complete()
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/http/proxy/server.py", line 350, in on_request_complete
    return self.intercept()
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/http/proxy/server.py", line 522, in intercept
    self.wrap_server()
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/http/proxy/server.py", line 549, in wrap_server
    self.server.wrap(text_(self.request.host), self.flags.ca_file)
  File "/usr/local/lib/python3.8/dist-packages/proxy.py-2.3.1-py3.8.egg/proxy/core/connection/server.py", line 44, in wrap
    self._conn = ctx.wrap_socket(
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1123)

[uuund3rfl0w]

proxy.py/proxy/core/connection/server.py

Lines 38 to 47 in df33b40

	def wrap(self, hostname: str, ca_file: Optional[str]) -> None:
	ctx = ssl.create_default_context(
	ssl.Purpose.SERVER_AUTH, cafile=ca_file)
	ctx.options |= ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3 | ssl.OP_NO_TLSv1
	ctx.check_hostname = True
	self.connection.setblocking(True)
	self._conn = ctx.wrap_socket(
	self.connection,
	server_hostname=hostname)
	self.connection.setblocking(False)

Here, by default ctx.verify_mode = ssl.CERT_REQUIRED, which will check certs.

This is a wrapper to SSL_CTX_set_verify, which can pass a callback function in C side, but cpython's ssl library does't export this.

https://github.com/openssl/openssl/blob/25959e04c350c2b82d545ea38b18ff714acf61ba/include/openssl/ssl.h.in#L1717

void SSL_CTX_set_verify(SSL_CTX *ctx, int mode, SSL_verify_cb callback);

https://github.com/python/cpython/blob/1f08d16c90b6619607fe0656328062ab986cce29/Modules/_ssl.c#L3057-L3090

/*
 * _SSLContext objects
 */

static int
_set_verify_mode(PySSLContext *self, enum py_ssl_cert_requirements n)
{
    int mode;
    int (*verify_cb)(int, X509_STORE_CTX *) = NULL;

    switch(n) {
    case PY_SSL_CERT_NONE:
        mode = SSL_VERIFY_NONE;
        break;
    case PY_SSL_CERT_OPTIONAL:
        mode = SSL_VERIFY_PEER;
        break;
    case PY_SSL_CERT_REQUIRED:
        mode = SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT;
        break;
    default:
         PyErr_SetString(PyExc_ValueError,
                        "invalid value for verify_mode");
        return -1;
    }

    /* bpo-37428: newPySSLSocket() sets SSL_VERIFY_POST_HANDSHAKE flag for
     * server sockets and SSL_set_post_handshake_auth() for client. */

    /* keep current verify cb */
    verify_cb = SSL_CTX_get_verify_callback(self->ctx);
    SSL_CTX_set_verify(self->ctx, mode, verify_cb);
    return 0;
}

Although we can simply change verify mode to CERT_NONE,

ctx.verify_mode = ssl.CERT_NONE
ctx.check_hostname = False

But then we will get empty cert info by using getpeercert().

proxy.py/proxy/http/proxy/server.py

Lines 555 to 556 in df33b40

	generated_cert = self.generate_upstream_certificate(
	cast(Dict[str, Any], self.server.connection.getpeercert()))

Maybe we can change dependency from ssl to pyOpenSSL, which provide set_verify with user definable callback function. So we can pass the site whose cert is self-signed in callback function.

[uuund3rfl0w]

urllib3 shows some good idea to port from ssl to pyOpenSSL

https://github.com/urllib3/urllib3/blob/main/src/urllib3/contrib/pyopenssl.py

[abhinavsingh]

@uuund3rfl0w Thank you for reporting this issue. I am looking into it this weekend. #638 has already landed tons of fixes.

Thinking about

So we can pass the site whose cert is self-signed in callback function.

How about we add a flag where users can pass list of upstream hostname which are expected to use self-signed certificates. Then within server.py, we can simply use CERT_NONE when expected. Wdyt?

[abhinavsingh]

BTW related / duplicate of #536

[abhinavsingh]

Closing as duplicate. Lets continue discussion on #536