Support WebSockets in LiteFS Proxy

[patte]

I successfully tested litefs on fly.io, but it seems that websocket connections are not working through the proxy.

The ws GET request arrives at the proxy, it gets proxied to the backend and in the browser I see a 101 Switching Protocols with the response headers and size 0. No messages are received. When I try to send a message from the browser the connection gets closed (Connection Closed: 1006).

2024-02-08T17:56:57Z app[178157ea245398] ams [info]level=INFO msg="proxy: GET /chat: node is primary, proxying to target"
2024-02-08T17:56:57Z app[178157ea245398] ams [info] INFO call: playground::chat: ams patte wants to connect ws

When I expose the backend port directly (so not going through the litefs proxy) everything works.

While writing this I found this thread in the forum, which seems to confirm that ws is not supported. It would be nice if this limitation is documented: Either litefs (with proxy) or websockets for one fly app.

Could I use litefs without the proxy and fly-replay write requests myself? How would I know if the local sqlitedb is synced up? I'd also be up to try to "fix" ws proxying in litefs if you think that would be a good idea to have and don't see any big roadblocks.

[benbjohnson]

@patte I added a PR (superfly/docs#1342) to document the WebSockets limitation. The tricky thing about supporting WebSockets is that messages don't have traditional HTTP verbs that indicate a read-only request versus read/write request so it's hard to know what to proxy.

[patte]

Thank you for updating the docs!
Yes, I see that problem. But wouldn't it be possible to make this decision at the time of the initial Upgrade request, based on the config passthrough or always-forward? If i understand proxy_server.go correctly the only change needed would be a httputil.NewSingleHostReverseProxy and then in proxyToTarget use the proxy for websocket requests (very minimal gist of what I mean). If all the rest of the targeting logic stays the same, a ws request with a path in always-forward (eg. /primary-chat) would still be fly-replayed to the primary and then be ws-proxied to the local machine (same for paths in passthrough).
This would have the advantages of a) not prevent the usage of websockets with litefs-proxy completely and b) even if in a simple manner, direct websockets locally or always to the primary.
The only situation where I see this would fail is if the primary changes, but it's machine doesn't stop. Thus not killing the ws connection and suddenly having an open connection that is not to the primary any more, but I don't know if this is a possibility with the current way litefs is implemented.
Do you see any other ways this could fail? Do you think it would be a good idea to have that?

[theronic]

Until this is solved, how to proxy secure websocket requests to my application? (wss://). This is a blocker for using Electric Clojure with LiteFS on Fly.io.

[liamkinne]

Got most of the way to getting Grafana working in a HA setup before I discovered this WS limiation. Unfortunate because it seemed like a silver bullet.