Arrow IPC via gRPC-Web into Typescript frontend.

[WhisperN]

Hello guys,

I am currently experimenting with IPC streams and also gRPC-Web.

My thought was that gRPC on h2 would reduce overhead and the parsing in the frontend for a much faster system.

I started to research about ways to load data from an in memory database (in my case duckdb) onto another system. There was Apache Arrow Flight, it sounded ideal for sharing columnized data via streams to dump directly into the clients memory.

So I tried to get a typescript frontend running that connects to the server and with a doGet it dumps some dataset into the clients memory. This way, at least I thought, I could build a frontend that utilizes client side processing instead of server side.

Now my problem is this:
The data gets prepared and shoved into the client extremely fast (~1.6s for a 134 MB dataset). If I also want to do something with the data I would have to parse it or have some sort of database in the frontend that just collects this payload and allows me to project it onto some kind of data visualization.

Does anyone have an idea on how to handle the data in my typescript web client?
Or alternative approaches for a super fast data vis system?

If I hit a dead end here next thing would be to send a .parquet file into a Typescript Duckdb instance and then process it in the frontend.

If you are interested in this project or you want to read the code. I have it all on my repo in the directories:
Backend: https://github.com/WhisperN/Go-Flight-Server.git
Frontend: https://github.com/WhisperN/gRPC-Web-Frontend.git

I am looking forward to hear from you!

[pitrou]

cc @lidavidm @zeroshade

[gianarb]

Hello! I am not an expert here but I did my research a few months ago. At the end I didn't need to implement something like that but I am wondering if you have looked at this project

https://github.com/hyparam/hyparquet

It deals with Parquet not IPC at least at the best of my knowledge

[WhisperN]

Thank you very much for your reply! I have just ran some tests using only a parquet file to transfer data and its a lot faster and I think I will go for this approach.

I think the bottleneck with my original approach is that parsing is just too slow or inefficient for gRPC-Web.

[lidavidm]

It seems duckdb-wasm supports ingesting Arrow data: https://shell.duckdb.org/docs/classes/index.AsyncDuckDBConnection.html#insertArrowFromIPCStream

vega/vega-lite support Arrow: https://github.com/vega/vega-loader-arrow

I'm not aware of anything else off the top of my head (I looked at Apache ECharts but that doesn't seem to support Arrow)

[WhisperN]

Thank you for your reply! Yes, indeed they do support it. The bottleneck in project is the gRPC-Web parsing. It is inefficient but I have ran a test with exposing an arrow file to an HTTP endpoint and this works a charme.

[lidavidm]

Ah, yeah. The arrow-http examples here may be more useful: https://github.com/apache/arrow-experiments/tree/main/http