Add binglog system table support for fluss tables

[MehulBatra]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

If you need to audit the changelog of the table, you can use the binlog system table. In the binlog system table, the update before and the update after will be packed in one row. (Similar to what we have in Paimon)

SELECT * FROM tablename$binlog;

/*
+------------------+----------------------+-----------------------+
|     rowkind      |       column_0       |       column_1        |
+------------------+----------------------+-----------------------+
|        +I        |       [col_0]        |       [col_1]         |
+------------------+----------------------+-----------------------+
|        +U        | [col_0_ub, col_0_ua] | [col_1_ub, col_1_ua]  |
+------------------+----------------------+-----------------------+
|        -D        |       [col_0]        |       [col_1]         |
+------------------+----------------------+-----------------------+
*/

Solution

No response

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[gyang94]

Good feature we may have.
Do we consider add other system tables like paimon has?

[MehulBatra]

Currently, we have plans to support two of the system tables (changelog, binlog)

[wuchong]

+1 for this feature. Since it involves packing a row into a column, I believe it depends on first supporting nested fields (see #835).

Additionally, we should discuss the naming of the system table. The name $binlog doesn't feel significantly different from $changelog. It might be worth investigating whether other systems use similar concepts and what naming conventions they follow.

[MehulBatra]

We should definitely choose a name that clearly reflects the functionality, making it immediately recognizable by its purpose. I initially considered 'binlog' to align with the Paimon system table we're developing, but 'binlog' feels too similar to 'changelog' and doesn't clearly communicate the value addition for the use cases. I'll research naming conventions used by other storage engines and explore.
Whether we should create our own terminology that better captures what we're building for in terms of Fluss.