How to configure both Fluss and Paimon for S3 storage.

[maarek]

I am just running the docker-compose quickstart and I was wondering if one could provide an example configuration to configure Fluss + Paimon for S3 storage. The docs are a little sparse on s3 and setting the config but there is no information on paimon configuration for s3.

[wuchong]

cc @luoyuxia , could you give some hints here?

[luoyuxia]

@maarek Hi, thanks for your attention and sorry for missing document about it. I think you can follow these steps:

1. Configure s3 for Fluss
   Follow the document fluss s3 to configure s3 for Fluss.
   If you want to read the Fluss with Flink, please remember to copy fluss-fs-os into FLINK_HOME/lib

2. Configure s3 for write into Paimon
   You'll only need to configure s3 filesystem in Flink since it'll use Flink's filesystem to write to paimon.. You can follow the document filesystem-s3 to configure s3 in Flink

Hope it can help.

[maarek]

@luoyuxia Thank you for your reply.

I am following the instructions here : https://alibaba.github.io/fluss-docs/docs/quickstart/flink/

I have added the following configuration to the docker containers (I wasn't sure tablet-server config vs coordinator config) to connect to s3 compatible local minio server.

First, I do not see it storing anything in s3 with these configs. This is with the fluss/fluss:0.5.0 docker container. I do see in the logs that it sees that the s3 access key is set, but it does not seem to do anything with it nor error out on connection. I see it still storing data locally within the data.dir. Is there some sort of internal syncing occurring between local tmp storage and s3? Am I configured incorrectly?

As well, when you say flink needs to be configured for s3 and paimon, what then is the configuration below doing when setting lakehouse.storage to paimon and setting the config metastore and warehouse to be on the local filesystem?

logs:

tablet-server-1  | 2025-02-28 01:11:41,332 INFO  com.alibaba.fluss.fs.s3.S3FileSystemPlugin                   [] - fs.s3a.access.key is set, using provided access key id and secret.

coordinator:

    environment:
      - |
        FLUSS_PROPERTIES=
        zookeeper.address: zookeeper:2181
        coordinator.host: fluss.coordinator-server
        remote.data.dir: s3://warehouse/fluss
        s3.endpoint: http://minio:9000/
        s3.access-key: admin
        s3.secret-key: password
        s3.region: us-phoenix-1
        table.datalake.enabled: true
        lakehouse.storage: paimon
        paimon.catalog.metastore: filesystem
        paimon.catalog.warehouse: /tmp/paimon

tablet-server:

    environment:
      - |
        FLUSS_PROPERTIES=
        zookeeper.address: zookeeper:2181
        tablet-server.host: fluss.tablet-server
        data.dir: /tmp/fluss/data
        kv.snapshot.interval: 0s
        remote.data.dir: s3://warehouse/fluss
        s3.endpoint: http://minio:9000/
        s3.access-key: admin
        s3.password: password
        s3.region: us-phoenix-1
        table.datalake.enabled: true
        lakehouse.storage: paimon
        paimon.catalog.metastore: filesystem
        paimon.catalog.warehouse: /tmp/paimon

[maarek]

I see it still storing data locally within the data.dir. Is there some sort of internal syncing occurring between local tmp storage and s3? Am I configured incorrectly?

How many segments in local per-bucket? Only when the number of segments is greater than 1, the local segment will be uploaded to remote.. If you did found the number of segments is greater than 1, is there any error/warn log in tablet server?

Can you elaborate on this? When does a segment increase in number for a bucket. Following the quickstart, I did not specify the number of buckets on a table. From what you're saying it sounds like log segments will only persist to s3 when >1 but I am not sure when this case would occur.

Also, for PrimaryKey tables backed by kv store, is this persisted to s3 as well? Are local storage mounts required to ensure fault tolerance?

what then is the configuration below doing when setting lakehouse.storage to paimon and setting the config metastore and warehouse to be on the local filesystem

When start the Lakehouse Tiering Service, it'll read the conf about paimon to create paimon catalog, thus create table in paimon when new table is created with datalake enabled is true

Can you elaborate? In the config above, it specifies paimon filesystem /tmp/paimon. How does this configuration translate into say a paimon catalog defined in flink. How is flink handling tiered storage internally? I hope this is clear, I am not familiar with paimon directly from flink but approaching this from using this for fluss tiered storage. The following would be a catalog config for paimon in flink back by s3 storage.

CREATE CATALOG my_catalog WITH (
    'type' = 'paimon',
    'warehouse' = 's3://<bucket>/<path>',
    's3.endpoint' = 'your-endpoint-hostname',
    's3.access-key' = 'xxx',
    's3.secret-key' = 'yyy'
);

[luoyuxia]

Can you elaborate on this? When does a segment increase in number for a bucket. Following the quickstart, I did not specify the number of buckets on a table. From what you're saying it sounds like log segments will only persist to s3 when >1 but I am not sure when this case would occur.

The defaul bucket number will be 1. When the size of log segment is greater than 1 G(configured by log.segment.file-size) by default, it'll roll a new segment to write. I think you can try to write more data to make log segment roll.

Also, for PrimaryKey tables backed by kv store, is this persisted to s3 as well? Are local storage mounts required to ensure fault tolerance?

Yes. It'll be persisted to s3. Local storage mounts is also required.

How is flink handling tiered storage internally?

As the document Tiered Storage said, Fluss have remote storage and lakehouse storage.

For remote storage, it'll just move the local segment to remote storage, like s3, hdfs, etc. Note it's just a simple copy, without any format conversion.

For lakehouse storage, currently, it required to start a Flink job as said in start-the-lakehouse-tiering-service to read Fluss and write to Paimon. It involves format conversion, converting Fluss native log format to Paimon's format(the parquet/orc file, and how the files is origanized as table).
Note the lakehouse storage is not necessary, if you don't care about tired to paimon for better analytics perfomacne, you can ignore configuring it.
Hope it's clear.

[michaelkoepf]

citing @maarek

to connect to s3 compatible local minio server

@luoyuxia is minio now supported? asking because of #211

[luoyuxia]

citing @maarek

to connect to s3 compatible local minio server

@luoyuxia is minio now supported? asking because of #211

Mninor is not supported.. I had test with minio, although fluss can tier log segment to minio, it can't read the tiered log segment in minio as the reason I saird in #211

[maarek]

citing @maarek

to connect to s3 compatible local minio server

@luoyuxia is minio now supported? asking because of #211

Mninor is not supported.. I had test with minio, although fluss can tier log segment to minio, it can't read the tiered log segment in minio as the reason I saird in #211

This sounds to me like any non-AWS provider would then be unavailable. Can using STS be made configurable? I am not too familiar with the inner workings of connection to S3 but paimon s3, hadoop s3, presto s3 fs drivers do not seem to require STS?