[VOTE] Release 0.7.0 RC2

[wuchong]

This is the Fluss 0.7.0 release RC2. You can verify the release using the signature (.asc files) or hashes (.sha256 files).

The complete staging area is available for your review, which includes:

• the official source release and binary convenience releases are attached, which are signed with the key with fingerprint 85BACB5AEFAE3202
• all artifacts to be deployed to the Maven Central Repository: https://oss.sonatype.org/content/repositories/comalibaba-5582/
• source code tag "v0.7.0-rc2": https://github.com/alibaba/fluss/tree/v0.7.0-rc2
• the documentation: https://alibaba.github.io/fluss-docs/docs/
• website pull request for release announcement blog post: TODO

The docker image for release-0.7.0 RC2 also has been uploaded with tag v0.7.0-rc2. Welcome to verify it.

• https://hub.docker.com/repository/docker/fluss/fluss/general
• https://hub.docker.com/repository/docker/fluss/quickstart-flink

To verify it, you can follow

1. The fluss quick start
2. Deploy it with docker and interacting with Fluss to verify some features documented in subpages of Engine Flink.

Nothing needed to change in the docker-compose file, as the docker image tag has been adapted for the RC test.

---

Full Changelog: https://github.com/alibaba/fluss/commits/v0.7.0-rc2

---

This discussion was created from the release Release 0.7.0 RC2.

[swuferhong]

+1
I run the Deploying Local Cluster and Deploying Distributed Cluster and all of queries in Engine Flink using local and distributed cluster.

[luoyuxia]

+1. I tried with quickstart, (paimon + s3) + local fluss cluster(not docker) + local flink 1.19, everything works well.

[wuchong]

+1

• verified signatures: OK
• verified hashes: OK
• build source code with JDK11 and Maven 3.8.6: OK
• Started local fluss cluster and verified logs: OK
• Go through the quickstart using local fluss cluster and local flink 1.18 cluster: OK

[MehulBatra]

+1

1. Verified signatures: OK
2. Verified hashes: OK
3. Build source code with JDK11 and Maven 3.8.6: OK
4. Started local fluss cluster and verified logs: OK
5. Used local fluss cluster and local flink 1.18 cluster: OK

[loserwang1024]

+1， verify:

1. the security including authentication and authorization: OK
2. acl operation: OK
3. docker startup with fluss image: OK

[leonardBang]

+1

• verified with quickstart in docker way, use paimon(local filestyem) + flink 1.20, everything works well.
• verified with quickstart in local cluster way, use paimon(Aliyun OSS) + flink 1.18, everything works well.

bangjiangxu fluss-0.7.0~  $ls -l plugins/paimon |grep -E 'flink|paimon'
-rw-r--r--@ 1 bangjiangxu  staff  43317025  6 12  2023 flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
-rw-r--r--@ 1 bangjiangxu  staff  41791028  6 12  2023 fluss-lake-paimon-0.7.0.jar
-rw-r--r--  1 bangjiangxu  staff  29155956  6 14 16:00 paimon-oss-1.0.1.jar

bangjiangxu flink-1.18.1~  $ls  -l lib|grep -E 'fluss|paimon|faker'
-rw-------@ 1 bangjiangxu  staff    3023222  6 14 15:36 flink-faker-0.5.3.jar
-rw-------@ 1 bangjiangxu  staff   68218905  6 14 16:37 fluss-flink-1.18-0.7.0.jar
-rw-r--r--  1 bangjiangxu  staff   41791028  6 14 15:25 fluss-lake-paimon-0.7.0.jar
-rw-r--r--@ 1 bangjiangxu  staff   48379580  1 24 17:34 paimon-flink-1.18-1.0.1.jar
-rw-r--r--  1 bangjiangxu  staff   29155956  1 24 17:40 paimon-oss-1.0.1.jar

[caozhen1937]

+1

1. Verified signatures: OK
2. Verified hashes: OK
3. Build source code with JDK11 and Maven 3.8.4: OK
4. Started local fluss cluster and verified logs: OK
5. Followed quickstart to verify local fluss cluster and local flink 1.18 cluster: OK

[michaelkoepf]

1. build source code with jdk 8 and jdk 11: OK
2. run flink quickstart guide: OK
3. test client backward compatibility using flink quickstart guide with fluss/quickstart-flink:1.20-0.6 image with old server-side config options ([coordinator|tablet-server].host; but doesn't work with new bind.listeners config option either): NOT OK.

steps to reproduce.

1. docker-compose.yaml

services:
  #begin Fluss cluster
  coordinator-server:
    image: fluss/fluss:v0.7.0-rc2
    command: coordinatorServer
    depends_on:
      - zookeeper
    environment:
      - |
        FLUSS_PROPERTIES=
        zookeeper.address: zookeeper:2181
        coordinator.host: coordinator-server  # OLD CONFIG OPTION!
        remote.data.dir: /tmp/fluss/remote-data
        datalake.format: paimon
        datalake.paimon.metastore: filesystem
        datalake.paimon.warehouse: /tmp/paimon
    volumes:
      - shared-tmpfs:/tmp/paimon
  tablet-server:
    image: fluss/fluss:v0.7.0-rc2
    command: tabletServer
    depends_on:
      - coordinator-server
    environment:
      - |
        FLUSS_PROPERTIES=
        zookeeper.address: zookeeper:2181
        tablet-server.host: tablet-server # OLD CONFIG OPTION!
        data.dir: /tmp/fluss/data
        remote.data.dir: /tmp/fluss/remote-data
        kv.snapshot.interval: 0s
        datalake.format: paimon
        datalake.paimon.metastore: filesystem
        datalake.paimon.warehouse: /tmp/paimon
    volumes:
      - shared-tmpfs:/tmp/paimon
  zookeeper:
    restart: always
    image: zookeeper:3.9.2
  #end
  #begin Flink cluster
  jobmanager:
    image: fluss/quickstart-flink:1.20-0.6 # OLD CLIENT!
    ports:
      - "8083:8081"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
    volumes:
      - shared-tmpfs:/tmp/paimon
  taskmanager:
    image: fluss/quickstart-flink:1.20-0.6 # OLD CLIENT!
    depends_on:
      - jobmanager
    command: taskmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 10
        taskmanager.memory.process.size: 2048m
        taskmanager.memory.framework.off-heap.size: 256m
    volumes:
      - shared-tmpfs:/tmp/paimon
  #end

volumes:
  shared-tmpfs:
    driver: local
    driver_opts:
      type: "tmpfs"
      device: "tmpfs"

2. docker compose up -d

3. docker container ls -a

Output:

CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS          PORTS                                                   NAMES
bd640efa6441   fluss/fluss:v0.7.0-rc2            "/docker-entrypoint.…"   27 seconds ago   Up 26 seconds                                                           verify-release-tablet-server-1
215c72b25309   fluss/fluss:v0.7.0-rc2            "/docker-entrypoint.…"   27 seconds ago   Up 26 seconds                                                           verify-release-coordinator-server-1
5b6251a4c951   fluss/quickstart-flink:1.20-0.6   "/docker-entrypoint.…"   27 seconds ago   Up 26 seconds   6123/tcp, 8081/tcp                                      verify-release-taskmanager-1
0e8b2f50b0c1   fluss/quickstart-flink:1.20-0.6   "/docker-entrypoint.…"   28 seconds ago   Up 27 seconds   6123/tcp, 0.0.0.0:8083->8081/tcp, [::]:8083->8081/tcp   verify-release-jobmanager-1
7ea2146f64a8   zookeeper:3.9.2                   "/docker-entrypoint.…"   28 seconds ago   Up 27 seconds   2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp                  verify-release-zookeeper-1

4. docker compose exec jobmanager ./sql-client

5. Create catalog

CREATE CATALOG fluss_catalog WITH (
    'type' = 'fluss',
    'bootstrap.servers' = 'coordinator-server:9123'
);

throws

[ERROR] Could not execute SQL statement. Reason:
java.lang.IllegalStateException: Failed to initialize fluss client connection to server because no bootstrap server is validate. bootstrap servers: [coordinator-server/172.18.0.4:9123]

[polyzos]

I think it’s known there is no backwards-compatibility, similar to the previous two versions.
I think for now it’s ok, as the project evolved at its core - 0.7 is the first release we consider “production-ready” I believe.

[michaelkoepf]

cross posting for tracking purposes. answer from @wuchong on the community slack:

It is expected to work. But we have found a bug in the old client that it can't connect to the new server unless we release a new version for 0.6

[wuchong]

@michaelkoepf thanks for reporting this. I have just created a v0.6.1 RC1 to include this fix. I verified locally and it works using the 0.6 client for 0.7 server. Please help to verify as well!

https://github.com/alibaba/fluss/releases/tag/v0.6.1-rc1

[michaelkoepf]

vote: +1

@wuchong

re-run verfication for 3. (testing backward compatibility with fluss flink connector v0.6.1-rc1)

• build fluss v0.6.1-rc1
• build fluss/quickstart-flink image with fluss flink connector v0.6.1-rc1 locally based on this repo

run quickstart guide with locally built quickstart image (w/o tiering service) for:

• old server-side configuration options ([coordinator|tablet-server].host): OK
• new server side configuration options introduced in v0.7.0 (bind.listeners): OK

[wuchong]

Thank you @michaelkoepf !

[zcoo]

I tried with new features in this release, including:

1. multi-level partitions OK
2. dynamic partitions OK
3. rack aware OK

[leonardBang]

Sorry for interrupted our vote, but we found a blocker issue， #1121

and I've raised a PR #1123 for fix is ASAP, CC @luoyuxia @wuchong

[wuchong]

@leonardBang thanks for reporting this. I closed this RC2 and will prepare RC3.