[Project Proposal] flink-scala-api

[novakov-alexey]

Repository Name
flink-scala-api

Description
Flink Scala API is a thin wrapper on top of Flink Java API which support Scala Types for serialisation as well the latest Scala version

Proposal

This project is a community-maintained fork of official Apache Flink Scala API, cross-built for Scala 2.12, 2.13 and 3.x. Official Apache Flink Scala API is going to be deprecated. The main features of the project are:

• New magnolia-based serialization framework
• Using a POJO-only flink serialization framework
• Closure cleaner from Spark 3.x
• Scala 3 support

Documentation

Information on flink-scala-api is currently located at https://github.com/findify/flink-scala-api/blob/master/README.md. It will be later divided into several pages and hosted on GitHub pages domain.

Initial Source

The initial source code for flink-scala-api is hosted at https://github.com/findify/flink-scala-api.

Project Admin

Roman Grebennikov (@shuttie)
Alexey Novakov (@novakov-alexey)

[wuchong]

+1 for the proposal.

What do you think @xintongsong @becketqin @MartijnVisser ?

[xintongsong]

+1

[MartijnVisser]

+1

[novakov-alexey]

@wuchong what will be the next step?

[shuttie]

Hi! Nice to meet you all, I'm the original author of the flink-scala-api package, and would like to provide a bit of context for the project current status:

• It was developed in the times when I was working at Findify - we were migrating to scala 2.13/3.x and Flink being 2.12 was always a blocker.
• In general it's just a fork of the original scala api code (which itself is a wrapper on top of java api), with minor changes, and magnolia-based serialization framework.

I've left Findify about a year ago, and Findify AFAIK has no internal motivation to continue maintaining this project, but I'm still wanting to support it - just don't have commit rights. The project itself is in quite a good shape - Findify was using it in production for some time without any issues. If someone from Findify wants to add 5c (@hicolour), you're welcome.

Some sort of internal roadmap for the project if it will be accepted to the incubator:

• Currently the serialization framework is a separate project (https://github.com/findify/flink-adt, historically made to be fully compatible with the original flink-scala, but with better support for ADTs and other scala-specific things), but it may be a good idea to have it merged with this project
• There's a bunch of github issues with serializing weird data structures, and it would be great to fix these
• We may also add scala-specific API for state processor
• A proper wire format compatibility testing suite, so we won't break someone's serialized state with new changes

@wuchong what are the next steps, considering all +1s above?

[wuchong]

Thank you all. I'm going to setup the repository.

[wuchong]

Hi @novakov-alexey , @shuttie , I have setup the repository https://github.com/flink-extended/flink-scala-api and grant the admin permission to you. You can invite other maintainers and synchronize the code and issues.

Thank you for your contribution!