[Python] Improve the pyarrow build system

[jorisvandenbossche]

The current build system of pyarrow consists of using setuptools with a custom setup.py that calls cmake to do the actual build. While this works, there are nowadays newer python "build backends" that we could benefit from to make this easier to maintain and use.
Specifically build backends that are meant to work with CMake projects, such as scikit-build, might be worth exploring.

[raulcd]

Thanks @jorisvandenbossche !

[h-vetinari]

Specifically build backends that are meant to work with CMake projects, such as scikit-build, might be worth exploring.

Scipy has pioneered meson as the backend (in the wider python ecosystem at least), which is now being used by numpy & pandas as well, and is being picked up by more and more projects. For a regular python project it's probably a better choice than CMake-based builds (the syntax is already nice and pythonic too), though if you have substantial CMake infrastructure then that integrates more easily with scikit-build.

[jorisvandenbossche]

though if you have substantial CMake infrastructure then that integrates more easily with scikit-build.

We do have that, yes: https://github.com/apache/arrow/blob/main/python/CMakeLists.txt

[h-vetinari]

I'm aware, I've been building arrow for a while now. ;-)
It was more a rhetorical statement - depends a bit on what one considers "substantial" too. :)

[jorisvandenbossche]

Also relevant issue: #36410

[jorisvandenbossche]

Major wish-list item: proper incremental / cached cython builds.

[vyasr]

I'll note that if scikit-build is still of interest, the modern solution here would be to use scikit-build-core

[pitrou]

So, caching should be usage for us in the next release of Cython: cython/cython#6070 (comment)

[WillAyd]

We have recently been adding Meson configuration files to the Arrow repository, so as @h-vetinari suggests meson-python may be a good option here. In addition to SciPy, NumPy, and pandas using it, arrow-nanoarrow was ported to use it a few months back

[pitrou]

Well, the main concern here is really to get rid of our clumsy setup.py. We could also migrate our CMake setup to meson, but that seems to complicate things even more?

Also, how well-maintained and flexible are our Meson configuration files, compared to CMake? Which one is easier to work with?

[pitrou]

Is anyone ensuring that the Meson build files are up-to-date when the CMake files change? Are they compatible with anything else than gcc/clang?

We don't need two build systems in the source tree. I'm ok with migrating from CMake to Meson if desired (there has to be a convincing argument, though), I'm less ok with us having to care about two different build systems. Unless someone is specifically willing to maintain the Meson files so that others don't have to care.