[Python] Gradually add type checks to Arrow, initial step

[rok]

This proposes adding type annotation to pyarrow by adopting pyarrow-stubs into pyarrow. To do so we copy a subset of pyarrow-subs's stubfiles into pyarrow tree. We then add annotation checks for some stubsfiles and some test files and make sure checks pass. Annotation checks should be expanded until all (or most) project files are covered in future work.

PR introduces:

1. adds pyarrow-stubs into arrow/python/pyarrow/
2. fixes pyarrow-stubs to pass mypy and pyright check
3. adds mypy and pyright check to CI (crudely)
4. adds a tool (update_stub_docstrings.py) to keep annotation docstrings in sync with source docstrings
5. adds docstring sync check to CI (crudely)

[pitrou]

Is this approach scalable? I notice this doesn't seem to include some extension types such as tensor arrays?

[rok]

Are you asking about annotating constructors per type? Overload approach is nice because it expresses the relationship between the argument and return types. We can use a single parameter type union and a single return union, but we then lose the relationship.

FixedShapeTensorArray is on line 4578, I'm not sure about other extension types. If we use type checkers to validate our tests we should get errors if our symbols are not annotated or if annotations disagree with tests. Our test coverage is pretty good and we can assume every symbol is tested at least once.

[rok]

Overloads are now removed and docstring sync is automated.
We should also add an automated alert for when symbols don't have annotations but should.

[pitrou]

Why are docstrings duplicated in the stubs files? Do they need to be kept in sync? Why (and how)?

[rok]

See my reply here. I am not yet sure we want to include docstrings here, but if we do we should definitely script their extraction. stubgen --include-docstrings does this but I suspect it doesn't read cython-only symbols.

[pitrou]

Is the __lib_pxi/xxx.pyi naming scheme documented anywhere?

[rok]

Is the __lib_pxi/xxx.pyi naming scheme documented anywhere?

It was adopted from pyarrow-stubs. Not really documented and we might want to change it.

[rok]

@pitrou

Is the __lib_pxi/xxx.pyi naming scheme documented anywhere?

Would __lib be a good naming scheme? Something else perhaps?

[pitrou]

Is the __lib_pxi/xxx.pyi naming scheme documented anywhere?

Would __lib be a good naming scheme? Something else perhaps?

Since we already have lib.pyi, why not also io.pyi etc.?

OTOH, we could put everything in a subdirectory (_typestubs or something). Would it be compatible with how type checkers work?

[pitrou]

As for docstrings: do they need to be in the repo, or can they be generated when building a wheel file?

[rok]

Is the __lib_pxi/xxx.pyi naming scheme documented anywhere?

Would __lib be a good naming scheme? Something else perhaps?

Since we already have lib.pyi, why not also io.pyi etc.?

OTOH, we could put everything in a subdirectory (_typestubs or something). Would it be compatible with how type checkers work?

I'll give these a shot, starting with a new folder approach.

As for docstrings: do they need to be in the repo, or can they be generated when building a wheel file?

I think that's doable and would make for nicer diffs. Tradeof would be docstring generation at build time would make wheel building less robust. Docsting generation is not terribly complex so I'd say it's worth it.

[rok]

Closing this in favor of apache#47609