Plan to support free-threaded Python

[rostan-t]

In free-threaded build, importing pydantic_core currently results in a segmentation fault. PyO3 v0.23 introduces support for free-threaded Python and addresses this issue. Is there any short-term plan to migrate to PyO3 0.23?

Currently I see the following blockers:

• Jiter support for free-threaded Python (support free-threaded Python jiter#165)
• Making SerializationCallable Sync, which isn't because of CollectWarnings and SerRecursionState contain RefCells
• Replacing IntoPy/ToPyObject by IntoPyObject
• Replacing calls to deprecated APIs (*_bound methods, into_py/to_object)

[davidhewitt]

Yes, I was testing PyO3 0.23 (when it was just a git branch) in #1450 and plan to proceed further with that upgrade this week.

There is potentially additional work required beyond your list to actually make free-threaded support ok, both here and in pydantic (where there are a bunch of caches which may or may not be thread-safe).

[davidhewitt]

PyO3 has now been bumped but there's a build failure on aarch64 on main. I'm trying to identify the cause. Before we can call free-threaded Python supported we should probably add extensive testing here and downstream in the pydantic main Python repo. I think this is realistically a Pydantic 2.11 feature cc @pydantic/oss

[samuelcolvin]

@sydney-runkle @Viicos this should go on the TODO list for Pydantic 2.11.

[davidhewitt]

For what it's worth, the build failures are fixed so we could move forward here in core relatively quickly. I fear that upstream pydantic is likely to be harder due to caches.

[davidhewitt]

We merged a test run in #1626, which is the first step. Build still fails, so fixing that will need to happen next.

[davidhewitt]

After #1628 the free-threaded test suite is all happy except for assertions dependent on inline-snapshot (where free-threaded support is pending). As far as I am concerned, free-threading now works fine in pydantic-core.

The work will now be in pydantic main package, to add testing etc. Follow at pydantic/pydantic#10483