Improve the code completion for python scene.

[damienmarchal]

Currently code completion on our binding is very poor.
This is connected to several issues that are explained later.

To see our progress here we are:

• code completion works by loading the modules (allows dynamic introspection in python interpreter)
•  add type hints stubs for our binary modules (allows static suggestion in code editor)
• fix bad c++ function signature (to improve the quality of suggestion on the binding)
• improve API design to favore code completion (improve the quality of suggestion on scene components)

Add type hints stubs for our binary modules

To correct this we simply need to automatically generate the typehints stub using a generation tools (eg: stubgen installed from MyMY )

To generate the typehint for the python module named "Sofa" you can simply do:

#> stubgen -p Sofa --include-docstrings  --inspect-mode

This is produce a set of files with extension ".pyi" in the "out" directory. You can then simply copy the files into the python module "Sofa".

FIx bad c++ function signature

When the typehints are generated their quality depend on how "rich' the type informations have been deduced from the c++ signature. When the c++ signature of a binded function is too vague to provide interesting completion (eg: py::object) we need to improve that by enriching the c++ binding.

The most common case for method that takes py::object as parameter or py::object as return type. To correct this we need to refined the type.

API design favoring code completion

The current Sofa API does not allow to get good insight on the type of manipulated object. In the following example,
the best the API allows is to infer that object1 is of type "Sofa.Core.Object". So there is no completion related to MechanicalObject.

node = Sofa.Core.Node("root")
object1 = node.addObject("MechanicalObject")

To help the type inference we need to expose rich types in python for the sofa scene objects. This was done for example in the Sofa.Component plugins.

from Sofa.Component.StateContainer.MechanicalObject import * 
node = Sofa.Core.Node("root")
object1 = node.addObject(MechanicalObject)      # Note that here we removed the " ... we are passing a real python object... and 
                                                                                       # the addObject method signature is saying that the type of the returned 
                                                                                       # object is the same as the one passed in parameters. 
object1.position.value  = [[1,2,3]]                             # Knowing that object1 is MechanicalObject allows suggestion for completion for both position and value

This may requires redesigning the SofaPython3 to make that very shiny.

Here is a small vidéo showing how code completion could work
https://www.youtube.com/watch?v=9ZTAcT-1C30

[damienmarchal]

Here is a small summary of the process

• Part 1 is trivial.
• Part 2 is kind of hack ! Because we fabricate python classes interfaces (.pyi files) for Sofa component (eg: MechanicalModel) while these component may not have a real python binding.
  This works because the type hints... are just "informative" and used by the code editor not by the actual python VM.

To improve part 1, the c++ binding code must be changed so it is more "narrow" on the input & output types of function. This can be done function by function once the part1 has been integrated in the packaging process of SofaPython3.

To improve part 2,
well there is a lot to do in term of identifying what is the best way to make these fake interface.
Eg: see the description of the generic type hints to describe that complex dynamic behavior like a sofa component, has a data field, that this data field as a "value" and that this type of the "value" is actually an array of float.

The code generation for part 2 is drafted in this branch: https://github.com/SofaDefrost/SofaComponents/tree/xp-component-stubgen

[damienmarchal]

I tried mymy's stubgen and pybind11-stubgen.

A first sight the latter produce better stubs...except it does not handle python's 3.12 feature for type generic.
This can be fixed by changing the regex to {func_name}([\S]){0,1}((?P.))\s(->\s*(?P.+))? in pybdin11-stubgen parser.py file.

[damienmarchal]

I made a long presentation to the DEFROST research team about code completion for SOFA.

The video is available at this address: https://youtu.be/tnrAkEd7unI