FieldsIO implementation for 0D to 2D cartesian grid fields, with MPI

[tlunet]

A small issue I encountered, in my attempt to use pySDC generated data to train a ML model for Rayleigh Benard convection, is that there was no efficient options (that I knew of) to write many (many many) field solutions when running many space-parallel simulations, hence a "tiny" technical problem.

So I quickly implemented a FieldsIO generic (abstract) class to write multiple field solution into a binary file, and specialized it to scalar, 1D and 2D fields written on Cartesian grids. All are compatible with Numpy arrays, and each class is as generic as possible, such that most problem class using one or more mesh class for their state variable can use it.

Also, I added the MPI-IO functionalities to the 1D and 2D grid classes, that can be used with any type of Cartesian block decomposition : user simply has to specify its own local bounds at start, and the classes automatically deals with the collective writing of the local fields.

Testing is done in sequential and parallel, and the parallel tests uses some basic BlockDecomposition class, that can be used up to 3D problems.

Note

This was mostly inspired from previous works during my PhD. It may be moved in a near future to a more generic package (as qmat), since there is also an high-order interpolation algorithm (also compatible with MPI cartesian decomposition, developed during the end of my PhD) that I would like to make as a standalone package, and was needing this specific part.

Bonus

CI uses now a "modern" python approach that installs pySDC in the test environment, so tests do not need some sys.path modification anymore, to ensure pySDC is in the path when running tests ...

[brownbaerchen]

Looks very nice! I really have just two main issues:

• need to write global grid but local fields in MPI version of 2d cartesian grid
• I would like protection against accidental overwrite of previous files

[tlunet]

All seems fine now, can be merged anytime

[pancetta]

Go ahead, can’t do this today.On 18. Jan 2025, at 10:37, Thibaut Lunet ***@***.***> wrote: All seems fine now, can be merge anytime —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>