Compiling Kratos with MPI support

Running Kratos with MPI support requires compiling and enabling:

Kratos' METIS Application, which provides an interface to the METIS graph partitioning library, to generate mesh partitions for MPI runs.
Kratos' Trilinos Application, which adapts components of the Trilinos project, providing distributed-memory matrix and vector classes and linear solvers.

Once these dependencies are installed (either by downloading or by compiling them), building Kratos itself requires the following tweaks in the building script, as shown in the following snippets.

Make Lapack and MPI libraries available:

# Load modules required for compilation
...
module load intel/oneapi2021/mkl/2021.1.1
module load openmpi
...

Add TrilinosApplication and MetisApplication to the list of applications to compile:

export KRATOS_APPLICATIONS=
...
add_app ${KRATOS_APP_DIR}/TrilinosApplication
add_app ${KRATOS_APP_DIR}/MetisApplication

Enable MPI and OpenMP parallelization, and hint cmake the location of Trilinos and Metis libraries (highly dependent on the system setup):

cmake -H"${KRATOS_SOURCE}" -B"${KRATOS_BUILD}" \
...
-DUSE_MPI=ON \
-DKRATOS_SHARED_MEMORY_PARALLELIZATION="OpenMP" \
-DKRATOS_SHARED_MEMORY_PARALLELIZATION="OpenMP" \
-DTRILINOS_ROOT="${HOME}/Projects/Trilinos/build" \
-DMETIS_ROOT_DIR="${HOME}/Projects/METIS/build" \

A complete working script can be found below. Also, don't forget to add the libraries path to the environment variables before executing Kratos.

Getting dependencies from packages

The easiest way to get the required libraries is by the Linux distribution's package repository.

Ubuntu:

sudo apt install metis-dev
sudo apt install trilinos-all-dev

TODO: confirm packages names, and add other systems.

Compiling dependencies

In some cases (e.g., no admin permissions, no packages or modules present, outdated versions) there may be necessary to build the libraries. This is a set of scrips for compiling GKlib, METIS and Trilinos libraries and their dependencies. Following the building instructions of their documentation is not enough for compiling them successfully for Kratos, as at the moment there are some workarounds the need to be applied.

The following scripts are just guidelines, and most likely they need to be adapted to each individual setup. For each package, a specific commit hash tag is cloned from its repository in GitHub, to allow reproducibility of the compilation.

We are assuming that they all are cloned and built in a WORK directory, for example:

export WORK=${HOME}/Projects

Tested with cmake v3.24, gcc v10.2

NOTE: As of July 6, 2022, there are a couple of files requiring a C++ standard >= 14 to compile. We won't fix this issue as Kratos is transitioning to C++17 at the moment, which will solve the problem automatically.

Step 1. Getting and building GKlib

GKlib is a dependency of METIS, so it has to be built first. The key step for building is to add the flag below to avoid compilation errors.

# GKlib
# - Last tested on July 5th, 2022. Commit a7f8172

# Clone
cd ${WORK}
git clone [email protected]:KarypisLab/GKlib.git

# Build
cd GKlib
cmake . \
-DBUILD_SHARED_LIBS=ON \
-DCMAKE_INSTALL_PREFIX=${WORK}/GKlib/build \
-DCMAKE_C_FLAGS="${CMAKE_C_FLAGS} -D_POSIX_C_SOURCE=199309L"
make install

Step 2. Getting and building METIS

Building METIS requires patching CMakeLists.txt file.

# METIS
# - Last tested on July 5th, 2022. Commit 94c03a6

# Download
cd ${WORK}
git clone [email protected]:KarypisLab/METIS.git 

# Compile
cd METIS/libmetis
echo "include_directories(.)" > CMakeLists.txt
echo "" >> CMakeLists.txt
echo "file(GLOB metis_sources *.c)" >> CMakeLists.txt
echo "" >> CMakeLists.txt
echo "add_library(metis \${METIS_LIBRARY_TYPE} \${metis_sources})" >> CMakeLists.txt
echo "target_link_libraries(metis PUBLIC GKlib)" >> CMakeLists.txt
echo "" >> CMakeLists.txt
echo "if(METIS_INSTALL)" >> CMakeLists.txt
echo "  install(TARGETS metis" >> CMakeLists.txt
echo "    LIBRARY DESTINATION lib" >> CMakeLists.txt
echo "    RUNTIME DESTINATION lib" >> CMakeLists.txt
echo "    ARCHIVE DESTINATION lib)" >> CMakeLists.txt
echo "endif()" >> CMakeLists.txt
cd ..
make config shared=1 prefix=${WORK}/METIS/build gklib_path=${WORK}/GKlib/build
make install

Step 3. Getting and building Trilinos

The following script builds Trilinos with only the packages required by the TrilinosApplication. Building requires LAPACK and an installation of MPI (OpenMPI in this case.) Intel's MKL is only used to provide BLAS and LAPACK.

# Trilinos
# - Last tested on July 4th, 2022
# - Commit 9c03d9b

# Clone
git clone [email protected]:trilinos/Trilinos.git

# Build
module load intel/oneapi2021/mkl/2021.1.1  # provides LAPACK
module load openmpi  # provides MPI

cd Trilinos
mkdir -p build
cd build
rm -rf CMakeCache.txt CMakeFiles/ cmake_install.cmake

export MAKEFLAGS=
cmake \
-D CMAKE_INSTALL_PREFIX=$WORK/Trilinos/build \
-D CMAKE_BUILD_TYPE="RELWITHDEBINFO" \
-D TPL_ENABLE_MPI=ON \
-D BUILD_SHARED_LIBS=ON \
-D Trilinos_ENABLE_Epetra=ON \
-D Trilinos_ENABLE_EpetraExt=ON \
-D Trilinos_ENABLE_Triutils=ON \
-D Trilinos_ENABLE_AztecOO=ON \
-D Trilinos_ENABLE_Ifpack=ON \
-D Trilinos_ENABLE_ML=ON \
-D Trilinos_ENABLE_Amesos=ON \
-D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
-D TPL_ENABLE_MKL=OFF \
-D BLAS_LIBRARY_DIRS="${MKLROOT}/lib/intel64" \
-D BLAS_LIBRARY_NAMES="mkl_rt" \
-D LAPACK_LIBRARY_DIRS="${MKLROOT}/lib/intel64" \
-D LAPACK_LIBRARY_NAMES="mkl_rt" \
$WORK/Trilinos
make -j 6 install