Skip to content

Using Spack to build Albany and MALI dependencies on unsupported machines

Jump to bottom
Xylar Asay-Davis edited this page Jan 31, 2023 · 18 revisions

On E3SM-supported machines (Cori, Perlmutter, Anvil, Chrysalis, Compy, and Chicoma), MALI dependencies are maintained in the compass python package, and instructions for compiling MALI on those machines are described on a separate page. Running MALI on other machines is not supported, but this page describes instructions and advice for how users of other machines can try to compile and run MALI on their own.

MALI dependencies

MALI has the following dependencies:

  • Albany: Albany is a multiphysics code that includes a thermomechanically-coupled 3d first-order ice-flow velocity solver. Generally speaking, MALI requires Albany velocity solver. (It is possible to run with MALI without Albany with a shallow-ice approximation velocity solver, but this is primarily used for testing and is not scientifically supported.)
  • SCORPIO: SCORPIO is a parallel I/O interface library required for
  • NetCDF/Parallel-NetCDF: low-level I/O libraries used by SCORPIO. One or both are required; Parallel-NetCDF is required for parallel I/O.

Compiling MALI dependencies

Each of the dependencies can be compiled manually, but the recommended method is to use the spack package manager. In particular, Albany requires a specific build of the Trilinos collection of reusable scientific software libraries, which itself requires a number of additional third party libraries. Building this software stack manually and consistently, can be time-consuming and error prone. Spack allows users to apply a verified recipe for Albany and all of its dependencies, as well as the other MALI dependencies.

The spack recipes required for MALI are maintained on the develop branch of a fork of the spack repository hosted by the E3SM-Project Github organization at https://github.com/E3SM-Project/spack.

Information about setting up and running spack to compile Albany can be found here: https://github.com/sandialabs/Albany/wiki/Building-Albany-using-Spack

Describing a Spack environment for MALI dependencies

We recommend that you build the dependencies for MALI together in a Spack environment. This allows the packages to be built together in a consistent way that is not guaranteed if you try to install dependencies one-by-one. In Spack parlance, this is known as unified concretization.

To do this, you will create an env.yaml file similar to the following example for an Ubuntu laptop:

spack:
  specs:
  - gcc
  - openmpi
  - hdf5
  - netcdf-c
  - netcdf-fortran
  - parallel-netcdf
  - [email protected]+pnetcdf~timing+internal-timing~tools+malloc
  - albany@develop+mpas

  concretizer:
    unify: true
  packages:
    all:
      compiler: [[email protected]]
    gcc:
      externals:
      - spec: [email protected]
        prefix: /usr
      buildable: false

  config:
    install_missing_compilers: false
  compilers:
  - compiler:
      spec: [email protected]
      paths:
        cc: /usr/bin/gcc
        cxx: /usr/bin/g++
        f77: /usr/bin/gfortran
        fc: /usr/bin/gfortran
      flags: {}
      operating_system: ubuntu20.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

Typically your system will already have compilers if nothing else, and this is what we assume here. Give the appropriate path (replace /usr with the appropriate path on your system). We have had better luck with gcc than other compilers like Intel so far so that's our recommendation. Use gcc --version to determine the version and replace 9.4.0 with this number.

Finally, you might need to update the target and operating_system. This is a bit of a "catch 22" in that you can use spack to find this out but the script that follows will install spack for you so we assume you don't have it yet. For now, make your best guess using the info on this page and we'll correct it later if necessary.

Building the Spack environment

In the same directory as env.yaml, create a file build.sh based on the following example:

#!/bin/bash

export TMPDIR=/tmp
export SPACKDIR=$HOME/data/spack
export SPACKENV=mali

export YAML=$PWD/env.yaml

set -e

# load system modules here if you need to, e.g.:
# module purge
# module load gcc

if [ -d $SPACKDIR/develop ]; then
  # update to the latest version of develop
  cd $SPACKDIR/develop
  git fetch origin
  git reset --hard origin/develop
else
  git clone -b develop [email protected]:E3SM-Project/spack.git $SPACKDIR/develop
  cd $SPACKDIR/develop
fi
source share/spack/setup-env.sh
spack env remove -y $SPACKENV &> /dev/null && \
  echo "recreating environment: $SPACKENV" || \
  echo "creating new environment: $SPACKENV"
spack env create $SPACKENV $YAML
spack env activate $SPACKENV
spack install
spack config add modules:prefix_inspections:lib:[LD_LIBRARY_PATH]
spack config add modules:prefix_inspections:lib64:[LD_LIBRARY_PATH]

Change the directory in $SPACKDIR to a convenient place for you to build the dependencies (they're large so make sure you have several GB of free space). You may change $SPACKENV if you would like (this name is arbitrary and just needs to be the same in the activation script described below). You may also need to point $TMPDIR to a location where you can store several GB of temporary files as the packages build.

You may need to load a system module to get the compilers. If this is the case, add them where the comment indicates. You can add them to env.yaml file as well but this shouldn't be necessary.

Run the script with:

/bin/bash build.sh

This will likely take several hours.

If dependencies don't build as expected, you may get an error message suggesting that your operating_system or target aren't right. You can use:

cd $SPACKDIR/develop
source share/spack/setup-env.sh
spack arch -o
spack arch -g

to get something close to what Spack wants. If you get something like x86_64_v4 for the target, use x86_64 instead.

If you are getting other error messages, do your best to debug them but also feel free to get in touch with the MALI team and we'll help if we can.

Using the compiled Spack libraries for compiling MALI

After the MALI dependencies have been compiled once using spack, they can be use for compiling MALI indefinitely, and only need to be rebuilt if the user has a need for a newer version of Albany or other dependencies. However, the spack-built libraries need to be made accessible to your environment before compiling MALI. They also need to be accessible to the environment for running MALI, because everything will be built with dynamic (runtime) linking.

Each time you would like to compile or run MALI, you will want to source an activation script similar this example load_mali_env.sh:

export CONDADIR=$HOME/mambaforge
export CONDAENV=compass_test
export SPACKDIR=/lcrc/group/e3sm/ac.xylar/spack_mali
export SPACKENV=mali
export SPACKVIEW=$SPACKDIR/develop/var/spack/environments/$SPACKENV/.spack-env/view

# load compass conda environment here
echo Loading conda environment...
source $CONDADIR/etc/profile.d/conda.sh
source $CONDADIR/etc/profile.d/mamba.sh
mamba activate $CONDAENV
echo Done.
echo

echo Loading Spack environment...

# If you need to purge system modules, do so before loading the Spack env:
# module purge

source $SPACKDIR/develop/share/spack/setup-env.sh
spack env activate $SPACKENV

# add system modules here, e.g.:
# module load gcc

echo Done.
echo

source $SPACKVIEW/export_albany.in
export NETCDF=$(dirname $(dirname $(which nc-config)))
export NETCDFF=$(dirname $(dirname $(which nf-config)))
export PNETCDF=$(dirname $(dirname $(which pnetcdf-config)))
export PIO=$SPACKVIEW
export MPAS_EXTERNAL_LIBS="${MPAS_EXTERNAL_LIBS} ${ALBANY_LINK_LIBS} -lstdc++"

export USE_PIO2=true
export HDF5_USE_FILE_LOCKING=FALSE
export HDF5_USE_FILE_LOCKING=FALSE

We assume you have created a compass conda environment, where you have Miniconda or Mambaforge installed in $CONDADIR and the compass environment is in $CONDAENV.

The variables $SPACKDIR and $SPACKENV are the same as used in build.sh above.

If you added system modules in the build.sh, you will probably also want to load the same modules in the activation script, as indicated by the comments in the script.

Note that MPAS_EXTERNAL_LIBS is a list of libraries and paths for the Albany installation and its own dependencies (e.g. Trilinos). This environment variable should be set to the contents of the file export_albany.in that is generated in Albany's installation directory. It may also be necessary to add -lmpi_cxx and either -lstdc++ (for most machines) or -lc++ (for OSX) to the list.

Once you have edited and sourced your activation script, you can proceed to build MALI following the standard instructions. Remember to include source load_mali_env.sh in batch jobs run on compute nodes.

Clone this wiki locally