Added venv infrastructure for running resilience project scripts on JUSUF #451
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When running the code on the cluster, you want to use modules as much as possible because they are installed in an optimised way. Installing MPI with conda, for instance, will give significantly lower performance compared to MPI modules.
However, there is not a module for everything and so you have to install some custom stuff using pip. In Jülich there are quotas for the number of files you can have in ~/home, which is quickly exceeded when installing large packages like numpy.
The support team recommended to use python venvs instead, which can be stored wherever you want. There is a handy template for this that let's you choose you own modules and packages and then easily activate the venv by running
source activate.sh. By adding this to the projects, others can see how to install on Jülich machines and adapt to their own machines if needed. So this is also good for reproducibility. I do recommend this to anyone running python on clusters!