GPU-enabled enroot containers fail after updating to latest nvidia-container-toolkit

[tpdownes]

Release 1.17.7 of the nvidia-container-toolkit contains a regression that impacts Slurm clusters that update to this version. See the following reports:

• [rpm] libnvidia-container-tools should pin to nvidia-container-toolkit version NVIDIA/nvidia-container-toolkit#1091
• Update to version 1.17.7-1 on Ubuntu breaks pyxis/enroot in Slurm NVIDIA/nvidia-container-toolkit#1093
• With most recent Ubuntu packages upgrade, enroot container load fails NVIDIA/enroot#232

We advise customers not to upgrade running systems to this package. All compute nodes with GPUs can do this by creating a file at /etc/apt/preferences.d/block-broken-nvidia-container:

Package: nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1
Pin: version 1.17.7-1
Pin-Priority: 100

We will integrate this workaround robustly into our solutions while NVIDIA work on a fix to their packaging and solution.

Additionally, we recommend disabling unattended-upgrades.service if it is not already disabled:

sudo systemctl disable unattended-upgrades.service
sudo systemctl stop unattended-upgrades.service

[tpdownes]

The mitigation to this is included in v1.51.1. I'm going to convert this to a discussion.