Set up the NVIDIA persistence daemon

This article explains how to set up the NVIDIA persistence daemon.

For applications to interact with an NVIDIA GPU, the NVIDIA kernel driver must be loaded, running, and connected to the application. If no application is using the GPU, the kernel driver may not be loaded and running. This can result in application startup delays and loss of persistent configurations/data in the driver.

NVIDIA provides a user-space daemon on Linux to support persistence of driver state across different CUDA compute applications. The daemon runs as a system service which mimics an external client of the GPU but does not actually use the GPU. We strongly recommend that you use the NVIDIA persistence daemon to ensure persistence of driver state on your hosts.

The following article explains how to install the NVIDIA persistence daemon as a permanently running system service.

This approach makes use of a permanently running system service. You must determine if this is a viable solution for your deployment and if it meets your system security requirements.

Service configuration method

Step 1: Create or choose a user

We recommend that you execute the service process as a normal user without sudo privileges.

Use the nvidia-persistenced user if it exists. Otherwise you can either create a new nvidia-persistenced user (recommended), or use another existing user. To check if the nvidia-persistenced user exists, use the following command:

cat /etc/passwd | grep nvidia-persistenced

Configure the user as an environment variable PERSISTENCED_USER to be used in the next step:

export PERSISTENCED_USER=nvidia-persistenced

Step 2: Configure/reconfigure the service

sudo mkdir -p /etc/systemd/system/nvidia-persistenced.service.d/
cat << EOF | sudo tee -a "/etc/systemd/system/nvidia-persistenced.service.d/override.conf"
[Service]
ExecStart=
ExecStart=/usr/bin/nvidia-persistenced --user ${PERSISTENCED_USER} --persistence-mode --verbose
EOF

sudo systemctl daemon-reload 
[[ $(sudo systemctl is-enabled nvidia-persistenced.service) == "disabled" ]] && sudo systemctl enable nvidia-persistenced.service
sudo systemctl restart nvidia-persistenced.service

For more details and information about security concerns when creating the service, see NVIDIA Device Persistent Daemon - Permissions and Security in the NVIDIA documentation.

Persistence mode (legacy)

We strongly recommend that you use the NVIDIA persistence daemon instead of the legacy persistence mode, which is less robust and is approaching end of support by NVIDIA. If system security requirements prevents you from installing a system service, using the legacy persistence mode is an option. However, persistence mode will eventually be deprecated and is not guaranteed to work in the long term.

For more information, see Persistence Mode (Legacy) in the NVIDIA documentation.