Proxmox: Plex LXC wait for Nvidia script

I've been having a persistent problem where Plex, running on Proxmox, was not able to see my GTX 1050 Ti on a fresh host boot until I ran nvidia-smi on the host. I even tried setting a startup delay for the container, and it still was not working.

Working through the issue with Copilot, it turns out that the nvidia-modprobe that was running as an lxc.hook.pre-start command in the container's config file was not bringing up all of the devices within /dev/nvidia*. I don't remember where I grabbed this from, but it was deep within one of the suggestions on how to add Nvidia support to a Proxmox LXC. Only nvidia-smi brings them all up. It seems like something needs to fully touch the card before it can be used.

So I'd figure I'd document what I did here.

Within my /etc/pve/lxc/ctid.conf, I have the following. The device numbers may differ for you.

lxc.hook.pre-start: /usr/local/bin/lxc-nvidia-wait.sh
lxc.environment: NVIDIA_VISIBLE_DEVICES=all
lxc.environment: NVIDIA_DRIVER_CAPABILITIES=all
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 511:* rwm
lxc.cgroup2.devices.allow: c 236:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-caps/nvidia-cap1 dev/nvidia-caps/nvidia-cap1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-caps/nvidia-cap2 dev/nvidia-caps/nvidia-cap2 none bind,optional,create=file
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file

This is all below the last line, which in my config, is:

unprivileged: 1

The emphasis is on this line:

lxc.hook.pre-start: /usr/local/bin/lxc-nvidia-wait.sh

This calls a script stored in /usr/local/bin/ called lxc-nvidia.wait.sh. This script looks like this:

#!/bin/bash

START=$(date +%s)


for i in $(seq 1 60); do
    /usr/bin/nvidia-smi >/dev/null 2>&1

    if [ -c /dev/nvidia0 ] &&
       [ -c /dev/nvidiactl ] &&
       [ -c /dev/nvidia-uvm ] &&
       [ -c /dev/nvidia-uvm-tools ] &&
       [ -c /dev/nvidia-caps/nvidia-cap1 ] &&
       [ -c /dev/nvidia-caps/nvidia-cap2 ]; then
        END=$(date +%s)
        echo "NVIDIA devices became available after $((END-START)) seconds"
        exit 0
    fi

    sleep 10
done

echo "NVIDIA devices did not appear within 600 seconds" >&2
exit 1

What this does is runs nvidia-smi within a loop (discarding its output), after which the devices are all checked. If the devices exist, then the script exits. If it does not, the loop sleeps for 10 seconds, then repeats. If the devices never show after ten minutes, the script exits with an error code, which errors out the starting of the container. I'd love to be able to output the messages directly to the tasks log in the webgui, but I don't know how. Oh well.