Skip to content

nvpassthrough: bind every PCI function of a GPU, not just the first auxiliary one - #115

Open
jjacobelli wants to merge 2 commits into
NVIDIA:mainfrom
jjacobelli:fix/bind-all-gpu-functions
Open

jjacobelli wants to merge 2 commits into
NVIDIA:mainfrom
jjacobelli:fix/bind-all-gpu-functions

Conversation

@jjacobelli

Copy link
Copy Markdown

Follow-up to NVIDIA/k8s-driver-manager#254.

Problem

VFIO assigns an entire IOMMU group to a guest and refuses the group unless every
device in it is bound to a vfio driver or to no driver at all. A discrete NVIDIA
GPU is a multi-function PCI device: .0 VGA/3D, .1 HDMI audio, and on
Turing-era boards .2 (VirtualLink USB xHCI) and .3 (USB-C UCSI).

getGraphicsAuxDev() could not find all of those:

  • it returned on the first match, so at most one auxiliary function was ever bound;
  • it discovered functions only via consumer:pci: device links. The HDA audio
    function creates one, but the VirtualLink xHCI and USB-C UCSI functions do not,
    so they were structurally invisible;
  • it returned early unless the device was PCIVgaControllerClass, so a GPU
    enumerating as a 3D controller (0x030200) got no auxiliary handling at all.

In practice this leaves xhci_hcd owning .2. The IOMMU group is then not
viable and passthrough fails with vfio: group N is not viable.

Fix

Replace it with getAuxDevices(), which returns every other function of the same
physical card. Functions are discovered both by enumerating siblings that share a
domain:bus:device and by following consumer links; the two sets are unioned and
de-duplicated.

Siblings are scoped to the physical card rather than to the IOMMU group
deliberately: where ACS is unavailable a group can span a whole root port, and
unrelated devices must not be unbound from their drivers. A vendor check against
nvpci.PCINvidiaVendorID guards against touching anything that is not an NVIDIA
function.

BindToVFIODriver() and Unbind() now iterate over all of them.

getAuxDevices() takes the devices root as a parameter so the discovery logic is
testable against a fake sysfs tree.

Tests

Adds table-driven tests for the discovery logic covering multi-function boards,
single-function GPUs, foreign-vendor siblings, devices on other slots, class
handling (VGA / 3D controller / NVSwitch), and the de-duplication of a function
reachable both as a sibling and via a consumer link. A second test asserts the
currently-bound driver is reported per function, which is what lets
BindToVFIODriver() skip functions already on the vfio driver.

make assert-fmt goimports vet check-vendor and go test ./pkg/nvpassthrough/
pass; no dependency or vendor changes.

…uxiliary one

VFIO assigns an entire IOMMU group to a guest and refuses the group unless every
device in it is bound to a vfio driver or to no driver at all. A discrete NVIDIA
GPU is a multi-function PCI device: .0 VGA/3D, .1 HDMI audio, and on Turing-era
boards .2 (VirtualLink USB xHCI) and .3 (USB-C UCSI).

getGraphicsAuxDev() could not find all of those:

  * it returned on the first match, so at most one auxiliary function was ever
    bound;
  * it discovered functions only via "consumer:pci:" device links. The HDA audio
    function creates one, but the VirtualLink xHCI and USB-C UCSI functions do
    not, so they were structurally invisible;
  * it returned early unless the device was PCIVgaControllerClass, so a GPU
    enumerating as a 3D controller (0x030200) got no auxiliary handling at all.

In practice this leaves xhci_hcd owning .2. The IOMMU group is then not viable
and passthrough fails with "vfio: group N is not viable".

Replace it with getAuxDevices(), which returns every other function of the same
physical card. Functions are discovered both by enumerating siblings that share
a domain:bus:device and by following consumer links, and the two sets are
unioned and de-duplicated. Siblings are scoped to the card rather than to the
IOMMU group deliberately: where ACS is unavailable a group can span a whole root
port, and unrelated devices must not be unbound from their drivers. A vendor
check guards against touching anything that is not an NVIDIA function.

BindToVFIODriver() and Unbind() now iterate over all of them.

Adds table-driven tests for the discovery logic covering multi-function boards,
single-function GPUs, foreign-vendor siblings, devices on other slots, and the
de-duplication of a function reachable both ways.

Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
@jjacobelli
jjacobelli marked this pull request as ready for review September 10, 2026 12:59
Comment thread pkg/nvpassthrough/nvpassthrough.go
Comment thread pkg/nvpassthrough/nvpassthrough.go
… devices

getAuxDevices() treated every failure while inspecting a candidate function as
"skip it": a stat error of any kind, and a vendor read or parse failure, all
returned nil.

Only one of those is benign. A function can disappear between being enumerated
and being inspected, and skipping it is correct. But a permission or I/O error
reading a device, or an unparseable vendor, means a function that may well need
binding is dropped without a word. BindToVFIODriver() then reports success while
leaving that function on its host driver, and the IOMMU group is rejected at VM
start with "vfio: group N is not viable" -- the very failure this code exists to
prevent, now with nothing in the logs to explain it.

Tolerate only fs.ErrNotExist, on both the stat and the vendor read, and return
every other error. This matches getDriver(), which already reports a missing
driver symlink as "no driver" and propagates the rest.

Adds tests for both halves: an unparseable vendor surfaces as an error, and a
consumer link naming an absent function is skipped silently.

Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants