nvpassthrough: bind every PCI function of a GPU, not just the first auxiliary one - #115
Open
jjacobelli wants to merge 2 commits into
Open
jjacobelli wants to merge 2 commits into
jjacobelli wants to merge 2 commits into
Conversation
…uxiliary one
VFIO assigns an entire IOMMU group to a guest and refuses the group unless every
device in it is bound to a vfio driver or to no driver at all. A discrete NVIDIA
GPU is a multi-function PCI device: .0 VGA/3D, .1 HDMI audio, and on Turing-era
boards .2 (VirtualLink USB xHCI) and .3 (USB-C UCSI).
getGraphicsAuxDev() could not find all of those:
* it returned on the first match, so at most one auxiliary function was ever
bound;
* it discovered functions only via "consumer:pci:" device links. The HDA audio
function creates one, but the VirtualLink xHCI and USB-C UCSI functions do
not, so they were structurally invisible;
* it returned early unless the device was PCIVgaControllerClass, so a GPU
enumerating as a 3D controller (0x030200) got no auxiliary handling at all.
In practice this leaves xhci_hcd owning .2. The IOMMU group is then not viable
and passthrough fails with "vfio: group N is not viable".
Replace it with getAuxDevices(), which returns every other function of the same
physical card. Functions are discovered both by enumerating siblings that share
a domain:bus:device and by following consumer links, and the two sets are
unioned and de-duplicated. Siblings are scoped to the card rather than to the
IOMMU group deliberately: where ACS is unavailable a group can span a whole root
port, and unrelated devices must not be unbound from their drivers. A vendor
check guards against touching anything that is not an NVIDIA function.
BindToVFIODriver() and Unbind() now iterate over all of them.
Adds table-driven tests for the discovery logic covering multi-function boards,
single-function GPUs, foreign-vendor siblings, devices on other slots, and the
de-duplication of a function reachable both ways.
Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
jjacobelli
marked this pull request as ready for review
September 10, 2026 12:59
tariq1890
reviewed
Sep 11, 2026
tariq1890
reviewed
Sep 11, 2026
… devices getAuxDevices() treated every failure while inspecting a candidate function as "skip it": a stat error of any kind, and a vendor read or parse failure, all returned nil. Only one of those is benign. A function can disappear between being enumerated and being inspected, and skipping it is correct. But a permission or I/O error reading a device, or an unparseable vendor, means a function that may well need binding is dropped without a word. BindToVFIODriver() then reports success while leaving that function on its host driver, and the IOMMU group is rejected at VM start with "vfio: group N is not viable" -- the very failure this code exists to prevent, now with nothing in the logs to explain it. Tolerate only fs.ErrNotExist, on both the stat and the vendor read, and return every other error. This matches getDriver(), which already reports a missing driver symlink as "no driver" and propagates the rest. Adds tests for both halves: an unparseable vendor surfaces as an error, and a consumer link naming an absent function is skipped silently. Signed-off-by: Jordan Jacobelli <jjacobelli@nvidia.com>
tariq1890
approved these changes
Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to NVIDIA/k8s-driver-manager#254.
Problem
VFIO assigns an entire IOMMU group to a guest and refuses the group unless every
device in it is bound to a vfio driver or to no driver at all. A discrete NVIDIA
GPU is a multi-function PCI device:
.0VGA/3D,.1HDMI audio, and onTuring-era boards
.2(VirtualLink USB xHCI) and.3(USB-C UCSI).getGraphicsAuxDev()could not find all of those:consumer:pci:device links. The HDA audiofunction creates one, but the VirtualLink xHCI and USB-C UCSI functions do not,
so they were structurally invisible;
PCIVgaControllerClass, so a GPUenumerating as a 3D controller (
0x030200) got no auxiliary handling at all.In practice this leaves
xhci_hcdowning.2. The IOMMU group is then notviable and passthrough fails with
vfio: group N is not viable.Fix
Replace it with
getAuxDevices(), which returns every other function of the samephysical card. Functions are discovered both by enumerating siblings that share a
domain:bus:deviceand by following consumer links; the two sets are unioned andde-duplicated.
Siblings are scoped to the physical card rather than to the IOMMU group
deliberately: where ACS is unavailable a group can span a whole root port, and
unrelated devices must not be unbound from their drivers. A vendor check against
nvpci.PCINvidiaVendorIDguards against touching anything that is not an NVIDIAfunction.
BindToVFIODriver()andUnbind()now iterate over all of them.getAuxDevices()takes the devices root as a parameter so the discovery logic istestable against a fake sysfs tree.
Tests
Adds table-driven tests for the discovery logic covering multi-function boards,
single-function GPUs, foreign-vendor siblings, devices on other slots, class
handling (VGA / 3D controller / NVSwitch), and the de-duplication of a function
reachable both as a sibling and via a consumer link. A second test asserts the
currently-bound driver is reported per function, which is what lets
BindToVFIODriver()skip functions already on the vfio driver.make assert-fmt goimports vet check-vendorandgo test ./pkg/nvpassthrough/pass; no dependency or vendor changes.