Skip to content

[AI Generated] network: verify SR-IOV VF debugfs diagnostics - #4707

Open
VENNELA2132 wants to merge 2 commits into
microsoft:mainfrom
VENNELA2132:ernis/new_testcases/debugfs
Open

VENNELA2132 wants to merge 2 commits into
microsoft:mainfrom
VENNELA2132:ernis/new_testcases/debugfs

Conversation

@VENNELA2132

@VENNELA2132 VENNELA2132 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Description

Adds verify_sriov_debugfs_diagnostics to the SR-IOV suite.

debugfs is the first thing anyone opens when an SR-IOV VF misbehaves in
production, so a file that a driver publishes there and then refuses to answer
is a defect regardless of what it was meant to report. This case mounts debugfs
if needed, resolves the VF's bound driver, then reads back every readable file
under that driver's debugfs tree, with each read bounded by a byte cap and a
timeout.

It asserts only what every driver owes its debugfs entries, which is that they
can be read. Layouts differ by orders of magnitude between drivers, so any rule
written around the size, naming, or structure of one driver's tree rejects the
others. Concretely, mlx5 keys its tree by PCI slot (mlx5/<slot>/...) while
mana keys it by ordinal (mana/1/vport0), and file counts range from 39 to
over 1700 on the same fleet.

Outcomes are graded so that only real defects fail:

  • A read that never answers fails. Nothing innocent explains it, and it
    blocks whoever is diagnosing the VF.
  • A read that returns an error is logged, not failed. Some entries are
    command interfaces that answer only once a command has been staged through
    them, and a kernel in lockdown refuses every debugfs entry that is not
    read-only.
  • A tree that is absent, or holds nothing but empty files, skips. The
    driver published nothing to read back.
  • A VF with no driver bound is split on kernel log evidence: a logged probe
    failure fails, because a driver was present, refused the device, and the
    VM has lost accelerated networking; silence skips, because no driver in
    that kernel ever claimed the VF.

Related Issue

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Refactoring
  • Documentation update

Checklist

  • Description is filled in above
  • No credentials, secrets, or internal details are included
  • Peer review requested (if not, add required peer reviewers after raising PR)
  • Tests executed and results posted below

Test Validation

Key Test Cases:
verify_sriov_debugfs_diagnostics

Impacted LISA Features:
Sriov, NetworkInterface

Tested Azure Marketplace Images:

  • canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest
  • canonical ubuntu-24_04-lts server latest
  • canonical ubuntu-24_04-lts server-gen1 latest
  • canonical ubuntu-24_04-lts server-arm64 latest
  • debian debian-11 11 latest
  • debian debian-12 12-gen2 latest
  • debian debian-12 12-arm64 latest
  • microsoftcblmariner azure-linux-3 azure-linux-3-gen2 latest
  • redhat rhel 810-gen2 latest
  • redhat rhel 9_5 latest
  • redhat rhel-arm64 9_5-arm64 latest
  • suse sles-15-sp6 gen2 latest
  • suse sles-15-sp6-arm64 gen2 latest

Test Results

Run across x64 gen1, x64 gen2, and arm64 matrices in westus3, covering both
NIC drivers Azure hands out (mana and mlx5_core).

Image VM Size Result
canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest Standard_D2ds_v5 PASSED, mana, 39/39 read
canonical ubuntu-24_04-lts server latest Standard_D2s_v3 PASSED, mlx5, 1475/1477 read
canonical ubuntu-24_04-lts server-gen1 latest Standard_D2ds_v5 PASSED, mana, 40/40 read
canonical ubuntu-24_04-lts server-arm64 latest Standard_D2pds_v5 PASSED, mlx5, 1475/1477 read
debian debian-12 12-arm64 latest Standard_D2pds_v5 PASSED, mlx5, 1481/1482 read
microsoftcblmariner azure-linux-3 azure-linux-3-gen2 latest Standard_D2ds_v5 PASSED, mlx5, 1419/1485 read, 66 refused under kernel lockdown
redhat rhel 810-gen2 latest Standard_D2ds_v5 PASSED, mlx5, 1490/1492 read
redhat rhel-arm64 9_5-arm64 latest Standard_D2pds_v5 PASSED, mlx5, 1487/1489 read
suse sles-15-sp6 gen2 latest Standard_D2ds_v5 PASSED, mana, 39/39 read
suse sles-15-sp6-arm64 gen2 latest Standard_D2pds_v5 PASSED, mlx5, 1490/1492 read
debian debian-12 12-gen2 latest Standard_D2ds_v5 SKIPPED, mana exposes no debugfs tree
redhat rhel 9_5 latest Standard_D2ds_v5 SKIPPED, mana exposes no debugfs tree
debian debian-11 11 latest Standard_D2ds_v5 SKIPPED, 5.10 kernel ships no driver claiming the VF

Every branch of the case was exercised on hardware rather than reasoned about:

Branch Evidence
Pass mana 1732/1732 files, mlx5 1475/1477 files
Skip, no debugfs tree debian-12, rhel 9_5
Skip, no driver in kernel debian-11, 5.10 kernel predates the driver
Fail, probe failed reproduced on hardware, plus a deterministic repro by unbinding the VF and injecting the probe-failure line
Fail, read never answers FIFO planted in the tree, reported ERR124 and failed as intended
Refused reads logged, not failed azure-linux-3 under kernel lockdown, 66 files refused, case still PASSED

Notes for reviewers:

  • The case mounts debugfs when it is absent and does not call mark_dirty. A
    debugfs mount is benign and is usually already present, but flagging it in
    case the suite's convention says otherwise.
  • The read timeout escalates to KILL (timeout -k 1), so a read that ignores
    TERM cannot stall the walk. A read wedged uninterruptibly in the kernel
    cannot be signalled at all and would hit LISA's own command timeout instead.
  • Several images skip because their mana build publishes no debugfs tree. That
    is reported rather than hidden, so the skip count is expected to be non-zero
    on any broad image matrix.

Add verify_sriov_debugfs_diagnostics to the SR-IOV suite. It mounts
debugfs if needed, resolves the VF's bound driver, then reads back every
readable file under that driver's debugfs tree, each read bounded by a
byte cap and a timeout. It asserts only what every driver owes its
debugfs entries, which is that they can be read, since layouts differ by
orders of magnitude between drivers and any rule written around the size
or naming of one of them rejects the rest. A tree that is absent, or
holds nothing but empty files, is skipped, because the driver published
nothing to read back. A read that returns an error is logged rather than
failed, since some entries are command interfaces that answer only once
a command has been staged through them and a kernel in lockdown refuses
every debugfs entry that is not read-only, while a read that never
answers is a failure.

A VF with no driver bound is split on the evidence in the kernel log. A
logged probe failure against the VF means a driver was present and
refused the device, so the VM has lost accelerated networking and the
case fails with that log line. Silence means no driver in the kernel
ever claimed the VF, so the distro predates it and the case skips. The
VF modalias is not used to make this split, because a driver built into
the kernel resolves to no module and would be mistaken for an absent one.

Generated by the LISA Test Writer prompt. Reviewers: verify AAA logic,
metadata, requirements, and node hygiene.

Signed-off-by: Erni Sri Satya Vennela <ernis@microsoft.com>
Copilot AI lite review requested due to automatic review settings September 8, 2026 14:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new test has a few failure/skip and timeout-handling issues that can cause misleading outcomes or premature command timeouts in large trees.

Key Test Cases:
verify_sriov_debugfs_diagnostics|verify_sriov_basic

Impacted LISA Features:
Sriov, NetworkInterface

Tested Azure Marketplace Images:

  • canonical 0001-com-ubuntu-server-jammy 22_04-lts-gen2 latest
  • canonical ubuntu-24_04-lts server latest

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new SR-IOV test case to validate that SR-IOV VF driver debugfs diagnostics are readable (bounded by per-file timeout and byte cap), aiming to catch “hung read” defects that block production triage.

Changes:

  • Add verify_sriov_debugfs_diagnostics to mount debugfs if needed, resolve the VF’s bound driver, locate the driver’s debugfs subtree, and read all readable files with bounds.
  • Add helper _read_debugfs_tree to enumerate readable files and capture per-file read outcomes (bytes read vs. error/timeout).
File summaries
File Description
lisa/microsoft/testsuites/network/sriov.py Adds the new debugfs diagnostics test and a helper to walk/read the driver debugfs tree.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lisa/microsoft/testsuites/network/sriov.py Outdated
Comment thread lisa/microsoft/testsuites/network/sriov.py
Comment thread lisa/microsoft/testsuites/network/sriov.py
@LiliDeng
LiliDeng marked this pull request as ready for review September 9, 2026 00:41
Comment thread lisa/microsoft/testsuites/network/sriov.py
Signed-off-by: Erni Sri Satya Vennela <ernis@microsoft.com>
Copilot AI review requested due to automatic review settings September 16, 2026 10:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved security and reliability issues remain in temporary-file handling, traversal status reporting, and timeout enforcement.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

lisa/microsoft/testsuites/network/sriov.py:1217

  • Major: the trailing rm -f makes the combined shell command return success even when find fails, so assert_exit_code(0) cannot detect an enumeration/traversal error and the test may silently inspect only a partial tree. Preserve the find status, clean the scratch file, and then exit with the saved status.
            f"-exec sh -c '{per_file}' _ {{}} '{scratch}' \\;; "
            f"rm -f '{scratch}'",

lisa/microsoft/testsuites/network/sriov.py:1186

  • Major: The raw timeout command is not checked before the walk. If it is absent or does not support -k, every head invocation produces an infrastructure error (for example, exit 127), which is encoded as a refused read; when this happens for the whole tree, the later SkippedException makes the case look like an expected driver with no readable content. Validate the Timeout tool up front or distinguish missing/invalid timeout failures from driver read errors so the test never passes by skipping without enforcing its bound.
        per_file = (
            f"if timeout -k {self.DEBUGFS_READ_KILL_GRACE_SECONDS} "
            f"{self.DEBUGFS_READ_TIMEOUT_SECONDS} "
            f'head -c {self.DEBUGFS_READ_CAP_BYTES} "$1" > "$2" 2>/dev/null; '
  • Files reviewed: 1/1 changed files
  • Comments generated: 2
  • Review effort level: Lite

) -> List[Tuple[str, int]]:
# Emits "<byte_count>\t<path>" per readable file and
# "ERR<exit_code>\t<path>" per file whose read fails.
scratch = "/tmp/lisa_debugfs_read"
Comment on lines +1209 to +1212
overall_timeout = (
candidate_count
* (self.DEBUGFS_READ_TIMEOUT_SECONDS + self.DEBUGFS_READ_KILL_GRACE_SECONDS)
+ self.DEBUGFS_TREE_TIMEOUT_PADDING_SECONDS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants