Summary
When using xdp: (and likely any multi-queue format relying on NIC RSS) on Intel X710/XL710 NICs with the in-tree i40e driver, hardware RSS may only ever deliver packets to the lower half of the receive queues. Parallel libtrace applications sized to the NIC's reported queue count will have half their processing threads silently idle. Nothing errors and every software-visible setting looks correct, which makes this miserable to diagnose — we lost several days to it on our own AF_XDP stack and are sharing the findings (plus a small detection patch) in case they save someone else the trip.
Symptom
With N combined channels configured and traffic with plenty of flow entropy, ethtool -S <if> shows per-queue RX counters like:
rx-0.packets: 773,707 rx-8.packets: 0
rx-1.packets: 774,313 rx-9.packets: 0
... ...
rx-7.packets: 773,752 rx-15.packets: 0
Everything a user would normally check is consistent and correct the whole time:
ethtool -l reports 16 combined channelsethtool -x shows a 512-entry indirection table evenly covering all 16 queues/sys/class/net/<if>/queues/ shows rx-0..rx-15- 4-tuple Toeplitz hashing confirmed via
ethtool -n <if> rx-flow-hash tcp4
- (under AF_XDP) all 16 XSK buffer pools register in dmesg
Root cause
The i40e firmware LLDP agent applies a default FCoE-aware DCB configuration at startup: two traffic classes, splitting the queue pairs evenly. From the driver's debugfs VSI dump:
tc_config: numtc = 2, enabled_tc = 0x3
tc_config: tc = 0, qoffset = 0, qcount = 8, netdev_tc = 0
tc_config: tc = 1, qoffset = 8, qcount = 8, netdev_tc = 1
i40e hardware RSS distributes within a traffic class, not across TC boundaries. All normal traffic lands in TC0, so RSS only ever uses queues 0–7. The kernel's cached view (channels, indirection table, sysfs) does not reflect the hardware TC partition.
Observed on: Intel XL710 4-port 10GbE [8086:1572], firmware 9.30/9.130.73618, in-tree i40e on kernel 5.15.0-181-generic. The agent's state is visible via ethtool --show-priv-flags <if> → disable-fw-lldp : off.
Fix / workaround
ethtool --set-priv-flags <if> disable-fw-lldp on
This is a persistent NVM write (survives reboots and driver reloads, but is per-device — replacement NICs regress). The driver collapses to a single TC immediately, no reload needed:
tc_config: numtc = 1, enabled_tc = 0x1
tc_config: tc = 0, qoffset = 0, qcount = 16, netdev_tc = 0
After the fix all 16 queues receive traffic within ~1.5% of each other.
Why this matters
linux_xdp_pstart_input() sizes perpkt_thread_count to the NIC's reported max queues and starts one stream per queue. On an affected NIC, threads 8–15 are fed by queues that hardware RSS never selects, so they run but process nothing — no error, no drop counter, just silently missing parallelism (and, for HASHER_BALANCE expectations, a skewed load distribution). The same applies to any RSS-dependent parallel format on this hardware.
Summary
When using
xdp:(and likely any multi-queue format relying on NIC RSS) on Intel X710/XL710 NICs with the in-treei40edriver, hardware RSS may only ever deliver packets to the lower half of the receive queues. Parallel libtrace applications sized to the NIC's reported queue count will have half their processing threads silently idle. Nothing errors and every software-visible setting looks correct, which makes this miserable to diagnose — we lost several days to it on our own AF_XDP stack and are sharing the findings (plus a small detection patch) in case they save someone else the trip.Symptom
With N combined channels configured and traffic with plenty of flow entropy,
ethtool -S <if>shows per-queue RX counters like:Everything a user would normally check is consistent and correct the whole time:
ethtool -lreports 16 combined channelsethtool -xshows a 512-entry indirection table evenly covering all 16 queues/sys/class/net/<if>/queues/shows rx-0..rx-15ethtool -n <if> rx-flow-hash tcp4Root cause
The i40e firmware LLDP agent applies a default FCoE-aware DCB configuration at startup: two traffic classes, splitting the queue pairs evenly. From the driver's debugfs VSI dump:
i40e hardware RSS distributes within a traffic class, not across TC boundaries. All normal traffic lands in TC0, so RSS only ever uses queues 0–7. The kernel's cached view (channels, indirection table, sysfs) does not reflect the hardware TC partition.
Observed on: Intel XL710 4-port 10GbE [8086:1572], firmware 9.30/9.130.73618, in-tree i40e on kernel 5.15.0-181-generic. The agent's state is visible via
ethtool --show-priv-flags <if>→disable-fw-lldp : off.Fix / workaround
This is a persistent NVM write (survives reboots and driver reloads, but is per-device — replacement NICs regress). The driver collapses to a single TC immediately, no reload needed:
After the fix all 16 queues receive traffic within ~1.5% of each other.
Why this matters
linux_xdp_pstart_input()sizesperpkt_thread_countto the NIC's reported max queues and starts one stream per queue. On an affected NIC, threads 8–15 are fed by queues that hardware RSS never selects, so they run but process nothing — no error, no drop counter, just silently missing parallelism (and, for HASHER_BALANCE expectations, a skewed load distribution). The same applies to any RSS-dependent parallel format on this hardware.