Fix NIC to GPU proximity detection on systems with multiple PCIe domains (candidate-1.70)

[paklui]

Candidate-1.70 counterpart of #352 (same fix, rebased onto candidate-1.70 instead of develop).

Motivation

On systems where each GPU sits in its own PCIe domain, TransferBench picked the wrong NIC as closest, thus every GPU mapped to a single front-end NIC that is not wired to any GPU, while the back-end NICs actually adjacent to each GPU mapped to nothing. The mapping decides which NIC a GPU uses for RDMA transfers (NIC_NEAREST executor, nic_rings preset). With incorrect mapping it would mean benchmarking using a NIC that is not attached to that GPU's root complex, producing misleading numbers. The causes are:

1. The proximity metric compared raw bus numbers. Bus numbers repeat across domains, so devices in different domains looked "close" purely because their bus numbers were numerically near.
2. When several NICs were equally close, only one was recorded, chosen by a round-robin counter, so a GPU with two adjacent NICs only ever reported one.

This PR fixes the topology detection so it works correctly on systems with multiple PCIe domains and multiple NICs per GPU (verified on a 4-GPU, 8 BE + 1 FE, or 2 BE NICs per GPU).

Technical Details

1. Use PCIe domain, not bus number, for the distance metric. ExtractBusNumber became ExtractDomain, and GetBusIdDistance became GetDomainDistance.

   The distance between two devices is now computed as follows:

   • If both devices sit in the same PCIe domain, the distance is 0. Within a single domain, the PCIe tree search already tells us how the two devices are related, so no extra tie-break is needed.
   • If the two devices sit in different domains, the distance is the difference between their domain numbers. That value is always at least 1, so a device in a different domain always ranks behind every device in the same domain.

   Bus numbers are no longer used to compare devices within a domain. Firmware assigns them, and they do not reliably indicate which device is physically closer. Using them would also separate NICs that are genuinely the same distance from a GPU, and reporting those ties is the point of this fix.

   An unparseable address (-1) is clamped to INT_MAX rather than compared directly, so it remains eligible as a last resort but can never incorrectly outrank a candidate with a known distance.

2. Record all equally-close NICs/GPUs. Replaces the assignedCount round-robin, which picked one NIC per GPU even when several were tied. Both directions now keep the full tie set, matching how GetClosestNicsToGpu() was already shaped.

   Trade-off: the removed round-robin was intended to spread multi-port NICs (ports sharing one busId) across GPUs. With ties recorded, nic_rings now uses all tied ports, better, but a bare NIC_NEAREST resolves slot 0 for every GPU rather than spreading. Use the explicit N<gpu>.<slot> form to select a specific port.

3. Update the CLI topology display. The display called the scalar GetClosestNicToGpu(), so it would have shown one NIC even after (2). Switched to the vector API. The NIC list is space-separated so the field stays a single column under OUTPUT_TO_CSV=1, where the separator is a comma.

   Both fallback paths (GPU->NIC and NIC->GPU) now read from ibvAddressList rather than the raw device list, so a NIC with a down port is never recorded as "closest" -- NicIsActive() would otherwise abort the Transfer with ERR_FATAL rather than falling back.

Test Plan

Built and ran on a candidate-1.70 base (92b0df7) on a gfx1250 rack node: 4 GPUs, 8 BE NICs + 1 FE NIC, 2 NICs wired via PCIe per GPU, each GPU in its own PCIe domain, ROCm 7.15.0.

Test Result

Currently with candidate-1.70 or develop HEAD:

 NIC | Device Name | Active | PCIe Bus ID  | NUMA | Closest GPU(s) | GID Index | GID Descriptor
-----+-------------+--------+--------------+------+----------------+-----------+-------------------
 0   | ionic_0     | Yes    | 0000:04:00.0 | 0    | 0,1,2,3        | 1         | RoCEv2 IPv4-mapped IPv6
 1   | ionic_1     | Yes    | 0001:44:00.0 | -1   |                | 1         | RoCEv2 IPv4-mapped IPv6
 2   | ionic_2     | Yes    | 0001:48:00.0 | -1   |                | 1         | RoCEv2 IPv4-mapped IPv6
 3   | ionic_3     | Yes    | 0003:44:00.0 | -1   |                | 1         | RoCEv2 IPv4-mapped IPv6
 4   | ionic_4     | Yes    | 0003:48:00.0 | -1   |                | 1         | RoCEv2 IPv4-mapped IPv6
 5   | ionic_5     | Yes    | 0002:44:00.0 | -1   |                | 1         | RoCEv2 IPv4-mapped IPv6
 6   | ionic_6     | Yes    | 0002:48:00.0 | -1   |                | 1         | RoCEv2 IPv4-mapped IPv6
 7   | ionic_7     | Yes    | 0004:44:00.0 | -1   |                | 1         | RoCEv2 IPv4-mapped IPv6
 8   | ionic_8     | Yes    | 0004:48:00.0 | -1   |                | 1         | RoCEv2 IPv4-mapped IPv6

        | gfx1250 | gfx1250 | gfx1250 | gfx1250 |
        | GPU 00 | GPU 01 | GPU 02 | GPU 03 | PCIe Bus ID  | #CUs | NUMA | #DMA | #XCC | NIC
--------+--------+--------+--------+--------+--------------+------+------+------+------+------
 GPU 00 |    N/A | XGMI-1 | XGMI-1 | XGMI-1 | 0001:01:00.0 | 256  | 0    | 16   | 8    | 0
 GPU 01 | XGMI-1 |    N/A | XGMI-1 | XGMI-1 | 0002:01:00.0 | 256  | 0    | 16   | 8    | 0
 GPU 02 | XGMI-1 | XGMI-1 |    N/A | XGMI-1 | 0003:01:00.0 | 256  | 0    | 16   | 8    | 0
 GPU 03 | XGMI-1 | XGMI-1 | XGMI-1 |    N/A | 0004:01:00.0 | 256  | 0    | 16   | 8    | 0

With ./TransferBench (no args):

 NIC | Device Name | Active | PCIe Bus ID  | NUMA | Closest GPU(s) | GID Index | GID Descriptor
-----+-------------+--------+--------------+------+----------------+-----------+-------------------
 0   | ionic_0     | Yes    | 0000:04:00.0 | 0    |                | 1         | RoCEv2 IPv4-mapped IPv6
 1   | ionic_1     | Yes    | 0001:44:00.0 | -1   | 0              | 1         | RoCEv2 IPv4-mapped IPv6
 2   | ionic_2     | Yes    | 0001:48:00.0 | -1   | 0              | 1         | RoCEv2 IPv4-mapped IPv6
 3   | ionic_3     | Yes    | 0003:44:00.0 | -1   | 2              | 1         | RoCEv2 IPv4-mapped IPv6
 4   | ionic_4     | Yes    | 0003:48:00.0 | -1   | 2              | 1         | RoCEv2 IPv4-mapped IPv6
 5   | ionic_5     | Yes    | 0002:44:00.0 | -1   | 1              | 1         | RoCEv2 IPv4-mapped IPv6
 6   | ionic_6     | Yes    | 0002:48:00.0 | -1   | 1              | 1         | RoCEv2 IPv4-mapped IPv6
 7   | ionic_7     | Yes    | 0004:44:00.0 | -1   | 3              | 1         | RoCEv2 IPv4-mapped IPv6
 8   | ionic_8     | Yes    | 0004:48:00.0 | -1   | 3              | 1         | RoCEv2 IPv4-mapped IPv6

        | gfx1250 | gfx1250 | gfx1250 | gfx1250 |
        | GPU 00 | GPU 01 | GPU 02 | GPU 03 | PCIe Bus ID  | #CUs | NUMA | #DMA | #XCC | NIC
--------+--------+--------+--------+--------+--------------+------+------+------+------+------
 GPU 00 |    N/A | XGMI-1 | XGMI-1 | XGMI-1 | 0001:01:00.0 | 256  | 0    | 16   | 8    | 1 2
 GPU 01 | XGMI-1 |    N/A | XGMI-1 | XGMI-1 | 0002:01:00.0 | 256  | 0    | 16   | 8    | 5 6
 GPU 02 | XGMI-1 | XGMI-1 |    N/A | XGMI-1 | 0003:01:00.0 | 256  | 0    | 16   | 8    | 3 4
 GPU 03 | XGMI-1 | XGMI-1 | XGMI-1 |    N/A | 0004:01:00.0 | 256  | 0    | 16   | 8    | 7 8

Each GPU is now correctly mapped to the NIC(s) in its own PCIe domain (GPU0->1,2, GPU1->5,6, GPU2->3,4, GPU3->7,8), and the front-end NIC (domain 0000) correctly shows no GPU mapping.

Also see #352 (same fix against develop), which was additionally verified against a real link-flap event: a NIC going down/up correctly changed a GPU from a tie to a single mapping and back.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[mustafabar]

Thanks Pak for this contribution. I have a couple of comments

1. On a topology where two GPUs are genuinely closest in PCIe addresses to the same pair of NICs, both GPUs now resolve to the same NIC, leaving the other idle (the round-robin approach was meant for those cases, and regressed your case, which I admin never occured to me)

2. I am a bit worried that we might run into cases that worked well with PCIe bus number as key, and now lack enough testing with the new metric using the PCIe domain number. Thinking about it again, I think it was sloppy to use only the bus_number in the first place but seems like we got lucky for some time. To add more resilience, would you consider adding a composite uint64_t key (with proper shifts and masks) and use that in the comparison instead within ExtractDomain?