From 9e74793ac4727722612ba2ab64a1c5e8e5df4620 Mon Sep 17 00:00:00 2001 From: Chris Tauchen Date: Fri, 21 Aug 2026 11:55:09 +0100 Subject: [PATCH 1/2] DOCS-2997: Write the L2 network troubleshooting guide Fill in the troubleshooting guide for L2 bridge networking, organized by symptom rather than by tool. The guide opens with where the signals are, because L2 problems report themselves somewhere unusual. Bridge problems appear only in the Felix log inside calico-node, with no Kubernetes event, no resource condition, and no metric. Attachment problems appear in the workload status and the CNI and Multus logs. The diagnostics section says that bridge state is not part of the standard bundle and has to be collected by hand, so a support case does not arrive without it. Entries cover the failures that are silent, late, or self-concealing: a bridge that does not meet its prerequisites, a workload with two interfaces called eth0 because the multi-interface mode was missed, a node taken off the network while its bridge was being configured, and a packet capture that masks the hardware filtering problem it was opened to investigate. Co-Authored-By: Claude Opus 5 (1M context) --- .../networking/l2-bridge/troubleshoot.mdx | 191 +++++++++++++++++- 1 file changed, 186 insertions(+), 5 deletions(-) diff --git a/calico-enterprise/networking/l2-bridge/troubleshoot.mdx b/calico-enterprise/networking/l2-bridge/troubleshoot.mdx index 069478104f..9b6604dc3c 100644 --- a/calico-enterprise/networking/l2-bridge/troubleshoot.mdx +++ b/calico-enterprise/networking/l2-bridge/troubleshoot.mdx @@ -6,44 +6,225 @@ description: Diagnose connectivity problems on Calico Enterprise L2 bridge netwo :::note -L2 bridge networking is a tech preview feature. APIs and behavior may change before GA. +L2 bridge networking is a tech preview feature. +APIs and behavior may change before GA. ::: -Most L2 problems fall into a handful of shapes. Find your symptom below. +Most problems on an L2 network fall into a handful of shapes. +Find your symptom below. ## Where the signals are -Read this first. L2 bridge problems report themselves in an unusual place, and looking anywhere else wastes time. +Read this first. +L2 problems report themselves somewhere unusual, and looking in the obvious places wastes time. + +Problems with the **bridge itself** appear only in the Felix log inside `calico-node`. +There is no Kubernetes event, no condition on the `Network`, no status on the workload, and no metric. +The messages are good — each names the bridge, the `Network`, and the command that fixes the problem — but you have to read the log to see them. + +```bash +kubectl logs -n calico-system -l k8s-app=calico-node --tail=200 | grep -i bridge +``` + +Problems **attaching a workload** appear in the workload's own status, and in the CNI and Multus logs on the node that was scheduled to run it. + +```bash +kubectl describe pod +``` + +On the node, CNI and Multus write to `/var/log/calico/cni/` and the container runtime's log. ## Collect diagnostic information -What to gather before opening a support case, including the bridge state that the standard bundle does not capture. +Gather this before opening a support case, because one part of it is not collected automatically. + +1. Run the standard diagnostics bundle: + + ```bash + kubectl exec -n calico-system -- calicoctl cluster diags + ``` + +2. Collect the bridge state by hand on each affected node. + The bundle does not include it: + + ```bash + ip -d link show type bridge + bridge link show + bridge vlan show + bridge fdb show + ``` + +3. If the problem is not obvious, raise Felix's log level and reproduce it: + + ```bash + kubectl patch felixconfiguration default --type=merge \ + -p '{"spec":{"logSeverityScreen":"Debug"}}' + ``` + +Set it back to `Info` afterwards. ## $[prodname] will not use the bridge I prepared +Workloads on the `Network` stay administratively down and never get connectivity, and the Felix log repeats a warning every few seconds. + +$[prodname] fails closed here by design. +Rather than attaching workloads to a bridge that cannot isolate them, it programs nothing at all. +That means no trunk enslavement, no VLAN membership, and no workload interfaces. + +Check the three properties $[prodname] requires: + +1. VLAN filtering is enabled: + + ```bash + ip -d link show | grep vlan_filtering + ``` + +2. The MAC address was set explicitly rather than inherited from a port. + If it was not, setting it to its current value is enough: + + ```bash + ip link set dev address + ``` + +3. The tag protocol is 802.1Q rather than 802.1ad: + + ```bash + ip -d link show | grep vlan_protocol + ``` + +The warning in the log names which of these failed and the command to fix it. +See [Prepare an existing bridge](byo-bridge.mdx) for the full requirements. + +Two side effects are worth knowing about. +A single broken bridge can intermittently delay unrelated `Network` resources, because $[prodname] stops processing on the first failure each cycle. +And the failure is per-node, so a `Network` that works on most nodes and fails on one reads like flakiness rather than a configuration error. + ## A workload came up with two eth0 interfaces and no connectivity +The multi-interface mode is not set. +Multus being installed is not sufficient on its own, and nothing reports the omission until this happens. + +```bash +kubectl get installation default -o jsonpath='{.spec.calicoNetwork.multiInterfaceMode}' +``` + +If the value is not `Multus`, set it as described in [Set the multi-interface mode](connect-vlan.mdx#set-the-multi-interface-mode), then delete and recreate the affected workloads. + ## The workload has no IP address, or will not start +The workload stays pending or failing, and the CNI log reports an address assignment failure. + +Check these in order: + +1. **No IP pool matches the VLAN's subnet.** + $[prodname] chooses a pool by checking which pools fall inside the subnet on the VLAN. + Nothing matching means no address, and $[prodname] fails rather than assigning an address from the wrong subnet. +2. **The pool is not marked for L2 use.** + A pool without `allowedUses: L2Workload` is not a candidate for an L2 interface. +3. **A requested address is already assigned.** + If you asked for a specific address with `ipAddrs`, and something already holds it, the request fails rather than duplicating it. +4. **A requested address is reserved.** + $[prodname] does not assign the first or last address of a subnet, or the gateway address. + ## The bridge was never created on this node +Other nodes have the bridge and this one does not. + +1. Check that a host configuration entry matches this node. + Remember that entries are evaluated in order and the first match wins — a broad entry earlier in the list shadows a more specific one after it. +2. Check that the trunk interface named in the matching entry actually exists on this node, with that exact name. + +Nothing validates interface names when you create the `Network`, because the API server cannot see the interfaces on your nodes. +A name that is wrong for one node shows up only in that node's Felix log. + ## The node lost connectivity while I was configuring the bridge +This is the failure that takes a node off the network, and it has three usual causes. + +- **VLAN filtering was enabled before VLAN membership existed.** + Turning filtering on makes the kernel enforce membership immediately, so an address on a bridge with no membership stops sending and receiving at that moment. +- **The bridge device has no membership of the host's VLAN in its own right.** + The bridge forwards that VLAN between its ports but delivers none of it to the host. + Add it with `bridge vlan add dev vid self`. +- **The bridge's MAC address changed.** + If it was never pinned, attaching the first workload can change it, and traffic to the old address stops being delivered. + +Recover from the console, not over the network. +See [Prepare an existing bridge](byo-bridge.mdx). + ## The workload cannot reach its gateway +The workload has an address but nothing beyond the node answers. + +1. Confirm the VLAN is listed in the `Network`. +2. Confirm the trunk port carries that VLAN, with `bridge vlan show`. +3. Confirm the switch port facing the node is configured as a trunk carrying that VLAN. +4. If the segment is presented untagged, confirm the native VLAN in the `Network` matches the switch's native VLAN. + A mismatch here silently puts traffic on the wrong VLAN. + ## Traffic works on one node but not across the fabric +Two workloads on the same node reach each other, and nothing off the node responds. +This points at the trunk and the physical path rather than at $[prodname]. + +Check the switch configuration first. +Then, on a bridge you prepared yourself, confirm the trunk interface is named in the `Network`. +Omitting it leaves the node unable to route workload traffic, because the reverse path check has no interface to validate against. + ## Source IP policy is not matching +Policy written against a workload's source address does not match traffic that came from it. + +Traffic between the $[prodname] pod network and an L2 network is routed through your external router, including when both workloads are on the same node. +If that router source-NATs between the two networks, policy on the L2 side sees the router's address instead of the workload's. + +Check for NAT on the inter-VLAN path. +This is a router configuration issue rather than something $[prodname] can work around. + ## A pod on an L2 network cannot reach cluster IP addresses +Connections from the pod to a Service address are dropped. + +Connect-time load balancing is a host-wide setting and cannot be turned off for one interface. +A Service address gets resolved and translated on the way out of the pod's primary interface, then dropped by reverse path filtering. + +VMs are unaffected, because they resolve Services inside their own kernel. +For a pod that needs both cluster Services and an L2 network, keep its primary interface on the $[prodname] pod network and attach the L2 network as an additional interface. + ## The MAC address is not what I asked for +1. **The VM does not start at all, and the CNI log reports a conflict.** + KubeVirt's `macAddress` field and the `hwAddr` annotation are both set, to different values. + $[prodname] rejects the interface rather than choosing. + Make them agree, or remove one. +2. **The address is not the one you asked for.** + Check the `network-status` annotation to see what $[prodname] actually applied, and confirm the annotation you set is scoped to the right interface name. +3. **The bridge's address changed rather than the workload's.** + On a bridge you prepared, an unpinned bridge MAC address changes when ports are added. + ## Capturing traffic hides the problem -On some network cards, the act of capturing changes the behavior you are trying to observe. +You capture on the trunk interface to investigate a connectivity problem, and the problem disappears while you are capturing. + +Putting an interface into promiscuous mode changes how some network cards filter incoming traffic, which can mask a hardware filtering problem for exactly as long as the capture runs. + +Capture without promiscuous mode, or capture from a different interface in the path: + +```bash +tcpdump -p -i +``` ## Workloads lost connectivity after a Network was deleted +Deleting a `Network` is not blocked while workloads are still attached to it, and doing so removes the bridge from under them. + +Recreate the `Network` with the same configuration, then delete and recreate the affected workloads so their interfaces are rebuilt. +Delete workloads before their `Network` in future. + ## Additional resources + +- [Prepare an existing bridge](byo-bridge.mdx) +- [Connect workloads to an existing VLAN](connect-vlan.mdx) +- [About L2 bridge networking](about-l2-bridge.mdx) +- [L2 bridge support and limitations](../../reference/l2-bridge-support.mdx) From f55060d21b3a576657540668880a1d65929f00ac Mon Sep 17 00:00:00 2001 From: Chris Tauchen Date: Fri, 21 Aug 2026 14:49:11 +0100 Subject: [PATCH 2/2] DOCS-2997: Record the user story for the troubleshooting guide Add a comment at the top of the page naming the user story it exists to serve. This page serves one story only. Diagnosis is framed by the reader's goal, which is why it is a how-to rather than a section appended to the setup guides. Co-Authored-By: Claude Opus 5 (1M context) --- calico-enterprise/networking/l2-bridge/troubleshoot.mdx | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/calico-enterprise/networking/l2-bridge/troubleshoot.mdx b/calico-enterprise/networking/l2-bridge/troubleshoot.mdx index 9b6604dc3c..bc9701e533 100644 --- a/calico-enterprise/networking/l2-bridge/troubleshoot.mdx +++ b/calico-enterprise/networking/l2-bridge/troubleshoot.mdx @@ -2,6 +2,13 @@ description: Diagnose connectivity problems on Calico Enterprise L2 bridge networks, from bridge prerequisites to VLAN and trunk misconfiguration. --- +{/* + User story, primary + S8: As an operator, I want to find out why a workload on an L2 network cannot reach its gateway, so that I can fix it without opening a support case. + + Story ladder and page plan: DOCS-2997. +*/} + # Troubleshoot L2 network connectivity :::note