Skip to content

hack: make IPv6-only kind clusters usable - #958

Open
Yuan Gao (ygao-g) wants to merge 2 commits into
agent-substrate:mainfrom
ygao-g:kind-ipv6-coredns
Open

hack: make IPv6-only kind clusters usable#958
Yuan Gao (ygao-g) wants to merge 2 commits into
agent-substrate:mainfrom
ygao-g:kind-ipv6-coredns

Conversation

@ygao-g

@ygao-g Yuan Gao (ygao-g) commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

An IPv6-only kind cluster is unusable today:

  1. the script corrupts the kubeconfig kind wrote by changing https://[::1]:PORT
    to https://localhost:PORT;
  2. nothing resolves from inside a pod.

For 1., remove the overwrite, which is only needed for a macOS client reaching
kind inside a Lima VM; run the loop inside the guest instead. For 2., forward to
an IPv6 resolver instead of the node's IPv4 one, and add a kind-registry:53 server
block so atelet can pull from its own network namespace.

This works around kubernetes-sigs/kind#4152 — kind's node entrypoint picks the
IPv4 default gateway as the node's nameserver — and moby/moby#41651 behind it.
Neither has a fix committed, so the patch stays until kind runs a resolver proxy
on the node.

Verified in a Lima guest on macOS. IP_FAMILY=ipv6 comes up clean; kind's untouched
https://[::1]:PORT answers ok; from a v6-only pod www.google.com,
kubernetes.default.svc.cluster.local and kind-registry. all resolve. IPv4 is
unaffected — the new block is gated on ipv6.

CI cannot verify this by itself. GitHub runners have no IPv6 egress, so a v6-only
pod there cannot reach an IPv6 upstream at all — an IPv6-only lane additionally
needs DNS64 and NAT64, which #1275 adds on top of this PR.

Part of #1099.
Part of #246.

🤖 Generated with Claude Code

@ygao-g Yuan Gao (ygao-g) mentioned this pull request Aug 14, 2026
Draft
@ygao-g
Yuan Gao (ygao-g) force-pushed the kind-ipv6-coredns branch 2 times, most recently from 45bac3d to ecf1fc6 Compare August 18, 2026 16:00
@ygao-g Yuan Gao (ygao-g) changed the title hack/create-kind-cluster.sh: fix DNS on IPv6-only kind clusters hack: fix DNS on IPv6-only kind clusters Aug 18, 2026
@ygao-g Yuan Gao (ygao-g) mentioned this pull request Aug 18, 2026
Closed
@ygao-g
Yuan Gao (ygao-g) force-pushed the kind-ipv6-coredns branch 3 times, most recently from e812e99 to 3350f55 Compare August 18, 2026 22:07
This was referenced Aug 18, 2026
Draft
Merged
@ygao-g Yuan Gao (ygao-g) added kind/bug Something isn't working / bugfixes area/network labels Aug 18, 2026
@ygao-g Yuan Gao (ygao-g) mentioned this pull request Aug 19, 2026
Open
@ygao-g
Yuan Gao (ygao-g) marked this pull request as ready for review August 19, 2026 00:32
@ygao-g

Yuan Gao (ygao-g) commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Testing

Manual Testing: IP_FAMILY=ipv6 hack/create-kind-cluster.sh → --deploy-ate-system → --deploy-demo-counter: atelet pulls the gVisor tarball from storage.googleapis.com over IPv6 from its own pod netns, the actor image pulls through kind-registry, and a golden actor runs and checkpoints.

CI has no IPv6 job and this PR does not add one — #1065 is a signal-only draft that produces that coverage on demand.

@ygao-g
Yuan Gao (ygao-g) marked this pull request as draft August 19, 2026 15:46
This was referenced Aug 19, 2026
Closed
Merged
@ygao-g
Yuan Gao (ygao-g) marked this pull request as ready for review August 19, 2026 16:45
@ygao-g Yuan Gao (ygao-g) mentioned this pull request Aug 19, 2026
Draft
@ygao-g
Yuan Gao (ygao-g) force-pushed the kind-ipv6-coredns branch 4 times, most recently from 9cc9703 to 3dbf64b Compare August 20, 2026 03:05
This was referenced Aug 27, 2026
Draft
Open
BenTheElder
Benjamin Elder (BenTheElder) reviewed Aug 27, 2026
View reviewed changes

@BenTheElder Benjamin Elder (BenTheElder) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bowei Du (@bowei) we're going to have to rewrite this in go too, sigh

this is meant to represent a portable cluster, if we customize it heavily then we're really not proving substrate works generically

can we please file upstream issues with the relevant projects for each thing we're working around, so we can actually fix them?

@ygao-g Yuan Gao (ygao-g) mentioned this pull request Aug 28, 2026
Open
@ygao-g
Yuan Gao (ygao-g) marked this pull request as draft August 28, 2026 16:14
@ygao-g Yuan Gao (ygao-g) changed the title hack: fix DNS on IPv6-only kind clusters [WIP] hack: fix DNS on IPv6-only kind clusters Aug 28, 2026
@ygao-g Yuan Gao (ygao-g) changed the title [WIP] hack: fix DNS on IPv6-only kind clusters hack: fix DNS on IPv6-only kind clusters Aug 28, 2026
@ygao-g
Yuan Gao (ygao-g) marked this pull request as ready for review August 28, 2026 16:17
This was referenced Aug 28, 2026
Closed
Open
haiyanmeng pushed a commit that referenced this pull request Aug 31, 2026
@ygao-g
atenet/egress: resolve upstream names on both address families (#1060)
bb83206 
Part of #246

The egress Envoy pinned `dns_lookup_family` to `V4_ONLY` in both the
dynamic
forward proxy filter and the dynamic forward proxy cluster. On an
IPv6-only
cluster every upstream connection failed: Envoy reported a DNS
resolution
failure and the actor got a 503. Seven sites across the two egress
manifests,
including the sdsmint variant, now use `ALL`, which returns both
families and
enables Happy Eyeballs.

This is a prerequisite for IPv6 egress, not the fix on its own. An
actor's
connection is redirected by nftables and atunnel recovers the original
destination with `getsockopt(SOL_IP, SO_ORIGINAL_DST)`, which returns
ENOENT
for a v6-redirected connection — so egress fails before Envoy is ever
asked to
resolve anything.

## Testing

`make verify` is clean. A new test walks the shipped manifests and
requires
`ALL` on every `dns_cache_config`, so a new egress variant cannot
reintroduce
the pin. It has already paid for itself: the seventh site arrived with
the
sdsmint MITM leg while this was in review, still pinned to `V4_ONLY`,
and the
test caught it on rebase.

Measured on an IPv6-only kind cluster with #911, #958, #979 and #753
applied.
Each value was deployed, Envoy restarted, and the live `/config_dump`
checked
before running the suites:

| `dns_lookup_family` | `TestActorEgress` / `TestActorEgressHTTPS` |
| --- | --- |
| `ALL` (shipped) | pass / pass, `code=200` to
`[2606:4700:10::ac42:93f3]:80` |
| `V4_ONLY` (before) | fail / fail, `code=503 flags=DF` |
| `AUTO` | pass / pass |

`AUTO` is not distinguishable from `ALL` on this path: the dynamic
forward
proxy is handed the IP literal that atunnel recovered, never a hostname,
so
`dns_lookup_family` only decides which literal families it accepts.
`ALL` is
chosen as the value that strands neither family.

The IPv4 lane is unaffected: `TestActorEgress` and
`TestActorEgressHTTPS` pass
with the change in place, across two runs of the standard e2e job.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@ygao-g
Yuan Gao (ygao-g) force-pushed the kind-ipv6-coredns branch 2 times, most recently from 7d9aa39 to 3c66a5d Compare August 31, 2026 21:31
@ygao-g

Yuan Gao (ygao-g) commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Bowei Du (Bowei Du (@bowei)) we're going to have to rewrite this in go too, sigh

this is meant to represent a portable cluster, if we customize it heavily then we're really not proving substrate works generically

can we please file upstream issues with the relevant projects for each thing we're working around, so we can actually fix them?

For the workaround in this PR, found existing kind issue github.com/kubernetes-sigs/kind/issues/4152 and added a comment there.

This was referenced Sep 1, 2026
Closed
Open
@ygao-g
hack: drop the IPv6 kubeconfig repoint
80db4de 
Before, on an IPv6-only cluster the script rewrote kind's
`https://[::1]:PORT` kubeconfig entry to `https://localhost:PORT`
unconditionally. That breaks any host whose `/etc/hosts` leaves
`localhost` off the `::1` line, including the Ubuntu cloud image Lima
runs. Remove the repoint logic, but note that limactl re-forwards the
published port to the host's v4 loopback only, so a macOS client cannot
directly access kind clusters run inside a Lima VM via `[::1]`. Instead,
operations should run inside the guest.

Tested: manual tests creating IPv6-only kind clusters on a local env
with IPv6 egress, namely macOS + Lima VM.
@ygao-g
hack: fix CoreDNS on IPv6-only kind clusters
6c9d0ce 
CoreDNS inherits the node's IPv4 resolver, which a v6-only pod cannot
reach. Change CoreDNS's Corefile to forward to an overridable IPv6
upstream. Also add a kind-registry:53 server block to CoreDNS. In
addition, `kind create` returns before the apiserver answers, so add
logic to wait for the control plane from inside the node first.

Tested: manual tests of creation and installation on a local env with
IPv6 egress, namely macOS + Lima VM. CI cannot verify this — the GitHub
runner is IPv4-only, so an IPv6-only cluster there additionally needs
DNS64 and NAT64.
@ygao-g Yuan Gao (ygao-g) changed the title hack: fix DNS on IPv6-only kind clusters hack: make IPv6-only kind clusters usable Sep 3, 2026
@ygao-g

Yuan Gao (ygao-g) commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Bowei Du (@bowei) This PR is ready for you to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BenTheElder Benjamin Elder (BenTheElder) BenTheElder left review comments
@bowei Bowei Du (bowei) Awaiting requested review from bowei
@haiyanmeng haiyanmeng Awaiting requested review from haiyanmeng

Assignees

@bowei Bowei Du (bowei)

Labels

area/network kind/bug Something isn't working / bugfixes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ygao-g @BenTheElder @haiyanmeng @bowei