tailscaled is unsupervised: a daemon panic leaves /etc/resolv.conf pointing at MagicDNS and breaks all DNS in the container

[nocnokneo]

What happened

tailscaled (1.102.3, installed by this feature) panicked inside a long-running dev container and was never restarted. Because it crashed rather than shut down, it did not restore /etc/resolv.conf, which it had pointed at MagicDNS. The container was left with:

nameserver 100.100.100.100
nameserver fd7a:115c:a1e0::53

and no daemon to answer it, so every DNS lookup in the container black-holed: getent hosts registry.npmjs.org failed, npm/pnpm failed with EAI_AGAIN, and curl hung for its full timeout. The container itself stayed up and looked healthy, so this went unnoticed for hours.

From /var/log/tailscaled.log:

health(warnable=derp-timed-out): error: Tailscale hasn't heard from the '<region>' relay
  server in 5m51s. The server might be temporarily unavailable, or your Internet
  connection might be down.
panic: ipnlocal: watchdog timeout

goroutine 240320 [running]:
tailscale.com/ipn/ipnlocal.(*LocalBackend).reportDeadlock(0x...?)
	tailscale.com@v1.102.3/ipn/ipnlocal/watchdog.go:146 +0xb2
created by time.goFunc
	time/sleep.go:215 +0x2d

Two containers on the same host, both using this feature, panicked within the same minute — one panic each, both preceded by the same DERP stall. So whatever the underlying deadlock is, it is reachable from an ordinary network blip and can hit every container at once.

Why this is a feature-level problem

The panic itself belongs to tailscale/tailscale, but the blast radius is this feature's, for two reasons:

1. Nothing supervises the daemon. tailscaled-devcontainer-start runs exec setsid /usr/local/sbin/tailscaled in a backgrounded subshell and never looks at it again, and tailscaled-entrypoint calls that script once before exec "$@". If tailscaled exits for any reason, it stays exited for the life of the container.
2. The crash takes DNS down with it, and the fix is already on disk. The feature's own backup, /etc/resolv.pre-tailscale-backup.conf, was intact and correct the whole time. Nothing consults it unless tailscaled shuts down cleanly.

The result is the worst failure shape: a container whose networking is silently broken in a way that looks like a registry outage, a proxy problem, or a broken image.

Suggestions

• Supervise tailscaled — a restart loop in the start script, or run it under whatever init the image has. A crash then costs seconds of connectivity instead of the rest of the container's life.
• Failing that, make the state self-healing: on start (and ideally periodically), if tailscaled is not running, /etc/resolv.conf names 100.100.100.100, and /etc/resolv.pre-tailscale-backup.conf exists, restore the backup. That alone converts "all DNS is dead" into "the tailnet is down", which is both recoverable and diagnosable.

Possibly the same root cause as #57 (rebuild leaves DNS broken, backup missing) and #11.

Environment

• Feature: ghcr.io/tailscale/codespace/tailscale
• tailscale/tailscaled 1.102.3 (commit 9329c3677031109ff6d0b80abee0cddc8f35ff6f)
• Base image: Ubuntu 22.04.5 LTS, Docker (not Codespaces), containers up 7-14 days

[bradfitz]

This is pretty painful-to-read slop and is already addressed by a half dozen features (use systemd-resolved to never hit this problem, this is why the tailscaled.service does ExecStopPost=/usr/sbin/tailscaled --cleanup, this is why we have the dnsresolvecache / dnsfallback as a last resort, to boot up and fix things...)

The real interesting part here would be why it crashed at all. That deadlock information would be the interesting part.

I guess in human speak, your request is that the codespace container should have a wrapper process around it to restore /etc/resolv.conf in case crashes (deadlocks in this case) so codespaces can continue to ... do something?

But really I care more about that truncated deadlock stack. Still have it?

[nocnokneo]

Part of this might be a misunderstanding of the goal of this project. I know the repo is called "codespace" but the about line says "Tailscale for devcontainers and codespaces" so I assumed that it's intended to support devcontainers more generally which is how I'm using it (directly with the devcontainer CLI). I have a simple container that execs sh. I could run dbus and resolved myself under something that keeps them alive, but I don't think I should have to. Besides, someone running something like Alpine wouldn't have a reasonable path to doing something like that.

your request is that the codespace container should have a wrapper process around it to restore /etc/resolv.conf in case crashes (deadlocks in this case) so codespaces can continue to ... do something?

Having Tailscale running inside my container enables access to specific resources on our tailnet, but the container is largely still functional with basic internet connectivity, so yes, something that restores /etc/resolv.conf would be useful (easier to recover and diagnose). But really, I would prefer if tailscaled were restarted (at least as first-order recovery attempt).

But really I care more about that truncated deadlock stack. Still have it?

container2-tailscaled.log
container1-tailscaled.log

[bradfitz]

Yeah, most of the features I mentioned aren't really relevant in the codespaces context.

In any case, the real problem is it crashing at all. A secondary dream is changing the codespaces environment to not use /etc/resolv.conf as a resource that multiple containers need to fight over, but we can tackle that later.

From your stacks, it appears that the logging system is blocked and we're trying to log while holding a mutex and therefore things get wedged and the watchdog kicks in.

When does this happen? I'm curious what's making logging freeze like that.

I suspect the VM/container is getting frozen and then upon resume, the watchdog is confused and using the wrong clock and the logging isn't actually frozen... just by wall clock it seems frozen. Looking into that now.

[nocnokneo]

From your stacks, it appears that the logging system is blocked

Yep, just took a look through the system logs. There was an OOM storm last night between 21:04:33 and 21:15 (someone was likely doing some R&D work; this is a shared machine). I didn't see any indication that tailscale was OOM-killed though so theoretically should have been able to survive if the watchdog didn't get confused and panic.

The devcontainer is running on a bare metal Linux machine, not a VM.