feat(gatewayapi): calico-system policy for namespaced data-plane proxies by electricjesus · Pull Request #4970 · tigera/operator

electricjesus · 2026-06-26T07:28:50Z

Description

Bug fix.

Since #4690 we run a single envoy-gateway controller in calico-system with deploy.type=GatewayNamespace. The data-plane proxies now run in each Gateway's own namespace, not in calico-system.

We render one gateway policy in the calico-system tier: calico-system.envoy-gateway. It lives in calico-system and selects the controller and certgen pods. The proxies run elsewhere and match no policy in that tier. When a Gateway namespace runs a default-deny tier, the proxy there has nothing to let its traffic through. This is the same kind of miss as the conformance MetalLB pool that stayed pinned to tigera-gateway (projectcalico/calico#13095).

This adds a GlobalNetworkPolicy, calico-system.envoy-gateway-proxy, that selects the proxy pods by k8s-app == 'calico-gateway-api-proxy'. The operator stamps that Calico-owned label on the proxy pods through the EnvoyProxy pod spec, so the policy keys off a label we control rather than Envoy Gateway's own gateway.envoyproxy.io/owning-gateway-name, which we do not own and which could change upstream. A GNP covers any Gateway namespace, including ones created later, with no re-render. It allows:

Egress to DNS, plus xDS (18000) and Wasm fetch (18002) to the controller in calico-system. The proxy dials the controller (see envoyproxy/gateway internal/infrastructure/kubernetes/proxy/resource.go and internal/xds/bootstrap). 18001 is the ratelimit path, not a proxy path, so it is left out.
Ingress on all inbound TCP. Listener ports are user-defined, so the proxy has to accept any port to work out of the box. This also covers the 19001 metrics scrape.

The same Calico-owned labelling now covers all three envoy-gateway components (controller, certgen, proxy) from one documented place in the gateway render. That k8s-app label is not set through the standard labeller in pkg/controller/utils/component.go on purpose: the labeller keys off the object name, and the proxy pods are created by the envoy-gateway controller at runtime, so the operator never renders them to label.

How I tested it

I built this operator image and ran it on an OSS master cluster (eBPF, namespaced mode). I put a Gateway, an HTTPRoute, and an nginx backend in a normal user namespace, scoped a Calico default-deny to the proxy pod, and curled the proxy from inside the cluster.

Step	Setup	Result
Baseline	no deny	HTTP 200
Reproduce	default-deny, stock operator (no GNP)	times out, proxy fully blocked
This PR	default-deny, our operator (GNP)	HTTP 503: proxy reachable and routing, only the backend hop is denied
Backend allow	user allows proxy to backend	HTTP 200

I also tried the narrower ingress idea (allow only 19001, then Pass). Under default-deny it times out: listener traffic falls through to the user's deny and the Gateway stops serving. So allowing all inbound TCP is the right default. The cost is that an Allow is terminal in this tier, so a user cannot narrow proxy ingress with their own policy. Scaling the operator back up showed it reverts any drift on the GNP and the Gateway recovers.

I re-ran the whole repro after switching the selector to the Calico-owned k8s-app label (OSS master, eBPF, namespaced mode). The operator stamps k8s-app: calico-gateway-api-proxy on the proxy pod, the rendered GNP calico-system.envoy-gateway-proxy selects it, and the table holds: baseline 200; under a Gateway-namespace default-deny the proxy returns 503 (reachable and routing, only the backend hop denied); allowing proxy to backend gives 200. Scaling the operator down and deleting the GNP drops the proxy to a full timeout, and scaling it back up re-renders the GNP and the Gateway recovers. One thing to watch when testing this by hand: the 503 lands after Envoy's ~10s upstream timeout, so a curl timeout under 10s reads as a connect failure even though the proxy is reachable..

Rebased onto master on 2026-08-19. Clean rebase, no conflicts. go build ./... passes and pkg/render/gatewayapi is 34 of 34 green.

One thing to know

The GNP covers the proxy's own needs: DNS, the control-plane link, and ingress. It does not open egress to backends, because backends are arbitrary user workloads. Its egress rules end in a Pass, so the decision falls to the user's own tier and they keep control of which backends the proxy may reach. A user who runs default-deny in a Gateway namespace has to allow the proxy to reach their backend themselves. Until they do, the proxy is up and configured but returns 503 on the upstream. This matches how the controller policy already works.

The docs side of that is tigera/docs#2802, which trims the reference policy on the create-an-ingress-gateway page down to the part the user still owns.

What moved out of this PR

An earlier push carried a second change that added the standard app.kubernetes.io/* labels to the pod templates of operator-rendered Jobs. It came out of Rene's review question about component.go. I pulled it out on its own branch, because it turned out not to be labels-only: making a Job's pod template non-nil also makes the shared tail in component.go stamp operator.tigera.io/host-networked on any hostNetwork pod template. The AWS security-group setup Job runs hostNetwork, and the podiprecovery controller deletes labelled hostNetwork pods whose status.podIPs do not match the node. That is a real side effect and it deserves its own review rather than riding along here.

Release Note

Add a `calico-system`-tier GlobalNetworkPolicy for Envoy Gateway data-plane proxies so Gateways work under a default-deny tier when proxies run in per-Gateway namespaces (deploy.type=GatewayNamespace).

For PR author

Tests for change.
If changing pkg/apis/, run make gen-files (n/a, no API change).
If changing versions, run make gen-versions (n/a).

Since deploy.type=GatewayNamespace (tigera#4690) the data-plane envoy proxies run in each Gateway's own namespace, not calico-system. The only calico-system gateway policy selects the controller/certgen pods in calico-system, so the proxies match nothing and have no policy punching through a default-deny tier in the namespaces they now run in. Add a GlobalNetworkPolicy selecting the EG proxy pods (label gateway.envoyproxy.io/owning-gateway-name) so it covers every Gateway namespace with no re-render: DNS + xDS(18000)/Wasm(18002) egress to the controller in calico-system, and all inbound TCP so a managed Gateway serves traffic out of the box under a default-deny tier. Backend egress is left to the user, matching the controller policy.

Per review: instead of selecting the data-plane proxy pods on Envoy Gateway's gateway.envoyproxy.io/owning-gateway-name label, which we do not own and which could change upstream without notice, stamp our own k8s-app=calico-gateway-api-proxy label on the proxy pods through the EnvoyProxy pod spec and point the calico-system-tier proxy policy at that. Any user-supplied pod labels from a custom EnvoyProxy are kept.

Put the Calico-owned k8s-app labeling for all three envoy-gateway components in one documented place. The operator-rendered controller and certgen pods go through setGatewayComponentLabel; the runtime-created proxy pods, which the operator never renders, go through the EnvoyProxy pod spec (ensureGatewayProxyLabel). Documents why this lives in the gateway render and not the standard labeler in component.go, which keys off the object name and cannot reach a runtime-created pod.

… cover tigera#2873 replaced the obsolete tigera-gateway namespace exclusion with a reference NetworkPolicy that users apply in each Gateway namespace. That policy allows DNS and the gateway controller hop itself. With tigera/operator#4970 the operator renders calico-system.envoy-gateway-proxy, which already allows inbound traffic to the proxy plus proxy egress to DNS and to the controller on 18000 and 18002. The reference policy would duplicate rules the operator now owns. What the operator's policy deliberately does not cover is the backend hop. It ends its egress rules with a Pass so the user keeps control of which backends the proxy may reach. Under default deny that hop stays denied, and the gateway returns 503 after the upstream connect timeout. So the step now covers only the user's part: - proxy egress to the backend, and backend ingress from the proxy. Both are needed, because default deny applies to the backend pod too. - on Enterprise, egress to the Kubernetes API server, for the log collector that runs in the proxy pod. The example selects proxies by k8s-app == "calico-gateway-api-proxy", the Calico-owned label that #4970 stamps on the proxy pod template.

Copilot

Pull request overview

This PR updates the Gateway API rendering to ensure Envoy Gateway data-plane proxy pods (which run in per-Gateway namespaces in namespaced mode) are covered by a calico-system-tier policy, so they can function under namespace default-deny.

Changes:

Add a GlobalNetworkPolicy (calico-system.envoy-gateway-proxy) that selects proxy pods via a Calico-stamped k8s-app label and allows required proxy ingress/egress under default-deny.
Stamp a Calico-owned k8s-app label onto proxy pods via the EnvoyProxy pod spec, and refactor controller/certgen label stamping into a helper.
Extend rendering tests to expect the new GlobalNetworkPolicy and verify the proxy label is stamped.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
pkg/render/gatewayapi/gateway_api.go	Adds proxy GlobalNetworkPolicy and stamps/standardizes `k8s-app` labeling for controller/certgen/proxy selection.
pkg/render/gatewayapi/gateway_api_test.go	Updates expectations for the new proxy policy and verifies proxy pod labeling behavior.

Suppressed comments (1)

pkg/render/gatewayapi/gateway_api.go:892

This comment says any user-supplied pod labels are preserved, but the function unconditionally overwrites "k8s-app" (which could also be user-supplied on a custom EnvoyProxy). Clarifying that only non-"k8s-app" labels are preserved would avoid confusion.

// ensureGatewayProxyLabel stamps the same Calico-owned k8s-app label on the data-plane
// proxy pods, via the EnvoyProxy pod spec, so gatewayAPIProxyPolicy can select them.
// Any user-supplied pod labels carried over from a custom EnvoyProxy are preserved.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+		proxyPolicy, err := rtest.GetResourceOfType[*v3.GlobalNetworkPolicy](objsToCreate, ProxyPolicyName, "")
+		Expect(err).NotTo(HaveOccurred())
+		Expect(proxyPolicy.Spec.Tier).To(Equal("calico-system"))
+		Expect(proxyPolicy.Spec.Selector).To(Equal(EnvoyProxyPolicySelector))
+		Expect(proxyPolicy.Spec.Selector).NotTo(ContainSubstring("owning-gateway-name"),
+			"proxy policy must select by our Calico label, not Envoy Gateway's owning-gateway-name")


caseydavenport · 2026-08-20T15:17:00Z

+// ensureGatewayProxyLabel stamps the same Calico-owned k8s-app label on the data-plane
+// proxy pods, via the EnvoyProxy pod spec, so gatewayAPIProxyPolicy can select them.
+// Any user-supplied pod labels carried over from a custom EnvoyProxy are preserved.
+func ensureGatewayProxyLabel(pod *envoyapi.KubernetesPodSpec) {


Alternatively, we could update the component handler so that it catches envoy proxy resources and labels them automatically too? WDYT?

caseydavenport · 2026-08-20T15:18:46Z

+// Gateway's own namespace (deploy.type=GatewayNamespace), not calico-system —
+// punch through any default-deny in those namespaces. It is a GlobalNetworkPolicy
+// rather than a per-namespace NetworkPolicy fanned out over GatewayNamespaces so
+// that it automatically covers new Gateway namespaces with no re-render.


Is "automatically with no re-render" actually important / desirable?

On its own I don't think this is a reason one way or another - but could there be reasons we want global vs. not?

i.e., differentiation in flow logging? or RBAC reasons?

caseydavenport · 2026-08-20T15:20:59Z

+			// Allow all inbound TCP from any source (this also covers the 19001 metrics
+			// scrape). Gateway listener ports are user-defined and dynamic, so a managed
+			// Gateway has to accept arbitrary ports to serve traffic out of the box,
+			// including under a default-deny tier. Verified on a cluster: the narrower
+			// alternative (allow only 19001, then Pass) lets listener ingress fall
+			// through to the user's default-deny and silently breaks every Gateway in a
+			// default-deny namespace. The cost of allowing all TCP is that an Allow is
+			// terminal in this tier, so a user cannot narrow ingress to the proxy with
+			// their own policy.


This comment is massive AI overcommenting - we especially don't need any "Verified on a cluster" type comments.

This should just explain why we want such an open ingress rule.

Do we actually want this? Or do we want to allow users to define where ingress can be allowed from?

Drop the narration around the proxy label helpers and the proxy policy, keeping only the parts a reader can't get from the code: why we select on our own label, why the proxy pod spec is the only labelling hook, and why the egress Pass is deliberate.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

marvin-tigera added this to the v1.44.0 milestone Jun 26, 2026

marvin-tigera added docs-pr-required release-note-required labels Jun 26, 2026

electricjesus added the docs-not-required label Jun 26, 2026

marvin-tigera removed the docs-pr-required label Jun 26, 2026

electricjesus force-pushed the seth/gatewayapi-proxy-allow-policy branch from d3d69a5 to 311699b Compare June 26, 2026 08:09

electricjesus marked this pull request as ready for review June 26, 2026 08:16

electricjesus requested a review from a team as a code owner June 26, 2026 08:16

electricjesus force-pushed the seth/gatewayapi-proxy-allow-policy branch from 311699b to fe034ab Compare June 26, 2026 10:13

electricjesus changed the title ~~feat(gatewayapi): allow-tigera policy for namespaced data-plane proxies~~ feat(gatewayapi): calico-system policy for namespaced data-plane proxies Jun 26, 2026

electricjesus mentioned this pull request Jun 26, 2026

docs(gateway): fix default-deny step for namespaced mode tigera/docs#2802

Open

5 tasks

rene-dekker reviewed Jul 7, 2026

View reviewed changes

Comment thread pkg/render/gatewayapi/gateway_api.go Outdated

electricjesus force-pushed the seth/gatewayapi-proxy-allow-policy branch from ed7bc96 to c549093 Compare July 29, 2026 17:28

danudey modified the milestones: v1.44.0, v1.45.0 Aug 17, 2026

electricjesus added 3 commits August 19, 2026 11:38

Copilot AI lite review requested due to automatic review settings August 19, 2026 10:47

electricjesus force-pushed the seth/gatewayapi-proxy-allow-policy branch from f50b72f to 1d372cd Compare August 19, 2026 10:47

Copilot started reviewing on behalf of electricjesus August 19, 2026 10:47 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

caseydavenport requested changes Aug 20, 2026

View reviewed changes

caseydavenport self-assigned this Aug 20, 2026

Copilot AI review requested due to automatic review settings August 20, 2026 19:57

Copilot AI reviewed Aug 20, 2026

View reviewed changes

Copilot started reviewing on behalf of electricjesus August 20, 2026 23:11 View session

Navigation Menu

Uh oh!

feat(gatewayapi): calico-system policy for namespaced data-plane proxies - #4970

feat(gatewayapi): calico-system policy for namespaced data-plane proxies#4970
electricjesus wants to merge 4 commits into
tigera:masterfrom
electricjesus:seth/gatewayapi-proxy-allow-policy

electricjesus commented Jun 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

caseydavenport Aug 20, 2026

Uh oh!

caseydavenport Aug 20, 2026

Uh oh!

caseydavenport Aug 20, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

electricjesus commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How I tested it

One thing to know

What moved out of this PR

Release Note

For PR author

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

caseydavenport Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

caseydavenport Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

caseydavenport Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

electricjesus commented Jun 26, 2026 •

edited

Loading