Add api toggle for the openframe managed policies

openframe/docs/README.md

@@ -39,6 +39,7 @@ The agent has its own switch, `--openframe-mode` / `ORBIT_OPENFRAME_MODE`.
 |-----|--------|
 | [architecture-host-assignments.md](architecture-host-assignments.md) | Direct host → policy/query targeting (`policy_hosts` / `query_hosts` join tables), gated by `FLEET_OPENFRAME_MODE`. Design & internals. |
 | [api-host-assignments.md](api-host-assignments.md) | REST API for the above (add/remove/replace/list hosts). |
+| [managed-policies.md](managed-policies.md) | `policies.openframe_managed` — platform-owned policies omitted from the policy list/count endpoints (and from GitOps deletion) while still running on hosts and reporting results. Gated by `FLEET_OPENFRAME_MODE`. |
 | [api-expose-osquery-host-id.md](api-expose-osquery-host-id.md) | Exposes `osquery_host_id` in the host JSON so the OpenFrame control plane can match agents. |
 | [query-results-ttl-cleanup.md](query-results-ttl-cleanup.md) | Time-based cleanup of `query_results` (keeps the Debezium CDC pipeline alive without unbounded growth). Gated by OpenFrame mode **and** a positive TTL. |
 | [redis-key-prefix.md](redis-key-prefix.md) | Per-tenant Redis key/channel prefix (`FLEET_REDIS_KEY_PREFIX`) so tenants can share one Redis. |

openframe/docs/fork-file-manifest.md

@@ -75,7 +75,8 @@ server/service/openframe/                           # agent token-auth pipeline
 server/datastore/mysql/migrations/openframe/        # separate goose client
 ├── migration.go
 ├── 20260301000001_AddPolicyHostsJoinTable.go
-└── 20260301000002_AddQueryHostsJoinTable.go
+├── 20260301000002_AddQueryHostsJoinTable.go
+└── 20260818000001_AddPoliciesOpenframeManagedColumn.go   # policies.openframe_managed
 
 server/datastore/redis/keyprefix.go                 # per-tenant Redis prefix
 server/fleet/openframe.go                           # IsOpenframeMode() gate

openframe/docs/managed-policies.md

@@ -0,0 +1,141 @@
+# OpenFrame-Managed Policies
+
+## Overview
+
+An **OpenFrame-managed policy** is a normal Fleet policy carrying `policies.openframe_managed = 1`.
+It is omitted from the policy **list** and **count** endpoints — the set the main UI renders and the
+set GitOps reconciles — while it keeps running on hosts and keeps recording results exactly like any
+other policy.
+
+The use case is platform-owned checks: OpenFrame needs its own compliance/telemetry policies on
+every tenant without them cluttering the tenant operator's Policies page.
+
+The column is named `openframe_managed` rather than something generic like `hidden` or `internal` so
+that it can never collide semantically with a field upstream Fleet may add later — the same
+reasoning as `teams.openframe_tenant_uuid`.
+
+This is fork-only behavior, but it is **not** gated on `FLEET_OPENFRAME_MODE`: that flag gates
+behavior such as host assignments, while the column is created by the OpenFrame migration pipeline,
+which `prepare db` runs unconditionally. The filter is therefore always on — and inert until
+something actually sets the flag.
+
+## What the flag does — and what it deliberately does not
+
+| Surface | Managed policy |
+|---|---|
+| `GET /policies`, `GET /teams/{id}/policies` (incl. inherited + `merge_inherited`) | **omitted** |
+| `GET /policies/count`, `GET /teams/{id}/policies/count` | **not counted** |
+| GitOps deletion pass (`fleetctl gitops`) | **invisible → never deleted** |
+| `GET /policies/{id}` (single policy) | returned in full |
+| `PATCH /policies/{id}` | accepted — including `openframe_managed: false` |
+| `POST /policies/delete` | accepted |
+| `POST /spec/policies` with a matching name | **silently overwrites the policy** |
+| Host details → `policies` array | returned |
+| Host failing-policies count / Issues column | counted |
+| Fleet Desktop ("My device") | counted |
+| Activity feed (`created_policy` / `edited_policy`) | recorded |
+| Automations (webhooks, Jira/Zendesk, install software, run script, calendar) | fire normally |
+| osquery agent config on the host | delivered (the query text is readable on the endpoint) |
+
+**This flag is decluttering, not access control.** It filters listings only; write paths and by-id
+reads never consult it. Verified against a running server: a user holding admin or maintainer can
+
+1. confirm a managed policy's name — `POST /policies` with that name returns
+   `Policy "<name>" already exists`;
+2. read it in full by id (ids are sequential and trivially enumerated);
+3. unhide and rewrite it — `PATCH /policies/{id} {"openframe_managed": false, "query": "..."}`;
+4. delete it — `POST /policies/delete {"ids":[N]}`;
+5. **overwrite it silently** — `POST /spec/policies` matches on `(team, name)` via
+   `INSERT ... ON DUPLICATE KEY UPDATE`, so the query/description/platform are replaced while
+   `openframe_managed` stays `1`. The operator ends up owning a policy they cannot see, and the
+   platform ends up running a query it did not write. This one fires by accident on a mere name
+   collision, no malice required.
+
+If any of that matters, the fix is guards on the write paths (`modifyPolicy`, both delete paths,
+`ApplyPolicySpecs`) returning 404 for managed policies, plus 404 on by-id reads. Full concealment
+from a tenant admin is not reachable while they hold write access to the same name space — the
+uniqueness key `(team_id, name)` always leaks existence — unless platform policy names get a
+reserved prefix.
+
+### GitOps interaction
+
+`fleetctl gitops` deletes team policies that are not in the YAML. It builds that "existing" set from
+`GetPolicies`, i.e. the list endpoint, so managed policies are invisible to it and survive a GitOps
+apply untouched. The apply half is the hazard described above.
+
+## API
+
+Create (both `POST /policies` and `POST /teams/{id}/policies`):
+
+```json
+{ "name": "openframe: disk encryption", "query": "SELECT 1 ...", "openframe_managed": true }
+```
+
+Modify takes `"openframe_managed": true|false`; omitting the field leaves the flag as-is. Every
+policy payload returns `"openframe_managed"` so the platform can tell them apart.
+
+There is deliberately **no `include_managed` query parameter** on the standard endpoints: the
+listing fails closed, and a user-supplied flag cannot widen it. The platform reads managed policies
+by id, or through a dedicated OpenFrame endpoint.
+
+## Implementation
+
+### Schema
+
+`server/datastore/mysql/migrations/openframe/20260818000001_AddPoliciesOpenframeManagedColumn.go`
+adds `policies.openframe_managed TINYINT(1) NOT NULL DEFAULT 0`. It ALTERs an upstream table from
+the OpenFrame pipeline — the same pattern as `teams.openframe_tenant_uuid`, and it is on the
+[semantic-conflict watchlist](upstream-sync-conflict-resolution.md).
+
+### Where it lives
+
+`policies.openframe_managed` is in `policyCols` like any other column, and `schema.sql` carries it
+too. That last part is the point worth remembering: `cmd/fleet/prepare.go` runs `MigrateOpenframe`
+**unconditionally**, so every real deployment has the column no matter what `FLEET_OPENFRAME_MODE`
+says — the mode flag gates behavior (host assignments), never schema. `schema.sql` is only used to
+build test databases, so it must reflect that same reality; without the column there, the test
+harness would diverge from production and every policy test touching `policyCols` would fail on
+`Unknown column`.
+
+The flag is written by the same `INSERT`/`UPDATE` statements as every other policy field
+(`newGlobalPolicy`, `newTeamPolicy`, `savePolicy`) — no separate write. `ApplyPolicySpecs` does not
+name the column, so a spec apply leaves it untouched.
+
+The only helper is `openframeManagedExclusion(alias)` in `server/datastore/mysql/policies.go`, under
+`OPENFRAME(managed-policies)` markers: it returns `AND <alias>.openframe_managed = 0` and is appended
+to the five listing/count queries.
+
+SEMANTIC-CONFLICT WATCHLIST: `make dump-test-schema` regenerates `schema.sql` from the upstream
+`tables/` migrations only and would drop the `openframe_managed` line. Re-add it after any such
+regeneration — see [upstream-sync-conflict-resolution.md](upstream-sync-conflict-resolution.md).
+
+### Touched files
+
+| File | Change |
+|------|--------|
+| `migrations/openframe/20260818000001_AddPoliciesOpenframeManagedColumn.go` | new column |
+| `server/fleet/policies.go` | `OpenframeManaged` on `PolicyData`, `PolicyPayload`, `NewTeamPolicyPayload`, `ModifyPolicyPayload` |
+| `server/fleet/api_policies.go` | `OpenframeManaged` on `GlobalPolicyRequest` |
+| `server/service/global_policies.go` | maps the flag into the create payload |
+| `server/service/team_policies.go` | maps the flag on team create and on modify |
+| `server/datastore/mysql/schema.sql` | the column, so test databases match production |
+| `server/datastore/mysql/policies.go` | `policyCols`, the two INSERTs + the UPDATE, and the exclusion in `listPoliciesDB`, `getInheritedPoliciesForTeam`, `ListMergedTeamPolicies`, `CountPolicies`, `CountMergedTeamPolicies` |
+| `server/datastore/mysql/policies_openframe_managed_test.go` | MySQL coverage for all of the above |
+
+## Host assignment interaction (open item)
+
+In OpenFrame mode `PolicyQueriesForHost` requires a `policy_hosts` row for every policy
+([architecture-host-assignments.md](architecture-host-assignments.md) claims a "no rows → all hosts"
+fallback that the code does not implement). A managed policy is subject to the same rule: without
+assignments it reaches no host, and hosts enrolling later are not backfilled.
+
+Making the flag bypass that requirement is a one-line change in both `policyQueriesForHostStmt` and
+`ListPoliciesForHost`:
+
+```sql
+AND (p.openframe_managed = 1 OR EXISTS (
+        SELECT 1 FROM policy_hosts ph WHERE ph.policy_id = p.id AND ph.host_id = ?
+))
+```
+
+It is **not** part of this change — it alters what agents execute, so it wants its own review.

openframe/scripts/verify.sh

@@ -46,7 +46,7 @@ rm -f vet.err
 # 3. Marker presence: if a merge silently dropped fork code, its OPENFRAME markers
 #    vanish too. A slug dropping to zero is a red flag worth a human look.
 step "OPENFRAME marker presence (dropped-fork-code detector)"
-for slug in host-assignments redis-key-prefix redis-seed-nodes query-results-ttl osquery-host-id agent-openframe-mode agent-json-content-type migration-race; do
+for slug in host-assignments managed-policies redis-key-prefix redis-seed-nodes query-results-ttl osquery-host-id agent-openframe-mode agent-json-content-type migration-race; do
   n=$(grep -rIl "OPENFRAME($slug" --include='*.go' --include='*.yaml' --include='*.tpl' . 2>/dev/null | wc -l | tr -d ' ')
   if [ "$n" -gt 0 ]; then ok "$slug — present in $n file(s)"; else bad "$slug — NO markers found (fork code may have been dropped in the merge)"; fi
 done

server/datastore/mysql/migrations/openframe/20260818000001_AddPoliciesOpenframeManagedColumn.go

@@ -0,0 +1,47 @@
+package openframe
+
+import (
+	"database/sql"
+	"fmt"
+)
+
+func init() {
+	MigrationClient.AddMigration(Up_20260818000001, Down_20260818000001)
+}
+
+// Up_20260818000001 adds `policies.openframe_managed` — the flag that keeps a policy out of the policy list
+// endpoints (the set the main UI renders) while it keeps running on hosts and keeps reporting
+// results. See openframe/docs/managed-policies.md.
+//
+// It is also the escape hatch for host-assignment scoping: an OpenFrame-managed policy runs on every in-scope
+// host without any policy_hosts rows, so a platform-owned check needs no per-host assignment
+// backfill for hosts that enroll later.
+//
+// Idempotent.
+//
+// SEMANTIC-CONFLICT WATCHLIST (openframe/docs/upstream-sync-conflict-resolution.md):
+// this ALTERs the upstream `policies` table from the OpenFrame pipeline. If a future upstream
+// release rebuilds that table, re-verify this migration after the sync.
+func Up_20260818000001(tx *sql.Tx) error {
+	const (
+		table  = "policies"
+		column = "openframe_managed"
+	)
+
+	hasCol, err := columnExists(tx, table, column)
+	if err != nil {
+		return fmt.Errorf("checking %s.%s column: %w", table, column, err)
+	}
+	if hasCol {
+		return nil
+	}
+
+	if _, err := tx.Exec("ALTER TABLE policies ADD COLUMN openframe_managed TINYINT(1) NOT NULL DEFAULT 0"); err != nil {
+		return fmt.Errorf("adding %s.%s column: %w", table, column, err)
+	}
+	return nil
+}
+
+func Down_20260818000001(tx *sql.Tx) error {
+	return nil
+}

server/datastore/mysql/policies.go

@@ -40,7 +40,8 @@ const policyCols = `
 	p.author_id, p.platforms, p.created_at, p.updated_at, p.critical,
 	p.calendar_events_enabled, p.software_installer_id, p.script_id,
 	p.vpp_apps_teams_id, p.conditional_access_enabled, p.type,
-	p.patch_software_title_id, p.continuous_automations_enabled
+	p.patch_software_title_id, p.continuous_automations_enabled,
+	p.openframe_managed
 `
 
 const (
@@ -122,10 +123,10 @@ func newGlobalPolicy(ctx context.Context, db sqlx.ExtContext, authorID *uint, ar
 	nameUnicode := norm.NFC.String(args.Name)
 	res, err := db.ExecContext(ctx,
 		fmt.Sprintf(
-			`INSERT INTO policies (name, query, description, resolution, author_id, platforms, critical, checksum) VALUES (?, ?, ?, ?, ?, ?, ?, %s)`,
+			`INSERT INTO policies (name, query, description, resolution, author_id, platforms, critical, openframe_managed, checksum) VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s)`,
 			policiesChecksumComputedColumn(),
 		),
-		nameUnicode, args.Query, args.Description, args.Resolution, authorID, args.Platform, args.Critical,
+		nameUnicode, args.Query, args.Description, args.Resolution, authorID, args.Platform, args.Critical, args.OpenframeManaged,
 	)
 	switch {
 	case err == nil:
@@ -406,6 +407,16 @@ func loadHostsForPolicies(ctx context.Context, db sqlx.QueryerContext, policies
 
 // <<< OPENFRAME(host-assignments)
 
+// >>> OPENFRAME(managed-policies): platform-owned policies kept out of the policy list endpoints — openframe/docs/managed-policies.md
+
+// openframeManagedExclusion is the WHERE fragment that keeps OpenFrame-managed policies out of a
+// listing; every listing/count query below aliases the table as `p`. Unconditional: `prepare db`
+// always runs MigrateOpenframe, so the column exists in every deployment — FLEET_OPENFRAME_MODE
+// gates behavior, not schema.
+const openframeManagedExclusion = ` AND p.openframe_managed = 0`
+
+// <<< OPENFRAME(managed-policies)
+
 func loadLabelsForPolicies(ctx context.Context, db sqlx.QueryerContext, policies []*fleet.Policy) error {
 	const sql = `
 		SELECT
@@ -610,11 +621,12 @@ func savePolicy(ctx context.Context, db sqlx.ExtContext, logger *slog.Logger, p
 			platforms = ?, critical = ?, calendar_events_enabled = ?,
 			software_installer_id = ?, script_id = ?, vpp_apps_teams_id = ?,
 			conditional_access_enabled = ?, continuous_automations_enabled = ?,
+			openframe_managed = ?,
 			checksum = ` + policiesChecksumComputedColumn() + `
 			WHERE id = ?
 	`
 	result, err := db.ExecContext(
-		ctx, updateStmt, p.Name, p.Query, p.Description, p.Resolution, p.Platform, p.Critical, p.CalendarEventsEnabled, p.SoftwareInstallerID, p.ScriptID, p.VPPAppsTeamsID, p.ConditionalAccessEnabled, p.ContinuousAutomationsEnabled, p.ID,
+		ctx, updateStmt, p.Name, p.Query, p.Description, p.Resolution, p.Platform, p.Critical, p.CalendarEventsEnabled, p.SoftwareInstallerID, p.ScriptID, p.VPPAppsTeamsID, p.ConditionalAccessEnabled, p.ContinuousAutomationsEnabled, p.OpenframeManaged, p.ID,
 	)
 	if err != nil {
 		return ctxerr.Wrap(ctx, err, "updating policy")
@@ -1184,6 +1196,10 @@ func listPoliciesDB(ctx context.Context, q sqlx.QueryerContext, teamID *uint, op
 		args = append(args, filterArgs...)
 	}
 
+	// >>> OPENFRAME(managed-policies): drop platform-owned policies from this listing — openframe/docs/managed-policies.md
+	query += openframeManagedExclusion
+	// <<< OPENFRAME(managed-policies)
+
 	// We must normalize the name for full Unicode support (Unicode equivalence).
 	match := norm.NFC.String(opts.MatchQuery)
 	query, args = searchLike(query, args, match, policySearchColumns...)
@@ -1232,6 +1248,10 @@ func getInheritedPoliciesForTeam(ctx context.Context, q sqlx.QueryerContext, tea
         WHERE p.team_id IS NULL
     `
 
+	// >>> OPENFRAME(managed-policies): drop platform-owned policies from this listing — openframe/docs/managed-policies.md
+	query += openframeManagedExclusion
+	// <<< OPENFRAME(managed-policies)
+
 	args = append(args, teamID)
 
 	// We must normalize the name for full Unicode support (Unicode equivalence).
@@ -1292,6 +1312,11 @@ func (ds *Datastore) CountPolicies(ctx context.Context, teamID *uint, matchQuery
 		args = append(args, *teamID)
 	}
 
+	// >>> OPENFRAME(managed-policies): keep platform-owned policies out of the paging/badge counts too — openframe/docs/managed-policies.md
+	// either — openframe/docs/managed-policies.md
+	query += openframeManagedExclusion
+	// <<< OPENFRAME(managed-policies)
+
 	if teamID != nil {
 		automationFilter, filterArgs, err := ds.createAutomationClause(ctx, automationType, *teamID)
 		if err != nil {
@@ -1325,6 +1350,10 @@ func (ds *Datastore) CountMergedTeamPolicies(ctx context.Context, teamID uint, m
 	var args []interface{}
 
 	query := `SELECT count(*) FROM policies p WHERE (p.team_id = ? OR p.team_id IS NULL)`
+	// >>> OPENFRAME(managed-policies): keep platform-owned policies out of the paging/badge counts too — openframe/docs/managed-policies.md
+	// either — openframe/docs/managed-policies.md
+	query += openframeManagedExclusion
+	// <<< OPENFRAME(managed-policies)
 	args = append(args, teamID)
 
 	automationFilter, filterArgs, err := ds.createAutomationClause(ctx, automationType, teamID)
@@ -1677,14 +1706,14 @@ func newTeamPolicy(ctx context.Context, db sqlx.ExtContext, teamID uint, authorI
 			`INSERT INTO policies (
 				name, query, description, team_id, resolution, author_id,
 				platforms, critical, calendar_events_enabled, software_installer_id,
-				script_id, vpp_apps_teams_id, conditional_access_enabled, checksum,
+				script_id, vpp_apps_teams_id, conditional_access_enabled, openframe_managed, checksum,
 				type, patch_software_title_id, continuous_automations_enabled
-			) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)`,
+			) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)`,
 			policiesChecksumComputedColumn(),
 		),
 		nameUnicode, args.Query, args.Description, teamID, args.Resolution, authorID, args.Platform, args.Critical,
 		args.CalendarEventsEnabled, args.SoftwareInstallerID, args.ScriptID, args.VPPAppsTeamsID,
-		args.ConditionalAccessEnabled, args.Type, args.PatchSoftwareTitleID, args.ContinuousAutomationsEnabled,
+		args.ConditionalAccessEnabled, args.OpenframeManaged, args.Type, args.PatchSoftwareTitleID, args.ContinuousAutomationsEnabled,
 	)
 	switch {
 	case err == nil:
@@ -1777,6 +1806,10 @@ func (ds *Datastore) ListMergedTeamPolicies(ctx context.Context, teamID uint, op
 		%s
     `, automationFilter)
 
+	// >>> OPENFRAME(managed-policies): drop platform-owned policies from this listing — openframe/docs/managed-policies.md
+	query += openframeManagedExclusion
+	// <<< OPENFRAME(managed-policies)
+
 	args = append(args, teamID, teamID)
 	if len(filterArgs) > 0 {
 		args = append(args, filterArgs...)