feat(collectors): periodically refresh google.projects.filter to pick up new/removed projects without a restart

[ntmspavan]

Problem & Impact

google.projects.filter is resolved to a concrete project list once at startup. In long-lived Kubernetes deployments, projects added to or removed from the matching org/folder are never reflected until the pod restarts — new projects silently go unmonitored and deleted ones are queried needlessly.

Fixes #538

Solution

Adds an opt-in google.projects.filter-refresh-interval flag (default 0, disabled — fully backward compatible). When set and google.projects.filter is configured, a background goroutine re-resolves the filter on that interval and atomically swaps the in-memory project list (lock-free atomic.Pointer[[]string], safe under concurrent scrapes). A transient API error or an unexpectedly empty result never overwrites a known-good list. The HTTP handler now rebuilds its collector registry per scrape (as the ?collect=-filtered path already did) so the refreshed list actually reaches /metrics.

Verification

• go build ./...
• go vet ./...
• go test ./... -race (new tests cover refresh success/error/empty-result handling, concurrent access, and a regression test for WithCache pointer sharing)

Live Testing

Verified end-to-end against a real GCP organization (not just unit tests), using a binary built from this branch (fix/issue-538-projects-filter-refresh, commit 5a92ca8).

Tool version: stackdriver_exporter built from commit 5a92ca81002d04a83fcc820e0fedf8feb24fc37f, go1.25.8, darwin/arm64.

Setup: created two throwaway GCP projects under a real organization and ran the exporter against the org with:

--google.projects.filter='parent.id:<org-id> lifecycleState:ACTIVE'
--google.projects.filter-refresh-interval=15s
--monitoring.metrics-prefixes=compute.googleapis.com/

Before: only project A exists

$ curl -s localhost:9255/metrics | grep -oE 'project_id="[^"]+"' | sort -u
project_id="sdexp-refresh-a-1788844202"

Add project B — no exporter restart

Created a second project (sdexp-refresh-b-1788844331) under the same org while the exporter kept running (same PID throughout). GCP's Resource Manager list API is eventually consistent — it took ~75s for the new project to become visible to projects.list, which the background refresh then picked up on its next tick:

time=...T22:11:55 msg="Starting stackdriver_exporter" ...
time=...T22:12:11 msg="refreshed project list from google.projects.filter" count=1
time=...T22:12:26 msg="refreshed project list from google.projects.filter" count=1
time=...T22:12:41 msg="refreshed project list from google.projects.filter" count=1
time=...T22:12:56 msg="refreshed project list from google.projects.filter" count=1
time=...T22:13:11 msg="refreshed project list from google.projects.filter" count=2   <-- project B picked up here

$ curl -s localhost:9255/metrics | grep -oE 'project_id="[^"]+"' | sort -u
project_id="sdexp-refresh-a-1788844202"
project_id="sdexp-refresh-b-1788844331"

Remove project B — no exporter restart

Deleted project B (still the same running process). The next refresh dropped it automatically:

time=...T22:14:11 msg="refreshed project list from google.projects.filter" count=2
time=...T22:14:26 msg="refreshed project list from google.projects.filter" count=1   <-- project B dropped here

$ curl -s localhost:9255/metrics | grep -oE 'project_id="[^"]+"' | sort -u
project_id="sdexp-refresh-a-1788844202"

This is the exact bug from #538 reproduced and fixed: previously, this whole cycle would have required restarting the exporter for either the addition or removal of project B to ever show up in /metrics. Here it self-heals within one refresh interval, with the exporter process never restarted (single PID for the full test).

[kgeckhart]

ATM the refresh has no timeout on it and if something stalls on the google side the refresh will be blocked with no real indication of why.

I think the best way to resolve this is create a function to build the cloudresourcemanager service similar to the monitoringservice,

stackdriver_exporter/collectors/service.go

Line 43 in ff409e6

func createMonitoringService(ctx context.Context, cfg *config.Config) (*monitoring.Service, error) {

. AFAICT this is safe to create once and hold like we do with the monitoringservice. This should allow us to construct it in the same way with the googleclient timeout + retries. I think universedomain should be skipped.

[ntmspavan]

Thanks @kgeckhart for the feedback! Addressed in latest commit:

• Added createResourceManagerService with consistent timeout/retry logic and CloudPlatformReadOnlyScope.
• Initialized service once in NewRuntime to reuse across refresh intervals instead of reconstructing per call.