Skip to content

fix: use cached client for handler to reduce audit log volume - #325

Draft
kseiter-rh wants to merge 2 commits into
openshift:masterfrom
kseiter-rh:fix/use-cached-client-reduce-audit-logs
Draft

fix: use cached client for handler to reduce audit log volume#325
kseiter-rh wants to merge 2 commits into
openshift:masterfrom
kseiter-rh:fix/use-cached-client-reduce-audit-logs

Conversation

@kseiter-rh

@kseiter-rh kseiter-rh commented Aug 6, 2026

Copy link
Copy Markdown

Summary

  • Switch the handler from an uncached client.New() to the manager's cached mgr.GetClient(), eliminating ~10+ direct API server GET calls per reconciliation cycle (every 5 minutes per cluster)
  • Expand the cache configuration with ByObject entries to cover cross-namespace resources the handler needs: Secrets in openshift-config (pull-secret) and ConfigMaps in openshift-monitoring (CAMO ConfigMap)
  • Cluster-scoped resources (Proxy, ClusterVersion) are already cacheable by default since DefaultNamespaces only restricts namespaced resources

Context

Investigation of HCMSEC-3645 (Splunk ingestion spike in osdsecuritylogs) identified openshift-ocm-agent-operator as a contributor. The handler was created with a separate uncached client because the manager's cache was namespace-scoped to the operator namespace and couldn't serve cross-namespace reads. Every reconciliation cycle made direct API calls that generated audit log entries flowing to Splunk. Across thousands of clusters, this adds up significantly.

Controller-runtime v0.23.3 supports ByObject cache configuration with per-object namespace overrides, which lets us expand the cache to cover the specific cross-namespace resources without watching entire namespaces.

What changes

Before After
Handler uses client.New() (uncached) Handler uses mgr.GetClient() (cached)
Every GET hits API server → audit log GETs served from informer cache → no audit entry
Cache only watches operator namespace Cache also watches Secrets in openshift-config and ConfigMaps in openshift-monitoring

Write operations (Create/Update/Delete) still hit the API server as expected — only read operations benefit from caching.

Test plan

  • go build ./... passes
  • go test ./... — all unit tests pass (no handler test changes needed since tests mock the client.Client interface, which is the same regardless of cached/uncached backing)
  • Deploy to a staging cluster and verify operator reconciles correctly
  • Verify pull-secret changes in openshift-config are still detected (cache watch delivers update events)
  • Monitor Splunk audit log volume for reduction

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Improved detection of configuration drift during periodic reconciliation, including pull-secret and proxy settings.
  • Performance & Reliability

    • Refined controller resource caching for relevant Secrets and ConfigMaps.
    • Streamlined Kubernetes client usage to improve consistency during reconciliation.

The handler previously used an uncached client.New() for all Kubernetes
API calls, causing every reconciliation cycle (~10+ GET calls every 5
minutes per cluster) to hit the API server directly. These direct API
calls generate audit log entries that flow to Splunk, contributing to
the ingestion spike tracked in HCMSEC-3645.

Switch the handler to use mgr.GetClient() (the manager's cached client)
and expand the cache configuration to cover the cross-namespace resources
the handler needs:

- Secrets in openshift-config (for pull-secret reads)
- ConfigMaps in openshift-monitoring (for CAMO ConfigMap)
- Cluster-scoped resources (Proxy, ClusterVersion) are already
  cacheable since DefaultNamespaces only restricts namespaced resources

With this change, steady-state GET operations are served from the
informer cache without hitting the API server, eliminating the
corresponding audit log entries. Write operations (Create/Update/Delete)
still go through the API server as expected.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 6, 2026
@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@kseiter-rh, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 52 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 982cf257-1f8a-4110-b166-474cdc522bc5

📥 Commits

Reviewing files that changed from the base of the PR and between e0b4512 and e6f4a97.

📒 Files selected for processing (1)
  • main.go

Walkthrough

The manager now caches selected Secrets and ConfigMaps. OCMAgentHandlerBuilder uses the manager client instead of a separate client. Reconciliation comments now describe pull-secret and proxy configuration drift detection.

Changes

OCM agent manager and reconciliation

Layer / File(s) Summary
Manager cache and client wiring
main.go
The manager caches Secrets in openshift-config and ConfigMaps in openshift-monitoring. The handler builder now receives the manager client.
Reconciliation drift documentation
controllers/ocmagent/ocmagent_controller.go
The periodic reconciliation comments describe pull-secret and proxy configuration drift detection.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: using the cached client to reduce audit log volume.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR changes no test files; all discovered Ginkgo titles use static string literals, and structural searches found no formatting, concatenation, or runtime-value titles.
Test Structure And Quality ✅ Passed The diff changes only main.go and the controller; it contains no *_test.go files or Ginkgo constructs, so these test-quality requirements do not apply.
Microshift Test Compatibility ✅ Passed The pull request changes only controller comments and manager/cache setup; it adds no new Ginkgo e2e tests requiring MicroShift API compatibility review.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR changes only two production Go files and adds no Ginkgo e2e tests, so no SNO multi-node compatibility issue applies.
Topology-Aware Scheduling Compatibility ✅ Passed The diff changes cache wiring and comments only; it adds no replicas, affinity, topology spread, node selectors, tolerations, or PDB scheduling constraints.
Ote Binary Stdout Contract ✅ Passed The PR is for an operator and does not add or modify an OTE/openshift-tests binary; its e2e binary runs as an osde2e Ginkgo suite, so this contract is not applicable.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The pull request changes only main.go and a controller implementation file. It adds no Ginkgo e2e tests or test networking, so this check is not applicable.
No-Weak-Crypto ✅ Passed The PR only changes Kubernetes cache/client wiring and comments. No MD5, SHA-1, DES, RC4, Blowfish, ECB, custom crypto, or secret comparison was added.
Container-Privileges ✅ Passed The PR changes only Go files and comments. No privileged, host namespace, SYS_ADMIN, or allowPrivilegeEscalation settings were added; the production image runs as UID 1001.
No-Sensitive-Data-In-Logs ✅ Passed The PR adds no logging statements or sensitive-value formatting; it only changes cache wiring and comments. Existing logs use resource names and generic errors, not secret contents or tokens.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kseiter-rh
Once this PR has been reviewed and has the lgtm label, please assign charlesgong for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Narrow the ByObject cache entries to watch only the exact resources
needed (pull-secret in openshift-config, ocm-agent in openshift-monitoring)
rather than all Secrets/ConfigMaps in those namespaces. This avoids
caching sensitive secrets the operator doesn't need access to.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant