Kafka ACL changes for firehoses - #123
Open
shruti-gojek wants to merge 158 commits into
Open
Conversation
fix: kube deployment name-limit handling
chore: change odpf references to goto
* fix: golanci config * fix: gci lint issues * fix: lint issues * fix: remove nosnakecase lint from disable list
* fix: deployment id name
* chore: add better error description * refactor: merge firehose module files
refactor: reduce firehose package
* feat: implement create, update, reset * feat: implement upgrade * feat: implement start, stop, scale * test: add lots of tests * feat: implement reset-sync * refactor: simplify kafka reset flow * feat: implement log * refactor: separate client & server CLI * feat: fix entropy client cli
* fix: use previous telegraf conf * refactor: remove old firehose module
fix: change commit author to bot
* fix: strip trailing colon in image tag * feat: add telegraf config templating
This reverts commit 2b799d7.
feat: firehose toleration affinity based on autoscaler
* fix: toleration affinity mode kube * fix: toleration affinity mode kube
* feat(firehose): sink based autoscaler config * feat: update custom transformers * feat: override triggers only * feat: update trigger override * feat: remove unused const * feat: update triggers
Co-authored-by: Ayushi Sharma <ayushi.sharma@gojek.com>
Co-authored-by: Ayushi Sharma <ayushi.sharma@gojek.com>
* feat: otel integration * fix: resolve non-constant format string issues - Fix WithCausef calls in pkg/validator/validator.go to use constant format strings - Fix WithCausef call in pkg/errors/errors.go to use constant format string - Fix multiple WithCausef calls in pkg/helm/helm.go to use constant format strings - All format functions now properly use '%s' as constant format with dynamic values as arguments - Resolves security and linting issues related to format string usage * fix: resolve all remaining non-constant format string issues - Applied comprehensive fix across entire codebase - Fixed WithCausef and WithMsgf calls to use constant format strings - Used automated script to fix patterns like err.Error() and strings.Join() - Manual fixes for edge cases in firehose and core modules - All format functions now properly use '%s' as constant format with dynamic values as arguments - Resolves all remaining security and linting issues related to format string usage * chore: update lint option * chore: update lint option * feat: enable newrelic * feat: enable grpc otel --------- Co-authored-by: Femi Novia Lina <feminovi@gmail.com>
feat: add dagger fs oss endpoint helm values
* feat(dagger): add support for tolerations and node affinity for dagger * feat: test dagger chart with existing autoscaler tolerations * add debug logs * feat: centralize node affinity interface conversion logic --------- Co-authored-by: Ayushi Sharma <ayushi.sharma@gojek.com>
* feat: make influx variables configurable --------- Co-authored-by: Ayushi Sharma <ayushi.sharma@gojek.com>
* feat: make telegraf container resource configurable * refactor: make telegraf resources conditional and use helm chart defaults Only include telegraf resources in helm values if explicitly specified by user, allowing the helm chart to provide default values via values.yaml. This ensures backward compatibility and follows the principle of letting helm charts manage their own defaults. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: generate unique cluster name per test to prevent conflicts Previously, TestClusterName was generated once at package load time and reused across all tests. This caused failures when multiple tests ran in parallel or in quick succession because KinD would fail with "node(s) already exist" error. Changes: - Generate a unique cluster name for each SetupTests() call - Pass the cluster name as a return value from SetupTests - Update all test suites to use the unique cluster name instead of global This ensures each test gets its own isolated cluster and prevents conflicts in CI environments. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: pass cluster name to BootstrapKubernetesResource function Update BootstrapKubernetesResource to accept clusterName as a parameter instead of using the now-removed global TestClusterName variable. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: update kafka_test.go to handle new SetupTests return value SetupTests now returns the cluster name as the 8th value. Update kafka_test.go to receive this value even though it doesn't use Kubernetes. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: make UsageSpec fields optional in validation Remove 'validate:\"required\"' tags from UsageSpec CPU and Memory fields to allow optional telegraf resources. The main firehose container limits/requests come from driver defaults and are always set, while telegraf resources are optional and can be omitted to use Helm chart defaults. Fixes validation error when creating modules with telegraf configuration. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Mirror the existing Kafka/Influx/BigQuery handling to make CSV a first-class Dagger sink type: - allow SINK_TYPE=CSV in the config JSON schema (both enums) - add SinkTypeCSV, SINK_CSV_* key constants, and the SinkCSV struct (embedded in Sink) - translate SINK_CSV_* into env vars in readConfig; require SINK_CSV_BASE_PATH and emit optional keys only when set so the Dagger app's own defaults are preserved - add config_test.go covering translation, the base-path guard, and the schema enum Co-authored-by: rajuGT <raju.gt@gojek.com>
* feat: mask sensitive values in resource and module API responses Mask configured sensitive values in resource and module API responses so secrets never leave Entropy in cleartext, while letting consumers detect when a specific credential changed via a keyed fingerprint. - pkg/masking: Masker (Mask/Restore/HMAC-SHA256 fingerprint), dot + trailing-* path walker, ValidatePaths, request-scoped Provider with per-module-URN cache over a narrow ModuleConfigLookup. - Resource read path: mask spec.configs + state.output across all six endpoints on a source-preserving copy; fail-open with warning; revisions resolve kind/project from the URN. - Module read path: mask configs in GetModule/ListModules, keeping sensitive_config visible. - Write path: core.Service WithMasking option restores stored secrets for masked-form inputs before Plan; masked-on-create rejected as invalid. - Module write validation of sensitive_config path syntax. - MaskingConfig.HMACKey (masking.hmac_key); empty disables masking. Threaded through cli/serve.go -> server.Serve. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(masking): simplify config lookup cache and write-path merge Replace the request-scoped Provider + per-URN cache with a process-level ConfigCache, evicted explicitly on module Create/Update instead of relying on per-request/TTL freshness. This removes the cache-object threading through every resource mapper call and collapses the duplicate moduleConfigLookup adapter into a single shared instance. Collapse the write-path merge into one rule with no error branch: a masked value with nothing stored (e.g. on Create) is now dropped rather than rejected via ErrMaskedWithoutStored/ErrInvalid. No change to external behavior: masked format, HMAC change-detection, and path syntax are unchanged. * refactor(masking): gate ListResources masking on with_spec_configs, drop state.output masking maskResource now masks spec.configs only; state.output is no longer masked since sensitive_config paths target spec.configs. On ListResources, masking runs only when with_spec_configs is set (spec.configs is not hydrated otherwise), skipping the per-resource ConfigCache lookup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(masking): rename well-known key sensitive_config to sensitive_configs Pluralize the module config key parsed by the masking layer and update validation messages, log lines, and tests accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…172) * feat(job): add per-container env_variables override in module config Dex re-dumps masked placeholder values into a new job's env vars on Create; masking's Restore only recovers a real secret on Update, so Create ships with no real secret. The job module driver config now accepts containers.<name>.env_variables overrides that are overlaid onto the matching container after the existing global env merge, with the module value always winning regardless of whether the client sent a real or masked value. Independent of masking/sensitive_configs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(kubernetes,flink): avoid nil-driver panic when module config is empty DriverFactory unmarshaled into &kd where kd was already *kubeDriver, so json.Unmarshal received a **kubeDriver. A null/empty module Configs reset kd itself to nil (per encoding/json's null-pointer semantics), returning a non-nil module.Driver interface wrapping a nil pointer. The first field dereference in Output (m.TolerationMode) then segfaulted — hit in production when a job resource's kube_cluster dependency forced a live Output call on a kubernetes module whose registration Configs was empty. Unmarshal into a local value instead, so the returned driver is always non-nil even with empty/null config; log a warning in that case. Applied the same fix to flink's DriverFactory, which had the identical pattern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
|
Important Review skippedToo many files! This PR contains 215 files, which is 115 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (10)
📒 Files selected for processing (215)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.