[GOBBLIN-XXXX] Surface snapshot/partition copy metrics in observability#4206
Open
pratapaditya04 wants to merge 1 commit into
Open
[GOBBLIN-XXXX] Surface snapshot/partition copy metrics in observability#4206pratapaditya04 wants to merge 1 commit into
pratapaditya04 wants to merge 1 commit into
Conversation
Add per-dataset snapshotsCommitted and partitionsCommitted counts to both the GaaSJobObservabilityEvent DatasetMetric and the job-status JobStatistics REST surface, so file/blob movement jobs (e.g. DDM Iceberg snapshot replication) can report table snapshots and partitions committed. DatasetMetric (GaaSJobObservabilityEvent.avsc) gains snapshotsCommitted and partitionsCommitted (long, default -1 = unsupported/unknown). DatasetTaskSummary carries the two counts as non-@nonnull fields so the existing 5-arg RequiredArgsConstructor used by AbstractJobLauncher is preserved; they populate via JSON deserialization for events that supply them, and toDatasetMetric maps them through. JobStatistics (executionStatistics) gains optional bytesWritten, recordsWritten, filesCommitted, snapshotsCommitted and partitionsCommitted. JobStatusRetriever reads them from the flat ddm.* copy-metric keys the JobSummary event persists into the job state, and FlowExecutionResource sets each only when reported, so jobs that do not emit them are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
Description
Adds per-dataset
snapshotsCommittedandpartitionsCommittedcounts to the observability surfaces so file/blob movement jobs (e.g. Iceberg snapshot replication) can report the number of table snapshots and partitions committed in a run.GaaSJobObservabilityEvent
GaaSJobObservabilityEvent.avsc: addsnapshotsCommitted+partitionsCommittedto theDatasetMetricrecord (long, default-1= unsupported/unknown).DatasetTaskSummary: carry the two counts as non-@NonNulllong fields so the existing 5-arg@RequiredArgsConstructor(used byAbstractJobLauncher) is preserved; they populate via JSON deserialization for events that supply them.toDatasetMetricmaps them onto the regeneratedDatasetMetric.Job-status REST API
JobStatistics.pdl(executionStatistics): add optionalbytesWritten,recordsWritten,filesCommitted,snapshotsCommitted,partitionsCommitted.JobStatusRetriever: read them from the flat per-dataset copy-metric keys the JobSummary event persists into the job state (-1= absent/unsupported).FlowExecutionResource: set each onJobStatisticsonly when reported (>= 0), so jobs that do not emit them are unchanged.Backward compatible: all new fields default to
-1/absent; native Gobblin call sites are unchanged.Tests
Schema/plumbing change validated by codegen + compilation:
generateAvroregeneratesDatasetMetric(7-arg ctor + fields);generateDataTemplateregeneratesJobStatisticssetters;:gobblin-runtime:compileJavaand the flow-config-service-server module compile pass. End-to-end population is exercised by the companion producer-side change and its unit tests.Commits