delta lake python/yaml by derrickaw · Pull Request #39052 · apache/beam

derrickaw · 2026-06-21T17:54:30Z

PreCommit Xlang Generated Transforms - https://github.com/apache/beam/actions/runs/28050009192
PreCommit YAML Xlang Direct - https://github.com/apache/beam/actions/runs/28049991009
PostCommit YAML Xlang Direct - https://github.com/apache/beam/actions/runs/28050031215

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2026-06-23T19:13:33Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces support for reading Delta Lake tables in Apache Beam's YAML SDK. It adds the necessary infrastructure to manage Delta Lake read transforms, including updating the expansion service, defining the read transform, and providing comprehensive integration tests to ensure functionality.

Highlights

Delta Lake Integration: Added support for reading Delta Lake tables within the Beam YAML SDK.
Testing Infrastructure: Implemented a new integration test for Delta Lake reading and added a corresponding temporary table fixture.
Expansion Service: Updated the expansion service configuration to include the Delta Lake IO module.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files

Ignored by pattern: .github/workflows/** (3)
- .github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml
- .github/workflows/beam_PreCommit_Xlang_Generated_Transforms.yml
- .github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

derrickaw · 2026-06-23T19:13:35Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for reading Delta Lake tables within Apache Beam, enabling the integration of the Delta Lake read schema transform into the Python SDK and YAML pipelines. The changes include adding Java tests, updating the expansion service, mapping the new transform in Python, and adding YAML integration tests. The review feedback recommends lazily importing optional dependencies like pyarrow to avoid import errors, specifying UTF-8 encoding when writing files, and using Dict instead of Mapping for type hints to prevent potential NameErrors.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist

Code Review

This pull request adds support for reading Delta Lake tables in Apache Beam, including Java-based schema transform provider tests, Python managed read transforms, and YAML integration tests. The review feedback suggests several improvements: replacing File.mkdirs() with Files.createDirectories() in Java tests to handle directory creation failures properly, importing pyarrow lazily in Python integration tests to avoid module-level import errors, and specifying encoding="utf-8" when opening files for writing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

codecov · 2026-06-23T19:34:19Z

Codecov Report

❌ Patch coverage is 14.28571% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.89%. Comparing base (e948ef6) to head (b2d710e).
⚠️ Report is 11 commits behind head on master.

Files with missing lines	Patch %	Lines
sdks/python/apache_beam/yaml/integration_tests.py	0.00%	17 Missing ⚠️
sdks/python/apache_beam/yaml/yaml_io.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #39052   +/-   ##
=========================================
  Coverage     55.89%   55.89%           
  Complexity     3913     3913           
=========================================
  Files          1320     1320           
  Lines        180971   181029   +58     
  Branches       2671     2671           
=========================================
+ Hits         101148   101187   +39     
- Misses        76931    76950   +19     
  Partials       2892     2892

Flag	Coverage Δ
python	`79.28% <14.28%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

gemini-code-assist

Code Review

This pull request introduces support for reading Delta Lake tables in Apache Beam, including Java tests, Python bindings, and YAML integration. The feedback recommends importing pyarrow locally within the integration test helper to prevent module-level ImportErrors when the library is not installed. Additionally, it suggests restricting the Parquet write transform in the Java tests to a single shard to avoid potential race conditions when writing to a static filename.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Abacn · 2026-06-24T03:59:46Z

        with:
          python-version: default
          java-version: |
-            17


"11 17" is effectively same as "17". Either change to 17 (and remove java17Home parameter) or add a "testJavaVersion=17" parameter and make the target honors it (like in https://github.com/apache/beam/pull/39064/changes#diff-0435a83a413ec063bf7e682cadcd56776cd18fc878f197cc99a65fc231ef2047)

Abacn · 2026-06-24T03:59:58Z

          java-version: |
-            17
            11
+            17


Same as above

derrickaw and others added 8 commits June 21, 2026 13:42

make changes to support delta

7cb8bbe

add transformprovider and yaml file

5908e88

add managed

d545063

fix lint

0eb47c7

Add Delta Lake source to the Java Managed API

19b86e5

match Cham's PR changes

ec43328

remove delta lake entry from external transforms

06cd805

set java home

b8e2a65

github-actions Bot added python java build io yaml delta labels Jun 21, 2026

derrickaw added 3 commits June 22, 2026 01:05

add back import

d8accd1

remove 11

ee3e447

Revert workflow changes

43017da

github-actions Bot removed the build label Jun 23, 2026

derrickaw added 2 commits June 23, 2026 13:50

few test fixes

98a15ce

force yaml to 17

0ffc468

github-actions Bot added the build label Jun 23, 2026

derrickaw added 2 commits June 23, 2026 15:32

fix another test

74c9928

revert improvements for now

b2d710e

derrickaw marked this pull request as ready for review June 23, 2026 19:12

derrickaw marked this pull request as draft June 23, 2026 19:12

gemini-code-assist Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread sdks/python/apache_beam/yaml/integration_tests.py

Comment thread sdks/python/apache_beam/yaml/integration_tests.py

Comment thread sdks/python/apache_beam/yaml/integration_tests.py

Comment thread sdks/python/apache_beam/yaml/yaml_io.py

gemini-code-assist Bot reviewed Jun 23, 2026

View reviewed changes

derrickaw marked this pull request as ready for review June 23, 2026 20:06

gemini-code-assist Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread sdks/python/apache_beam/yaml/integration_tests.py

Comment thread sdks/python/apache_beam/yaml/integration_tests.py

Comment thread ...o/delta/src/test/java/org/apache/beam/sdk/io/delta/DeltaReadSchemaTransformProviderTest.java

derrickaw requested review from Abacn and ahmedabu98 June 23, 2026 20:14

derrickaw changed the title ~~[wip] delta lake yaml~~ delta lake yaml Jun 23, 2026

derrickaw changed the title ~~delta lake yaml~~ delta lake python/yaml Jun 23, 2026

Abacn reviewed Jun 24, 2026

View reviewed changes

Conversation

derrickaw commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist Bot commented Jun 23, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

derrickaw commented Jun 23, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 23, 2026

Codecov Report

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Abacn Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

Abacn Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

derrickaw commented Jun 21, 2026 •

edited

Loading