Skip to content

[feat](thirdparty) add arrow-adbc to the thirdparty build - #66358

Merged
morningman merged 2 commits into
apache:masterfrom
morningman:adbc-thirdparty
Aug 2, 2026
Merged

[feat](thirdparty) add arrow-adbc to the thirdparty build#66358
morningman merged 2 commits into
apache:masterfrom
morningman:adbc-thirdparty

Conversation

@morningman

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related Issue: #65615

Related PR: #66331

Problem Summary:

Split out of #66331, which adds an adbc catalog type that reads an external source
through an Arrow Database Connectivity driver. This PR
carries only that PR's thirdparty/ half, so the dependency can be reviewed and the
build-env image rebuilt before the code that links against it lands.

Nothing in the tree consumes these artifacts yet -- this PR adds one package to the
thirdparty build and declares its license, and changes nothing else.

What comes out of it

Artifact How Used by
libadbc_driver_manager.a built from source to be statically linked into doris_be (#66331)
libadbc_driver_jni.so built from source to be loaded by the FE ADBC connector (#66331)
libadbc_driver_sqlite.so built from source tests only, not shipped
libadbc_driver_flightsql.so prebuilt, from the official release wheel tests only, not shipped

Doris ships no ADBC driver to users; a deployment supplies its own. The two drivers above
exist so the ADBC code paths can be tested at all.

Three things upstream does that do not carry over

  • The SQLite driver needs a system SQLite3 development package, which Doris does not
    ship and most build hosts lack. The source tree vendors the amalgamation but never
    references it from CMake, so it is compiled here into a scratch static library, handed
    to FindSQLite3, and dropped afterwards. It ends up statically inside the driver,
    leaving no sqlite artifacts in thirdparty.

  • The JNI bridge header is generated by shelling out to Maven
    (java/driver/jni/CMakeLists.txt runs mvn -Pjni,javah compile). Doing that would make
    this the first thirdparty package to require Maven, a Maven Central connection and a
    JDK 11+, while the build-env image runs this script with JAVA_HOME on JDK 8. The
    javah output is checked in as a patch instead and jni_wrapper.cc is compiled against
    it directly, needing nothing but jni.h. The patch header records how to regenerate it
    on a version bump.

    The prebuilt JNI binary inside upstream's Maven jar is not used either: it requires
    GLIBC_2.34 and GLIBCXX_3.4.31, which excludes CentOS 7/8, Rocky 8 and Ubuntu 20.04.

  • The Flight SQL driver is written in Go and no bare shared library is published, so it
    is taken from the official release wheel -- a zip the existing download step already
    knows how to unpack -- rather than adding a Go toolchain to the thirdparty build. It is
    skipped on platforms upstream publishes no prebuilt binary for, the same way hyperscan
    is.

On the version pin

The source tree is tag apache-arrow-adbc-24, which is release C/Go 1.12.0 (the tag
carries neither number). The prebuilt Flight SQL driver is pinned to that same release,
and that is not cosmetic: ADBC partition descriptors are driver-private bytes, so every
process that hands one to another must have loaded the very same driver build.

dist/LICENSE-dist.txt gets the corresponding Apache-2.0 entry. It is the only file
outside thirdparty/ here.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

morningman and others added 2 commits August 2, 2026 14:03
Builds three artifacts from one source tree:
- libadbc_driver_manager.a  : to be statically linked into doris_be
- libadbc_driver_jni.so     : for the FE side; self-built because the upstream
                              prebuilt binary in the Maven jar needs GLIBC_2.34
                              and GLIBCXX_3.4.31, which excludes CentOS 7/8,
                              Rocky 8 and Ubuntu 20.04
- libadbc_driver_sqlite.so  : tests only, not shipped

Two things upstream does that do not carry over:

The SQLite driver needs a system SQLite3 development package, which Doris does
not ship. The source tree vendors the amalgamation but never references it from
CMake, so build it here into a scratch static library, hand the paths to
FindSQLite3, and drop it afterwards. It ends up inside the driver, leaving no
sqlite artifacts in thirdparty.

The JNI bridge header is generated by shelling out to Maven. Doing that would
make this the first thirdparty package to require Maven, a Maven Central
connection and a JDK 11+, and the build-env image runs this script with
JAVA_HOME on JDK 8. Check the javah output in as a patch instead and compile
jni_wrapper.cc directly, which needs nothing but jni.h.

Nothing links against these artifacts yet; the consumer is apache#66331.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RgJjuW5w4jEKF9HorENTur
The only ADBC driver that implements executePartitioned -- splitting one query
into partitions several readers can take in parallel -- is the Flight SQL one,
and nothing on a build host has it: upstream writes that driver in Go and
publishes no bare shared library.

Take it from the official release wheel, which is a zip the existing download
step already knows how to unpack, and install it next to the SQLite driver.
Same release as the arrow-adbc source tree built here, which is not cosmetic:
partition descriptors are driver-private bytes, so every process that handles
one has to load the very same file. Like the SQLite driver it is a test
artifact and is not shipped, and like hyperscan it is skipped on platforms
upstream publishes no prebuilt binary for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RgJjuW5w4jEKF9HorENTur
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 2, 2026
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.16% (25628/43323)
Line Coverage 43.24% (257201/594855)
Region Coverage 38.93% (203927/523790)
Branch Coverage 40.27% (93037/231037)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.74% (32003/42254)
Line Coverage 60.39% (356813/590879)
Region Coverage 56.95% (299324/525557)
Branch Coverage 58.40% (134939/231075)

@morningman
morningman merged commit 0c01156 into apache:master Aug 2, 2026
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants