[feat](thirdparty) add arrow-adbc to the thirdparty build - #66358
Merged
Conversation
Builds three artifacts from one source tree:
- libadbc_driver_manager.a : to be statically linked into doris_be
- libadbc_driver_jni.so : for the FE side; self-built because the upstream
prebuilt binary in the Maven jar needs GLIBC_2.34
and GLIBCXX_3.4.31, which excludes CentOS 7/8,
Rocky 8 and Ubuntu 20.04
- libadbc_driver_sqlite.so : tests only, not shipped
Two things upstream does that do not carry over:
The SQLite driver needs a system SQLite3 development package, which Doris does
not ship. The source tree vendors the amalgamation but never references it from
CMake, so build it here into a scratch static library, hand the paths to
FindSQLite3, and drop it afterwards. It ends up inside the driver, leaving no
sqlite artifacts in thirdparty.
The JNI bridge header is generated by shelling out to Maven. Doing that would
make this the first thirdparty package to require Maven, a Maven Central
connection and a JDK 11+, and the build-env image runs this script with
JAVA_HOME on JDK 8. Check the javah output in as a patch instead and compile
jni_wrapper.cc directly, which needs nothing but jni.h.
Nothing links against these artifacts yet; the consumer is apache#66331.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RgJjuW5w4jEKF9HorENTur
The only ADBC driver that implements executePartitioned -- splitting one query into partitions several readers can take in parallel -- is the Flight SQL one, and nothing on a build host has it: upstream writes that driver in Go and publishes no bare shared library. Take it from the official release wheel, which is a zip the existing download step already knows how to unpack, and install it next to the SQLite driver. Same release as the arrow-adbc source tree built here, which is not cosmetic: partition descriptors are driver-private bytes, so every process that handles one has to load the very same file. Like the SQLite driver it is a test artifact and is not shipped, and like hyperscan it is skipped on platforms upstream publishes no prebuilt binary for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RgJjuW5w4jEKF9HorENTur
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Gabriel39
approved these changes
Aug 2, 2026
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related Issue: #65615
Related PR: #66331
Problem Summary:
Split out of #66331, which adds an
adbccatalog type that reads an external sourcethrough an Arrow Database Connectivity driver. This PR
carries only that PR's
thirdparty/half, so the dependency can be reviewed and thebuild-env image rebuilt before the code that links against it lands.
Nothing in the tree consumes these artifacts yet -- this PR adds one package to the
thirdparty build and declares its license, and changes nothing else.
What comes out of it
libadbc_driver_manager.adoris_be(#66331)libadbc_driver_jni.solibadbc_driver_sqlite.solibadbc_driver_flightsql.soDoris ships no ADBC driver to users; a deployment supplies its own. The two drivers above
exist so the ADBC code paths can be tested at all.
Three things upstream does that do not carry over
The SQLite driver needs a system SQLite3 development package, which Doris does not
ship and most build hosts lack. The source tree vendors the amalgamation but never
references it from CMake, so it is compiled here into a scratch static library, handed
to
FindSQLite3, and dropped afterwards. It ends up statically inside the driver,leaving no sqlite artifacts in thirdparty.
The JNI bridge header is generated by shelling out to Maven
(
java/driver/jni/CMakeLists.txtrunsmvn -Pjni,javah compile). Doing that would makethis the first thirdparty package to require Maven, a Maven Central connection and a
JDK 11+, while the build-env image runs this script with
JAVA_HOMEon JDK 8. Thejavahoutput is checked in as a patch instead andjni_wrapper.ccis compiled againstit directly, needing nothing but
jni.h. The patch header records how to regenerate iton a version bump.
The prebuilt JNI binary inside upstream's Maven jar is not used either: it requires
GLIBC_2.34andGLIBCXX_3.4.31, which excludes CentOS 7/8, Rocky 8 and Ubuntu 20.04.The Flight SQL driver is written in Go and no bare shared library is published, so it
is taken from the official release wheel -- a zip the existing download step already
knows how to unpack -- rather than adding a Go toolchain to the thirdparty build. It is
skipped on platforms upstream publishes no prebuilt binary for, the same way hyperscan
is.
On the version pin
The source tree is tag
apache-arrow-adbc-24, which is release C/Go 1.12.0 (the tagcarries neither number). The prebuilt Flight SQL driver is pinned to that same release,
and that is not cosmetic: ADBC partition descriptors are driver-private bytes, so every
process that hands one to another must have loaded the very same driver build.
dist/LICENSE-dist.txtgets the corresponding Apache-2.0 entry. It is the only fileoutside
thirdparty/here.Release note
None
Check List (For Author)
Test
Build Third Party Librariesworkflow; this exact content passed itsLinux, macOS and macOS-arm64 jobs in [feat](catalog) support ADBC catalog that reads external sources over Arrow #66331
(run).
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)