[feat](catalog) support ADBC catalog that reads external sources over Arrow - #66331
Open
morningman wants to merge 42 commits into
Open
[feat](catalog) support ADBC catalog that reads external sources over Arrow#66331morningman wants to merge 42 commits into
morningman wants to merge 42 commits into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
morningman
changed the title
morningman
force-pushed
the
wt-adbc-catalog
branch
from
August 2, 2026 05:38
77ee3bd to
a6e8d66
Compare
Contributor
Author
|
run buildall |
16 tasks
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
morningman
added a commit
that referenced
this pull request
Aug 2, 2026
### What problem does this PR solve? Related Issue: #65615 Related PR: #66331 Problem Summary: Split out of #66331, which adds an `adbc` catalog type that reads an external source through an [Arrow Database Connectivity](https://arrow.apache.org/adbc/) driver. This PR carries only that PR's `thirdparty/` half, so the dependency can be reviewed and the build-env image rebuilt before the code that links against it lands. Nothing in the tree consumes these artifacts yet -- this PR adds one package to the thirdparty build and declares its license, and changes nothing else. **What comes out of it** | Artifact | How | Used by | |---|---|---| | `libadbc_driver_manager.a` | built from source | to be statically linked into `doris_be` (#66331) | | `libadbc_driver_jni.so` | built from source | to be loaded by the FE ADBC connector (#66331) | | `libadbc_driver_sqlite.so` | built from source | tests only, **not shipped** | | `libadbc_driver_flightsql.so` | prebuilt, from the official release wheel | tests only, **not shipped** | Doris ships no ADBC driver to users; a deployment supplies its own. The two drivers above exist so the ADBC code paths can be tested at all. **Three things upstream does that do not carry over** - *The SQLite driver needs a system SQLite3 development package*, which Doris does not ship and most build hosts lack. The source tree vendors the amalgamation but never references it from CMake, so it is compiled here into a scratch static library, handed to `FindSQLite3`, and dropped afterwards. It ends up statically inside the driver, leaving no sqlite artifacts in thirdparty. - *The JNI bridge header is generated by shelling out to Maven* (`java/driver/jni/CMakeLists.txt` runs `mvn -Pjni,javah compile`). Doing that would make this the first thirdparty package to require Maven, a Maven Central connection and a JDK 11+, while the build-env image runs this script with `JAVA_HOME` on JDK 8. The `javah` output is checked in as a patch instead and `jni_wrapper.cc` is compiled against it directly, needing nothing but `jni.h`. The patch header records how to regenerate it on a version bump. The prebuilt JNI binary inside upstream's Maven jar is not used either: it requires `GLIBC_2.34` and `GLIBCXX_3.4.31`, which excludes CentOS 7/8, Rocky 8 and Ubuntu 20.04. - *The Flight SQL driver is written in Go and no bare shared library is published*, so it is taken from the official release wheel -- a zip the existing download step already knows how to unpack -- rather than adding a Go toolchain to the thirdparty build. It is skipped on platforms upstream publishes no prebuilt binary for, the same way hyperscan is. **On the version pin** The source tree is tag `apache-arrow-adbc-24`, which is release C/Go 1.12.0 (the tag carries neither number). The prebuilt Flight SQL driver is pinned to that same release, and that is not cosmetic: ADBC partition descriptors are driver-private bytes, so every process that hands one to another must have loaded the very same driver build. `dist/LICENSE-dist.txt` gets the corresponding Apache-2.0 entry. It is the only file outside `thirdparty/` here.
Nothing references the ADBC symbols yet, so the archive contributes no objects to the binary until AdbcDriverRegistry lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Doris serdes only accept the Arrow variants Doris itself emits: the string serde takes STRING, BINARY and FIXED_SIZE_BINARY and nothing else. Third-party ADBC drivers emit others -- DuckDB emits string_view, Go-based drivers may emit large_* and dictionary. Convert those before materialization, and fail with the offending type named when no Doris column can hold it, because silently wrong data is far worse than a loud error. Normalization loops rather than converting once: decoding dictionary<int32, large_utf8> leaves large_utf8 behind, which still needs converting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The driver manager dlopens on every AdbcDatabaseInit and keeps no cache of its own, so without this every scan range would reload the driver. Load each resolved path at most once and never dlclose: drivers carry global state and background threads -- Go runtimes especially -- so unloading one is a use-after-free hazard. Failed loads are cached too, so a bad path does not retry the dlopen once per scan range, and the failure message carries the path the user configured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows the string-map shape jdbc_params and es_params already established. The partition descriptor is opaque binary, so it travels base64-encoded rather than as a typed field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mirrors remote_doris_reader: the Arrow-stream-to-Block materialization is identical, only the stream source changes from Arrow Flight to ADBC. Each column is normalized before materialization, since third-party drivers emit Arrow variants the serdes reject. The reader drives the driver's own function table rather than the driver manager's free functions. Those re-dlopen on every AdbcDatabaseInit, which would defeat AdbcDriverRegistry. Databases are not pooled across scan ranges. That is a throughput optimization which only pays off once multiple partitions run concurrently, and adding an untestable caching layer now would only obscure the functional path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ADBC only has a v2 reader, and enable_file_scanner_v2 is a fuzzy=true session variable the regression harness flips at random, so honoring it would make ADBC queries fail on a coin flip. Force adbc onto v2, mirroring the existing transactional_hive force-to-v1. Loads are excluded: there is no ADBC load path, so widening the rule to cover them would only route them somewhere they still cannot run. is_supported also has to accept adbc under FORMAT_ARROW, otherwise the scanner would refuse the very ranges the operator forces onto it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First cut of the FE side: a new connector module registered in the reactor, its plugin-zip assembly, the ServiceLoader entry and AdbcConnectorProvider claiming catalog type "adbc". Metadata is not wired yet -- getMetadata fails loud until the following commits land it. The provider carries one non-obvious obligation, pinned by AdbcConnectorProviderIsolationTest: neither its static initializer nor its no-arg constructor may load org.apache.arrow.adbc.*. DirectoryPluginRuntimeManager instantiates every discovered factory before it rejects a duplicate type name, so a connector present both as a classpath built-in and as a directory plugin gets constructed once in a second classloader and then discarded. Reaching an ADBC class there would run a second System.load of the JNI shim from that second loader and throw UnsatisfiedLinkError, losing the copy that had loaded fine. Verified by mutation: a static reference to AdbcDriver.PARAM_URI makes the test fail and names the class. Dependency versions go in fe/pom.xml. Three of the four are new to FE: adbc-driver-manager comes with adbc-driver-jni (JniDriverFactory implements its SPI) and arrow-c-data had no FE consumer before. adbc 0.24.0 pins Arrow Java 19.0.0, matching arrow.version, so no second Arrow enters the build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four pieces the connector cannot work without, none of which the module's own
pom can supply.
build.sh: the connector deployment loop is a hardcoded module list, so adbc has
to be added there by hand. Missing it is a nasty failure -- the module still
compiles and the zip is still produced, it just never reaches
output/fe/plugins/connector/, and the only symptom is CREATE CATALOG reporting
an unknown catalog type. It also creates plugins/adbc_drivers on both FE and BE
(the drop point for the driver a user supplies) and copies the thirdparty-built
JNI shim into fe/lib. The shim inside the adbc-driver-jni jar is deliberately
not used: it needs GLIBC 2.34 / GLIBCXX 3.4.31, which the supported build hosts
do not have, while the thirdparty build needs only GLIBC 2.7.
conf/fe.conf: arrow.adbc.driver.jni.library.path is set to ${DORIS_HOME}/lib.
Note this is a DIRECTORY, not a file -- adbc-driver-jni's resolver appends
System.mapLibraryName("adbc_driver_jni") to it. Verified that ${DORIS_HOME} is
already exported when start_fe.sh reads fe.conf, so the value expands (LOG_DIR
in the same file is the existing precedent).
Config.java + DefaultConnectorContext: adbc_drivers_dir and
adbc_driver_secure_path, threaded to the plugin through the connector
environment because a plugin cannot read FE Config. The JDBC equivalents are not
reusable here -- ConnectorValidationContext#validateAndResolveDriverPath
resolves against jdbc_drivers_dir and enforces a .jar grammar.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
driver_url keeps the name the JDBC catalog uses, but accepts local references only: a bare file name resolved under adbc_drivers_dir, a file:// URL, or an absolute path. Remote schemes are refused with a message that says why, because "unsupported" alone would read as an arbitrary limitation. The reason is that FE and every BE must load the identical driver library -- ADBC partition descriptors are driver-private opaque bytes, and two official implementations of the same protocol already serialize incompatible protobuf messages that parse into each other silently instead of erroring. A URL each node fetches for itself cannot promise sameness, and the resulting failure surfaces as an unreadable partition, far from its cause. The rest of the resolver is the JDBC security rule adapted to shared libraries: no traversal (checked on the decoded path, so %2e%2e cannot survive), a bare name may contain no separator, file:// may carry no authority/query/fragment, and adbc_driver_secure_path is matched by path component so that /opt/drv-evil is not admitted by an allowance for /opt/drv. The not-found message is deliberately long. Doris ships no ADBC drivers, so a missing file is the most likely first experience of this catalog type, and a bare "dlopen failed" would leave the user with nothing to act on. It names the exact path, says every BE needs the same file, and gives both places to obtain one. adbc.*-prefixed properties pass through with their names intact: ADBC option names already begin with "adbc.", so stripping the prefix would hand the driver a name it does not know and the option would be ignored in silence. BE applies the same rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AdbcClient owns one catalog's Arrow allocator and AdbcDatabase and lends out short-lived connections. It goes through adbc-driver-jni rather than a pure-Java driver so FE and BE run the same C driver manager over the same driver library; the pure-Java Flight SQL driver would be lighter here but does not implement getTableSchema at all, which would leave FE deriving column types from XDBC type codes while BE reads real Arrow types. Nothing opens in the constructor. An FE follower replaying the edit log builds every catalog and its filesystem layout need not match the leader's, so touching the driver file during construction would let one missing library stop FE from starting rather than fail the single catalog that cannot work. Errors carry the ADBC status, SQLSTATE and vendor code, and the driver's own message is appended only when it says something: the SQLite driver answers NOT_IMPLEMENTED with the literal text "(unknown error)", so forwarding it as the error would name neither the operation nor the cause. Tests run against the real JNI bridge and the real SQLite driver built by thirdparty -- that native path is the entire reason for choosing the JNI driver, so stubbing it would leave the risky part unverified. They skip loudly when the libraries are absent, stating that the native path was NOT exercised, so a skipped run cannot be misread as a pass. Two facts were confirmed empirically while writing them: driver_entrypoint does reach the driver manager (a bogus value fails with "dlsym(...) undefined symbol"), and the JNI library property names a directory rather than a file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three pieces sit between what a driver reports and what Doris shows. AdbcNamespace projects ADBC's three naming levels onto Doris's two. The uri property is required to pin the remote catalog, which keeps this a projection rather than a join: at most one level varies, so the Doris database name never has to be built by concatenation. That matters because a catalog name may legally contain a dot, so a concatenated name could not be split back unambiguously -- which is why the remote parts are carried, never re-derived. Empty string and null are the same absent level: SQLite reports its missing schema level as "", other drivers as null, and treating them differently would name the same source's database differently per driver build. AdbcObjectsReader walks the nested getObjects result. It stops at the table layer on purpose: the column layer carries XDBC integer type codes rather than Arrow types, so deriving Doris column types from it would reintroduce the two-step translation ADBC exists to remove, and would disagree with the real Arrow arrays BE reads. It also filters rows to the requested namespace, because getObjects filters are advisory and a driver may answer with everything it has. AdbcTypeMapper converts Arrow types. Unsigned integers widen one step (Doris has none, and same-width would wrap the upper half of the range with no error); nanosecond timestamps truncate to Doris's 6 digits rather than making a table unreadable; struct field names are lowercased at every level because BE indexes struct children by lowercase key and a mixed-case child crashes it. Anything with no Doris equivalent is rejected here, naming the column and the Arrow type -- a wide table gives a user nothing to act on otherwise. The getObjects tests run against the real SQLite driver, so the nested parsing is checked against output a driver actually produces. That required opening java.base/java.nio to Arrow in surefire: without it any test that materializes Arrow data fails on DirectByteBuffer access. Same flag conf/fe.conf and fe-core's surefire config already carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SHOW DATABASES / SHOW TABLES / DESC now work against an ADBC source. AdbcTableHandle keeps the remote catalog, schema and table separately. The Doris name it also carries is for display and lookup only and is never parsed back: either remote level may itself contain a dot, so a joined name has several readings and the wrong one addresses a table that does not exist. The schema fallback exists because neither way of asking is guaranteed. The ADBC API's getTableSchema has a default that throws NOT_IMPLEMENTED and drivers do leave it there (the Java Flight SQL driver); executeSchema is no safer in the other direction, since the SQLite driver rejects it while implementing getTableSchema fine. So each backs the other, and which one works is remembered per catalog rather than re-probed per table. Only NOT_IMPLEMENTED triggers that fallback. Writing the test for it is what exposed why: any other status means the driver did try and the table is at fault -- a missing table answers NOT_FOUND -- so falling back on every error turned a plain "no such table" into "this driver implements neither method", pointing the user at their driver instead of their table name. There is a test for exactly that shape now. getObjects' column layer is deliberately not a third fallback: it carries XDBC integer type codes rather than Arrow types, so it would answer in a different type system than the one BE reads the data in. buildTableDescriptor emits the Hive descriptor, as the other connectors reading through the generic file-scan path do. The SPI default (null) is not neutral here: fe-core would fall back to SCHEMA_TABLE and BE would build a SchemaTableDescriptor instead of the one the scan path expects. To be re-checked against BE when the scan path lands. Tests run the whole surface against the real SQLite driver through the JNI bridge. One fixture detail is load-bearing and documented in place: SQLite is dynamically typed and its ADBC driver derives the Arrow schema from the values present, not the declared column types -- on an empty table it reports every column as int64, so the fixture inserts a row. Found by the assertion failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plugin-driven scan node translates a connector's file-format name into the thrift enum BE selects a reader with, and had no case for "arrow": a connector that hands BE Arrow record batches rather than a file fell to the FORMAT_JNI default. That default is a working reader, not an error, so the mismatch produces no failure at this layer at all -- the scan simply lands in the JNI scanner, which has no branch for an Arrow table format, and fails there against a message about the wrong reader. The mapping stays connector-agnostic: it is a format name, and the switch carries no knowledge of which connector produced it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An ADBC source is any driver implementing the C ABI, so the set of SQL spellings this connector must produce cannot be closed the way the JDBC connector closes it over an enum of drivers. The dialect is therefore an interface with a registry: adding a source costs a class and a registration, and the query builder must never grow a branch for one. A test drives the builder with a dialect defined inside the test to keep that true. Only the ANSI dialect ships. The others the design names wait for a source to verify them against; an unverified dialect is a set of guesses about someone else's SQL, and a wrong guess is either a syntax error at scan time or a predicate that quietly selects different rows. The builder is conservative for the same reason. Comparisons, null tests, IN and the boolean connectives are pushed; functions, LIKE, BETWEEN and null-safe equality are not, because each would assert that an unidentified source evaluates it as Doris does. A predicate is translated whole or left behind whole -- half of a predicate is a different predicate, not a weaker one -- and BE re-applies everything anyway, which is what makes pushing purely a speed-up. Two choices decide rows rather than speed: - The projection is always explicit. BE matches returned columns to query slots by name and rejects one it did not ask for, so SELECT * fails the scan outright rather than merely over-reading. An empty projection is COUNT(*) pushed down and selects a constant, so a row count does not drag the table width across the wire. - A row limit is emitted only when every predicate went with it. BE filters whatever comes back, so a source that truncated to n rows before applying the predicates Doris kept would answer with fewer than n. The schema fallback in the metadata path now names its table through the dialect too. Two spellings of one table name is how a source ends up working for queries and failing for DESC. Verified against the real SQLite driver: the generated statements are run through it, so the quoting, the literal spellings and the placement of LIMIT are checked for acceptance by a source rather than only for text. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One range, one statement. The design splits a query into the driver's own partition descriptors and hands one to each backend, but BE has no reader for those yet, so planning several would hand out work nothing can execute. The shape here is the one the JDBC connector has always had, and the partitioned path replaces the body of planScan without disturbing anything around it. The parameter names on the range are a contract with C++: BE looks them up literally, nothing checks the two sides agree at build time, and a rename on one side surfaces as a scan failing at run time about a missing parameter. Two are easy to get wrong and are asserted by name -- the credentials travel as ADBC's own username/password rather than this connector's user property, and a driver option keeps its "adbc." prefix, which is part of the option name rather than a namespace this connector added. populateRangeParams has to be overridden rather than inherited: the inherited one writes jdbc_params, which BE's ADBC reader never reads, so the scan would arrive with no driver path at all. The driver path is sent as FE resolved it. BE loads it verbatim -- it has no drivers directory of its own to resolve a bare name against -- so FE and every BE need the same file at the same path, which is what the deployment already asks for and how a JDBC catalog's driver reference already behaves. EXPLAIN never calls planScan, so it regenerates the statement; a test pins that the two agree, because a divergence would have EXPLAIN describe a query that is not the one run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A pushed-down COUNT(*) leaves the scan with no slots at all, so every column the source returns is unrequested by definition and the reader's unknown-column check rejected the first one. A query asking for nothing but a number therefore failed, and FE cannot avoid it: no SQL statement returns zero columns. Only the empty projection is special-cased. An unrequested column arriving alongside requested ones still fails, because that state means FE and this reader disagree about the projection and this check is the only signal the disagreement exists. Tolerating unknown columns generally would have fixed the count case by removing that signal. NOT COMPILED OR RUN: this machine cannot build BE. The two tests added here have never executed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers the scan path end to end against this cluster's own Arrow Flight SQL endpoint: catalog creation, SHOW/DESC, a full read, a projection, each predicate shape in the pushable set, one outside it, a limit beside an unpushable predicate, and COUNT(*). NEVER RUN. It was written on a machine that cannot build BE, so nothing in it has executed even once. Every expectation is a claim to check on the first real run, not a passing baseline, and the suite says so at the top. It also carries no .out: one written without running would be a guess presented as a verified result, and a guess that happened to be right would be indistinguishable from one that was checked. It skips loudly, and says the ADBC path went untested, unless an ADBC driver is configured -- Doris ships none, and FE and every BE must load the same file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Flight SQL suite beside this one needs a driver Doris does not ship and that is not present on any developer machine by default, so nothing has yet exercised the scan path outside FE unit tests. SQLite is not the source this connector targets, but it travels the identical path -- FE through the JNI bridge to the C driver manager to a driver .so, BE through the same driver manager to the same .so -- and thirdparty already builds it, so this runs without downloading anything. The properties that decide rows are asserted directly rather than through qt_: with no baseline, a -genOut run records whatever came back, so a wrong result would be blessed rather than caught. qt_ is left only for DESC, where the point is to notice future drift rather than to state today's answer. Three fixture details are load-bearing. Every column carries at least one non-null value, because the SQLite ADBC driver derives the Arrow schema from the values present rather than the declared types and an all-null column comes back as int64. One row holds a quote, without which the escaping assertion answers zero whether the literal was escaped correctly or never pushed at all. And the LIKE-plus-LIMIT query has exactly one match so that a limit wrongly pushed past a predicate Doris kept shows up as a missing row. Nothing asserts the limit through EXPLAIN, deliberately: EXPLAIN regenerates the statement with no limit at all, as the JDBC connector's does, so such an assertion would hold or fail for a reason unrelated to what it claims to test. NEVER RUN, like the suite beside it, and it carries no .out for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…laims Found by the first real end-to-end run: `SELECT id FROM t1 WHERE name IS NULL` failed with "Unsupported arrow type for string column: 9" (9 = arrow::Type::INT64) against a TEXT column. The projection is id, but BE re-evaluates the predicate -- which is the premise this connector's pushdown rests on -- so name is a query slot too and FE selects it. The one surviving row has name NULL, so the column comes back with no non-null value in it, and a source that infers Arrow types from the VALUES it returns has nothing left to infer from. Measured on the SQLite driver: the same column is utf8 for `SELECT id, name FROM t1` and int64 for the same query plus `WHERE name IS NULL`. FE cannot prevent this. It cannot know which rows a filter will leave, and the column type it published came from the source's own schema call. N nulls are what such an array means whatever type it claims, so they are materialized directly. The branch is narrow on purpose: a column with even one non-null value keeps its real type and a genuine mismatch still fails loudly, because that is the signal that FE and the source disagree about the schema rather than about one result set. It is also restricted to a nullable target -- substituting defaults into a NOT NULL column would turn a source that wrongly sent nulls into silently wrong data. The regression suite that caught this now says why that query is load-bearing, and carries the DESC baseline the run generated. BE NOT COMPILED: this machine cannot build BE, so neither the fix nor the two tests added with it have been compiled or executed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scan path's headline feature is reading one query as several partitions across BEs, and the only driver that implements executePartitioned is the Flight SQL one. It could not be exercised at all: upstream writes that driver in Go and publishes no bare shared library, so nothing on a build host has it. Take it from the official release wheel, which is a zip the existing download step already knows how to unpack, and install it next to the SQLite driver. Same release as the arrow-adbc source tree built here, which is not cosmetic: partition descriptors are driver-private bytes, so FE and BE must load the very same file. Like the SQLite driver it is a test artifact and is not shipped, and like hyperscan it is skipped on platforms with no prebuilt binary. The flight sql suite now finds it there instead of demanding a hand-placed path. That suite still has never been run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A scan against another Doris over Flight SQL killed the BE outright: ../src/arrow/c/helpers.h:164:: ArrowArrayStreamRelease did not cleanup release callback The Arrow C data interface requires a release callback to clear itself, and Arrow C++ does not merely complain when one does not -- it calls abort(), so a single scan takes the process down. The Flight SQL driver's stream does not clear it. Measured both ways with the same query: through the ADBC driver manager's wrapper the callback comes back cleared, through the driver's own entry points it does not. The BE calls those directly, because the driver registry owns driver lifetime, so it sees the raw stream. The FE never hit this: its JNI bridge goes through the driver manager. Wrap the driver's stream in one that honours the contract before Arrow ever sees it, for every driver rather than this one: this connector loads third-party drivers, and none of them should be able to abort the BE. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first Flight SQL scan against another Doris sent SELECT "id", "name", "score" FROM "db"."t1" and the source answered `no viable alternative at input 'FROM "db"'`. Doris reads a double-quoted name as a string literal, so ANSI quoting there does not just look foreign: nothing parses, and the connector cannot read the one kind of source phase one exists to replace. Only the ANSI dialect shipped, so the vendor name a Doris source reports -- DorisFE, its Flight SQL server name -- was claimed by nothing and fell back to ANSI silently. Add a dialect that keeps everything ANSI renders (literals, the two-part table name, LIMIT, all of which Doris accepts unchanged) and changes identifier quoting, and let it claim that vendor. Scoped to Doris alone. Backticks are the whole MySQL family's spelling and MySQL or StarRocks would very likely work, but neither has been run against, and a dialect claimed on family resemblance is a guess about someone else's SQL. They can still name this dialect explicitly. The end-to-end suite runs for the first time with this. Its baseline comes from a fully passing run, was checked line by line against the fixture, and then re-run without -genOut so the values are compared rather than recorded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A scan planned one range holding one statement, so one backend did all the reading no matter how much the source could have handed out in parallel. Ask the driver to split the statement instead: each partition it reports becomes its own range, and Doris's existing assignment spreads them over the cluster. A driver that answers NOT_IMPLEMENTED gets the statement path back, and the answer is remembered per catalog rather than re-probed per query -- whether the method exists is a property of the driver. Two failures deliberately do not fall back. More partitions than max_partitions allows, and a driver reporting none at all, both fail the query: by the time the descriptors exist the source has already executed the statement, so planning it again as a statement would execute it a second time while the first result set sits unread. Planning zero ranges would be worse still -- a silently empty answer to a query that has one. enable_partitioned_read turns the whole thing off for a source that pays badly for the planning-time round trip, which on a Flight SQL source IS the query's execution. That same fact is why EXPLAIN must never take this path, pinned by a test that plans with a client which refuses to open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
EXPLAIN plans a scan for real -- that is where its inputSplitNum comes from. Its explain level is NORMAL, which is not a plan level, so NereidsPlanner does not return before splitFragments, and the fragment translator finalizes every scan node, which calls planScan. test_paimon_predict already relies on this: it asserts inputSplitNum=9 inside an explain block. That was harmless while planning only listed files or built a string. It is not harmless for ADBC, where asking the driver to partition a query executes that query on the source: EXPLAIN would run the very query it was asked to describe, and leave the result set sitting unread on the source until it timed out. A comment in the ADBC connector claimed EXPLAIN never reaches planScan; it does. Carry the fact on the scan request instead, read from the parsed statement the explain command marks before planning. Connectors that only list files never read it and plan identically. ADBC reads it and plans the statement, which is what a partitioned scan splits anyway -- so EXPLAIN still describes the real scan, only with a range count that a query nobody ran cannot have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Flight SQL driver partitions, so every query in this suite now takes the partitioned path. Add a second catalog with enable_partitioned_read=false and run the same queries through both, comparing values rather than a baseline -- a baseline passes when both paths break the same way, and the question here is whether they agree. This does not prove cross-backend parallelism. The source is a single-backend cluster, which reports one partition per query, so the partition path is exercised at N=1 and the comment says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A scan range now carries either the statement to run or one partition of a statement the source has already run, and the reader takes the matching branch: a partition is base64-decoded and read straight off the connection with ConnectionReadPartition, creating no statement at all. The two parameters are mutually exclusive and a range carrying both or neither is refused up front. That is not tidiness: reading a partition means the source has already executed the query, so a range that said both would let this reader execute it a second time depending on which branch the code happened to check first. Both branches still pass their stream through enforce_stream_release_contract before Arrow sees it -- ReadPartition's stream breaks the release contract exactly like ExecuteQuery's does, and Arrow aborts the process over it. Compiles clean, but NOT yet executed: the unit test build was stopped partway by machine contention, so the three tests added here have never run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The registry hands out AdbcDriver pointers documented to stay valid for the life of the process, and it never releases a driver: dlclosing one that owns background threads, as a Go-based driver does, is a use-after-free. But the registry itself was a function-local static, so static destruction tore the map down anyway. Every pointer already handed out dangled from that moment, and the driver manager's per-driver state -- freed only by the release callback nobody calls -- lost its last reference. LeakSanitizer sees exactly that and reported it, since its check runs after every static destructor. Allocate it and never free it, which is what the class already claims to do. This is what made the first-ever BE unit test run of this connector fail at exit; it has been there since the registry was added, unseen because the tests had never run. They do now: 30 pass, no leaks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build.sh carries the connector list twice: once to pick the maven modules to build, once to deploy each module's archive as a plugin directory. The adbc connector was only ever added to the second one. That is not a no-op. The deploy step takes whatever archive it finds in the module's target/, so every build shipped the plugin left over from whenever someone last packaged that module by hand -- and skipped it silently when there was none. An end-to-end run against a freshly built FE therefore exercised a connector hours older than the source tree, and passed, because the property that would have changed its behaviour was simply unknown to the old code and ignored. Add adbc to the build list and say in a comment why the two must match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The downgrade line reported only that partitioned execution was unavailable, never the exception behind it. Which layer refused -- driver, driver manager, or the JNI bridge -- then has to be established by hand, and the same log line appears whether the refusal is expected (SQLite has no partitions) or a misconfiguration worth fixing. Carry the exception out of the lambda and put its status and message in the line, matching what the metadata path already logs when it degrades. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A driver without partitioned execution is downgraded to the single-statement path, which returns exactly the same rows from one backend instead of many. That invisibility is a problem for anything that depends on the parallelism: a test written for the partitioned path passes while exercising the fallback, and the pass is indistinguishable from the real thing. This round shipped such a test and it took a log-timestamp comparison to tell which path it had actually taken. Replace enable_partitioned_read with partitioned_read = auto | disabled | required. Three states rather than two booleans, because enable=false + require=true contradicts itself and would need a rule to reject. Nothing is released yet, so renaming costs nothing. required fails the query naming the driver's own answer, on the first scan and on every later one -- the refusal is remembered per catalog, so it must not quietly succeed once the memo is set. EXPLAIN is exempt: it never asks for partitions by design, and refusing to describe a query helps nobody. The Flight SQL suite now pins its catalog to required, so a driver that stops partitioning fails the run; the SQLite suite pins the other half, since that driver genuinely refuses and is the only one here that can prove the mode does what it says. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The connector asks getObjects for base tables only, and a Doris source answers with everything it has: its Flight SQL endpoint recognises the literal "VIEW" as a type filter and treats every other value, including the "table" ADBC sends, as no filter at all. So an ADBC catalog over Doris listed views alongside tables, and DESC and SELECT on one of them worked -- which is why nothing looked wrong. The table_type each object comes back with is accurate even there, so the filter is applied where the answer is read. A source that honours the request has nothing left to drop. A type that is neither TABLE nor BASE TABLE is dropped rather than kept. The forgiving rule is the wrong one here: a leaked view scans fine, so it never announces itself, while a source that spells its tables some third way lists nothing at all and is noticed immediately. A missing type is not an unrecognised one and is still kept. The unit test drives a real SQLite driver with no type filter at all, which reproduces through a real driver what Doris does to the filter the connector does send. The end-to-end suite grows a view for the same reason: Doris is the source that gets this wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
test_adbc_catalog_scan covers the rows a partitioned scan returns. What it cannot say is how many partitions there were: it reads a three-row, one-bucket table, and a run where the source stopped splitting scans entirely would look exactly the same. This suite reads the count back out of the connector instead of assuming one. A catalog capped at one partition either answers the query, which it can only do if there was one, or refuses and names the number it found. Nothing here hardcodes a number or a theory of how a source plans: on one backend the count is pinned at 1, and on several it must be at least one per backend, which is the cross-backend case the feature exists for. The fixture gives the table eight buckets per backend so that premise holds. Reading every partition is then checked against the source table rather than against a second read of the same catalog, because the right answer is known here and a fault shared by both read paths cannot hide in an agreement. count(*) alone would miss a partition read twice that also lost one, and distinct and sum would miss a duplicate of a whole partition; together they do not. A single-backend run says out loud that it did not cover any of that, because every other assertion passes either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
The engine builds a ConnectorMetadata per statement, so every query on an ADBC catalog opened with three remote round trips before planning had begun: list the databases, list one database's tables, read one table's schema. Two of those list every object in the source, so the cost grew with the source rather than with the query. The answers now live on the connector, whose lifetime is the catalog's, in the shared meta-cache framework the other connectors use. Defaults are on, 1000 entries, and a ten-minute TTL where the framework's own default is a day: an ADBC source is another live database whose tables other people's DDL alters at any time, and nothing tells Doris when, so this bounds how long someone who forgot to REFRESH stays wrong. It is deliberate, and the test that pins it says so. REFRESH was a no-op for this connector until now, which was harmless while nothing was remembered and would have been the worst possible default once something was. All three hooks are implemented, and REFRESH TABLE drops its database's table listing along with the table's schema -- otherwise a table created remotely could never be brought in by the statement a user would reach for, and their REFRESH would appear to do nothing. A cached listing is not allowed to answer "there is no such table". Resolving a name that the listing lacks re-reads the listing first, so a table created a second ago is found rather than denied; SHOW TABLES still serves what was remembered, because a report may lag the source while a lookup that errors may not. The remote call falls only on the path that was about to raise an error anyway. The native tests change the source behind Doris's back and then ask what Doris sees, which is the only evidence that separates a remembered answer from a fresh one -- and unlike a call counter, cannot be satisfied by a cache that stores what it never reads. Both directions were checked by mutation: dropping the schema entry and dropping the re-read each fail exactly one test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
The engine does not ask the connector whether a table exists. It loads its own name list from listTableNames and decides from that -- and when a name is not in that list it clears the list, asks again, and only then reports the table missing (ExternalDatabase.buildTableForInit). The same shape one level up for databases. Serving those two calls from the cache added yesterday made that last chance a formality: the second listing came back from memory, identical to the first, so a table created remotely a moment ago was unreachable until an entry expired -- and unreachable is where it would have stayed for a user who had no reason to suspect a cache. That is precisely the staleness this cache was supposed not to introduce. They are now read from the source every time, and refresh what is remembered as they go. Nothing is lost by it: the engine caches both listings itself, so they were never the per-query cost. What a query actually repeats is resolving a database and a table name and reading a schema, and those still come from memory. Verified by mutation: pointing either listing back at the cache fails its test, one showing a table created behind Doris's back, the other a planted database the source does not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
Three things the unit tests cannot reach, because each needs the engine sitting in front of the connector. A table created behind Doris's back, by the sqlite3 CLI the fixture is already built with, must be queryable with no REFRESH at all. That works only if the connector answers the engine's re-list from the source; it is the end-to-end half of the rule, and it fails against a connector that serves that listing from memory. REFRESH CATALOG must reach the connector, asserted through a schema change rather than a new table: the engine clears its own schema copy either way, so a stale answer after the refresh can only have come from the connector. Without the hook the catalog would serve the schema it first read until the TTL expired, and REFRESH CATALOG does not rebuild the connector. The cache knobs, both directions: a catalog with the cache turned off still works, and an unparseable ttl fails at CREATE CATALOG. The second doubles as proof that the deployed plugin is the new one -- an older build ignores an unknown property, so the CREATE would succeed and the assertion would fail rather than pass quietly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
driver_checksum has been a property name with nothing behind it since the connector was written: a user could set it and Doris would ignore it. It is now checked at CREATE CATALOG, by MD5, against the file this FE resolves. It earns its place from what this catalog type asks of an operator. Doris ships no ADBC driver, so the library is placed by hand on every node and the copies have to stay identical -- and a wrong or stale copy announces nothing. It loads, it answers, and whatever it does differently arrives later as a query failure that never mentions a file. The property remains optional and stays honest about its reach: it sees this FE's copy and no BE's, so it does not verify that the nodes agree. What it gives an operator is a way to state which build the catalog was written for and have one node check itself against that. A checksum that cannot be computed fails rather than passes, or the property would be quietly optional on exactly the node that could not read the file. Not routed through the validation context's own checksum service: that one resolves against jdbc_drivers_dir and enforces a .jar grammar, neither of which applies here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
When a source refuses the pushed-down statement, its answer arrives in its own words -- typically a syntax error pointing at a quote character. Nothing in that says Doris wrote the statement, and nothing says which SQL Doris writes is a catalog property. That is the first wall anyone pointing this connector at something other than Doris walks into, because the default dialect is conservative ANSI and a source that wants something else rejects the very first query. The planning-time failure now names the dialect the statement was generated in and the property that changes it. Only the partitioned path can carry this: asking the driver to split a statement executes it, so FE is where that rejection lands. The single-statement path runs on BE and its error is BE's to improve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
EXPLAIN's QUERY line named more columns than the statement that runs. The property cache is warmed in init() -- initSchemaParams asks for the path partition keys, which come from these properties -- and at that moment the scan tuple still carries every column of the table. Nereids prunes it when the project above the scan is translated, and only then does planScan build the statement that executes. So a connector that renders remote SQL from the column list (adbc, jdbc) had a cached statement that over-projects against the one BE runs, and EXPLAIN described a query nobody would send. The display is re-asked with the CURRENT columns and the ORIGINAL filter. The filter has to be the original: by this point the pushed-down conjuncts have been removed from the node's list, so rebuilding it would drop the pushed predicates out of the WHERE clause shown. Nothing else moves -- what goes to BE, the MVCC/rewrite/top-n pins, and the conjunct pruning all keep using the cached result, so a connector that puts schema dictionaries in these properties is untouched. Costs nothing when the projection did not narrow: the slot ids are compared first, and a connector that renders no remote statement returns before the comparison. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
…ment A TIMESTAMPTZ literal reaches the dialect already converted to UTC: in a +08:00 session, ts > '2023-01-01 00:00:00' arrives as the wall clock 2022-12-31 16:00:00. Standard SQL's TIMESTAMP '...' spelling carries no zone, so the source reads that UTC wall clock as its own local time and compares it against its column. On a source east of UTC that only widens the match and Doris narrows it again on what comes back. On one west of UTC it NARROWS the match, and the rows the source drops are rows the query wanted -- a scan cannot ask for rows the source never sent. There is no portable spelling that says which instant is meant, so the comparison stays with Doris, for the same reason NaN, null-safe equality and LIKE already do. Also records why this connector does not consult the catalog property enable.mapping.timestamp_tz when mapping a zoned arrow timestamp: ExternalCatalog stamps that property as "false" into every external catalog that does not name it, so reading it cannot tell a user who asked for wall clocks from one who said nothing, and honouring it would force DATETIMEV2 on every adbc catalog. This connector's default is TIMESTAMPTZ; making the property settable needs fe-core to let a connector supply its own default first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
Twenty suites over a live catalog: type mapping and semantics, complex and binary types, scan edges, predicate pushdown, column pruning, query shapes, source table models, metadata operations, nested catalogs, cross source joins, large data, outfile, MTMV, and negative cases. Several assertions are written against what the planner actually delivers rather than against what the SQL says, because Nereids normalises the predicate before any connector sees it: a one-column disjunction becomes an IN list, NOT is pushed through comparisons and connectives by De Morgan, column-plus-literal arithmetic is folded, and <=> against a non-null literal becomes =. Pinning the SQL spelling would have been asserting on the optimizer, so each of those keeps a shape the optimizer cannot rewrite -- a two-column OR, two-column arithmetic, two nullable columns of one type -- to hold the connector's own behaviour. Likewise count(*) is asserted as one narrow column, not zero: pruning an empty scan tuple puts the smallest slot back. Three expectations record losses this connector cannot avoid. IPV4 arrives as the address's 32 bits read signed, because Doris encodes it as int32 on both sides of the wire. A source datetime arrives as TIMESTAMPTZ, so comparisons cast it back before checking the instant. DBL_MAX cannot reach the test client at all -- Doris prints a double with 16 significant digits, which parses back as infinity -- so that row is compared inside Doris with a null-safe join instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
Both of this connector's deployment-level settings move off fe.conf and into the plugin's own adbc.conf -- the channel apache#66347 introduced -- as drivers_dir and driver_secure_path, read through ConnectorConf.get. Neither has an fe.conf half. The two @ConfFields they used to be (adbc_drivers_dir, adbc_driver_secure_path) and the two DefaultConnectorContext env entries that forwarded them are removed rather than kept as a fallback: this connector has never shipped, so no deployment configured them anywhere else, and a key in fe-core is an engine change per connector setting -- which is what that channel exists to stop. The default drivers directory is computed in the connector from the doris_home the engine already publishes, so it stays <DORIS_HOME>/plugins/adbc_drivers. build.sh needs no change: it seeds a live <name>.conf from any *.conf.template found in a plugin zip. AdbcConnectorConfTest pins the template's name against ConnectorProvider.name() -- a template under any other name deploys a file the engine never opens, with every setting in it silently ignored -- and pins that an environment still carrying the old fe.conf keys does not resurrect a channel fe-core no longer feeds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RgJjuW5w4jEKF9HorENTur
morningman
force-pushed
the
wt-adbc-catalog
branch
from
August 2, 2026 15:51
a6e8d66 to
4315a84
Compare
morningman
marked this pull request as ready for review
August 2, 2026 15:52
morningman
requested review from
CalvinKirs,
Gabriel39 and
yiguolei
as code owners
August 2, 2026 15:52
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related Issue: #65615
Problem Summary:
Doris reads an external database through JDBC today: the source's rows become JDBC
objects and then Doris columns, one value at a time, across JNI. The source is also
read by a single backend, because JDBC has no notion of how a result is partitioned.
This PR adds a new catalog type,
adbc, that reads an external source through anArrow Database Connectivity driver. Data arrives as
Arrow record batches and is handed to BE's scanner without a per-value conversion, and
a scan is split into the driver's own result partitions so that N backends read N
partitions in parallel. Neither is reachable through JDBC at all.
CREATE CATALOG remote_source PROPERTIES ( "type" = "adbc", "driver_url" = "libadbc_driver_flightsql.so", "uri" = "grpc://remote-doris:8070", "user" = "root", "password" = "" ); SELECT count(*) FROM remote_source.some_db.some_table;Phase one targets Arrow Flight SQL sources, which makes this the intended replacement
for the
remote_doriscatalog. Other sources need only their driver's.soand, wheretheir SQL differs from ANSI, a dialect implementation -- no architectural change.
Supported features
What an ADBC catalog gives you from the outside -- what you can do with it and what it does
for you without being asked. The boundaries are in Known limits further down.
CREATE CATALOG ... PROPERTIES("type" = "adbc", ...), then query the source's tables like any other external table. Phase one targets Arrow Flight SQL sources, including another Doris cluster.uri,user,passwordpartitioned_read=auto/disabled/required(auto),max_partitions(1024)EXPLAINprints that same statement.=,!=,<,<=,>,>=,IS [NOT] NULL,[NOT] INandAND/OR/NOTover them become the remoteWHERE. Anything else -- functions, arithmetic,LIKE,BETWEEN-- stays in Doris. Doris re-applies every predicate regardless, so pushdown changes speed only, never the rows you get.LIMITpushdownWHEREwas pushed, so the source never truncates ahead of a filter Doris still has to apply.COUNT(*)without column valuesSHOW DATABASES,SHOW TABLES,DESC,SHOW CREATE TABLEandinformation_schemaover the source's own databases, tables and columns.ARRAY/MAP/STRUCT,DECIMAL, date, and timestamp with and without a zone.REFRESHREFRESH CATALOG/DATABASE/TABLEdrops what is remembered.meta.cache.adbc.metadata.enable/.ttl-second(600) /.capacity(1000)ORDER BY,UNION, subqueries,SELECT ... INTO OUTFILE, and an MTMV built on an ADBC table.sql_dialect(auto-detected,ansifallback;dorisprovided)"adbc.adbc.snowflake.sql.db" = "...".adbc.*propertiesCREATE CATALOGrather than as a puzzling query failure later.driver_url,driver_checksum,driver_entrypointDesign decisions worth knowing before reviewing
.so. FE goes through the ADBC JNI bridge, whichwraps the same C driver manager BE links statically, and BE dlopens the identical
file. This is a measured constraint, not a preference: partition descriptors are
driver-private bytes, and two official implementations of the same protocol already
serialize incompatible messages and mis-parse each other silently rather than
erroring.
adbc_drivers_diron FE andbe/plugins/adbc_driverson every BE.driver_urltherefore accepts local references only (bare name,
file://, or an absolute path);remote schemes are rejected because a per-node download cannot promise the nodes
agree. "Driver file not found" is written as a first-class error for that reason, and
driver_checksumcan pin the build.dialect interface a source can claim by vendor name or by the
sql_dialectproperty.Predicates are pushed all-or-nothing per conjunct from a whitelist; BE re-applies
every predicate regardless, so pushdown is pure acceleration.
statement executes it. That makes this the first connector for which
EXPLAINhad tobe told not to plan for real -- see the SPI addition below.
Changes outside the connector
ConnectorScanRequest.isExplainOnly()(SPI) +PluginDrivenScanNodefills itEXPLAINreachesplanScanfor real (its explain level isNORMAL, soNereidsPlanner.distribute()does not return early). For this connector that would execute the very query it was asked only to describe.PluginDrivenScanNodere-asks for the display statement with the current columnsinit(), before Nereids prunes the scan tuple, soEXPLAIN'sQUERY:line named more columns than the statement that actually runs. Affects any connector that renders remote SQL from the column list (adbc, jdbc). What goes to BE is unchanged.PluginDrivenScanNode.mapFileFormatType()learns"arrow"TTableFormatFileDesc.adbc_paramsbe/src/vec/exec/format/adbc/*+file_scanner_v2gateFlightSqlSchemaHelper(FE, Arrow Flight SQL server)GetTablesdescribed aDATEV2column as date64 while BE writes date32, and described array/map/struct with placeholder children. A client that types its columns from that schema fails on the first batch. Visible to every Arrow Flight SQL client, not only this connector.DataTypeTimeStampTzSerDe::read_column_from_arrow(BE)thirdparty:arrow-adbcdoris_be; the JNI bridge is built for FE (upstream's prebuilt one needs GLIBC 2.34). The SQLite and Flight SQL drivers are fetched for tests only and are not shipped.Config.adbc_drivers_dir/adbc_driver_secure_path,conf/fe.confJVM options,build.shBehaviour to be aware of
filter ADBC sends, so the filter is applied where the answer is read.
partitioned_readisautoby default: split the scan when the driver can, read itas one statement when it cannot.
requiredforbids that downgrade, which is whatkeeps a test from going green while quietly exercising the fallback;
disabledis theescape hatch.
(
meta.cache.adbc.metadata.*), and everyREFRESHstatement drops it. A newlycreated remote table is reachable without any refresh.
INSERT,CREATE/DROP TABLEand the other write statements against anADBC catalog are rejected. An MTMV over an ADBC table does build and refresh.
Test coverage
fe-connector-adbc: 186 unit tests, no skips. About 25 of them drive the real SQLiteADBC driver through the same Java -> JNI -> C driver manager -> driver
.sopath FEtakes in production;
fe-connector-apicovers the new SPI field.the scanner gate and the TIMESTAMPTZ serde.
regression-test/suites/external_table_p0/adbc/): 20 suites against a livecatalog -- type mapping and semantics, complex and binary types, scan edges, predicate
pushdown, column pruning, query shapes, source table models, metadata operations,
nested catalogs, cross-source joins, large data, outfile, MTMV, negative cases, and
partitioned vs. single-statement reads compared row for row. Each suite returns early
when the driver is absent, so a cluster without one does not fail.
Known limits
them over several backends has only been reasoned from the code, not observed -- the
test environment has one backend, and
test_adbc_multi_backendreturns early there.IPV4from a Doris source arrives as a bareINT(both sides encode it as int32), anda source
DATETIMEarrives asTIMESTAMPTZ. Both are asserted, not worked around.driver_checksumverifies FE's copy only.Release note
Support ADBC catalog, which reads an external source through an Arrow Database
Connectivity driver: data is transferred as Arrow record batches and a scan is split
across backends by the driver's own result partitions. Phase one targets Arrow Flight
SQL sources. The driver library is supplied by the operator; Doris ships none.
Also fixes two defects on the Arrow Flight SQL path that are visible to existing
clients:
GetTablesreported the wrong Arrow type forDATEV2and placeholder childrenfor
ARRAY/MAP/STRUCT, and aTIMESTAMPTZcolumn read from Arrow was decoded bycopying its bits instead of converting the epoch.
Check List (For Author)
Test
Behavior changed:
adbc; no existing catalog type changes.DATEV2and for nested types (previously a client that trustedGetTablesfailed on the first batch).
EXPLAINon a plugin-driven catalog prints the projection the scan really asksfor; the statement sent to BE is unchanged.
Does this need documentation?
Check List (For Reviewer who merge this PR)