feat: Support PostgreSQL export for export-metadata

[JWuCines]

Description

The export-metadata tool only supports exporting from Derby metadata stores, because it relies on Derby's native SYSCS_EXPORT_TABLE procedure. This PR adds support for exporting from PostgreSQL by implementing a generic JDBC export in SQLMetadataConnector, which PostgreSQL (and any future connector) inherits automatically. While doing so, it also fixes a few correctness issues in the tool that affect Derby exports as well.

Generic JDBC export in SQLMetadataConnector

Added exportTable(tableName, outputPath, columns) and a protected exportTableWithJdbc() that exports any table to CSV over plain JDBC:

• Binary columns (BINARY, VARBINARY, LONGVARBINARY, BLOB and PostgreSQL BYTEA) are hex-encoded, booleans are written as true/false and NULLs as empty fields, matching what the rewrite stage of the tool expects.
• Values are escaped as per RFC 4180 (csvEscape()): values containing a comma, double quote, \n or \r are quoted, with inner quotes doubled. A NULL is written as an unquoted empty field while an empty string is written as a quoted empty field, so that the two remain distinguishable on import.
• The export runs inside a transaction (so the connection has autoCommit=false) with getStreamingFetchSize() applied to the Statement, which is what the PostgreSQL driver requires to stream a result set with a cursor instead of buffering it entirely in memory. The value is passed through as-is, since drivers such as MySQL's use a sentinel (Integer.MIN_VALUE) to request streaming.
• The query is qualified with getMetadataTableSchema(), a new overridable hook that returns the schema Druid's tables live in. PostgreSQLConnector overrides it to return the configured dbTableSchema, so that both the export query and the column lookup below agree with its tableExists. The schema is quoted, since it is the name as stored in the database, while the table name is left unquoted so that it is folded like in every other Druid statement.

DerbyConnector keeps overriding exportTable with the native export, using SYSCS_EXPORT_QUERY when an explicit column list is given.

Deterministic segments column order

Added SQLMetadataConnector.getTableColumns(), which reads a table's columns from DatabaseMetaData, scoped to the same schema as the export query and matching the table name ignoring case, since the database folds unquoted identifiers (to uppercase in Derby, to lowercase in PostgreSQL). The schema is a JDBC search pattern, in which _ and % are wildcards, so the returned TABLE_SCHEM is compared to it exactly rather than relying on the pattern match.

ExportMetadata uses it to export the segments table in a canonical column order (id, dataSource, created_date, start, end, partitioned, version, used, payload, used_status_last_updated, indexing_state_fingerprint, upgraded_from_segment_id[, schema_fingerprint, num_rows]), instead of relying on the physical column order, which depends on the order in which ALTER TABLE added the newer columns. Unknown columns are appended at the end, and the export fails with an ISE rather than falling back to SELECT * if the column list cannot be read. Each column is quoted with the database's identifier quote string, so reserved words such as end work.

Fixes in ExportMetadata

• Table names are no longer unconditionally uppercased; uppercasing is applied only for Derby (detected via the jdbc:derby URI prefix), since PostgreSQL uses lowercase table names.
• rewriteSegmentsExport hardcoded columns 0–8 and silently dropped used_status_last_updated, indexing_state_fingerprint, upgraded_from_segment_id, schema_fingerprint and num_rows. All columns after payload are now passed through.
• All five rewrite methods re-escape non-payload fields via csvEscape(). Previously, fields containing commas or double quotes were parsed correctly but written back unquoted, producing malformed output.
• The rewrite stage read the intermediate CSV with the default OpenCSV CSVParser, which treats backslash as an escape character and silently dropped it from values such as segment ids and datasource names (backslashes are permitted by Druid's id validation). It now uses an RFC4180Parser, matching the output written by the export stage.
• The rewrite methods read one physical line at a time, so a quoted value containing a newline or carriage return was misparsed. They now share an openCsvReader() helper and iterate with readNext(), so a record spanning several lines is handled as a single record.
• readRecord() validates the field count of each record and fails with the row number, file name and expected arity, instead of throwing an ArrayIndexOutOfBoundsException on a malformed raw CSV. A CsvValidationException from the parser is reported as an IOException naming the offending file.
• Updated the @Command description to mention PostgreSQL support.

Documentation

• export-metadata.md — documented that a PostgreSQL import must go through staging tables with TEXT payload columns and decode(payload, 'hex'), using an export run with --use-hex-blobs: the payload columns are BYTEA, and COPY parses a BYTEA field with the bytea input syntax, in which the backslashes of the rewritten JSON payloads would either fail or silently change the value. Also removed the Derby-only limitation and added a PostgreSQL section under "Running the tool" with the required -Ddruid.extensions.loadList and -Ddruid.metadata.storage.type flags. Documented the exported segments column order and how to adjust the import commands for tables with fewer optional columns, including how to import a legacy nine-column export into a target whose used_status_last_updated column is NOT NULL, and explained how NULLs (written as empty fields) must be handled per database: Derby imports them as NULL, PostgreSQL COPY needs FORCE_NULL, and MySQL LOAD DATA needs NULLIF on user variables since it would otherwise store '' and coerce numeric columns such as num_rows to 0. The Derby commands use SYSCS_IMPORT_DATA with explicit column lists (quoting the reserved word end) so that the documented adjustment applies, and the MySQL commands use ESCAPED BY '', since the exported CSV is RFC 4180 where a backslash is an ordinary character.
• metadata-migration.md — updated the intro and the export tool reference to include PostgreSQL.
• deep-storage-migration.md — updated the export tool reference and noted that no running processes are needed when migrating from PostgreSQL.

Tests

SQLMetadataConnectorTest exercises the generic JDBC path via a new TestDerbyConnector.exportTableGeneric() (bypassing the Derby override), plus the native Derby path and the column lookup:

• testExportTable — hex-encoded BLOBs and true/false booleans
• testExportTableWithSpecialCharacters — CSV quoting of commas, double quotes and plain values
• testExportTableWithNullValues — NULL columns produce empty fields, empty strings produce quoted empty fields
• testExportTableWithExplicitColumnOrder — explicit column order with a reserved-word column
• testExportTableWithDerbyNativeExport — DerbyConnector's native SYSCS_EXPORT_QUERY path
• testGetTableColumns — column lookup for a table name in either case, and for a non-existent table

ExportMetadataTest (JUnit 5, using TemporaryFolderExtension and TestDerbyConnector.DerbyConnectorRule) covers the rewrite stage: the canonical column ordering, RFC 4180 escaping, segments rewrite with all columns and with only the nine legacy columns, special characters, backslash preservation, the failure on a truncated row, and testExportAndRewriteSegments_withMultilineFields, an end-to-end test that exports a Derby table containing newlines, carriage returns and commas through the generic JDBC path and verifies that every record and field round-trips through the rewrite.

Release note

The export-metadata tool now supports exporting from PostgreSQL metadata stores in addition to Derby. When exporting from PostgreSQL, pass -Ddruid.extensions.loadList='["postgresql-metadata-storage"]' -Ddruid.metadata.storage.type=postgresql on the command line along with the appropriate --connectURI.

The segments table is now exported in a fixed column order instead of the physical column order of the source table, and all of its optional columns (such as used_status_last_updated and upgraded_from_segment_id) are preserved, where they were previously dropped. Exported CSV files follow RFC 4180; see the documentation for the matching import commands.

---

Key changed/added classes in this PR

• SQLMetadataConnector — added exportTable, exportTableWithJdbc, getTableColumns, getMetadataTableSchema, makeExportSelectList and csvEscape for generic, transactionally streamed JDBC CSV export
• DerbyConnector — native export now supports an explicit column list via SYSCS_EXPORT_QUERY
• PostgreSQLConnector — overrides getMetadataTableSchema() so that the export query and column lookup use the configured schema
• ExportMetadata — added isDerby(), conditional table name casing, canonical segments column order, record-aware RFC 4180 CSV reading with field-count validation, and all-column preservation in the segments rewrite
• TestDerbyConnector — added exportTableGeneric() to exercise the generic JDBC path
• SQLMetadataConnectorTest — added export and column lookup tests
• ExportMetadataTest — added tests for column ordering, CSV escaping and the segments rewrite

---

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

[FrankChen021]

Severity	Findings
P0	0
P1	2
P2	1
P3	0
Total	3

Reviewed 7 of 7 changed files.

Found three export-migration risks: buffered PostgreSQL reads, dropped current-schema segment columns, and lost CSV escaping during rewrites.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 8 of 8 changed files. PostgreSQL streaming is fixed, but segment import compatibility and CSV backslash preservation remain broken.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 9 of 9 changed files. Stable segment ordering and RFC CSV parsing fix the prior issues, but legacy schemas still conflict with the documented import list and mixed-case PostgreSQL bases fail column discovery.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Severity	Findings
P0	0
P1	1
P2	1
P3	0
Total	2

Reviewed 9 of 9 changed files. Mixed-case PostgreSQL identifiers are now handled, but legacy Derby imports and configured PostgreSQL schema discovery remain incorrect.

---

This is an automated review by Codex GPT-5.6-Sol

[FrankChen021]

Severity	Findings
P0	0
P1	2
P2	1
P3	0
Total	3

Severity	Findings
P0	0
P1	2
P2	1
P3	0
Total	3

Reviewed 10 of 10 changed files. The prior mixed-case and RFC CSV parsing fixes are present, but configured PostgreSQL schema resolution and legacy-schema import compatibility remain incomplete; MySQL backslash compatibility is also unresolved.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	2
P3	0
Total	3

Reviewed 10 of 10 changed files.

Validation: git diff --check passed; no builds or tests were run.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 10 of 10 changed files.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

LGTM.

but I think the original design of exporting data is not good, all 'export' related logic should be in the ExportMetadata class, the SQLMetadataConnector only defines how we can retrieve data from databases. and because of that, we can see that the csvEscape which has nothing to do with SQLMetadataConnector has to be defined in this class in this PR. If you have time to help us improve this part in another PR, I will appreciate it.

[JWuCines]

I will create a following PR for the improvement of exporting data tooling!

[FrankChen021]

Severity	Findings
P0	0
P1	1
P2	2
P3	0
Total	3

The full current diff was reviewed because the previous reviewed SHA was unavailable locally. Earlier streaming, ordering, parsing, mixed-case, and schema-qualification concerns appear addressed; the findings below are current residual import and qualification issues.

Reviewed 3 of 10 changed files with findings; all 10 changed files were reviewed.

Validation: focused git diff --check passed. Builds, tests, and database execution were not run.

---

This is an automated review by Codex GPT-5.6-Luna(max)

[FrankChen021]

Severity	Findings
P0	0
P1	2
P2	0
P3	0
Total	2

Reviewed 10 of 10 changed files.

Validation: focused git diff --no-ext-diff --check from the previous reviewed commit to current HEAD passed. Builds and tests/database execution were not run.

Findings that could not be attached inline:

• services/src/main/java/org/apache/druid/cli/ExportMetadata.java:496 - P1 CSV export loses the distinction between NULL and empty strings. RFC4180ParserBuilder().build() uses the default NEITHER null-string configuration, so an unquoted NULL field and a quoted empty string are both parsed as "". The exporter then re-emits both as quoted empty strings. Nullable segment fields such as num_rows can therefore fail PostgreSQL imports or lose NULL semantics. Configure parsing so empty separators represent null while quoted empty strings remain empty, and add a round-trip test.
• docs/operations/export-metadata.md:199 - P1 Documented Derby import column names do not resolve. The documented Derby SYSCS_IMPORT_DATA commands use lowercase or mixed-case insert-column names, including the literal "end", but the metadata table columns are resolved by Derby as uppercase unquoted identifiers (ID,DATASOURCE,...,END). The documented commands therefore fail before importing data. Use Derby's actual metadata names or the correct quoted identifier syntax in the documentation.

---

This is an automated review by Codex GPT-5.6-Luna(max)