[feat](connector) give each connector plugin its own conf file

.licenserc.yaml

@@ -59,6 +59,16 @@ header:
     - "fe/fe-filesystem/fe-filesystem-spi/src/test/resources/filesystem-plugin-surface.txt"
     - "fe/fe-authentication/fe-authentication-spi/src/test/resources/authentication-plugin-surface.txt"
     - "fe/fe-core/src/test/resources/lineage-plugin-surface.txt"
+    # Connector plugin settings templates. build.sh seeds each connector's live
+    # <name>.conf from its template verbatim (cp -n), so the template's content IS
+    # the file an administrator edits in plugins/connector/<dir>/. Matched by name
+    # rather than by a **/*.conf.template glob, so a new one is a deliberate entry
+    # here rather than something a wildcard silently absorbs.
+    - "fe/fe-connector/fe-connector-hive/src/main/resources/hms.conf.template"
+    - "fe/fe-connector/fe-connector-iceberg/src/main/resources/iceberg.conf.template"
+    - "fe/fe-connector/fe-connector-jdbc/src/main/resources/jdbc.conf.template"
+    - "fe/fe-connector/fe-connector-paimon/src/main/resources/paimon.conf.template"
+    - "fe/fe-connector/fe-connector-trino/src/main/resources/trino-connector.conf.template"
     # Golden 4.1.3 upgrade fixtures, emitted by Gen413Fixtures running real 4.1.3
     # bytecode (see the sibling PROVENANCE.txt) and stamped "do not edit by hand".
     # These are the generator's only non-binary outputs -- its .bin files are skipped

build.sh

@@ -1082,8 +1082,17 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
         fi
         mkdir -p "${conn_plugin_target}"
         unzip -o "${conn_zip}" -d "${conn_plugin_target}/"
+        # A connector's own settings file. The zip carries only <name>.conf.template; the live
+        # <name>.conf is seeded from it here and never overwritten, so an upgrade that unzips a new
+        # plugin build over this directory refreshes the jars and the template but leaves whatever the
+        # administrator configured. Deliberately generic (globbed on *.conf.template, no connector
+        # named): a new connector ships a template and needs no change here.
+        for conn_conf_tpl in "${conn_plugin_target}"/*.conf.template; do
+            [ -e "${conn_conf_tpl}" ] || continue
+            cp -n "${conn_conf_tpl}" "${conn_conf_tpl%.template}"
+        done
     done
-    unset CONN_PLUGIN_DIR conn_module conn_plugin_target conn_module_dir conn_zip
+    unset CONN_PLUGIN_DIR conn_module conn_plugin_target conn_module_dir conn_zip conn_conf_tpl
 
     # RC-4: self-contain the paimon connector plugin for OSS. The connector sets
     # fs.oss.impl=com.aliyun.jindodata.oss.JindoOssFileSystem; that impl lives in the jindofs jars,

fe/fe-connector/README.md

@@ -211,7 +211,23 @@ metastore/shade/cache). For a write path, the richest example is
     (example: `RecordingConnectorContext`). Never touch
     `connector-metadata-methods.txt` unless you changed the shared SPI
     surface itself.
-14. **Packaging.** Add `src/main/assembly/plugin-zip.xml` (copy from es or
+14. **Deployment-level settings** (if any). A value that is one-per-FE rather
+    than one-per-catalog goes in your plugin's own settings file, NOT in
+    fe.conf: ship `src/main/resources/<name>.conf.template`, add it to your
+    assembly's `<files>` at the zip root, and read it with
+    `ConnectorConf.get(context, "<key>", null, "<default>")`. `<name>` is
+    `ConnectorProvider.name()`, which is **not** necessarily your plugin
+    directory name — `plugins/connector/hive/` holds `hms.conf` and
+    `plugins/connector/trino/` holds `trino-connector.conf`. Guard that with a
+    test asserting `name() + ".conf.template"` is on the classpath (copy
+    `IcebergConnectorConfTest#theConfTemplateIsNamedAfterTheProvider`); a
+    template under any other name deploys a file the engine never opens.
+    `build.sh` seeds the live `.conf` from the template generically, so it
+    needs no change. Do NOT add a key to `Config.java` or to
+    `DefaultConnectorContext.buildEnvironment` — that is an engine change per
+    setting, and the keys still there are only the ones several connectors
+    share plus the fe.conf fallbacks kept for existing deployments.
+15. **Packaging.** Add `src/main/assembly/plugin-zip.xml` (copy from es or
     paimon). Verify your module through `package`/`install`, not just
     `test-compile` — shades and the plugin zip only materialize then.
 15. **Gates and e2e.** Your module must pass the forbidden-import gate (runs

fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/package-info.java

@@ -160,8 +160,8 @@
  *
  * <h2>Rule 7 — where a connector's tunable knobs live</h2>
  *
- * <p>Pick the channel by the SCOPE of the value. Only the second one obliges anyone to touch the engine, so
- * a knob that can be catalog-scoped should be.</p>
+ * <p>Pick the channel by the SCOPE of the value. None of them requires an engine change any more, so a
+ * knob that can be catalog-scoped should be, and one that cannot belongs in the plugin's own conf file.</p>
  *
  * <ul>
  * <li><b>Per catalog</b> &rarr; a key in {@code CREATE CATALOG ... PROPERTIES(...)}. The engine hands the
@@ -174,9 +174,22 @@
  *     declared in {@code HiveConnectorProperties} and read in {@code HiveScanPlanProvider} /
  *     {@code HiveConnector}; those key strings appear nowhere in {@code fe-core}.</li>
  * <li><b>Per FE process</b> (one deployment-level value for every catalog, e.g. a driver directory)
- *     &rarr; an {@code fe.conf} field forwarded through {@code ConnectorContext.getEnvironment()} by
- *     {@code DefaultConnectorContext.buildEnvironment}. This is the one knob shape that requires an engine
- *     change per key, so use it only when the value genuinely is not per catalog.</li>
+ *     &rarr; a key in the plugin's own {@code <name>.conf}, read with {@code ConnectorConf.get}. The engine
+ *     locates and parses {@code <pluginDir>/<name>.conf} generically
+ *     ({@code ConnectorPluginManager.loadPlugins}) and serves it back through
+ *     {@code ConnectorContext.getConnectorConfig()}, so no key name of yours reaches {@code fe-core} and
+ *     adding one costs the engine nothing. Do <b>not</b> prefix these keys — the file name already
+ *     namespaces them. Ship {@code src/main/resources/<name>.conf.template} and add it to your assembly's
+ *     {@code <files>} at the zip root; {@code build.sh} seeds the live {@code .conf} from it. {@code <name>}
+ *     is {@code ConnectorProvider.name()} and need not equal your plugin directory name
+ *     ({@code plugins/connector/hive/} holds {@code hms.conf}).</li>
+ * <li><b>Per FE process, legacy</b> &rarr; an {@code fe.conf} field forwarded through
+ *     {@code ConnectorContext.getEnvironment()} by {@code DefaultConnectorContext.buildEnvironment}. This is
+ *     the one shape that requires an engine change per key, and it is <b>closed to new keys</b>. What is
+ *     still there is either shared by several connectors or not a connector setting at all
+ *     ({@code doris_home}, {@code doris_version}); the connector settings among them are kept as the
+ *     fallback {@code ConnectorConf.get} consults after the plugin conf, so deployments configured before
+ *     the conf files existed keep working untouched.</li>
  * <li><b>Per session</b> &rarr; read the query's session variables from
  *     {@link ConnectorSession#getSessionProperties()}. The connector does not declare them; it looks up the
  *     names it cares about.</li>

fe/fe-connector/fe-connector-hive/src/main/assembly/plugin-zip.xml

@@ -40,6 +40,17 @@ under the License.
             <source>${project.build.directory}/${project.build.finalName}.jar</source>
             <outputDirectory>/</outputDirectory>
         </file>
+        <!--
+          The connector's settings template. Only the .template ships; build.sh seeds the live hms.conf
+          from it with cp -n, so unzipping a newer plugin build over a deployed directory refreshes this
+          file but never the administrator's copy. The name must equal ConnectorProvider.name() +
+          ".conf.template" - note that name() is "hms" while this plugin's directory is "hive";
+          HiveConnectorMetadataDdlTest fails if the two drift apart.
+        -->
+        <file>
+            <source>src/main/resources/hms.conf.template</source>
+            <outputDirectory>/</outputDirectory>
+        </file>
     </files>
 
     <dependencySets>

fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java

@@ -67,6 +67,7 @@
 import org.apache.doris.connector.hms.HmsPartitionInfo;
 import org.apache.doris.connector.hms.HmsTableInfo;
 import org.apache.doris.connector.hms.HmsTypeMapping;
+import org.apache.doris.connector.spi.ConnectorConf;
 import org.apache.doris.connector.spi.ConnectorContext;
 import org.apache.doris.connector.spi.ConnectorStorageContext;
 import org.apache.doris.filesystem.FileSystem;
@@ -1618,7 +1619,8 @@ public void createTable(ConnectorSession session, ConnectorCreateTableRequest re
         }
         Map<String, String> env = context.getEnvironment();
         String fileFormat = userProps.getOrDefault(HiveConnectorProperties.CREATE_FILE_FORMAT,
-                env.getOrDefault(HiveConnectorProperties.ENV_HIVE_DEFAULT_FILE_FORMAT,
+                ConnectorConf.get(context, HiveConnectorProperties.CONF_DEFAULT_FILE_FORMAT,
+                        HiveConnectorProperties.ENV_HIVE_DEFAULT_FILE_FORMAT,
                         HiveConnectorProperties.DEFAULT_FILE_FORMAT));
 
         // Metastore table parameters: lower-case every key and stamp the file_format / location keys under a
@@ -1673,11 +1675,14 @@ public void createTable(ConnectorSession session, ConnectorCreateTableRequest re
         // enable gate first, then the hash requirement.
         ConnectorBucketSpec bucketSpec = request.getBucketSpec();
         if (bucketSpec != null) {
-            boolean bucketEnabled = Boolean.parseBoolean(env.getOrDefault(
+            boolean bucketEnabled = Boolean.parseBoolean(ConnectorConf.get(context,
+                    HiveConnectorProperties.CONF_ENABLE_CREATE_BUCKET_TABLE,
                     HiveConnectorProperties.ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE, "false"));
             if (!bucketEnabled) {
-                throw new DorisConnectorException(
-                        "Create hive bucket table need set enable_create_hive_bucket_table to true");
+                throw new DorisConnectorException("Create hive bucket table need set '"
+                        + HiveConnectorProperties.CONF_ENABLE_CREATE_BUCKET_TABLE + "' in hms.conf (or "
+                        + HiveConnectorProperties.ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE
+                        + " in fe.conf) to true");
             }
             if (HiveConnectorProperties.BUCKET_ALGO_RANDOM.equals(bucketSpec.getAlgorithm())) {
                 throw new DorisConnectorException("External hive table only supports hash bucketing");

fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java

@@ -73,6 +73,13 @@ private HiveConnectorProperties() {
             new HashSet<>(Arrays.asList(CREATE_FILE_FORMAT, CREATE_LOCATION)));
     public static final String DORIS_PROP_PREFIX = "doris.";
 
+    // -- deployment-level settings, read from this plugin's own hms.conf --
+    // The file is named after ConnectorProvider.name() ("hms"), NOT after the plugin directory
+    // ("plugins/connector/hive"). Each key falls back to the ENV_ name below it, which is the fe.conf key
+    // it used to live under and still works.
+    public static final String CONF_DEFAULT_FILE_FORMAT = "default_file_format";
+    public static final String CONF_ENABLE_CREATE_BUCKET_TABLE = "enable_create_bucket_table";
+
     // -- environment keys threaded from fe-core DefaultConnectorContext (must stay byte-identical there) --
     public static final String ENV_HIVE_DEFAULT_FILE_FORMAT = "hive_default_file_format";
     public static final String ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE = "enable_create_hive_bucket_table";

fe/fe-connector/fe-connector-hive/src/main/resources/hms.conf.template

@@ -0,0 +1,21 @@
+# Hive (HMS) connector plugin configuration.
+#
+# build.sh seeds hms.conf from this file on first deploy and never overwrites it, so an upgrade that
+# unzips a new plugin build over this directory keeps whatever you configured here.
+# This file must exist on EVERY FE node -- it is not replicated through Doris metadata.
+# Changes take effect after an FE restart.
+#
+# The file name is the connector's name (ConnectorProvider.name() == "hms"), NOT the directory name:
+# this plugin is deployed under plugins/connector/hive/ but its settings file is hms.conf.
+#
+# Every setting below is optional. Left commented out, each falls back to the fe.conf key named
+# after it, and then to the built-in default -- so an existing deployment needs no change here.
+
+# Storage format for a CREATE TABLE that does not name one itself.
+# Falls back to fe.conf's hive_default_file_format, whose default is orc. A CREATE TABLE may override
+# it per table with the file_format property, which wins over both.
+# default_file_format=orc
+
+# Whether CREATE TABLE may create a bucketed hive table. Off by default.
+# Falls back to fe.conf's enable_create_hive_bucket_table.
+# enable_create_bucket_table=false

fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeConnectorContext.java

@@ -24,9 +24,10 @@
 import java.util.Map;
 
 /**
- * Minimal {@link ConnectorContext} test double: carries a fixed catalog identity and an environment map (the
- * channel through which fe-core threads the FE-global CREATE TABLE defaults). Everything else uses the
- * interface defaults.
+ * Minimal {@link ConnectorContext} test double: carries a fixed catalog identity plus the two maps a
+ * deployment-level setting can arrive in — this plugin's own {@code hms.conf}
+ * ({@link #getConnectorConfig()}) and fe.conf as forwarded by the engine ({@link #getEnvironment()}).
+ * Everything else uses the interface defaults.
  */
 public class FakeConnectorContext implements ConnectorContext, ConnectorStorageContext {
 
@@ -40,6 +41,7 @@ public ConnectorStorageContext getStorageContext() {
     private final String catalogName;
     private final long catalogId;
     private final Map<String, String> environment;
+    private final Map<String, String> connectorConfig;
 
     public FakeConnectorContext() {
         this("test_catalog", 0L, Collections.emptyMap());
@@ -50,9 +52,15 @@ public FakeConnectorContext(Map<String, String> environment) {
     }
 
     public FakeConnectorContext(String catalogName, long catalogId, Map<String, String> environment) {
+        this(catalogName, catalogId, environment, Collections.emptyMap());
+    }
+
+    public FakeConnectorContext(String catalogName, long catalogId, Map<String, String> environment,
+            Map<String, String> connectorConfig) {
         this.catalogName = catalogName;
         this.catalogId = catalogId;
         this.environment = environment == null ? Collections.emptyMap() : environment;
+        this.connectorConfig = connectorConfig == null ? Collections.emptyMap() : connectorConfig;
     }
 
     @Override
@@ -69,4 +77,9 @@ public long getCatalogId() {
     public Map<String, String> getEnvironment() {
         return environment;
     }
+
+    @Override
+    public Map<String, String> getConnectorConfig() {
+        return connectorConfig;
+    }
 }

fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataDdlTest.java

@@ -112,6 +112,67 @@ public void createTableFallsBackToOrcWhenEnvMissing() {
         Assertions.assertEquals("orc", client.lastCreateTable.getFileFormat());
     }
 
+    // ==================== createTable: the plugin's own hms.conf ====================
+
+    @Test
+    public void createTableFileFormatFromPluginConfBeatsFeConf() {
+        RecordingHmsClient client = new RecordingHmsClient();
+        // WHY: the point of the plugin conf channel. An administrator who sets default_file_format in
+        // hms.conf must get it even though fe.conf still names the old value -- reverse the precedence
+        // and migrating a deployment to the new file silently does nothing.
+        Map<String, String> conf = Collections.singletonMap(
+                HiveConnectorProperties.CONF_DEFAULT_FILE_FORMAT, "parquet");
+        Map<String, String> env = Collections.singletonMap(
+                HiveConnectorProperties.ENV_HIVE_DEFAULT_FILE_FORMAT, "orc");
+
+        metadataWithConf(client, conf, env).createTable(session(), request().build());
+
+        Assertions.assertEquals("parquet", client.lastCreateTable.getFileFormat());
+    }
+
+    @Test
+    public void createTableUserFileFormatStillBeatsThePluginConf() {
+        RecordingHmsClient client = new RecordingHmsClient();
+        // WHY: the new channel is deployment-level; it must not outrank a per-catalog CREATE TABLE
+        // property. Only the two deployment channels reordered relative to each other.
+        Map<String, String> conf = Collections.singletonMap(
+                HiveConnectorProperties.CONF_DEFAULT_FILE_FORMAT, "parquet");
+
+        metadataWithConf(client, conf, Collections.emptyMap()).createTable(session(),
+                request().properties(Collections.singletonMap("file_format", "orc")).build());
+
+        Assertions.assertEquals("orc", client.lastCreateTable.getFileFormat());
+    }
+
+    @Test
+    public void createTableBucketGateCanBeOpenedFromThePluginConfAlone() {
+        RecordingHmsClient client = new RecordingHmsClient();
+        // WHY: the second migrated setting, and the one where getting the channel wrong is expensive --
+        // a deployment that opts in through hms.conf but is still gated by fe.conf's default 'false'
+        // would find bucketed creates rejected with no indication which file is in charge.
+        Map<String, String> conf = Collections.singletonMap(
+                HiveConnectorProperties.CONF_ENABLE_CREATE_BUCKET_TABLE, "true");
+        ConnectorBucketSpec bucket = new ConnectorBucketSpec(
+                Collections.singletonList("id"), 8, "doris_default");
+
+        metadataWithConf(client, conf, Collections.emptyMap())
+                .createTable(session(), request().bucketSpec(bucket).build());
+
+        Assertions.assertEquals(Collections.singletonList("id"), client.lastCreateTable.getBucketCols());
+        Assertions.assertEquals(8, client.lastCreateTable.getNumBuckets());
+    }
+
+    @Test
+    public void theConfTemplateIsNamedAfterTheProvider() {
+        // WHY: the engine reads <name>.conf, and this connector's name is "hms" while its plugin
+        // directory is "hive". A template under any other name deploys a file nothing ever opens --
+        // silently, with every setting in it ignored. Renaming getType() must break here.
+        String expected = new HiveConnectorProvider().name() + ".conf.template";
+        Assertions.assertNotNull(
+                HiveConnectorMetadataDdlTest.class.getClassLoader().getResource(expected),
+                "the plugin must ship " + expected + " on its classpath");
+    }
+
     // ==================== createTable: transactional rejection ====================
 
     @Test
@@ -368,6 +429,12 @@ private static HiveConnectorMetadata metadata(RecordingHmsClient client,
         return new HiveConnectorMetadata(client, catalogProps, new FakeConnectorContext(env));
     }
 
+    private static HiveConnectorMetadata metadataWithConf(RecordingHmsClient client,
+            Map<String, String> conf, Map<String, String> env) {
+        return new HiveConnectorMetadata(client, Collections.emptyMap(),
+                new FakeConnectorContext("test_catalog", 0L, env, conf));
+    }
+
     /**
      * Columns are nullable: hive rejects a {@code NOT NULL} column up front (validateColumns runs before every
      * other createTable check), so a NOT NULL fixture column would short-circuit every test in this class before

fe/fe-connector/fe-connector-iceberg/src/main/assembly/plugin-zip.xml

@@ -36,6 +36,20 @@ under the License.
         </fileSet>
     </fileSets>
 
+    <files>
+        <!--
+          The connector's settings template, at the ZIP ROOT (not lib/): the engine looks for
+          <pluginDir>/<name>.conf. Only the .template ships; build.sh seeds the live iceberg.conf from
+          it with cp -n, so unzipping a newer plugin build over a deployed directory refreshes this
+          file but never the administrator's copy. The name must equal ConnectorProvider.name() +
+          ".conf.template" - IcebergConnectorConfTest fails if it drifts.
+        -->
+        <file>
+            <source>src/main/resources/iceberg.conf.template</source>
+            <outputDirectory>/</outputDirectory>
+        </file>
+    </files>
+
     <dependencySets>
         <dependencySet>
             <outputDirectory>lib</outputDirectory>