[tiering] introduce the option of `lake.tiering.io.tmpdir` to avoid disk exhaustion by zuston · Pull Request #3569 · apache/fluss

zuston · 2026-07-03T10:04:04Z

Purpose

Linked issue: close #3568

Brief change log

Tests

API and Format

Documentation

…isk exhaustion

zuston · 2026-07-03T10:05:05Z

link the PR of #3193 . Could you help review this? @luoyuxia

zuston · 2026-07-03T10:05:52Z

 public class MergeTreeWriter extends RecordWriter<KeyValue> {

-    // the option key to configure the temporary directory used by fluss tiering
-    private static final String FLUSS_TIERING_TMP_DIR_KEY = "fluss.tiering.io-tmpdir";


We don't need to reserve this option that it haven't been valid.

zuston · 2026-07-06T05:51:16Z

cc @luoyuxia

Copilot

Pull request overview

This PR adds a configurable temporary I/O directory for lake tiering (lake.tiering.io.tmpdir) and wires it through the Flink tiering pipeline into the Paimon MergeTree writer, so tiering can avoid exhausting the default local temp disk.

Changes:

Introduce ConfigOptions.LAKE_TIERING_IO_TMP_DIR (lake.tiering.io.tmpdir) and propagate an ioTmpDir value via WriterInitContext.
Plumb the temp dir from the Flink tiering source into TieringSourceReader/TieringSplitReader and finally into PaimonLakeWriter → MergeTreeWriter IOManager creation.
Add unit tests covering the new ioTmpDir plumbing and the tmp-dir selection behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/tiering/mergetree/MergeTreeWriterTest.java	Adds a unit test for `MergeTreeWriter` tmp-dir selection.
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/PaimonLakeWriter.java	Passes `ioTmpDir` from `WriterInitContext` into `MergeTreeWriter`.
fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/mergetree/MergeTreeWriter.java	Adds an `ioTmpDir`-aware constructor and uses it to create Paimon `IOManager`.
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/utils/FlinkConnectorOptionsUtilTest.java	Adds tests for Flink tmp-dir derivation and lake tiering tmp-dir option resolution.
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/tiering/source/TieringWriterInitContextTest.java	Adds a unit test for `TieringWriterInitContext.ioTmpDir()`.
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/utils/FlinkConnectorOptionsUtils.java	Adds helper methods to derive tmp dirs and resolves `lake.tiering.io.tmpdir`.
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/TieringWriterInitContext.java	Extends the init context with an `ioTmpDir` field and exposes it via `WriterInitContext`.
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/TieringSplitReader.java	Carries `ioTmpDir` through to `TieringWriterInitContext` used for lake-writer creation.
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/TieringSourceReader.java	Adds constructors/plumbing to pass `ioTmpDir` down to split readers.
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/TieringSource.java	Passes resolved lake-tiering tmp dir into the source reader.
fluss-common/src/test/java/org/apache/fluss/lake/writer/WriterInitContextTest.java	Extends test coverage for the new default `ioTmpDir()` method.
fluss-common/src/main/java/org/apache/fluss/lake/writer/WriterInitContext.java	Adds a new default method `ioTmpDir()` for lake writers.
fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java	Introduces `lake.tiering.io.tmpdir` configuration option and description.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    public static String getFlinkIoTmpDir(
+            org.apache.flink.configuration.Configuration flinkConfig) {
+        if (flinkConfig.contains(TMP_DIRS)) {
+            return new File(flinkConfig.get(CoreOptions.TMP_DIRS), "/fluss").getAbsolutePath();
+        }
+        return System.getProperty("java.io.tmpdir") + "/fluss";
+    }


luoyuxia

Left three inline comments on configuration propagation, multiple Flink temporary directories, and documentation for the new option.

luoyuxia · 2026-07-18T05:46:02Z

+                sourceReaderContext,
+                connection,
+                lakeTieringFactory,
+                getLakeTieringIoTmpDir(flussConf, sourceReaderContext.getConfiguration()));


Could we pass or read this option from lakeTieringConfig instead? FlussLakeTiering puts CLI keys starting with lake.tiering. into lakeTieringConfig, while flussConf only contains --fluss.* keys after removing the fluss. prefix. LakeTieringJobBuilder currently passes only flussConfig to TieringSource, so a standard --lake.tiering.io.tmpdir /data/tmp argument is absent here and always falls back to the Flink/JVM temporary directory. Please wire lakeTieringConfig (or the resolved option) into the source.

luoyuxia · 2026-07-18T05:46:02Z

+    public static String getFlinkIoTmpDir(
+            org.apache.flink.configuration.Configuration flinkConfig) {
+        if (flinkConfig.contains(TMP_DIRS)) {
+            return new File(flinkConfig.get(CoreOptions.TMP_DIRS), "/fluss").getAbsolutePath();


CoreOptions.TMP_DIRS can contain multiple directories separated by comma, pipe, or the platform path separator. This code treats the entire raw value as one File parent, and Paimon later calls IOManager.create(String), which also treats it as one directory. For example, /disk1/tmp,/disk2/tmp is handled as one combined path instead of two temporary directories. Please parse the value with Flink ConfigurationUtils.parseTempDirectories() and either select one valid directory or pass the resulting directories to Paimon as an array. A multiple-directory test would catch this case.

got it. But I think maybe we should merge this PR #3617 firstly, and then I will change this on that baseline.

luoyuxia · 2026-07-18T05:46:02Z

                                    + ConfigOptions.TABLE_DATALAKE_AUTO_EXPIRE_SNAPSHOT
                                    + " is false.");

+    public static final ConfigOption<String> LAKE_TIERING_IO_TMP_DIR =


Could we also add this user-facing option to the Datalake Tiering Service Options table in website/docs/maintenance/tiered-storage/lakehouse-storage.md, together with a CLI example? That table currently lists only lake.tiering.auto-expire-snapshot, so users have no documentation showing how to set the new lake.tiering.io.tmpdir option.

luoyuxia

@zuston Thanks for the pr. Left minor comments

luoyuxia · 2026-07-18T06:01:35Z


    private final Connection connection;

    public TieringSourceReader(


public TieringSourceReader(
FutureCompletingBlockingQueue<RecordsWithSplitIds<TableBucketWriteResult>>
elementsQueue,
SourceReaderContext context,
Connection connection,
LakeTieringFactory<WriteResult, ?> lakeTieringFactory) {
this(elementsQueue, context, connection, lakeTieringFactory, (String) null);
}

is not called anymore, seems can be removed.

luoyuxia · 2026-07-18T06:02:12Z

+            SourceReaderContext context,
+            Connection connection,
+            LakeTieringFactory<WriteResult, ?> lakeTieringFactory,
+            @Nullable String ioTmpDir) {


When it'll be null? Can we remove Nullable notation?

[tiering] introduce the option of 'lake.tiering.io.tmpdir' to avoid d…

c291201

…isk exhaustion

zuston marked this pull request as ready for review July 3, 2026 10:04

zuston changed the title ~~[tiering] introduce the option of 'lake.tiering.io.tmpdir' to avoid disk exhaustion~~ [tiering] introduce the option of lake.tiering.io.tmpdir to avoid disk exhaustion Jul 3, 2026

zuston commented Jul 3, 2026

View reviewed changes

luoyuxia requested a review from Copilot July 18, 2026 03:43

Copilot started reviewing on behalf of luoyuxia July 18, 2026 03:43 View session

Copilot AI reviewed Jul 18, 2026

View reviewed changes

luoyuxia reviewed Jul 18, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[tiering] introduce the option of `lake.tiering.io.tmpdir` to avoid disk exhaustion#3569

[tiering] introduce the option of `lake.tiering.io.tmpdir` to avoid disk exhaustion#3569
zuston wants to merge 1 commit into
apache:mainfrom
zuston:fixtmp2

zuston commented Jul 3, 2026

Uh oh!

zuston commented Jul 3, 2026

Uh oh!

zuston Jul 3, 2026

Uh oh!

zuston commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

luoyuxia left a comment

Uh oh!

luoyuxia Jul 18, 2026 •

edited

Loading

Uh oh!

luoyuxia Jul 18, 2026

Uh oh!

zuston Jul 20, 2026

Uh oh!

luoyuxia Jul 18, 2026

Uh oh!

luoyuxia left a comment

Uh oh!

luoyuxia Jul 18, 2026

Uh oh!

luoyuxia Jul 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		private final Connection connection;

		public TieringSourceReader(

Uh oh!

Conversation

zuston commented Jul 3, 2026

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

zuston commented Jul 3, 2026

Uh oh!

zuston Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

zuston commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

luoyuxia Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luoyuxia Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

zuston Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

luoyuxia Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

luoyuxia Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

luoyuxia Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

luoyuxia Jul 18, 2026 •

edited

Loading