Core, Spark: Clean up uncommitted files when a staged table is aborted

[wombatu-kun]

Summary

StagedSparkTable.abortStagedChanges() was an empty // TODO: clean up. Spark calls it to roll back an atomic CTAS/RTAS, and it is also used by the snapshot/migrate actions. Iceberg's internal transaction cleanup only runs when commitTransaction() is actually invoked and then fails. When a failure happens after the staged write but before commitStagedChanges() is called (for example, SnapshotTableSparkAction/MigrateTableSparkAction failing while importing data, or an interrupted CTAS), nothing cleaned the manifest list and manifests already written into the uncommitted transaction. For a staged CREATE the table is never registered, so those orphans have no table metadata pointing at them and are unreachable by removeOrphanFiles - they leak permanently.

Changes

• Add a best-effort default void abortTransaction() to the Transaction API. The default is a documented no-op, so existing implementations are unaffected.
• Override it in BaseTransaction to run the existing cleanUp() (cleanAllUpdates() + deleteUncommittedFiles()) - the same cleanup Iceberg already performs when a create/replace transaction's own commit fails. The override is skipped once commitTransaction() has been attempted (see below).
• Delegate it in CommitCallbackTransaction; the post-commit callback is intentionally not run on abort.
• Wire StagedSparkTable.abortStagedChanges() to transaction.abortTransaction() in spark/v4.1. The Spark-side change is identical for 3.5 and 4.0; a backport PR will follow.
• Add the corresponding java.method.addedToInterface entry to .palantir/revapi.yml.

A no-op default (rather than throwing) is used because abortTransaction() runs from catch/finally blocks where a secondary exception would mask the original failure. The underlying deletion is best-effort and idempotent (CatalogUtil.deleteFile swallows NotFoundException, cleanAllUpdates() suppresses failures).

abortTransaction() does nothing once commitTransaction() has been attempted. Spark calls abortStagedChanges() from the catch block wrapping commitStagedChanges(), so an abort can follow a commit that threw CommitStateUnknownException after succeeding; cleaning up there would delete metadata the committed snapshot references. The commit path already performs whatever cleanup is safe, guarded by CommitStateUnknownException and TableOperations.requireStrictCleanup().

Out of scope

Executor-written data files in the write-succeeds-then-commit-fails path are not deleted here. That matches Iceberg's existing create-transaction behavior, and those files are handled by SparkWrite.abort() on write-job failure.

Testing

• New abort case in core TestCreateTransaction (runs across all format-version templates): stages a create transaction, performs an append, asserts the manifest and manifest list exist, calls abortTransaction(), asserts they are removed and the table is not created, and verifies a second abortTransaction() does not throw.
• New TestStagedSparkTable in spark/v4.1: stages a CREATE via SparkCatalog.stageCreate, routes an append into the staged transaction, calls abortStagedChanges(), and asserts the uncommitted manifest/manifest-list files are deleted and the table is not created. The session-catalog parameterization is skipped (it produces RollbackStagedTable, not StagedSparkTable). 3 configs (hive/hadoop/rest) pass, 1 skipped, 0 failures.
• New TestReplaceTransaction.testAbortTransactionAfterUnknownStateKeepsCommittedFiles: runs a create transaction whose commit succeeds but reports CommitStateUnknownException, then calls abortTransaction() and asserts the committed manifest and manifest list survive.
• ./gradlew revApiCheck, spotlessApply -DallModules, and the tests above all pass.

---

AI Disclosure

• Model: Claude Opus 5 (1M context)
• Platform/Tool: Claude Code
• Human Oversight: fully reviewed
• Prompt Summary: Implement StagedSparkTable.abortStagedChanges, which was an empty "// TODO: clean up", so aborted staged tables clean up their uncommitted files.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[wombatu-kun]

not stale

[wombatu-kun]

@RussellSpitzer @rdblue this has been open since May 18 with no review (it already collected a stale-bot warning). CI is green (56/56) and it still merges cleanly on current main, so it is only waiting on a reviewer.

StagedSparkTable.abortStagedChanges() is an empty // TODO: clean up, so a failed atomic CTAS/RTAS leaks the manifests and manifest list already written into the uncommitted transaction. For a staged CREATE the table is never registered, so removeOrphanFiles cannot reach those files and they leak permanently. The fix adds a best-effort no-op Transaction.abortTransaction() default and wires it up.