[SPARK-59631][SQL] Preserve cached V1 time-travel snapshots after table writes

[joelrobin18]

What changes were proposed in this pull request?

This PR adds FileIndex.isTimeTravel, defaulting to false, so V1 file data sources can
identify immutable time-travel snapshots.

CacheManager honors this signal during write-driven cache invalidation by table name and path.
InsertIntoHadoopFsRelationCommand marks its path-based recache as write-driven. Existing explicit
refresh operations continue to include time-travel snapshots.

Why are the changes needed?

SPARK-53732 changed write-driven cache
invalidation from plan matching to table-name matching with includeTimeTravel = false. V2
relations honor that flag through DataSourceV2Relation.timeTravelSpec, but V1
LogicalRelationWithTable was matched only by table name.

For providers that lower a pinned read to LogicalRelation(HadoopFsRelation(FileIndex)), the
time-travel state survives only in the FileIndex. A write to the live table therefore cleared and
rebuilt the pinned cache. The DataFrame remained registered as cached, but the next action
re-executed its plan, including expensive or non-idempotent UDFs.

The new signal closes this V1 gap without coupling Spark to a specific data source.

JIRA: SPARK-59631

Does this PR introduce any user-facing change?

Yes. For V1 data sources that identify an immutable snapshot, appending to or overwriting the live
table no longer recomputes an already materialized cache for that snapshot. Live-table caches are
still refreshed after writes, and explicit refresh or uncache operations still include pinned
snapshots.

Before:

is_cached=true
udf_calls=10->20
tokens_same=false

After the coordinated Spark and provider changes:

is_cached=true
udf_calls=10->10
tokens_same=true

How was this patch tested?

Added V1TimeTravelCacheSuite with six cases covering name- and path-based invalidation for pinned
and live V1 relations, including automatic write invalidation and explicit refresh.

build/sbt "sql/testOnly org.apache.spark.sql.V1TimeTravelCacheSuite"

All six tests passed. The existing V2 time-travel cache test, SQL MiMa, and main/test Scalastyle
checks also passed.

End-to-end testing reproduced 10->20 UDF executions with unpatched Spark and preserved 10->10
with the coordinated Spark/provider changes. The provider integration matrix passed all four tests
in both AUTO and NONE modes.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (gpt-5.6-sol)

[cloud-fan]

Review summary

The cache-invalidation design is coherent and mirrors the existing V2 immutable-snapshot policy; I did not find a production correctness defect. The remaining items are two regression-sensitivity gaps at the production V1 caller and V2 path branch, plus an inaccurate Scaladoc contract on the new overload. They are bounded, non-blocking follow-ups and do not call for a broader refactor.

Findings

3 total: 0 P0, 0 P1, 2 P2, 1 P3.

Non-blocking (P2)

• Exercise the production V1 write handoff — sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala:216 — see inline.
• Cover V2 time-travel path recache — sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:633 — see inline.

Nit (P3)

• Document the time-travel exclusion in recacheByPath — sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:600 — see inline.

[cloud-fan]

Nit (P3): This overload says it tries to recache all matching entries, but includeTimeTravel = false intentionally leaves matching immutable snapshots loaded through both the V2 relation guard and FileIndex.isTimeTravel. Please document the parameter's inclusive/exclusive meaning and qualify the all-entries claim.

[cloud-fan]

Non-blocking (P2): The added V1TimeTravelCacheSuite invokes recacheByPath(..., includeTimeTravel = false) directly; it never executes this production caller. Removing or changing this argument would restore inclusive invalidation after real V1 writes while all six new tests still pass. Please add a regression that performs a real V1 file write with an already materialized immutable-snapshot cache and verifies that the snapshot stays loaded while a live cache refreshes.

[cloud-fan]

Non-blocking (P2): This is a new V2 path-invalidation exclusion, but the added suite only builds V1 LogicalRelation fixtures and the existing V2 time-travel test exercises recacheTableOrView. Please add path-based coverage that reaches an ExtractV2Table time-travel relation, distinguishes write-driven from inclusive refresh, and fails if this timeTravelSpec guard is removed.