[GLUTEN-13044][CORE] Fix exceedsMaxMemoryUsage to use actual used heap instead of committed heap

[r7raul1984]

Fixes #13044

DynamicOffHeapSizingMemoryTarget.exceedsMaxMemoryUsage() previously used Runtime.getRuntime().totalMemory() (committed heap) as the on-heap memory pressure indicator. Committed heap is the memory the JVM has reserved from the OS, which includes a potentially large
amount of free pages that have not yet been returned. As a result, the check overstates memory pressure between Spark stages and triggers spurious OOMs.

Example of the failure:

• Stage 0 runs with 2 concurrent tasks (~2 GB peak each); JVM commits ~8.9 GB of a 9 GB heap
• Stage 0 finishes; actual heap usage drops to ~2 GB, but totalMemory() remains ~8.9 GB because the JVM has not yet returned the committed pages to the OS
• Stage 1 starts and requests 8 MB: 8 MB + 0 + 8.9 GB >= 9 GB → OOM triggered
• Actual free heap was ~6.5 GB (freeMemory()); the allocation should have succeeded

Fix:

Replace totalMemory() with totalMemory() - freeMemory() (actual used heap) at all four call sites of exceedsMaxMemoryUsage() — three in borrow() and one in shouldTriggerAsyncOnHeapMemoryShrink(). The parameter is renamed from totalOnHeapMemory to usedOnHeapMemory for
clarity.

// Before
exceedsMaxMemoryUsage(totalHeapMemory, usedOffHeapMemory, size, 1.0)

// After
exceedsMaxMemoryUsage(totalHeapMemory - freeHeapMemory, usedOffHeapMemory, size, 1.0)

Note: freeHeapMemory is already read from Runtime.getRuntime().freeMemory() at the top of borrow(), so no additional JVM calls are introduced.

How was this patch tested?

Existing unit tests in DynamicOffHeapSizingMemoryTargetTest (added in #12425) continue to pass. New tests covering the spurious-OOM scenario between stages will be added as a follow-up.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Sonnet 4.6, Anthropic

[github-actions]

Run Gluten Clickhouse CI on x86