Resolve rest-of-RAM heaps to the main thread's stack base

[TheTharin]

Problem

The SetupHeap/EndOfHeap syscall handlers and the runtime's guest-heap machinery capped the guest heap at a fixed 0x01F00000 (31MB). On real hardware there is no such ceiling: the kernel resolves a "rest of RAM" heap (heap_size = -1) to the main thread's stack base, which sits at the top of the 32MB.

Games commonly link their CRT with heap_size = -1 and then size a master allocation from whatever EndOfHeap reports. With a fixed 31MB cap such games lose memory they own on retail hardware — and the cap was also internally inconsistent: SetupThread here already places the default main stack at PS2_RAM_SIZE - stack_size (0x1FD0000 for a typical 192KB stack), so there was an 832KB dead zone between the heap ceiling and the stack that neither could use.

What the retail kernel does

• SetupThread(gp, stack, stack_size, ...) — with stack = -1, the stack is placed at the top of RAM; the kernel records the stack base in the thread state.
• SetupHeap(heap_base, heap_size) — with heap_size = -1, the heap end resolves to that recorded stack base. An explicit size resolves to heap_base + heap_size, unclamped.
• EndOfHeap() — returns the resolved heap end.

References:

• ps2tek's BIOS EE syscall documentation (reverse-engineered from the retail kernel) states it directly (EE Syscalls):

  0x3C InitMainThread: "Returns the stack pointer of the thread. If stack == -1, the stack pointer equals the end of RDRAM - stack_size. Else, it equals stack + stack_size."

  0x3D InitHeap: "Initializes the current thread's heap. If heap == -1, the end of the heap resides at the thread's stack pointer. Else, the end of the heap is heap + heap_size. Returns the end of the thread's heap."

  0x3E EndOfHeap: "Returns the current thread's heap [end]."

• Play! — the only other complete open-source EE-kernel HLE, validated against a large retail compatibility corpus — implements the same rule in Source/ee/PS2OS.cpp:

  // sc_SetupThread
  thread->stackBase = stackAddr - stackSize;
  // sc_SetupHeap
  if(heapSize == 0xFFFFFFFF) { thread->heapBase = thread->stackBase; }
  else                       { thread->heapBase = heapBase + heapSize; }
  // sc_EndOfHeap
  m_ee.m_State.nGPR[SC_RETURN].nV[0] = thread->heapBase;

  with a comment on the stack placement that "some games rely on the stack and heap being a very precise size".
• Retail software is the empirical proof: Rogue Galaxy ships with SetupHeap(base, -1) and a master allocation that only fits if the heap runs to the stack. It boots on real hardware and on PCSX2 — which performs no kernel HLE at all (it requires a BIOS dump and executes the original kernel's MIPS code), so its behavior is the retail kernel's. A fixed 31MB ceiling would make this retail disc unbootable on a real PS2.
• ps2sdk's crt0.c (reimplementation of the standard SCE startup) calls SetupHeap(&_end, (int)&_heap_size) with _heap_size conventionally -1, i.e. "rest of RAM up to the stack" is the default contract for real software.

Note on the exact resolution point: ps2tek describes InitHeap(-1) as resolving to "the thread's stack pointer" (for stack=-1 that is end of RDRAM - stack_size), while Play! resolves to the stack base below a 4KB top pad. This runtime's SetupThread already computes the ps2tek value (PS2_RAM_SIZE - stack_size), so this PR records and reuses exactly that — consistent with ps2tek's wording and conservative (never above the stack) in the explicit-stack case.

Reproducible failure case: Rogue Galaxy (SCUS-97490)

The game's CRT calls SetupHeap(0x528200, -1) with a 192KB main stack. Early in boot, GameMain sizes a master arena as "total budget minus two pools" = 0x18B0000 bytes (25.9MB) and allocates it with memalign — at that point the heap break is already at ~0x6B9230, so the heap must reach 0x1F69230. Under the 31MB cap the allocation fails by ~430KB; the game does not check the result, stores the null pointer into its arena descriptor, and its arena-clear loop memsets 26MB starting at guest address 0 — wiping the game's own loaded image (the recompiled code keeps running, so the failure shows up much later as an infinite sceCdSearchFile("") spin over a zeroed path table).

With this change the allocation succeeds and the game proceeds to load its first overlay (BIN/TITLE.BIN).

Implementation

Following Play!'s shape (record the stack base, resolve -1 against it):

• SetupThread now records the main thread's stack base in the runtime (PS2Runtime::setGuestMainStackBase). It already computed the value; this just stores it.
• SetupHeap passes heapLimit = 0 ("rest of RAM") to configureGuestHeap for heap_size 0/-1, and heap_base + heap_size (clamped only to PS2_RAM_SIZE) for explicit sizes.
• configureGuestHeap's limit resolution (clampGuestHeapLimit) treats 0 as "resolve to the recorded stack base"; before SetupThread has run it falls back to the old bounded default (kGuestHeapFallbackLimit = 0x01F00000, now used only as that fallback).
• EndOfHeap returns guestHeapLimit() as before — which now reports the resolved value.
• The async-callback stack floor is raised to the configured heap limit in configureGuestHeap, so runtime-reserved callback stacks always stay above the heap.

Behavior for explicit heap sizes and for the pre-SetupThread window is unchanged.

Tests

Two new cases in ps2xTest/src/ps2_runtime_kernel_tests.cpp:

• "SetupHeap with size -1 runs the heap up to the main stack base" — SetupThread(stack=-1, size=0x30000) then SetupHeap(base, -1): EndOfHeap must return PS2_RAM_SIZE - 0x30000, and a 25.9MB master arena (Rogue Galaxy's exact allocation) must succeed.
• "SetupHeap with size -1 stays bounded before SetupThread runs" — without a recorded stack base, EndOfHeap reports the bounded fallback.

The full suite passes with this change (427/427 on the final run). During testing, one unrelated case — "VU0 macro mappings cover all S1/S2 enums" — failed intermittently both on unmodified main and with this change applied, so it appears flaky independently of this PR.

Verified on a real game as well: with this change Rogue Galaxy (SCUS-97490) gets its 25.9MB arena (memalign -> 0x6B9230, arena descriptor showing base 0x6B9230 / count 0x18B000 quadwords) and boots past the point where it previously destroyed itself, proceeding to load BIN/TITLE.BIN.