feat(rocm): AMD ROCm support (gfx1100) + Qwen3.5-MoE GGUF loader - #217
Open
samuelishida wants to merge 7 commits into
Open
feat(rocm): AMD ROCm support (gfx1100) + Qwen3.5-MoE GGUF loader#217samuelishida wants to merge 7 commits into
samuelishida wants to merge 7 commits into
Conversation
added 7 commits
August 25, 2026 19:15
Bring FreeToken up on AMD ROCm (RX 7900 XTX / gfx1100) alongside CUDA via a thin device seam. Includes: - Device detection / architecture gating (is_rocm), build toolchain for tvm-ffi/HIP JIT, --offload-arch=gfx1100 pinning, and HIP backend dispatch. - Pinned-memory + graph-capture gating (ROCm settles to kernel-launch decode). - device_api.h seam, GGUF kernel HIP port, quant mapping, nvfp4->mxfp4. - qwen3_5_moe GGUF loader (config/dense weights/expert offload). - fix(gguf): de-interleave the GDN mrope_interleaved value heads (in_proj_qkvz v/z rows, out_proj cols, in_proj_ba, conv1d v-channels, dt_bias; A_log stored as A=-exp(A_log)) so weights match HF and the model serves correct text. - torch attention backend (ground-truth reference) + ROCm regression tests. - AOT CI (rocm.yml) and docs/install-amd.md. Tests: ROCm suite passes; qwen35voe de-interleave + torch-backend tests added.
…ncoding
The GGUF->fast-tokenizer converter relies on BPE merges to keep special
strings whole: <im_start>-style tokens happen to be merge entries and
survive, but think tags / tool-call tags are NOT in the merge table, so
they silently split into plain pieces ('<th','ink','>'). The model then
receives garbage ids it never saw in training and answers with gibberish
+ EOS -- observed as 'reasons a little, returns empty content' on Qwen3.6
GGUF checkpoints.
_register_control_tokens re-registers every ggml token_type 2/3/4 string
that does not already encode atomically, against its EXISTING vocab id:
vocab size and id assignments never change, already-atomic tokens are
untouched (no-op on healthy setups), and safetensors/HF checkpoints never
touch this path. Hardware-independent; safe for CUDA by construction.
Also: chat-template resolution now mirrors official HF configs --
chat_template.jinja sidecar next to the .gguf wins, then
FT_CHAT_TEMPLATE_REPO (hf_hub_download), then the embedded metadata.
CPU-only regression tests included (synthetic BPE tokenizer, no GPU or
download needed).
_infer_tool_call_parser only special-cased qwen3_5/coder names, so a Qwen3.6-* filename fell through to qwen25 (legacy JSON grammar). The 3.5/ 3.6 hybrid family instructs the XML invoke-block format in its chat template (<function=name> + <parameter=k> blocks -- the Qwen3-Coder grammar), so the model emitted well-formed calls the wrong detector could not see: output swallowed, empty stop. Add qwen3_6/qwen3.6 markers.
- serve-qwen-moe.sh: single-line array-based launch (backslash-newline continuations get mangled by VS Code shells), setsid detach so a cancelled task cannot kill the server mid-load, CRLF guard, status/log subcommands, 128k KV default (~2.5 GiB on the hybrid arch). - mirror-hf-configs.sh: mirror official HF chat_template/generation_config next to a local GGUF without touching the file. - kill-freetoken.sh: hard-stop helper. - .vscode/: tasks (serve start/stop/status/log, fast tests), debugpy launch config for the server, LF pinning for shell scripts. - .gitignore: nohup.out, .plans/.
… penalties Root cause of the intermittent empty/loop responses and the non-deterministic 'greedy' decode: the GGUF lm_head (GGUFLinear) returned FULL prefill logits [total, vocab] without gathering each request's final row, while the engine contract is [batch.size, vocab] (the token to sample after each request's prompt). ParallelLMHead and Nvfp4LMHead already gather via attn_metadata.get_last_indices(); the GGUF path did not. Consequences fixed: - The first generated token was sampled from POSITION 0's logits (after '<|im_start|>' -> 'user') instead of the last prompt position (after '...assistant\n thinking\n' -> 'Here'). - Worse, a fresh prefill (142-token batch, row 0 = prompt token 0) and a radix-cache continuation (14-token batch, row 0 = prompt token 128 -> '<|im_end|>' -> immediate stop, EMPTY content) sampled different rows, so the SAME greedy request gave different outputs depending on cache state -- the 'model thinks then returns empty' report. Fix: qwen3_5_moe/model.py gathers last_indices on prefill before the GGUF lm head (matches ParallelLMHead/Nvfp4LMHead; shared code path for both backends). Also in this change: - Presence/frequency penalties were accepted by the API but IGNORED by the sampler. Implemented end-to-end (core.py SamplingParams + Req.prompt_len, generation.py/openai_api.py pass-through, engine/sample.py apply_penalties over generated tokens only, engine.py passes the batch to sample()). Applies to greedy too. Breaks reasoning loops by penalizing repeated tokens. - Serve script: --max-output-tokens 65536 default (FT_MAX_OUTPUT knob) so the reasoning model has room to finish; keeps the Inc-1 diagnostic knobs (moe stats, prefill overlap, cpu layers). Verified: - 28k-token prompt now ANSWERS (finish=stop, content='Four'), no empty, no loop. - Tool calls still return tool_calls with correct args. - TRUE greedy (top_k=1, top_p=1.0) deterministic within a cache state; the first-after-startup fresh run differs from radix hits only at ~1 bf16 ULP (continuation GEMM batch shapes) -- documented residual, both outputs valid. - tests/engine + tests/server: 618 passed, same 15 pre-existing failures.
- Remove .github/workflows/rocm.yml (no GH Actions for ROCm) - Remove kernel/tinygrad_fallback.py (dead: nothing imports it) - Remove .vscode/launch.json (hardcoded machine model path) - kernel/utils.py: drop shadowed duplicate _build_stamps definition - kernel/backend.py: drop unused _CUDA_ONLY_PACKAGES - models/qwen3_5_moe/gguf.py: drop unused _q8_0_down_row_bytes - moe/nvfp4_to_mxfp4.py: drop unused _nearest_e2m1_codes/_FP4_SORT_SIGN; assert K is a block multiple instead of silently truncating - moe/expert_banks.py: remove dead dummy-path import in _gguf_banks - models/qwen3_5_moe/gdn.py: rename _fp8 -> _split_proj (also covers GGUF) - engine/engine.py: drop redundant local is_rocm imports; build infos once - utils/graph_gate.py, engine/graph.py, attention/torch.py, tests: strip plan-increment (Inc N) references - docs/install-amd.md, pyproject.toml: drop plan refs; align ROCm torch index - kernel/gguf.py: honor FREETOKEN_KERNEL_CACHE_GFX for the JIT offload-arch - scripts: remove hardcoded /media/smk model paths (require FT_MODEL/arg) - utils.cuh: fix stray [[unlikely]]; statement
…-support # Conflicts: # python/freetoken/engine/engine.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds AMD ROCm (HIP) support targeting gfx1100 GPUs, plus a GGUF loader for Qwen3.5-MoE.
Highlights
device_api.h,utils.cuh), HIP toolchain plumbing (_toolchain.py,backend.py), torch fallback attention backend, pinned-tensor & batch-memcpy HIP pathsmodels/qwen3_5_moe/gguf.py), dequantization support, special-token registration fixesscripts/serve-qwen-moe.sh,mirror-hf-configs.sh,kill-freetoken.shdocs/install-amd.mdCommits
9741646feat(rocm): AMD ROCm support (gfx1100) + qwen35moe GGUF loader6f4974efix(gguf): register unmerged CONTROL/USER_DEFINED tokens for atomic encodingbc62324fix(server): route Qwen3.6 to the qwen3_coder tool-call parser25d7bd8chore(local): ROCm serve scripts + VS Code tasks/debug configs729789afix(qwen3.5-moe): GGUF lm_head last-token gather + presence/frequency penalties1f76e88chore(rocm): drop CI workflow + tinygrad fallback; review cleanup