Conversation
* ui : add browser-style conversation tabs store Track open conversation tabs in order, persisted to localStorage and pruned against the loaded conversation list on init. The chat layout syncs the route's tab on every navigation, so any way of reaching a conversation opens a tab for it. * ui : add temporary new-chat tabs New-chat tabs are unsaved conversations carrying a temporary id used directly as the route (#/chat/<id>). They live in memory and are only persisted to the database - keeping the same id so the route and tab stay stable - when the first message is sent. Deleting one drops it without confirmation, and deleting conversations now closes their tabs. * ui : render conversation tab bar in chat layout Desktop-only tab bar above the chat screen, one tab per open conversation or new-chat tab. The active tab follows the route id; clicking navigates, middle-click or the close button closes (switching to the left neighbor), and a trailing + starts a new chat. Tabs appear only on chat-id routes; the bare #/ new-chat view has none. The bare route stays put unless a prompt/model deep-link routes it to a new-chat tab. * ui : route new-chat entry points through tabs The sidebar New chat item, Cmd+Shift+O, the search page and the arrow-key fallback now open a new-chat tab instead of navigating to the ?new_chat URL, which is removed. New chat is no longer a special route but a tab like any other conversation. * ui : track sidebar expanded state in a shared ui store Move the desktop sidebar expanded/collapsed state out of deviceStore into a dedicated uiStore so the chat tab bar can react to it. Assisted-by: pi * chat : add opt-in conversation tabs setting Add a Display setting that turns browser-style conversation tabs on or off, enabled by default. Assisted-by: pi * chat : add browser-style conversation tabs with a new-chat screen Track open conversations as tabs above the chat, one per open chat, plus a single New chat tab for the bare `#/` route. New chat is just the `#/` screen - no temporary conversations - and its tab is dropped when navigating away. Sending the first message creates a real conversation and opens a tab for it. Assisted-by: pi * chat : turn tab bar into a horizontally scrollable carousel Make the tab bar a horizontally scrollable carousel with edge scroll buttons and active-tab centering, and align its styling with the sidebar. Assisted-by: pi * chat : restyle the scroll-to-bottom button to match tab styling Assisted-by: pi * chat : add close-tab keyboard shortcut Assisted-by: pi * chat : soften tab bar fade and dim inactive tabs Assisted-by: pi * feat: Add stop button to tabs * refactor: Componentize * ui : fix carousel scrollability detection Observe the content wrapper as well as the container, since adding overflowing items does not change the container's own box size. Also expose an onScrollableChange callback. Assisted-by: pi * ui : add unified ScrollCarousel component Single carousel component with top/center variants, gap and scroll options, and hover-revealed chevrons. Rename the HorizontalScrollCarousel accessibility story accordingly. Assisted-by: pi * ui : migrate carousels to ScrollCarousel Switch the settings mobile header, attachments list, thumbnail strip, and MCP resources to the unified component, and drop HorizontalScrollCarousel. Assisted-by: pi * ui : improve chat tabs carousel UX Scroll newly added tabs into view, fade overflowing tabs at the edges, and hide the New chat button while a new-chat tab is open. Assisted-by: pi * refactor: Naming * chat : add keyboard shortcut to jump between conversation tabs Shift+Cmd/Ctrl+Left/Right cycles the open tabs, mirroring the existing Shift+Cmd/Ctrl+Up/Down conversation navigation. Assisted-by: pi * chat : make the whole tab item act as a link The full tab is now a link instead of only the inner label button, while the stop and close buttons stay interactive by swallowing their clicks. Assisted-by: pi * chat : adjust tab bar width and use a shared offset variable Widen the tab bar for the expanded sidebar and rename the tab bar height variable to --chat-tabs-offset with a smaller value so the chat screen min-height accounts for the overlay without overshooting. Assisted-by: pi * chat : account for the tab bar offset in the assistant min-height Subtract the tab bar offset when it is shown so the last assistant message does not overflow the available viewport space. Assisted-by: pi * refactor: Post-review fixes * ui : restore deep links on the chat start page - handle ?model selection, with ?load=true eager router loading - ?q now creates a conversation, sends the prompt, and clears the params - show the not-available-model dialog for unknown models - never block mount on the conversation list Assisted-by: pi * ui : fix tab item link nesting and centralize tab constants - the tab anchor covers the whole item while stop/close stay siblings, so interactive elements are never nested inside the anchor - cmd/ctrl/middle clicks are left to the browser (new window) - extract the tab labels, the active-tab data attribute, and the sidebar-offset max widths into constants Assisted-by: pi * ui : tidy scroll carousel hook and keep mobile header arrows on - drop the dead scrollLeft/scrollRight helpers and the unused onScrollableChange/scrollBy props - init the carousel once instead of inside a derived - restore items-start on the center variant - always show the settings header arrows on touch Assisted-by: pi * ui : keep the new-chat tab across reloads and fall back on close - the new-chat sentinel is no longer pruned on init, so reloading on the bare new-chat route keeps the tab the user is on - closing the active conversation falls back to the new-chat screen when Conversation tabs are off Assisted-by: pi * ui : don't block startup on the conversation list - prune persisted tabs after the list loads in the background instead of awaiting it during init - openNewChat now returns void; its return value was never read Assisted-by: pi * ui: fix routing nits * chore: Update doc comments * refactor: Mark fire-and-forget openNewChat calls as `void` * chat: fix the deep-linked prompt, the tab width and the tab shortcuts The chat start page creates the conversation and hands the prompt over to the chat route, which still sees it in the query string. Sending it on both sides queues the second copy as a pending message, which shows up as a stray user bubble once the answer lands and vanishes on reload since it never reaches the database. The tab bar takes the max width of the collapsed sidebar while it is expanded, and the other way round. The tab list is pruned against a snapshot of the loaded conversations, so a conversation created while that list is still loading loses its tab even though the route just opened it. The active tab then falls out of the list and the cycling shortcut jumps to an edge on every keypress instead of moving one tab over. Tabs synced from the route are kept as they are, only the persisted ones are pruned. The rich chat input claims ctrl or alt with shift and an arrow for its badge-aware word jump, which now belongs to the tab cycling shortcut. Holding shift hands the key combination over, the plain word jump is unchanged. The close-tab shortcut consumes the event before checking whether the setting is on, and the logo background loses its importance flag. --------- Co-authored-by: Pascal <admin@serveurperso.com>
…7574) * ggml : fix meta tensor split state propagation * Add test-llama-archs to CI
* DeepseekV4: fix rollback with multi-seq * fix model loading * make pending rollback single use * only clear cache for seq_id for full load * add assert for compress ratio * make graph topology static * pass true instead of flags in clear_compressed * cont : clean-up + TODOs --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…org#26692) The device_info loop iterates over the discovered devices and gets the available and total memory counts. With the CUDA backend (and possibly others too) this requires creating a GPU context, which, in case of CUDA, results in a 550 MB VRAM allocation. For this information to be used in any way, the log verbosity must be set to LOG_LEVEL_TRACE. If it's not, including in the default configuration, the contexts get created, memory sizes get queried, then the log function quietly discards the data. In certain cases the user may not want to use any GPU resources. The device_loop iteration is the only place touching the GPU that cannot be skipped. Fix by checking the verbosity level and skipping the loop if there would be no output.
Run test-llama-archs with 1 to 4 GGML_METAL_DEVICES, mirroring the existing CUDA runs, and dispatch the job unconditionally since the per-backend guards now decide what to run. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
…gml-org#27594) * mtmd: use pillow-accurate resize algo, correct resize_algo for all models * speed optimization
* tests : fix tests for multi-gpu environment * cont : not needed
* readme : update links * readme : update maintainer PRs list Add the new members of the `ggml-org` `maintainers` team to the author filter of the maintainer PRs link (nikwen, marty1885, Titaniumtown), keeping the canonical team ordering. The list now matches the team exactly (35 members). Assisted-by: pi:llama.cpp/Qwen3.8-27B
* DSV4: sm tensor * set coarser granularity for head splits * fix dspark * add model saving for dsv4 + allow dflash to return on specific device * add comment about dsv4 seq_rm * simplify * add shared expert delayed allreduce * remove special test for dsv4
…gml-org#27513) * mamba2 : flatten mamba2 in/out projections to dispatch gemm instead of gemv * mamba2 : remove redundant output reshape
* ggml : fix ggml_clamp * cont : update ggml-alloc
…ml-org#27602) * ci : apply ccache-clear with older/min/dry-run to all ccache jobs Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : install gh in ccache-clear if missing (container jobs) The ccache-clear action relies on the gh CLI, which is not present in container-based jobs. Install it on demand so those jobs can clear caches. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : install gh via apt repo in ccache-clear The install.sh script used previously is no longer served (404). Switch to the official GitHub CLI apt repository, which is still available. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : pass --repo to gh cache commands in ccache-clear In container jobs gh cannot auto-detect the repository from git, so gh cache list/delete fail with 'failed to run git: not a git repository'. Pass the repository explicitly via --repo using GITHUB_REPOSITORY. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : drop -new suffix from vulkan ccache key The -new suffix was only needed to force a fresh cache. With ccache-clear now evicting stale caches, the original key can be used again. The old ccache-vulkan-ubuntu-24.04-arm-new entries still match the ccache-clear key prefix and are cleaned up automatically. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : fix ccache-clear date parsing on macOS (BSD date) macOS ships BSD date, which has no -d option. The older cutoff check was silently disabled there: 'date: illegal option -- d' errors in the log and the loop was only stopped by the min limit, risking deletion of caches not older than the cutoff (e.g. saved by a concurrent job). Parse the ISO-8601 timestamps with GNU date when available and fall back to BSD date otherwise (TZ=UTC, fractional seconds dropped). Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : extract ccache-clear logic into scripts/ccache-clear.sh The composite action now consists of a dedicated step that installs the GitHub CLI when missing (e.g. in container jobs) and a thin step that calls the new script. The script follows the make-release-checks.sh conventions (usage/env header, set -euo pipefail, CLI flags) and only checks that gh is available. The action inputs are unchanged, so the workflow steps are untouched. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * ci : remove unused apple ccaches
* mtmd: video: fix moov at the end of file Co-authored-by: rkfg <rkfg@rkfg.me> * fix SIGPIPE * windows: handle broken pipe case --------- Co-authored-by: rkfg <rkfg@rkfg.me>
…ay be defined as K in flash_attn_decls.tmpl if KV_OVERLAP (ggml-org#27545) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* ggml : shorten virtual device naming in CUDA and Metal Assisted-by: llama.cpp:DeepSeek-V4-Flash-0731 * ggml-metal : build device description at init Assisted-by: llama.cpp:DeepSeek-V4-Flash-0731 * cont : naming
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
…27659) `repetition_penalty` is standard HF key for repetion penalty. Currently, only `penalty_repeat` is mapped, read `repetition_penalty` and map it to `metadata.sampling_penalty_repeat`.
* metal : per-op source split + parallel compile (ggml-org#24021) * preliminary extract common header * op source split * split metallib into 8 libs && load in parallel * derive kernel->library routing from functionNames * x-macro lib list + underscore filenames, dedup QK_NL, MRC fixes * op source split 8 to 20 * improve robustness of source fallback * clean up * change bool -> atomic_bool * only prepend headers that source actually includes * no semaphore, use GCD global queue * dedup library compile path, fix NSError lifetime, rename gla * relocate upstream concat/rope_back/repeat kernel changes into split files * move ggml-common.h from common.h into dequantize.h to shrink binary size --------- Co-authored-by: lvyichen <lvyichen@stepfun.com> * metal: add col2im_1d op (f32/f16/bf16) (ggml-org#25176) * metal : add set_rows with src0 f16 (ggml-org#25434) * metal : add CONV_2D_DW (depthwise convolution) support (ggml-org#21565) * metal : add Q2_0 support (ggml-org#25419) * metal: fuse snake activation (mul, sin, sqr, mul, add) (ggml-org#25459) * ggml-metal: FWHT kernel for metal backend (ggml-org#25924) * metal : port new kernels into the split sources Move the kernels added on master after the split (lightning indexer, DSv4 hyper-connections, silu_back, f16 bin ops, TQ2_0, the flash-attn KV dequantization pass, rope offset/inplace, ssm_scan rollback, packed q8_0 dequantization and the tensor-API mat-mat K clamp) into the corresponding kernels/*.metal sources. Copied verbatim, no functional change. --------- Co-authored-by: lvyichen <lvyichen@stepfun.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* metal : per-device tuned (Q, NE) for flash-attn vec (ggml-org#25750) * rebase Q-generic FA vec body from 01dc936 (ggml-org#23114) * add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133) * add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback * add FA vec (Q,NE) perf sweep * fill tuning result * fold family table into a per-family representative SKU * refactor tuning result format * extend FA vec tuning to quantized KV caches * sync fa vec tuner bucketing with runtime, use pointwise tuning regret * update tuned table * format and cleanup * prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active * add device id -> token lookup for the offline tuning tool * add ggml-metal-tuning skeleton * add op-agnostic perf cell + median timing for the tuner * add FA-vec graph build + tensor init to the tuner * tools : add FA-vec (Q,NE) sweep, compression and table emit * cool down and re-measure the dirty window on thermal drift * test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice * tools : document the Metal tuner, point the table comment at it * abort on unknown KV type, single-source fa_vec_legal_ne * cleanup * honor -o in the FA vec (Q,NE) slice * retune FA-vec (Q, NE) under a pointwise no-harm gate * cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…ml-org#27538) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* grammar : accept "\-" escape in character classes gbnf_escape_char_class() escapes '-' as "\-" but parse_char() rejected that escape, so generated tool-call grammars failed to parse. Assisted-by: Claude Code <claude@anthropic.com> * tests : add parser test for "\-" in char classes Assisted-by: Claude Code <claude@anthropic.com> * tests : add integration test for "\-" in char classes Assisted-by: Claude Code <claude@anthropic.com> * tests : drop integration and parser tests
* ggml : bump version to 0.22.0 * scripts : update default release desc
* move more jobs to ccache-buckets * add venv deps * also jq
* opencl: add Adreno xmem SDPA path Assisted-by: Codex * Removed the Adreno-specific queue profiling override * Clean up formatting * 修复数值误差优化gqa/mask attn Assisted-by: Codex * add env GGML_OPENCL_XMEM_SDPA Assisted-by: OpenAI Codex --------- Co-authored-by: happyyzy <happyyzy@users.noreply.github.com>
…g#27633) * opencl: add extended elementwise unary ops (sgn, step, elu, hardswish, hardsigmoid, floor, ceil, round, trunc) Adds nine GGML_UNARY_OP_* elementwise ops that were falling back to CPU on the OpenCL backend, following the same variant shape as the existing ABS op: f32, f32_4 (vec4), f16, f16_4 (vec4), and stride-addressed f32_nc / f16_nc for non-contiguous inputs. New kernels/unary_ext.cl (macro-generated), a shared ggml_cl_unary_ext dispatch helper mirroring ggml_cl_abs, the supports_op cases, and the compute-forward cases. Values are computed in float (the f16 variants read/write half and convert), so the conditional ops (step, elu) match the CPU reference; the vec4 forms use select() for the branch. Validated with test-backend-ops on Adreno 840 and 850 (E17): all nine ops pass every case including the vec4 and non-contiguous variants (8/8 or 14/14). * opencl: dispatch a contiguous f32 copy over the whole device kernel_cpy_f32_f32 maps one workgroup to each (i01,i02,i03) row and strides the row across that workgroup's lanes, and the host launches ne01*MIN(64,ne00) work items. A tensor with few long rows therefore runs on a single workgroup. The mamba2 and gated-delta-net recurrent state cache is one row of 524288 floats, copied once per layer per graph, and lands on 64 work items. When both sides are contiguous the copy is a linear move, so dispatch it over the whole device: one work item per float4. Gated on ggml_is_contiguous for both tensors and equal element counts, so copies already spread over many rows keep the existing path. The kernel is created optionally, so a driver that rejects it falls back rather than aborting. vload4/vstore4 rather than a float4 cast: they require only the scalar type's alignment, and these buffers carry an arbitrary 4-byte view offset. CPY, DUP and CONT are 217/217 on Adreno 840 and 740 with the path enabled and disabled. GGML_OPENCL_CPY_FLAT=0 forces the old kernel. * opencl: support all easy-copy types in CONCAT CONCAT was F32-only. Extend it to every "easy-copy" type -- any non-quantized type with a block size of 1 and an element size of 1, 2, 4 or 8 bytes, i.e. f16/bf16/i8/i16/i32/i64 as well as f32. The kernels are keyed by element SIZE rather than by type, which is what CUDA already does for the same op: one kernel per byte width (b1/b2/b4/b8) plus the packed b4 fast path, instead of one per ggml type. supports_op gates on the same property, so a new type of a supported width is picked up with no further work. Validated with test-backend-ops on Adreno 840 / A8X and X2-90 / X2E.
…25789) (ggml-org#28089) * ui : fix MCP image attachments not displayed in tool block (ggml-org#25789) Fixes regression from ggml-org#25450 where ChatMessageAgenticContent passed message.extra instead of section.toolResultExtras to tool blocks, leaving tool images invisible. Also fixes TOOL_RESULT_JSON_OPEN_REGEX which misclassified "[Attachment saved: ...]" as JSON. Fixes ggml-org#25789 Assisted-by: Muse Spark * Addressed PR comments: 1.- Removed ·?? mesage?extra· as it has no case left to cover 2.- Added ·[\· to cover the case of ·[[1, 2], [3, 4]]· case suggested in the PR comment 3.- Added unit test for covering up this regex case * ui : fix MCP image attachments not displayed in tool block (ggml-org#25789) - Addressed lint error on regex (redundant \)
* addition of m3 in fa_vec_tuned_table * adding q4_0,q4_1,q5_0,q5_1 in ggml-metal-tuning * Fix formatting in ggml-metal-tuning.cpp
…-org#27631) define two new environment variables to better understand how much memory is being allocated, and when. This has been invaluable in inproving the --fit algorithm, and is likely to be useful when debugging other memory-related issues. `-lv 4` will be required to enable the following: GGML_SYCL_MEMTRACE=1 will show per-site memory usage, updated whenever it increases by more than 64MiB. GGML_SYCL_MEMTRACE=2 will show every allocation and deallocation. To change the default 64MiB threshold for reporting memory usage increases, use GGML_SYCL_MEMTRACE_STEP. A sample log line: [SYCL-MEMTRACE] device memory query (dev): total 59493 MiB, free 4494, in use 54998; allocated 0 (buffers 0 + scratch 0), peak 0 MiB
…T support (ggml-org#28016) (ggml-org#28254) * Reapply "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (ggml-org#28184) This reverts commit c845263. * tests : fix unused variable M in test-backend-ops * tests: fix trailing space error and isolate kronecker tests for sycl backend only
Remove the build-time C++ helper and external gzip dependency, simplifying cross-compilation. Keep the generated C++ in templates for readability and preserve fully embedded UI assets. Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* common: add --log-jsonl * rename unknown to none
* ui : update active conversation fields in place updateCurrentNode, applyConversationUpdate, updateConversationTimestamp and the pin toggle replaced the whole activeConversation object, so its identity changed on every send, tool result and rename. ChatMessages tracks that identity to refresh sibling info, so each replacement triggered a full refetch of every message in the conversation. Write the changed fields instead, mirroring updateMessageAtIndex. Assisted-by: pi:zai-org/GLM-5.3 * ui : reuse the conversation load read for sibling info Opening a conversation read every message from the database twice: once in loadConversation for the active path, once in ChatMessages for the sibling map. Hand the freshly read array over once so the chat screen builds sibling info from it, and set the conversation and its messages in one sync block so effects never see the new conversation paired with the previous one's messages. Assisted-by: pi:zai-org/GLM-5.3 * ui : memoize leaf walks in sibling map build buildSiblingInfoMap resolves each sibling's leaf by walking the last-child chain, once per sibling per message, so the walk repeats along the same chains for every message in the conversation ( O(messages^2) on long chats ). Memoize leaf resolution per build with path compression so each edge is walked once. Assisted-by: pi:zai-org/GLM-5.3 * ui : skip sibling refetch for in-place message edits refreshAllMessages refetches every message of the conversation just to rebuild sibling info, but preserve-responses and non-branching assistant edits never create branches, so the sibling map stays valid. Refresh only after actions that branch (editWithBranching kept) or delete. Assisted-by: pi:zai-org/GLM-5.3 * ui : drop unused currentResponse reactive writes Nothing reads chatStore.currentResponse, but setChatStreaming reassigned it on every streamed chunk, so each token paid a reactive write and string assignment for nothing. Remove the field and the clearUIState wrapper that only reset it. Assisted-by: pi:zai-org/GLM-5.3 * ui : reuse completed agentic turn sections during streaming deriveAgenticSections runs in a $derived invalidated per streamed chunk, but re-derived every turn of the session each time, so per-chunk cost grew with session length. Cache completed turns keyed by their assistant message plus reference checks on every field that feeds derivation; only the streaming turn recomputes. Cache hits return the same section objects, so tool block props stay stable and skip their per-chunk re-derive. Assisted-by: pi:zai-org/GLM-5.3 * ui : share markdown block infrastructure Every markdown block duplicated shared work: a full copy of the hljs theme CSS per instance, and the remark/rehype plugin chain rebuilt on every processMarkdown call ( once per block at mount, again per coalesced chunk while streaming ). Use the single theme style element already maintained by SyntaxHighlightedCode, and build pipelines once - shared process-wide for attachment-less blocks, cached by attachments identity otherwise. Assisted-by: pi:zai-org/GLM-5.3 * ui : measure assistant layout only for the last message Every assistant message ran getComputedStyle, getBoundingClientRect and a ResizeObserver over the previous user bubble at mount, even off-screen ones, forcing a layout pass per message while a long conversation renders. The measured vars only feed the :last-child min-height rule, so gate the effect on isLastAssistantMessage; one measurement and one observer remain, and the effect re-runs when the last message changes. Assisted-by: pi:zai-org/GLM-5.3 * ui : trim whole-blob scans in tool block headers Tool block headers parsed their entire blobs at mount, even collapsed, and most tool results and args are large plain text or embedded file content: skip JSON.parse unless the blob starts with a JSON container, prefilter search-result extraction with a Title:/URL: substring check, and match the end-anchored exit-code marker against only the tail of exec outputs. Assisted-by: pi:zai-org/GLM-5.3 * ui : parse write_file and edit_file titles without the content blob Both block headers parsed the full args JSON at mount, even collapsed, and write_file and edit_file args embed the whole file content or edit strings, so every block paid a full-blob JSON parse just to read the path. Split the meta into a title tier that extracts the path with a targeted key match (full parse only as fallback) and a body tier that keeps the full parse; Svelte deriveds are lazy, and the body snippet renders only while the block is expanded, so collapsed blocks no longer parse args. Assisted-by: pi:zai-org/GLM-5.3 * ui : mount chat messages lazily near the viewport Every message row mounted its full component tree on load, so the cycle collector, GC and layout invalidation kept walking every live object and DOM node even for rows the user never scrolls to - which dominated the profile of long conversations. Wrap each row in a placeholder with an IntersectionObserver ( two viewport heights of runway ) that swaps in the real ChatMessage when the row approaches the viewport; the row shell keeps the content-visibility sizing, and rows stay mounted once realized. Rows targeted by the pending-edit flow mount eagerly. Assisted-by: pi:zai-org/GLM-5.3 * ui : smooth the chat navigation animations Slide the centered new-chat form to the bottom edge with a transform instead of a bottom offset - layout-property transitions need the main thread every frame and stutter while a long conversation loads, while transform transitions run on the compositor. Fade the message list in with a CSS animation keyed to the conversation id, disabled under prefers-reduced-motion. Assisted-by: pi:zai-org/GLM-5.3 * ui : follow the svelte runes guidance in chat message code Two effects detected changes with manual previous-value refs and reset flags. The permission request carries object identity, so its dismissal is now a derived comparing the dismissed request; the continue request is a bare boolean, so its dismissal only shrinks to a reset while no request is pending. Also drop a dead if (browser) guard in the markdown theme loader - effects never run on the server. Assisted-by: pi:zai-org/GLM-5.3 * test : pin the chat perf invariants in the unit suite Cover the fixes whose silent regression would be stale or wrong UI rather than a crash: the turn-section cache must reuse unchanged turns yet recompute on every field it compares; the sibling map must resolve the same leaves after the leaf-walk memoization; the active conversation must keep its identity through field updates; and the blob gates ( exec tail window, plain-text result gate, search prefilter ) must keep accepting what they gate. Only the risky invariants are pinned - no coverage for coverage's sake. Assisted-by: pi:zai-org/GLM-5.3 * refactor : address review remarks Name the tool-arg string-field pattern, move the file tools' path field aliases and the JSON container gates into lib/constants, and export the write_file / edit_file meta types from $lib/types instead of the parser modules. Assisted-by: pi:zai-org/GLM-5.3
* Add Spark3 Model * rename spark3 -> spark2_5 Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> Co-authored-by: dongjiang <dongjiang2010@gmail.com>
* models: use flash-linear-attention's l2norm for gated delta net q/k
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
* Update src/models/models.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggml-org#27765) * vulkan: add TQ1_0 support (mm, mat-vec, dequant, get_rows) * vulkan: pack TQ1_0 powers of 3 into a 32-bit constant Replaces the constant array with a packed 32-bit value (7 bits per entry, max 81 < 128) extracted with shift/mask, as suggested in review — avoids a constant array that may not be kept in registers. test-backend-ops on gfx1151: tq1_0 MUL_MAT 11/11, MUL_MAT_ID 6/6, GET_ROWS 4/4, unchanged. * vulkan: address review - shared TQ1_0 decode helpers, fix standalone dequant shader Review feedback from jeffbolznv, all points: - Move the packed-pow3 decode into shared helpers in types.glsl (tq1_0_byte_of / tq1_0_digit_of / tq1_0_trit) and use them from dequant_funcs.glsl, mul_mm_funcs.glsl, dequant_funcs_cm2.glsl and dequant_tq1_0.comp instead of repeating the logic. The cm2 path also drops its constant array for the packed-constant extraction. - Translate all remaining comments to English. - dequant_tq1_0.comp: use dequant_head.glsl. The shader previously declared its own single-field push constant while the pipeline is created with the 5-field layout, so p.ne read the wrong field - confirmed broken, as suspected in review. - Fix wg_denoms for the standalone dequant pipeline: one invocation decodes 4 elements with local_size 256, so a workgroup covers 256*4 elements, not 256*16. With the old value the dispatcher launched a quarter of the required workgroups. Verified by temporarily forcing the dequant + f16 matmul path for TQ1_0 (hack not committed): test-backend-ops MUL_MAT passes through the rewritten standalone shader, and the standard MUL_MAT / MUL_MAT_ID / GET_ROWS tq1_0 cases still pass on Vulkan (AMD gfx1151). * vulkan: address review — English comments, shared tq1_0_trit, trim TQ1_0 test cases - mul_mat_vec_tq1_0.comp: drop leftover non-English comment and the local POW3_PACKED constant; all decode sites now call tq1_0_trit() from types.glsl - types.glsl / dequant_funcs_cm2.glsl: ASCII-only, drop stale reviewer note - test-backend-ops: remove the oversized MUL_MAT_ID case (432 MiB A tensor, ~172 GFLOP reference); move the two remaining ones next to the other backend-specific mul_mat_id one-offs and document why they are needed * metal: decline TQ1_0 for GET_ROWS and mat-mul in supports_op The new TQ1_0 cases in test-backend-ops exposed that the Metal backend claimed support for GET_ROWS/MUL_MAT/MUL_MAT_ID with TQ1_0 sources while having no such kernels (ggml_metal_library_compile_pipeline aborted on the missing kernel_get_rows_tq1_0). Decline the type so the ops fall back to the CPU, matching the existing NVFP4 handling on the same lines. Assisted-by: Claude Fable 5 * vulkan: trim the TQ1_0 comments Addresses @0cc4m's review: keep only what the code does not already say. Removed the block-format recaps (the layout is right there in the struct) and the step-by-step decode walkthrough. Kept the two facts a reader cannot infer: the 8-bit truncation is part of the format, not an optimisation, and the powers of 3 are packed into one uint so they do not end up in a constant array that may miss the registers. No functional change. * vulkan: address review — trim comments, fold Metal check, drop unused _v Per @0cc4m's review: - dequant_funcs.glsl, dequant_funcs_cm2.glsl: drop the "see types.glsl" pointers — they apply to every quant and say nothing specific. - dequant_tq1_0.comp: drop the wg_denoms note. It is a precondition, not information. - mul_mm_funcs.glsl: same pointer removed. - types.glsl: the comment on tq1_0_trit is down to the one fact the code cannot show — the 8-bit truncation is part of the format, matching the C reference, not an optimisation. - dequant_funcs_cm2.glsl: removed dequantFuncTQ1_0_v and its define. You were right that it is optional: it wrapped four scalar decodes and vectorised nothing, and mul_mm_cm2.comp already guards the path with `#if defined(dequantFuncA_v)` (DATA_A_F32 omits it the same way). - ggml-metal-device.m: folded TQ1_0 into the existing NVFP4 check instead of a separate block, and dropped both comments. - test-backend-ops.cpp: the two mul_mat_id cases stay — they cover the block-stride loop and the per-expert base offset that k == 256 alone never reaches — but the comment is now one line instead of five. Kept: the one-line labels on the three block regions in mul_mat_vec_tq1_0.comp and on tq1_0_byte_of(). Those state the 5-trits-per-byte packing, which the loop bounds do not show. Happy to remove them too if you prefer. Re-verified on AMD gfx1151 (Vulkan), test-backend-ops, 2/2 backends passed: MUL_MAT 9 TQ1_0 cases, MUL_MAT_ID 5, GET_ROWS 4 — all OK, no failures. The coopmat2 path is unchanged apart from the removed _v define.
This commit contains a suggestion for handling container images which are currently not semver tagged, they only have build numbers in there tags. The proposed solution here is to first add a check to make sure that there are container images built for the build number of the release and if not fail the build. The container images are build nightly but they can be triggered manually as well. If the the container images check passes then the make-release workflow will re-tag the images with the semver.
Support RMS_NORM + MUL + ADD (+ MUL) and RMS_NORM + VIEW + SET_ROWS. Extend ROPE + VIEW + SET_ROWS to support IMROPE. Worth around 4% in gemma4 on my system.
* ggml-cuda: fix divergent barrier in f16 flash attention * ggml-cuda: avoid duplicate metadata pointer setup
…gml-org#28208) Problem - Loader prefers `<arch>.attention.recurrent_layers`, falls back to `full_attention_interval` if missing - Converter only ever writes the interval. gguf-py has no constant/writer for the array - Interval can only describe evenly spaced full-attention layers. Any non-uniform `layer_types` gets reconstructed wrong - No error, no warning. Model loads, runs, wrong layers get wrong ops. Full-attn layers marked recurrent lose their KV cache - Every published Qwen3.5 checkpoint is uniform so nobody's hit it yet Repro 12 layers, periods 4/3/5: layer: 0 1 2 3 4 5 6 7 8 9 10 11 actual: L L L F L L F L L L L F loader: L L L F L L L F L L L F ^ ^ Layer 6 is full attn, loaded as recurrent. Layer 7 the reverse. 52-layer non-uniform stack: 15/52 mis-typed. Fix - `constants.py`: add `Keys.Attention.RECURRENT_LAYERS` (name already registered in llama-arch.cpp) - `gguf_writer.py`: add `add_recurrent_layers()`, same shape as `add_rope_pattern()` - `conversion/qwen.py`: emit array from `layer_types` in `Qwen3NextModel.set_gguf_parameters` (covers 3-Next, 3.5, 3.5-MoE) Notes - Array is padded with `false` for MTP blocks. `get_key_or_arr` checks length against `n_layer_all`, which includes MTP. Matches the fallback's `i < n_layer()` guard - Interval is still written. Old builds only understand the interval - `layer_types` length != `num_hidden_layers` now raises in converter instead of producing a GGUF that fails at load Tested - End-to-end on a 62-layer non-uniform Qwen3.8-27B (2 linear layers removed). Loader reads the array, 62 blocks, 0 mismatches. Without fix: interval fallback, mis-typed - MTP padding NOT tested on a real MTP model. Reasoned from qwen35.cpp + get_key_or_arr. Would appreciate a check Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…gml-org#28538) This commit adds the printing of the ggml version and commit to the test-cmake example. The motivation is just to be able to quickly verify that the correct version of ggml is being used. Example output: ```console test-cmake] llama.cpp version: 0.4.0-dev, build: 10837 (5202104) [test-cmake] ggml version: 0.23.0, commit: 5202104 [test-cmake] Initializing backend... ... ```
* vulkan: fall back to CPU for GET_ROWS with misaligned offsets
The Vulkan GET_ROWS shader asserts when a tensor's backing-buffer offset
plus view_offs is misaligned w.r.t. minStorageBufferOffsetAlignment
(see init_pushconst_tensor_offsets). Previously this caused a hard crash
on models using ggml_view + ggml_get_rows (e.g. Qwen3-TTS, Qwen3-VL).
Return false from supports_op() in the misaligned case so the scheduler
falls back to CPU, matching the existing pattern for PAD_REFLECT_1D and
other unsupported op/shape combinations.
Repro: llama-tts -m Qwen3-TTS-*.gguf -mm mmproj-*.gguf -ngl 99
Crash: GGML_ASSERT(dst->op != GGML_OP_GET_ROWS || (a_offset == 0 && ...)) failed
* vulkan: trim comment for GET_ROWS misalign fallback
* vulkan: fix file corruption in gated_linear_attn struct
* vulkan: properly handle misaligned offsets in GET_ROWS quantized path
- get_rows_quant.comp was missing get_aoffset()/get_boffset()/get_doffset()
calls that are already present in get_rows.comp, causing GGML_ASSERT crashes
when GET_ROWS operates on views with non-zero view_offs, as produced by
KV cache slices in Qwen3-TTS and Qwen3-VL.
- Remove the defensive misalignment GGML_ASSERT in init_pushconst_tensor_offsets
for the binary push-constants specialization, since both get_rows.comp and
get_rows_quant.comp now correctly apply per-tensor base offsets.
- Remove the workaround CPU fallback in supports_op() for GET_ROWS, since the
Vulkan backend now handles misaligned offsets natively (no more bailout).
- Add backend test coverage with view_src0=true (ggml_view_4d into a padded
tensor) for F32, F16, Q4_0, Q4_K, Q8_0, and I32 types, exercising both the
non-quantized (get_rows.comp) and quantized (get_rows_quant.comp) paths
with non-zero view_offs that reproduce the original Qwen3-TTS crash.
* tests: trim redundant comments in test_get_rows vs0 region
* tests: trim redundant comments in test_get_rows vs0 region (follow-up)
* vulkan: bind tensor base for binary ops, pass full view_offs via push constants
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, MUL, etc.),
bind the view_src base and pass the full view_offs divided by type_size via
push constant misalign_offsets. This avoids truncation when misalign_bytes is
not a multiple of quantized block size.
ggml_vk_tensor_subbuffer gains a use_view_offs parameter. When false, the
binding points to vk_tensor_offset (base) and size includes view_offs.
init_pushconst_tensor_offsets<binary> computes a/b/d_offset directly from
tensor->view_offs, which is always row-aligned and therefore exact.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).
* vulkan: bind aligned offset for binary ops, pass adjusted misalign via push constants
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.
ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types).
ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (target - adjusted_misalign) instead of the view_src base, keeping the
offset small enough for 16-bit/8-bit push constant fields.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA RTX 5060 Ti).
* vulkan: bind aligned offset for binary ops, fix UMA offset mismatch
For ops using vk_op_binary_push_constants (GET_ROWS, ADD, SUB, etc.), bind
the buffer to an aligned position near the view offset (not the tensor base)
and pass the adjusted misalignment via push constants.
Added ggml_vk_tensor_physical_offset to unify physical offset lookup across
UMA and non-UMA devices. On UMA, resolves via ggml_vk_host_get(tensor->data);
otherwise uses vk_tensor_offset(t) + t->view_offs. Both get_misalign_bytes and
the new ggml_vk_get_adjusted_misalign helper build on top of this function,
so buffer bindings and push constant offsets are always consistent regardless
of device memory model.
ggml_vk_get_adjusted_misalign finds the smallest misalign that is both a
multiple of minStorageBufferOffsetAlignment and type_size, ensuring
misalign/type_size is exact (no truncation for quantized block types) while
remaining small enough for 16-bit/8-bit push constant fields
(adjusted_misalign < lcm(align, type_size)).
ggml_vk_tensor_subbuffer gains use_view_offs parameter. When false, binds
to (physical_offset - adjusted_misalign) on both UMA and discrete GPUs,
fixing a bug where the UMA host_get path previously skipped the adjusted
misalign binding and returned the target offset directly.
Added non-zero view offset (offset_rows=3) backend tests for GET_ROWS across
all_types with be1={1,7}, v={false,true}, skipping gradient setup for view
tensors (GGML_OP_VIEW fails ggml_set_param).
All 223 GET_ROWS tests pass on Vulkan (NVIDIA GeForce RTX 5060 Ti).
* finish misalignment fix
* supports_op changes for openvino/webgpu
---------
Co-authored-by: AiChiTuDouPian <15327701848@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync AMD-Ecosystem master with ggml-org/llama.cpp upstream master through 972d231.\n\nPreserves the fork deprecation README and the GGML_HIP_UNSAFE_MATH option.\n\nValidation: representative CPU test targets and the HIP gfx1151 test-backend-ops target build successfully.