-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
chore(release): bump dev version + trigger the release workflow on release/* branches
#2446
opened Sep 21, 2026 by
SBrandeis
Collaborator
Loading…
fix(tk-encode): longest-prefix matching in PrecompiledNormalizer
#2430
opened Sep 20, 2026 by
SIDDARTHAREDDY8
Loading…
docs: fix 'deserilized' typo in the utils doc comment
#2429
opened Sep 18, 2026 by
simpleqt
Loading…
docs(python): drop the dead free-threading-audit.md link
#2428
opened Sep 18, 2026 by
simpleqt
Loading…
Respect BPE cache_capacity when reading JSON
#2420
opened Sep 17, 2026 by
LysandreJik
Member
Loading…
bpe: do not slice merge tokens at a byte offset that may not be a boundary
#2398
opened Sep 10, 2026 by
apollo-2006
Loading…
Fix BERT normalization of CJK Extension E boundary characters
#2395
opened Sep 8, 2026 by
mottopanikeiku
Loading…
fix: Unigram prune_sentence_pieces uses alternatives[id].len() not alternatives.len()
#2394
opened Sep 8, 2026 by
VirajMishra1
Loading…
fix(python): forward unk_id in SentencePieceUnigramTokenizer
#2393
opened Sep 7, 2026 by
Con-Benksl
Loading…
fix(bpe): drop heap entries with non-positive pair_counts before u64 cast
#2389
opened Sep 3, 2026 by
okxint
Loading…
perf(decode): index the vocab slab by id, not through the MPHF slot
#2387
opened Sep 3, 2026 by
ArthurZucker
Collaborator
Loading…
fix: correctly decode numpy UCS-4 arrays instead of treating as UTF-8
#2386
opened Sep 3, 2026 by
VirajMishra1
Loading…
v1.0.0-rc0: the train/encode split, repacked into 53 commits
#2384
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(normalizers): fast-path BertNormalizer for non-CJK input
#2383
opened Sep 2, 2026 by
bsachart
Loading…
Fix UnicodeScripts pre-tokenizer dropping leading spaces
#2382
opened Sep 2, 2026 by
Kayvan-Zahiri
Loading…
perf(parallelism): default the pool to physical cores, not SMT siblings
#2380
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(train): select top-k in the trainers instead of ordering everything
#2379
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(word-cache): scan the tag window with shrn, not an emulated movemask
#2377
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(pre_tokenizers): eliminate per-token heap allocations in ByteLevel decoder
#2376
opened Sep 1, 2026 by
bsachart
Loading…
Fix BPE training across consecutive line endings
#2370
opened Aug 30, 2026 by
Alphaxiaoteng
Loading…
Previous Next
ProTip!
Follow long discussions with comments:>50.