GPT-2-style LLM built from scratch in C/CUDA with hand-written backprop, BPE tokenizer, FlashAttention, pretraining, and SFT.

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

Kotlin multiplatform BPE tokenizer library for OpenAI models

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

High performance unsupervised text tokenization for Ruby

A Python/Cython implementation of Pruned BPE, a Byte Pair Encoding tokenizer that improves visible vocabulary efficiency by pruning low-exposure tokens while preserving internal merge structure.

Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

R package for Byte Pair Encoding based on YouTokenToMe

Byte-level byte pair encoding (BPE) in Haskell

An educational Python project for learning tokenization step by step by building character-level, byte-level, and BPE tokenizers from scratch.

Feature extraction from sequential data

This is a tool that encrypts a sequence of words (or pieces of texts) using the AES-256 algorithm and encodes the encrypted result into a PNG image by linking each byte value to a specific color. It also decodes the before image to get back the original sequence of words

Code repo for the paper "AutoGO: Automated Computation Graph Optimization for Neural Network Evolution", accepted to NeurIPS 2023.

Byte-Pair Encoding tokenizer for training large language models on huge datasets

Lightweight, header-only Byte Pair Encoding (BPE) trainer in modern C++17. Produces HuggingFace-compatible vocabularies for transformers and integrates with Modern Text Tokenizer.

Генерация новостных заголовков

Byte Pair Encoding (BPE)

A lightweight, from-scratch implementation of Byte Pair Encoding (BPE) tokenization in Python.

The fastest tokenizer for LLMs, up to 30x+ faster.