A PHP implementation of OpenAI's BPE tokenizer tiktoken.

Byte-Pair Encoding tokenizer for training large language models on huge datasets

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

(1) Train large language models to help people with automatic essay scoring. (2) Extract essay features and train new tokenizer to build tree models for score prediction.

Teaching transformer-based architectures

🐍This is a fast, lightweight, and clean CPython extension for the Byte Pair Encoding (BPE) algorithm, which is commonly used in LLM tokenization and NLP tasks.

[Rust] Unofficial implementation of "SuperBPE: Space Travel for Language Models" in Rust

BPE tokenizer for LLMs in Pure Zig

processing de LANguage NATural

C89, single header, nostdlib byte pair encoding algorythm

High performance Byte-Pair Encoding tokenizer for large language models

This project implements a Byte Pair Encoding (BPE) tokenization approach along with a Word2Vec model to generate word embeddings from a text corpus. The implementation leverages Apache Hadoop for distributed processing and includes evaluation metrics for optimal dimensionality of embeddings.

TF-IDF Calculation

✨ BPE-Tokenizer for university module Foundational Generative Models.

Naive implementation of byte pair encoding (BPE) tokenization algorithm in golang

A modular Byte Pair Encoding (BPE) tokenizer implemented in Python

Transformer Models for Humorous Text Generation. Fine-tuned on Russian jokes dataset with ALiBi, RoPE, GQA, and SwiGLU.Plus a custom Byte-level BPE tokenizer.

The basic BPE-tokenizer in NLP

From-Scratch High Valyrian MiniGPT — Build and train a GPT-like transformer model fully from scratch, including custom tokenizer, dataloader, transformer, training, and inference! Learn the internals of LLMs layer-by-layer.

Tokenizer Chopper is a implementation of a text tokenizer and detokenizer using Byte Pair Encoding (BPE) for modern LLM systems.