#tokenization (11 Repositories)
Ranked open-source repositories tagged with #tokenization, scored by pull request acceptance likelihood and maintainer engagement velocity.
13.1%
34.1h
11 repositories tagged #tokenization
toon-format/toon
🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
WorksApplications/sudachi.rs
Sudachi in Rust 🦀 and new generation of SudachiPy
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
lunasec-io/lunasec
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
Apil-Shrestha/token-efficiency-lex
Token Cost Parity: Multilingual LLM Efficiency Analysis 2026
JuliaText/WordTokenizers.jl
High performance tokenizers for natural language processing and other related tasks
adobe/NLP-Cube
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
marcelroed/gigatoken
Language model tokenization at GB/s
RavenProject/Ravencoin
Ravencoin Core integration/staging tree