#tokenization (8 Repositories)
Ranked open-source repositories tagged with #tokenization, scored by pull request acceptance likelihood and maintainer engagement velocity.
0.0%
0.0h
8 repositories tagged #tokenization
JuliaText/WordTokenizers.jl
High performance tokenizers for natural language processing and other related tasks
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
toon-format/toon
🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
WorksApplications/sudachi.rs
Sudachi in Rust 🦀 and new generation of SudachiPy
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
lunasec-io/lunasec
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
Apil-Shrestha/token-efficiency-lex
Token Cost Parity: Multilingual LLM Efficiency Analysis 2026