#tokenizer (20 Repositories)
Ranked open-source repositories tagged with #tokenizer, scored by pull request acceptance likelihood and maintainer engagement velocity.
4.7%
4.9h
20 repositories tagged #tokenizer
lindera/lindera
A multilingual morphological analysis library.
Chevrotain/chevrotain
Parser Building Toolkit for JavaScript
niieani/gpt-tokenizer
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
open-korean-text/open-korean-text
Open Korean Text Processor - An Open-source Korean Text Processor
ongjin/garu
์ด๊ฒฝ๋ ํ๊ตญ์ด ํํ์ ๋ถ์๊ธฐ โ 1MB ๋ชจ๋ธ๋ก ๋ธ๋ผ์ฐ์ ์์ ์คํ(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
JuliaLang/Tokenize.jl
Tokenization for Julia source code
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
neurosnap/sentences
A multilingual command line sentence tokenizer in Golang
daac-tools/vaporetto
๐ฅ Vaporetto: Very accelerated pointwise prediction based tokenizer
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
guillaume-be/rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
sugarme/tokenizer
NLP tokenizers written in Go language
NLPOptimize/flash-tokenizer
EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
mck89/peast
JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
CogComp/cogcomp-nlp
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
timtadh/lexmachine
Lex machinary for go.
daac-tools/vibrato
๐ค vibrato: Viterbi-based accelerated tokenizer
bytexenon/Tiny-Lua-Compiler
๐ธ Possibly the smallest Lua compiler ever
lovit/soynlp
ํ๊ตญ์ด ์์ฐ์ด์ฒ๋ฆฌ๋ฅผ ์ํ ํ์ด์ฌ ๋ผ์ด๋ธ๋ฌ๋ฆฌ์ ๋๋ค. ๋จ์ด ์ถ์ถ/ ํ ํฌ๋์ด์ / ํ์ฌํ๋ณ/ ์ ์ฒ๋ฆฌ์ ๊ธฐ๋ฅ์ ์ ๊ณตํฉ๋๋ค.