#tokenizer (30 Repositories)
Ranked open-source repositories tagged with #tokenizer, scored by pull request acceptance likelihood and maintainer engagement velocity.
22.5%
21.1h
30 repositories tagged #tokenizer
lindera/lindera
A multilingual morphological analysis library.
ikawaha/kagome
Self-contained Japanese Morphological Analyzer written in pure Go
microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
natasha/natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
hplt-project/sacremoses
Python port of Moses tokenizer, truecaser and normalizer
mck89/peast
JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
glayzzle/php-parser
:herb: NodeJS PHP Parser - extract AST or tokens
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
bytexenon/Tiny-Lua-Compiler
🌸 Possibly the smallest Lua compiler ever
luauminati/Tiny-Lua-Compiler
🌸 Possibly the smallest Lua compiler ever
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
niieani/gpt-tokenizer
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
ongjin/garu
초경량 한국어 형태소 분석기 — 1MB 모델로 브라우저에서 실행(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
CogComp/cogcomp-nlp
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
timtadh/lexmachine
Lex machinary for go.
NLPOptimize/flash-tokenizer
EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
guillaume-be/rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
neurosnap/sentences
A multilingual command line sentence tokenizer in Golang
JuliaLang/Tokenize.jl
Tokenization for Julia source code
Chevrotain/chevrotain
Parser Building Toolkit for JavaScript
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
open-korean-text/open-korean-text
Open Korean Text Processor - An Open-source Korean Text Processor
sugarme/tokenizer
NLP tokenizers written in Go language
mathewsanders/Mustard
🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.
lovit/soynlp
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
wangfenjin/simple
支持中文和拼音的 SQLite fts5 全文搜索扩展 ｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin
no-context/moo
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
marcelroed/gigatoken
Language model tokenization at GB/s
dqbd/tiktokenizer
Online playground for OpenAPI tokenizers