Releases · mlc-ai/tokenizers-cpp

What's Changed

Add npm package by @tqchen in #3
Update CMakeLists.txt by @songkq in #2
Fix readme typo by @erjanmx in #4
Fix windows build by linking against ntdll by @junrushao in #5
Update CMakeLists.txt by @luiyen in #6
Update rust to support latest tokenizers by @tqchen in #11
add rwkv world tokenizer by @BBuf in #14
msgpack as 3rdparty library instead of fetch_context by @BBuf in #15
[FIX] Remove boost dependency from submodule msgpack by @Hzfengsy in #16
Update to reduce memory realloc by @tqchen in #18
fix rwkv world tokenzier by @BBuf in #19
Update build.sh to make web tokenizer work with chrome extension by @manuongithub in #20
Add support for querying vocabulary from tokenizer by @Ubospica in #22
[Web] Expose getVocabSize and idToToken to web, bump version to 0.1.3 by @CharlieFRuan in #25
Allow parameters related to special tokens for HFTokenizer by @Abhishek8394 in #27
Set value of TOKENIZERS_CPP_CARGO_TARGET based on ANDROID_ABI by @yuxuanchiadm in #28
Added EncodeBatch interface by @vinx13 in #33
Remove incorrect final for non-virtual func by @vinx13 in #34
[Rust] Bump HuggingFace tokenizer version by @MasterJH5574 in #36
[CMake] Support disable SentencePiece tokenizer by @MasterJH5574 in #38
[Web] Set TOKENIZERS_PARALLELISM to false for HFTokenizer by @CharlieFRuan in #42
[CMake] Enable SentencePiece tokenizer by default by @MasterJH5574 in #47
[Rust] Bump huggingface tokenizer to 0.20.0 by @MasterJH5574 in #49
Bump CMake minimum to v3.19 by @afuller-TT in #59
Fix for loading tokenizers using non-utf8 strings by @ThomasProg in #55
updating tokenizer version from 0.20.0 to 0.21.0 by @NaveenMittal0 in #62
Add a way to force the macos target in case it is set by @ykhrustalev in #52
Fix CMake Configuration for MacOS Support by @Darce-One in #65
[Web] Bump web-tokenizer to 0.1.6 by @CharlieFRuan in #67
Fixed compilation for aarch64 by @NaveenMittal0 in #66