Releases: mlc-ai/tokenizers-cpp
Releases · mlc-ai/tokenizers-cpp
v0.1.1
What's Changed
- Add npm package by @tqchen in #3
- Update CMakeLists.txt by @songkq in #2
- Fix readme typo by @erjanmx in #4
- Fix windows build by linking against ntdll by @junrushao in #5
- Update CMakeLists.txt by @luiyen in #6
- Update rust to support latest tokenizers by @tqchen in #11
- add rwkv world tokenizer by @BBuf in #14
- msgpack as 3rdparty library instead of fetch_context by @BBuf in #15
- [FIX] Remove boost dependency from submodule msgpack by @Hzfengsy in #16
- Update to reduce memory realloc by @tqchen in #18
- fix rwkv world tokenzier by @BBuf in #19
- Update build.sh to make web tokenizer work with chrome extension by @manuongithub in #20
- Add support for querying vocabulary from tokenizer by @Ubospica in #22
- [Web] Expose getVocabSize and idToToken to web, bump version to 0.1.3 by @CharlieFRuan in #25
- Allow parameters related to special tokens for HFTokenizer by @Abhishek8394 in #27
- Set value of TOKENIZERS_CPP_CARGO_TARGET based on ANDROID_ABI by @yuxuanchiadm in #28
- Added EncodeBatch interface by @vinx13 in #33
- Remove incorrect final for non-virtual func by @vinx13 in #34
- [Rust] Bump HuggingFace tokenizer version by @MasterJH5574 in #36
- [CMake] Support disable SentencePiece tokenizer by @MasterJH5574 in #38
- [Web] Set TOKENIZERS_PARALLELISM to false for HFTokenizer by @CharlieFRuan in #42
- [CMake] Enable SentencePiece tokenizer by default by @MasterJH5574 in #47
- [Rust] Bump huggingface tokenizer to 0.20.0 by @MasterJH5574 in #49
- Bump CMake minimum to v3.19 by @afuller-TT in #59
- Fix for loading tokenizers using non-utf8 strings by @ThomasProg in #55
- updating tokenizer version from 0.20.0 to 0.21.0 by @NaveenMittal0 in #62
- Add a way to force the macos target in case it is set by @ykhrustalev in #52
- Fix CMake Configuration for MacOS Support by @Darce-One in #65
- [Web] Bump web-tokenizer to 0.1.6 by @CharlieFRuan in #67
- Fixed compilation for aarch64 by @NaveenMittal0 in #66
New Contributors
- @tqchen made their first contribution in #3
- @songkq made their first contribution in #2
- @erjanmx made their first contribution in #4
- @junrushao made their first contribution in #5
- @luiyen made their first contribution in #6
- @BBuf made their first contribution in #14
- @Hzfengsy made their first contribution in #16
- @manuongithub made their first contribution in #20
- @Ubospica made their first contribution in #22
- @CharlieFRuan made their first contribution in #25
- @Abhishek8394 made their first contribution in #27
- @yuxuanchiadm made their first contribution in #28
- @vinx13 made their first contribution in #33
- @MasterJH5574 made their first contribution in #36
- @afuller-TT made their first contribution in #59
- @ThomasProg made their first contribution in #55
- @NaveenMittal0 made their first contribution in #62
- @ykhrustalev made their first contribution in #52
- @Darce-One made their first contribution in #65
Full Changelog: v0.1.0...v0.1.1