File tree Expand file tree Collapse file tree 2 files changed +13
-1
lines changed Expand file tree Collapse file tree 2 files changed +13
-1
lines changed Original file line number Diff line number Diff line change @@ -5,6 +5,17 @@ All notable changes to this project will be documented in this file.
5
5
The format is based on [ Keep a Changelog] ( https://keepachangelog.com/en/1.0.0/ ) ,
6
6
and this project adheres to [ Semantic Versioning] ( https://semver.org/spec/v2.0.0.html ) .
7
7
8
+ ## [ v0.5.1] - 2024-10-02
9
+
10
+ ### Added
11
+
12
+ - Add new ByteLevel normalizer (` Tokenizers.Normalizer.byte_level/0 ` ).
13
+
14
+ ### Changed
15
+
16
+ - Reduce memory copies when encoding.
17
+ - Bump Rust tokenizers to v0.20.0.
18
+
8
19
## [ v0.5.0] - 2024-04-24
9
20
10
21
### Added
@@ -97,6 +108,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
97
108
98
109
First release.
99
110
111
+ [ v0.5.1 ] : https://github.com/elixir-nx/tokenizers/compare/v0.5.0...v0.5.1
100
112
[ v0.5.0 ] : https://github.com/elixir-nx/tokenizers/compare/v0.4.0...v0.5.0
101
113
[ v0.4.0 ] : https://github.com/elixir-nx/tokenizers/compare/v0.3.2...v0.4.0
102
114
[ v0.3.2 ] : https://github.com/elixir-nx/tokenizers/compare/v0.3.1...v0.3.2
Original file line number Diff line number Diff line change @@ -2,7 +2,7 @@ defmodule Tokenizers.MixProject do
2
2
use Mix.Project
3
3
4
4
@ source_url "https://github.com/elixir-nx/tokenizers"
5
- @ version "0.6.0-dev "
5
+ @ version "0.5.1 "
6
6
7
7
def project do
8
8
[
You can’t perform that action at this time.
0 commit comments