Skip to content

Commit aaf0db5

Browse files
committed
Release v0.5.1
1 parent b3cd84b commit aaf0db5

File tree

2 files changed

+13
-1
lines changed

2 files changed

+13
-1
lines changed

CHANGELOG.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,17 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [v0.5.1] - 2024-10-02
9+
10+
### Added
11+
12+
- Add new ByteLevel normalizer (`Tokenizers.Normalizer.byte_level/0`).
13+
14+
### Changed
15+
16+
- Reduce memory copies when encoding.
17+
- Bump Rust tokenizers to v0.20.0.
18+
819
## [v0.5.0] - 2024-04-24
920

1021
### Added
@@ -97,6 +108,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
97108

98109
First release.
99110

111+
[v0.5.1]: https://github.com/elixir-nx/tokenizers/compare/v0.5.0...v0.5.1
100112
[v0.5.0]: https://github.com/elixir-nx/tokenizers/compare/v0.4.0...v0.5.0
101113
[v0.4.0]: https://github.com/elixir-nx/tokenizers/compare/v0.3.2...v0.4.0
102114
[v0.3.2]: https://github.com/elixir-nx/tokenizers/compare/v0.3.1...v0.3.2

mix.exs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ defmodule Tokenizers.MixProject do
22
use Mix.Project
33

44
@source_url "https://github.com/elixir-nx/tokenizers"
5-
@version "0.6.0-dev"
5+
@version "0.5.1"
66

77
def project do
88
[

0 commit comments

Comments
 (0)