GPT Tokenizer

Recreating the Byte Pair Encoding algorithm used in tokenizers, along with GPT styled Regex Tokenizer.

• BPEtokenizer.ipynb :

  • basically a follow along notebook from Karpathy's tokenizer video.

• BPETokenizer.py:

  • a cleaned up version of the notebook, BasicTokenizer of the Karpathy exercise.

• GPTTokenizer.py

  • Regex Tokenizer from the Karpathy Exercise, also implemented handling of special tokens.

TO-DO

• Implement sentence piece