Byte-pair encoding
2 stories tagged Byte-pair encoding on Learning Fanatic.
Lesson 2.1: Tokenisation: characters, words and subwords
How text is cut into tokens, and why the cut matters: characters, words and subwords compared on vocabulary size, sequence length, unseen words and cost.
Lesson 2.2: Byte-pair encoding from scratch
Write byte-pair encoding in about fifty lines: learn merges from text, encode and decode losslessly, and test it on unseen words and accented text.