Jun 277 min readTokenizationByte Pair Encoding: Cracking the Subword CodeByte Pair Tokenization: Efficient subword segmentation. Merges frequent character pairs, handles unseen words, scales to sentences.
