Tokenizer Comparison

Tokenizers are the components that convert raw text into the numerical token sequences that large language models process. Different tokenization algorithms and vocabularies produce significantly different token counts for the same input text, directly impacting cost, context window utilization, and multilingual performance. ¹⁾

BPE is the most widely used subword tokenization method. It builds a vocabulary by iteratively merging the most frequent adjacent byte pairs in training text. ²⁾

The algorithm works as follows:

1. Initialization: Represent the corpus as a list of individual bytes (UTF-8 encoded characters)
2. Frequency counting: Identify the most common adjacent byte pairs (e.g., “t” + “h” becomes “th”)
3. Merging: Replace all occurrences of that pair with a new token in the vocabulary; repeat for a fixed number of merges (typically 30,000 to 100,000+ steps)
4. Encoding: To tokenize new text, greedily apply merges from most to least frequent, falling back to individual bytes for unknown sequences

BPE creates compact representations for common words while handling rare words via subword decomposition (e.g., “tokenization” becomes “token” + “ization”). It is used in GPT models via tiktoken.

SentencePiece treats tokenization as a probabilistic segmentation problem rather than a merge-based one. ³⁾

• Learns a vocabulary by scoring subword candidates via a unigram probability distribution
• Prunes low-probability tokens using an EM (Expectation-Maximization) algorithm to reach a target vocabulary size
• Uses the Viterbi algorithm at inference time to find the highest-probability segmentation, allowing backtracking unlike BPE's greedy approach
• Language-agnostic (no spaces required), making it ideal for multilingual text

SentencePiece is used in T5, Llama, and several Google models.

tiktoken is OpenAI's fast, Rust-based BPE tokenizer library supporting multiple encodings. It handles regex preprocessing and is approximately 10x faster than pure Python BPE implementations. ⁴⁾

Key encodings:

• cl100k_base: 100,421 vocabulary tokens, used by GPT-4 and GPT-3.5-turbo. Approximately 1 token per 4 English characters.
• o200k_base: Approximately 200,000 vocabulary tokens, used by GPT-4o, GPT-4.1, and o-series models. 15-20% more efficient than cl100k_base, especially for non-English text and code.

The same text produces different token counts across models due to vocabulary size, training data, and algorithm differences. Typical variance is 10-50%. ⁵⁾

Early English-centric tokenizers like cl100k_base use 2-4x more tokens for Asian languages due to byte-level splitting. Modern tokenizers (o200k_base, Llama, Gemini) dedicate vocabulary entries to non-Latin scripts, significantly reducing this overhead. ⁶⁾

For example, Chinese characters may require 4-6 tokens with cl100k_base but only 1-2 tokens with o200k_base or SentencePiece-based tokenizers. This means non-English prompts can cost 1.5-3x more with older tokenizers.

Token count directly scales API costs. A 100K-token prompt at $2/M input tokens costs $0.20. More efficient tokenizers reduce both cost and context window consumption: ⁷⁾

• o200k_base and Llama tokenizers save 15-30% over cl100k_base on mixed workloads
• Multilingual savings compound further for global applications
• Always test with the model-specific tokenizer for precise token counts before deployment

• Prompt Caching
• Embedding Models Comparison