Subword tokenization is a text processing technique that bridges word-level and character-level tokenization approaches. Rather than treating entire words as indivisible units or breaking text down to individual characters, subword tokenization decomposes words into smaller, reusable linguistic units called subword tokens. This strategy enables language models to maintain compact vocabularies while effectively handling rare words, morphological variations, and out-of-vocabulary terms through compositional decomposition ¹⁾ Word-level tokenization treats each word as a token but struggles with rare words, while character-level tokenization handles any text but creates very long sequences that strain computational resources ²⁾.

Traditional word-level tokenization creates a fundamental vocabulary size problem: large vocabularies (100,000+ tokens) increase computational costs and memory requirements, while small vocabularies fail to represent rare or domain-specific terms. Character-level tokenization solves the rare word problem but produces extremely long sequences that strain model attention mechanisms and increase computational costs proportionally to sequence length.

Subword tokenization offers a middle ground by recognizing that words contain frequently-recurring components. Common morphemes, prefixes, suffixes, and letter combinations can be shared across many words. For example, the word “unbelievable” might decompose into [“un”, “believ”, “able”] or [“un”, “be”, “liev”, “able”], allowing a model to understand unseen words like “unbelievably” by recognizing shared subword components ³⁾

Byte Pair Encoding (BPE) represents one of the earliest subword approaches. BPE begins with character-level tokenization and iteratively merges the most frequently co-occurring adjacent token pairs in the training corpus. After training on a large text corpus, this process produces a vocabulary of predetermined size containing both characters and frequent subword sequences. The algorithm is deterministic and reproducible, making it widely adopted in models like GPT-2 and RoBERTa ⁴⁾

WordPiece tokenization, developed by Google for BERT, uses a similar greedy approach but optimizes for language model likelihood during merging rather than simple frequency counting. WordPiece greedily combines character sequences into longer tokens when doing so maximizes the likelihood of the training data under the current vocabulary. This likelihood-based approach often produces more semantically meaningful subword units compared to frequency-based BPE ⁵⁾

SentencePiece provides a language-agnostic tokenization framework that treats tokenization and training as a single unified process. Unlike BPE and WordPiece which typically require preprocessing to identify word boundaries, SentencePiece operates directly on raw text and learns the optimal way to segment text into subwords. This approach proves particularly valuable for languages without clear word boundaries, such as Chinese and Japanese ⁶⁾

Unigram language model tokenization probabilistically determines the optimal tokenization by training a unigram language model over subword candidates. Rather than using greedy merging, this method maintains probability distributions over all possible tokenizations and selects the most probable decomposition for each word. This probabilistic approach can handle ambiguous tokenization decisions more gracefully than deterministic methods.

Subword tokenization fundamentally shapes model behavior in several ways. The granularity of tokenization directly affects sequence length: fine-grained tokenization produces longer sequences that increase computational cost, while coarse-grained tokenization may fail to capture important linguistic distinctions. Different language pairs require different optimal vocabulary sizes; low-resource languages and agglutinative languages benefit from smaller subword units, while English typically uses medium-sized vocabularies of 20,000-50,000 tokens.

Modern language models including BERT, GPT-2, GPT-3, and T5 rely on subword tokenization schemes. The specific tokenization algorithm influences how models represent and process language; models trained with BPE may develop different internal representations than models trained with WordPiece for identical text ⁷⁾

Code switching and multilingual scenarios present specific challenges for subword tokenization. When text mixes multiple languages, fixed vocabularies may handle different languages with widely varying efficiency. Some subword tokenization approaches maintain separate vocabularies per language, while others attempt to develop unified multilingual vocabularies that balance representation quality across many languages simultaneously.

Tokenization misalignment creates problems where meaningful linguistic units split across subword boundaries, potentially hindering model interpretability and performance. The optimal vocabulary size and tokenization granularity remain empirical questions without principled guidelines; different downstream tasks may benefit from different tokenization schemes applied to identical source text.

Domain-specific tokenization presents ongoing challenges. Scientific text, programming code, and specialized technical domains often contain rare terms and unconventional word formations that general-purpose tokenization schemes handle inefficiently. Developing domain-adaptive tokenization approaches remains an active research area.

• WordPiece
• Tokenization
• Tokenizer Comparison
• SentencePiece
• English vs Chinese Tokenization