Tokens and tokenizers
Distinguishes characters, bytes, subwords, and integer token IDs, and explains BPE merge rules and subword tokenization pipelines.
· Last updated: 2026-11-20 · 7 min read
Overview
A token is the discrete unit a language model uses to represent and process text inputs and outputs, and a tokenizer is the component that maps raw strings to and from sequences of tokens and integer token IDs. Depending on vocabulary design, a single token may correspond to a whole word, a subword fragment, a character, or a raw byte sequence. Consequently, writing 고양이 → 귀엽다 illustrates a word-level mapping for human readers; showing actual token boundaries requires specifying which tokenizer and vocabulary are used.
A token ID is the integer index assigned to a token by a vocabulary table . The integer value itself carries no semantic ordering; instead, the model looks up row vector in an embedding matrix and feeds that vector into its context layers. Even for the exact same input text, changing the normalization, pre-tokenization, subword splitting rules, or vocabulary produces a different token sequence and ID list1.

The figure above shows the miniature merge rules and vocabulary table constructed specifically for the code example below rather than the output of a production LLM tokenizer.
Merging smaller pieces into subword tokens
Byte Pair Encoding (BPE)—originally introduced by Philip Gage (1994) for data compression2 and adapted to subword tokenization—learns an ordered list of merge rules by repeatedly finding the most frequent adjacent symbol pair in a training corpus and replacing it with a new symbol . By contrast, unigram subword models (supported in tools such as SentencePiece) select the segmentation from candidate segmentations that maximizes the sum of subword log-probabilities.
Rather than running corpus-wide frequency counting, the Python and C++20 examples below take three fixed merge rules—("l", "o"), ("lo", "w"), and ("e", "r")—and apply them in order to the character sequence of "lower" before looking up integer IDs in a small vocabulary.
def merge_pair(symbols, pair):
merged = []
i = 0
while i < len(symbols):
if i + 1 < len(symbols) and tuple(symbols[i:i + 2]) == pair:
merged.append(symbols[i] + symbols[i + 1])
i += 2
else:
merged.append(symbols[i])
i += 1
return merged
if __name__ == "__main__":
symbols = list("lower") + ["</w>"]
rules = [("l", "o"), ("lo", "w"), ("e", "r")]
for pair in rules:
symbols = merge_pair(symbols, pair)
vocabulary = {"low": 0, "er": 1, "</w>": 2}
ids = [vocabulary[symbol] for symbol in symbols]
assert symbols == ["low", "er", "</w>"]
assert ids == [0, 1, 2]
print(symbols, ids)Output:
['low', 'er', '</w>'] [0, 1, 2]An equivalent C++20 implementation of sequential pair merging and vocabulary ID lookup is shown below:
#include <iostream>
#include <string>
#include <unordered_map>
#include <utility>
#include <vector>
std::vector<std::string> mergePair(
const std::vector<std::string>& symbols,
const std::pair<std::string, std::string>& rule) {
std::vector<std::string> merged;
for (std::size_t i = 0; i < symbols.size();) {
if (i + 1 < symbols.size() && symbols[i] == rule.first &&
symbols[i + 1] == rule.second) {
merged.push_back(symbols[i] + symbols[i + 1]);
i += 2;
} else {
merged.push_back(symbols[i]);
++i;
}
}
return merged;
}
int main() {
std::vector<std::string> symbols{"l", "o", "w", "e", "r", "</w>"};
const std::vector<std::pair<std::string, std::string>> rules{
{"l", "o"}, {"lo", "w"}, {"e", "r"}};
for (const auto& rule : rules) {
symbols = mergePair(symbols, rule);
}
const std::unordered_map<std::string, int> vocab{
{"low", 0}, {"er", 1}, {"</w>", 2}};
for (const auto& sym : symbols) std::cout << sym << ' ';
std::cout << "->";
for (const auto& sym : symbols) std::cout << ' ' << vocab.at(sym);
std::cout << '\n';
return 0;
}Output:
low er </w> -> 0 1 2merge_pair (mergePair) takes an already segmented symbol list and a length-2 merge pair. Here </w> marks the end of a word, and the IDs 0, 1, 2 are defined only for this toy vocabulary. Attempting to look up a symbol absent from the vocabulary raises a KeyError in Python or std::out_of_range in C++; handling arbitrary unseen text therefore requires a tokenizer whose base vocabulary covers all characters or all 256 UTF-8 bytes (or provides an explicit unknown-token <unk> fallback).
Despite the word "byte" in BPE, not every BPE implementation operates on raw UTF-8 bytes. The subword BPE formulation of Sennrich et al. (2016) merges Unicode characters, whereas byte-level BPE (used in GPT-2 and related models) starts from the 256 raw byte values so any UTF-8 string can be tokenized without out-of-vocabulary fallbacks.
Merge rules in the original subword paper
Sennrich, Haddow, and Birch, Neural Machine Translation of Rare Words with Subword Units (ACL 2016), introduced BPE subword segmentation to represent rare and unseen words as sequences of subword units3. The figure below shows the miniature BPE merge example from §3.2 of their paper.

The excerpt above is Figure 1 from Sennrich et al. (2016) (PDF page 4, paper page 1718; source page, © 2016 Association for Computational Linguistics, ACL copyright terms). Each step merges the left-hand symbol pair into the new symbol on the right, including the end-of-word marker (·). The Python and C++20 examples earlier use a simplified three-rule list for exposition.
Distinguishing characters, bytes, and token counts
The Korean string 고양이 consists of 3 Unicode code points and 9 UTF-8 bytes (3 bytes per Hangul syllable). Neither number alone determines how many tokens a given LLM will consume: one tokenizer’s vocabulary might include 고양이 as a single token, while another splits it into syllables, jamo, or byte fragments. Moreover, for complex emojis or combining sequences, even the number of Unicode code points can differ from the number of user-perceived characters (grapheme clusters).
Kudo and Richardson’s SentencePiece (EMNLP 2018) trains and applies subword models (both BPE and unigram) directly on raw text without language-specific pre-tokenizers, encoding whitespace as the visible meta-symbol ▁ (U+2581) to allow lossless detokenization4. However, when Unicode normalization rules such as NFKC are enabled in the pipeline, decoding back from tokens is not guaranteed to reproduce the original string byte-for-byte.
When budgeting context window limits or API token costs, always run the exact tokenizer version matching the target model and account for special control tokens (such as <|im_start|> or <|eot_id|>) and chat-template framing added outside the raw message body.
See also
Footnotes
- Hugging Face Tokenizers Documentation, “The Tokenization Pipeline.” Documentation. ↩
- Philip Gage, “A New Algorithm for Data Compression,” The C Users Journal 12(2), 1994, pp.23–38. Archived article, ACM record. ↩
- Rico Sennrich, Barry Haddow, Alexandra Birch, “Neural Machine Translation of Rare Words with Subword Units,” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), 2016, §3.2, Figure 1, pp.1715–1725. ACL Anthology, DOI. ↩
- Taku Kudo, John Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing,” Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2018, §2–3, pp.66–71. ACL Anthology, DOI. ↩