TokEval: A Tokenizer Evaluation Suite

Overview§

TokEval provides a systematic framework for evaluating language model tokenizers beyond standard metrics like fertility and compression rate. The paper introduces intrinsic metrics targeting linguistically and structurally meaningful tokenizer properties (e.g., UTF-8 character boundary integrity, digit place-value alignment) and validates them against actual downstream model performance through controlled pretraining experiments.

Core Methodology§

Intrinsic Metrics§

The authors propose several categories of metrics:

• Information-theoretic: entropy $H(T)$ and cross-entropy over token sequences, which relate to compression and bits-per-byte (BPB).
• Structural sensitivity: measures how well the tokenizer preserves character boundaries, digit place-value structure (e.g., splitting numbers into tokens that align with decimal positions), and line-break boundaries in code.

A key metric is digit boundary alignment. For a number like 12345, a good tokenizer should produce tokens that respect place values (e.g., 12 and 345) rather than arbitrary splits. This is formalized as:

$$\text{DigitBoundaryScore} = \frac{1}{N} \sum_{n=1}^{N} \mathbb{1}[\text{tokenization preserves place-value boundaries for number } n]$$

Similarly, UTF-8 boundary integrity measures whether a byte-level tokenizer ever splits a multi-byte character, which can hinder multilingual performance.

Validation via Controlled Pretraining§

Small Transformer LMs are pretrained from scratch with different tokenizers (varying training data mixture, pretokenization, and algorithm) while keeping all other hyperparameters fixed. Resulting models are evaluated on:

• BPB (tokenizer-agnostic perplexity)
• Benchmarks for linguistic understanding, math reasoning, and code generation.

The central finding is that intrinsic metrics correlate with downstream performance in a task-dependent way:

• Information-theoretic metrics predict language modeling well (Spearman $\rho$ up to 0.80 for BPB).
• Structure-sensitive metrics (digit alignment, line-break handling) correlate with math and code task accuracy.

Implementation Sketch§

Below is a Python snippet illustrating how one might compute the digit boundary alignment metric for a tokenizer.

import re
from transformers import AutoTokenizer

def digit_alignment_score(tokenizer, sample_numbers):
    """Compute fraction of numbers whose tokenization preserves place-value boundaries."""
    correct = 0
    for num in sample_numbers:
        tokens = tokenizer.tokenize(str(num))
        # Concatenate tokens and remove special tokens/whitespace
        joined = "".join(t.replace("Ġ", "") for t in tokens)
        # Check if the joined string matches the original and every split is at a decimal boundary
        if joined == str(num):
            # Verify that each token length is a multiple of powers of 10? Simplified: no split inside a digit group
            if all(len(t) == len(str(num)) or len(t) % 3 != 0 for t in tokens):
                correct += 1
    return correct / len(sample_numbers)

# Example usage
tokenizer = AutoTokenizer.from_pretrained("gpt2")
print(digit_alignment_score(tokenizer, [12345, 100000, 3.14159]))

Note that the actual implementation in the paper is more rigorous, but this conveys the spirit.

Implications§

TokEval offers a cheap proxy for model quality, potentially reducing the need for expensive pretraining sweeps. However, the authors caution that intrinsic metrics are not universally predictive—they must be chosen based on the target task (language modeling vs. math vs. code). This opens the door to task-specific tokenizer optimization and more transparent tokenizer selection.