Stage 2: The Context Window Budget
Models have a finite input-and-output token budget called the context window. This toy bucket drops the oldest chunks to visualize one truncation strategy. Real applications may truncate different content, summarize it, retrieve it later, or reject an overlong request.
Stage 3: The Prediction Engine
At its core, a generative AI model is just a massive probability calculator. It looks at the current tokens and calculates the percentage chance of every possible next token. You are the AI now: Pick the most statistically probable next token to continue the sentence. (Percentages are synthetic toy numbers for illustration).
Calculate next token probabilities:
Stage 4: The Economics of Language
Tokens are not equivalent to words. Segmentation and token count depend on the specific tokenizer and text: some languages, scripts, symbols, or code may use more or fewer tokens than an English rendering under a particular model.
Note: The counts below come from this page's educational approximation, not a real tokenizer. They cannot establish the relative efficiency or cost of a language.
Required
Frequently Asked Questions
Why don't models just read letters?
Tokenizer design is a model-specific engineering tradeoff among vocabulary size, sequence length, coverage, and efficiency. Tokens may be words, subwords, characters, bytes, or combinations; character-level and byte-level approaches also exist.
Do all models use the same tokens?
No. Different models use different token dictionaries (vocabularies). A word might be 1 token in GPT-4 but 3 tokens in an older model.
Why does AI struggle with spelling or counting letters?
Subword tokenization can make character-level tasks less direct because generation operates over token IDs, but tokenized text still encodes the underlying spelling. Performance depends on the model, tokenizer, training, and whether it uses suitable reasoning or tools.
How does tokenization affect price?
Some API providers price usage by input and output tokens. For a particular model and tokenizer, two equivalent texts can have different token counts, which may affect cost and processing time. No language is inherently more expensive: the result varies by tokenizer, text, model, and provider pricing.