Every number here was measured,never claimed.

Move the cap and watch the layers fit,then read the run behind the model.

13 layers under the cap
1.9B tokens, one GPU
lm-evaluation-harness
Runs in your browser

Budget explorer

Fifty million, and the word list counts.

At our width of 512, GPT-2's 50,257-token vocabulary spends 51.5% of the cap before a single layer exists and leaves room for 7 layers. Ours has 16,384 tokens, spends 16.8% and fits 13. Everything in the calculator below is arithmetic on the cap, recomputed as you move it.

Try another word list

Start from a tokenizer, then move anything. This part is arithmetic on the cap, not a measurement.

GPT-2 Small50,257 × 7681 layer GPT-2 vocabulary, our width50,257 × 5127 layers Llama tokenizer32,000 × 51210 layers Ours this run16,384 × 51213 layers Tiny vocabulary8,192 × 512Less text fits in each 1,024-token window14 layers

Layers that fit

13 layers

vs 7 with GPT-2's vocabulary

Spent before any layer

16.8% of the cap

vs 51.5% with GPT-2's vocabulary

49,296,896 of 50,000,000 parametersUnder the cap, 703,104 spare

The whole cap, drawn to scale

One block per transformer layer, on the word list's cost.

Vocabularytokens

Rule 01 16,384 tokens × 512 wide = 8,388,608 before any layer exists.

Widthwide

One layer at this width costs 3,146,752.

Feed-forward ratio× width

Feed-forward is 2,097,152 of every layer.

Options

Tie the output head to the embedding

Rule 02 Tied: +3 layers over a separate head.

Rotary positions, not a learned table

Rule 03 Rotary: saves 524,288, under one layer here.

Where every parameter goes

Every row is Each × Count. The groups are the parts of the drawing, and they add up to the total, checked against the cap.

Parameter ledger for the current settings: each part's size, how many are paid for, and the parameters it costs.
Part	Each	Count	Parameters
Before any layer16.8% of the cap			8,388,608
Token embedding16,384 tokens × 512 wide8,388,608 × 1	8,388,608	1	8,388,608
Output headTied to the embedding: saves 8,388,6088,388,608 × 0 · tied	8,388,608	0	0
PositionsRotary: computed, not stored, saves 524,288524,288 × 0 · rotary	524,288	0	0
13 layers3,146,752 each · 81.8% of the cap			40,907,776
Attention4d²: query, key, value and output1,048,576 × 13	1,048,576	13	13,631,488
Feed-forward8d²: two matrices at 4× the width2,097,152 × 13	2,097,152	13	27,262,976
Norms2d: two weight-only norms1,024 × 13	1,024	13	13,312
Final normd: after the last layer512 × 1	512	1	512
Total98.6% of the cap			49,296,896
Unused			703,104
Cap			50,000,000

The cap counts the token embeddings and the output head.

Runs on this device

Try the model.

The same 49,296,896 parameters the calculator lands on, exported to run in your browser. Load once, then write.

It writes fluently and is often wrong. This checkpoint is from step a step not yet published of a training run that is still going.

embedding-tax-50m

Not loaded

One download, kept by your browser. Nothing you type leaves this page.

embedding-tax-50m base model

Pick an opening or type your own. The model's words follow yours in darker type as they arrive.

A base model, not a chat assistant: it continues whatever you type. It writes fluently and is often wrong.

Sampling

PresetFocused

Balanced Focused Creative Published

Uses Balanced's sampler but stops at the end of the first finished passage (about 55 tokens). Shorter, faster, and with less room to drift off topic.

Temperature

Higher is wilder. 0.6 to 1.4, the range tested with the guards on.

Top-k

Sample from the k likeliest tokens. 40 to 200, the values tested with the guards on.

Max new tokens

Stops sooner at end of text. Up to 90, the longest length tested; longer runs drift further from the prompt.

Unlimited

At most 256 new tokens. It stops sooner at the end of the first finished passage, when it starts repeating, or if a very long prompt fills its 1,024-token context. With the guards on, stopping at the end of the passage scored more coherent in blind tests than running to 90 tokens: Focused is Balanced with this on. Published with this on was not part of those tests, so it is not recommended there.

GuardsOn

Every preset except Published adds these rules, and so do your own settings made from one of them; Published, with or without Unlimited, has none. They only change which token can come next; the model's words are never edited.

Stays on your prompt. Each step also runs the model with no prompt at all, then favours tokens your prompt made likelier (context-aware decoding, strength 1). Two passes per token, so about half the speed.
Skips long shots. Drops any token less than about a tenth as likely as the top choice (min-p, set from the temperature; stricter below 1).
Repeats less. Tokens already in the last 64, your prompt included, are made less likely (repetition penalty 1.2).
No blank start. The first 3 tokens cannot be a line break or the end of the text.
Breaks after sentences. A line break can only follow a finished sentence.

Seed

Same seed, same text, on the same backend.

Same seed

Focused · temperature 0.7 · top-k 40 · until done, at most 256 tokens · guards on, min-p 0.1 · seed 1337

Shape: 13 layers · 16,384-token vocabulary
Weights: not loaded yet
Download: not loaded yet
Backend: not loaded yet