It refuses to be quietly wrong
Writing a matrix product over its own input would corrupt it with no error, so MatMul returns one instead. A NaN stops at Softmax rather than reaching a sampler.
An LLM engine, written from scratch, running in this tab.
Ember is C++20 with no ML framework, compiled to WebAssembly. The grid here is not an animation. Your browser is computing it right now, with the same math attention runs on.
It loads a 15M parameter Llama and writes stories with it, in this tab, on your machine. The model is small, so the stories are simple. The engine underneath is the point.
Point at a coal or tap one to read its number.
Made-up queries and keys through the real MatMul and Softmax, with each row scaled to its own hottest cell.
Loading the engine
It writes here, and shows where it was unsure.
Every word below comes from the C++ engine running in your browser, with no server. Each token is shaded by the entropy of the model's choice: pale where it was sure, dark where many next words looked about as good.
That signal is free to an engine that owns its logits, and it is what a local-first router needs to decide when to hand a request to a bigger model.
A 15M parameter Llama trained on children's stories, fetched from Hugging Face at a pinned revision and checked against its SHA-256 before the engine touches it.
Look inside while it reads.
The engine can record what it does on the way: every attention head's weights, and what the model would predict if it stopped after each layer. Both come from the same forward pass that writes the stories above.
Train one from scratch, here.
Ember also learns. Its own autograd computes every gradient, checked against finite differences in its tests, and AdamW updates the weights. Press train and a small transformer goes from random noise to something shaped like Shakespeare.
A 111,040 parameter transformer starts from random weights and learns Shakespeare one character at a time. Every gradient is computed by Ember's own autograd, in this tab.
Softmax turns scores into chances. Try to break it.
A language model scores every possible next word, and this one step decides how those scores become odds. Raise the temperature and the odds even out. Lower it and one word takes over.
The coals were still
glowing
n/a
warm
n/a
burning
n/a
dark
n/a
singing
n/a
Loading the engine.
The order of three loops is worth more than a factor of twelve.
MatMul and the textbook loops add the same numbers in the same order, but only MatMul reads memory in a straight line. Attention multiplies by a transposed matrix, which breaks that line, so the first benchmark changed how Ember will build attention: copy first.
- MatMul
- MatMul, transposed input copied first
- MatMul, transposed input
- Textbook loops
| Matrix size | This browser, WebAssembly | Apple M4, native, saved |
|---|---|---|
| 64 x 64 | not run | 0.018 ms |
| 256 x 256 | not run | 1.0 ms |
| 1024 x 1024 | not run | 65.2 ms |
Fastest of up to 5 runs, on one thread. The tab can pause for a moment at the largest size. The WebAssembly build has no SIMD yet, and two different machines are not a fair race.
Saved numbers come from one run of the repo's benchmark script, before any hand-written SIMD. They are a starting point, not a claim of being fast. The comparison with llama.cpp is further down.
A real model, checked against Hugging Face.
Ember loads a 15M parameter Llama trained on children's stories and runs it to the last logit. After “Once upon a time” it picks a comma, and so does Hugging Face's own library, with scores that agree to within 0.00003. A separate Llama written in NumPy agrees with every one of the 32,000.
- 3.0e-5
- largest difference from Hugging Face's own logits, checked on every push
- 156 / 156
- positions where Ember and Hugging Face pick the same next token
- 846
- tokens a second writing on one thread, with 147 MB for the whole process
- 91.9%
- of the engine's lines run by its 431 tests
- NumPy with Apple's tuned BLAS
- Ember
NumPy is still 7.6 times faster, because it hands the math to a library tuned by hand for this chip. Writing the SIMD registers out by hand made Ember 1.6 times faster with answers identical to the bit, in the Nov 8 run, which timed both commits. Every number here comes from the repo's results script, which anyone can rerun.
A 0.6B model, next to llama.cpp.
The same engine runs Qwen3 0.6B on the laptop, matches Hugging Face on it, and loads it in 8 bits. llama.cpp, the engine most local AI runs on, is still more than twice as fast. Here is by how much.
- 1.0e-4
- largest difference from Hugging Face across the top 10 logits, same next token at 73 of 73 positions
- 2.2x
- faster with 8-bit weights: 58.9 tokens a second against 26.2 in f32, 6 threads
- 2377 MB
- peak memory in 8 bits, against 3545 MB in f32
- 94.2%
- of positions where 8 bits picks the same next token as f32, mean KL 0.011 nats
- llama.cpp
- Ember
- llama.cpp
- Ember
Ember runs at 46% of llama.cpp's speed writing and 40% reading, with the same model in the same 8-bit format. It was 33% and 19% before integer dot products and a lock-free thread pool. The 58.0 here and the 58.9 above are two runs by two scripts on the same build. The browser runs only the 15M model, because Qwen3 needs more memory than a WebAssembly tab should take.
It can tell when it is likely wrong.
Qwen3 0.6B answered all 1319 GSM8K test questions in Ember. The highest per-token entropy in each answer, read straight from the logits, flags the wrong ones better than question length does, so a router can keep easy questions local and send the rest to a bigger model.
- 62.1%
- of GSM8K questions Qwen3 0.6B gets right on its own, in Ember
- 28.9%
- of held-out questions escalated by a threshold set on the other half
- 51.2%
- of the local model's mistakes caught by those escalations, 95% CI 45.0 to 57.5
- 73.3%
- accuracy on the questions kept local, against 61.2% keeping everything
- Highest entropy in the answer
- Question length
- Random
The gap over question length is +0.053 AUROC, 95% CI +0.023 to +0.094. No cloud model was called: escalated questions are scored with the answer key, so this measures whether the signal finds the local model's mistakes, not what a combined system would score. One model, one benchmark, greedy decoding.
Built so that you can check it instead of trusting it.
The source, tests, and results are on GitHub. The engine also ran these on itself in this tab when the page loaded.
- 32-bit memory checks
- running
- Math checks
- running
Machines check every change to main
CI builds with GCC and runs the tests under AddressSanitizer and UBSan. UBSan caught a real null pointer bug in MatMul on Nov 6, before that code was ever pushed.
It gets checked against the definition
Every layer and the whole model run against a second version written straight from the formulas. Breaking RoPE, the attention scale, or the causal mask on purpose makes those tests fail, so they catch real bugs.
Model files are treated as hostile
A downloaded file can lie about its own sizes. Every truncation of a model and thousands of corrupted headers have to fail cleanly under AddressSanitizer, and a fuzzer attacks the loader on every push.