Tiny Computers GitHub

tiny-attention

Live visualizer

A FlashAttention engine in SystemVerilog, checked bit-for-bit against a golden model. It computes attention one block at a time, without storing the whole score matrix.

Open live visualizerView source on GitHub
A recording of the online softmax sketch below.

Explore the idea.

Online softmax over eight scores, in blocks of two. Each block updates the running max, sum and output. At the end, O ÷ l gives the exact softmax-weighted answer, the same as the direct formula.

Interactive illustration · Simplified model

What it shows.

Standard attention writes a full score matrix to memory. FlashAttention keeps only a few running numbers per row, so less data moves.

The live visualizer runs the compiled RTL and compares it with a golden model.

One block at a time.

Frames from the sketch as each block of scores arrives. The running max, sum and output update at every step.

After one block: The running max is the largest of the first two scores.
After one block. The running max is the largest of the first two scores.
After two blocks: Earlier sums are rescaled to the new max, not recomputed.
After two blocks. Earlier sums are rescaled to the new max, not recomputed.
After three blocks: Three blocks in, the running numbers are still three numbers.
After three blocks. Three blocks in, the running numbers are still three numbers.
Done: O ÷ l equals the direct softmax-weighted average.
Done. O ÷ l equals the direct softmax-weighted average.