Keyword spotting on a microcontroller used to justify throwing the model away. Our 4.2-bit BERT reaches 94.1% accuracy inside 256 KB of RAM. Here is what the accuracy-vs-bits curve actually looks like.
The bit-width cliff
Post-training quantization stays flat through 8 bits, wobbles at 4, and falls off a cliff at 3. What matters in practice is where every task sits on that curve. For keyword spotting on a BERT-tiny-class architecture, we found the curve stays flat much further than common wisdom suggests — but only if the per-layer bit allocation is allowed to differ.
A uniform 4-bit shrink costs roughly nine points of accuracy on our benchmark. A mixed-precision assignment that keeps attention projections at 8 bits and lets feed-forward blocks drop to 2 recovers almost all of it inside the same memory budget.
Where the memory actually goes
On a Cortex-M4 the weights are only half the story. Activation buffers, the quantization scale tables and the tokenizer vocabulary each conspire to consume multiples of what the network needs. The winning trick was a vocabulary-pruned, integer-indexed tokenizer that removed more bytes than every quantization trick combined.
Reproduce it
The toolkit, the per-layer bit assignments and the hardware benchmark harness are open source. If you deploy models on microcontroller-class hardware, the accuracy-vs-latency sweep across targets is the part we most want people to check.