Mote is a community project for running capable language models on small, common hardware. Here it is on a Raspberry Pi 5: four Arm Cortex-A76 cores, 8 GB of RAM, no accelerator. Speculative decoding, hand-written A76 kernels, and Arm KleidiAI raise general-text throughput from 6.6 to 11.2 tokens per second, and the default mode leaves the model's output unchanged.
The Intelligence of Things. The capability that lives in datacenters should also run on the processors already around us: a board on a bench, a controller in a robot, a sensor in a field. Mote is the inference stack to put it there, starting with the Raspberry Pi 5 and the Arm cores that permeate much of the embedded world.
Every number is measured on a Raspberry Pi 5 with the Q4_0 weights and four threads. The baseline is the identical GGUF served by a stock library with single-stream decoding, the path you get out of the box.
Stock single-stream decode vs. the default quality mode.
General-text decode, built up one layer at a time.
Distribution-faithful: every token is a true argmax of the full model without quality loss.
Accepts near-miss draft tokens. Trades a small amount of fidelity for more throughput on prose.
Adds an n-gram drafter for repeated spans. Built for logs, data dumps, and templated output.
Decode on a CPU is bound by memory bandwidth: each token streams the full weight set through RAM. Everything here either amortizes that read or removes a stall the stock engine leaves on Arm.
Gemma-4-E2B ships a multi-token-prediction head that drafts several tokens ahead. The main model verifies the whole draft in one batched pass and keeps the tokens that match its own greedy output. Decode is bound by memory bandwidth: every step streams the entire weight set through RAM, and that read dominates the time. The A76's int8 matmul reads those weights once for each group of four output rows, so the draft is set three tokens deep. The verify batch is then four wide, three drafts plus the token already committed, and lands on exactly one four-row group, so a single pass over the weights returns all four positions. A fourth draft would widen the batch to five, spilling a row into a second pass that streams the whole matrix again, which costs more than the token it buys.
The A76 has no bf16 dot-product instruction, so the stock engine runs the model's bf16 projection weights
through a scalar loop. The replacement,
ggml_vec_dot_bf16,
widens bf16 to f32 in register and accumulates on the NEON pipeline, following the same arithmetic as the
x86 vector path. The second,
ggml_compute_forward_pad_f32,
rewrites the per-step pad over the 256K-token vocabulary, once a per-element bounds-checked copy, as a
row-wise memcpy parallelized across the vocabulary.
KleidiAI is Arm's library of microkernels tuned for cores like the A76. Enabling it routes the Q4_0 weight matmuls, the bulk of every forward pass, through KleidiAI's int8 dot-product GEMM, which outruns the engine's own repacked kernel on this core.
The engine is compiled with link-time optimization. The dequantization and dot-product inner loops, called across file boundaries on every token, inline into their callers, and the compiler schedules the result for the A76 pipeline with the whole program in view rather than one file at a time.
Builds the tuned server, fetches the model, installs the gemma command, and applies the system tuning. Then it serves an OpenAI-compatible API and a web chat UI on port 8080.
# clone, then build + install in one step:
git clone https://github.com/mkturkcan/mote && cd mote
bash scripts/install.sh
# run it
gemma start # quality mode, lossless
gemma chat # or open http://<pi>:8080 in a browser
gemma start turbo # tuned for repetitive output