Quantization explained with two real models

Gemma Gemini's local side runs from two quantized files. Gemma 4 E2B answers messages from a 4-bit Q4_0 file. EmbeddingGemma 2, which reads Auto-mode messages so the router can pick Gemma or Gemini 4 Argon, runs from an 8-bit Q8_0 file. Those suffixes say how each file packs its weights, and what that costs.

Gemma runs on two of Offline Base's Jetson Orin Nano boards, and one of those also runs EmbeddingGemma 2. Each board has 8 GB of memory shared by its CPU and GPU.

What quantization does

A model is mostly weights, the numbers it learned in training. Google publishes both models in BF16, a 16-bit floating-point format. Quantization stores each weight in fewer bits. llama.cpp's quantize README says it "reduces the precision of model weights (e.g., from 32-bit floats to 4-bit integers), which shrinks the model's size and can speed up inference," and that it "may introduce some accuracy loss," measured with perplexity or KL divergence.

ggml-org's Gemma 4 E2B repository shows the size effect: 9.31 GB in BF16, 4.97 GB in Q8_0 and 2.84 GB in Q4_0.

Reading a GGUF name

GGUF is the file format of llama.cpp, the engine Ollama runs these models on. One file holds the weights, the tokenizer and loading settings, and the suffix says how the weights are packed.

Name	How weights are stored	Bits per weight
BF16	16-bit floating point, as published	16
Q8_0	Blocks of 32: one 16-bit scale plus 32 8-bit integers	8.5
Q4_0	Blocks of 32: one 16-bit scale plus 32 4-bit integers	4.5
Q4_K_M	Mostly Q4_K, some tensors in Q6_K	4.89 on Llama 3.1 8B

The number after the Q is the width of each stored integer. In the _0 types, each weight is rebuilt as its integer times its block's scale, and that shared scale is why a 4-bit format costs 4.5 bits per weight. A K marks the k-quants, added in 2023, which group blocks into super-blocks. Hugging Face's GGUF page lists Q4_0 and Q8_0 as "Legacy quantization method (not used widely as of today)."

A name can also hide exceptions. ggml-org's Q4_0 Gemma file keeps its token embeddings at Q8_0, and its conversion log puts the whole file at 4.88 bits per weight.

What quantization costs in quality

llama.cpp's quantize tool lists reference figures for Llama 3 8B, where lower perplexity is better. The sizes are copied exactly as the tool prints them:

Type	Size (as llama.cpp prints it)	Added perplexity
Q8_0	7.96G	+0.0026
Q4_K_M	4.58G	+0.1754
Q4_0	4.34G	+0.4685

On that model, 8 bits costs almost nothing and 4 bits costs an amount you can measure. Q4_K_M takes about 6% more space than Q4_0 for less than half the added perplexity.

Training can shrink the 4-bit cost. On June 5, 2026, Google released quantization-aware training (QAT) checkpoints for Gemma 4 in Q4_0: "By simulating quantization during training, QAT minimizes quality loss when the model is compressed." ggml-org builds its Q4_0 E2B file from that checkpoint, so we run a legacy format on purpose. Local models in 2026 covers which Gemma 4 sizes fit which hardware.

Gemma 4 E2B at Q4_0

Google estimates the memory to load Gemma 4 E2B's weights at 11.4 GB in BF16, 5.7 GB in its 8-bit SFP8 format and 2.9 GB in Q4_0, including 20% overhead and leaving out the context window. An 8 GB board also holds the operating system and the fleet agent, and one of our two Gemma boards now holds the embedding model too, so Q4_0 leaves the room we need.

We run it under Ollama at about 25 to 29 tokens per second (Gemma 4 E2B on a Jetson Orin Nano). On the board that also runs the router, ollama ps lists Gemma at 1.9 GB, 100% GPU.

That is smaller than the 2.84 GB file. Our reading of the gap: the file's largest tensor is the per-layer embedding table, 1,260 MiB of its 2,695 MiB of weights. Gemma 4 E2B's model card says "These embedding tables are large but are only used for quick lookups." llama.cpp files that table under the input layer, which its source code always keeps on the CPU because "there is very little benefit to offloading the input layer." If Ollama counts only what sits on the GPU, that would leave the table out of the 1.9 GB, while the table still sits in the board's shared memory. We have not confirmed how Ollama counts it. We plan with Google's 2.9 GB, plus the context.

EmbeddingGemma 2 at Q8_0

EmbeddingGemma 2 turns each message into 768 numbers, and two small classifiers in the app turn those into a routing decision (We replaced our cloud router with EmbeddingGemma 2). ggml-org publishes its text model as BF16 (about 558 MB) and Q8_0 (about 310 MB), with no 4-bit file. The Q8_0 is a direct conversion of Google's BF16 weights, at 8.68 bits per weight across the file (conversion log). Ollama supports it starting with version 0.40.4.

For an embedding model, quality means each vector points the same way it would at full precision, and cosine similarity measures that: 1.0 is the same direction. We embedded our 118 labeled training messages twice, once with the Q8_0 file in Ollama and once with the original weights in Sentence Transformers. Every pair scored 0.998 or higher.

We then trained the router's classifiers on the Q8_0 vectors themselves. On the board, vectors from that file match our Mac's at 0.9998 or higher.

On the board it runs from a CPU-only copy, listed at 640 MB in ollama ps, and an embedding takes 36 ms (median), against about 20 ms on an Apple M4 Max laptop. Running EmbeddingGemma 2 next to Gemma on an 8 GB board explains why we keep it on the CPU.

Picking a quantization

Chat models

• Use the maker's QAT build when one exists. For Gemma 4 on llama.cpp or Ollama, that is Q4_0.
• Otherwise start at Q4_K_M.
• Move up to Q8_0 when memory allows.
• Budget for the context on top of the weights. Google notes that "Larger context windows require significantly more VRAM on top of the base model weights."
• Try a build on your own prompts before you rely on it.

Embedding models

Start at 8 bits. Embedding models are small, so the extra bits are cheap: our Q8_0 embedding file is about a ninth the size of the chat model's Q4_0 file. Google's QAT results for the 2025 EmbeddingGemma at 768 dimensions show what each step cost:

EmbeddingGemma (2025)	MTEB Multilingual v2	MTEB English v2	MTEB Code v1
Full precision	61.15	69.67	68.76
Q8_0 (QAT)	60.93	69.49	68.70
Q4_0 (QAT)	60.62	69.31	67.99

Q8_0 stayed within 0.22 points of full precision, and Q4_0 within 0.77. Those are first-generation QAT numbers, and our EmbeddingGemma 2 file is a plain conversion, so we measured it ourselves. The same check works for any embedding model:

• Embed a sample of real inputs at full precision and at your quantization, and compare cosine similarity pair by pair.
• Keep stored vectors and new queries on one model and one quantization. Ollama's docs say "Use the same embedding model for both indexing and querying." After a change, re-embed what you stored and retrain anything built on the old vectors.
• Run the full-precision reference in bfloat16 or float32. The EmbeddingGemma 2 model card says float16 overflows the model's activations and returns NaN or silently degraded embeddings.