Running EmbeddingGemma 2 next to Gemma on an 8 GB board

Since October 13, 2026, the router on Gemma Gemini has been EmbeddingGemma 2, running on one of Offline Base's Jetson Orin Nano boards. That board was already serving Gemma 4 E2B, so two models now share its 8 GB of memory. These are our notes from setting it up, including the first attempt, after which the board dropped offline.

How the router turns an embedding into a decision is covered in We replaced our cloud router with EmbeddingGemma 2.

What the board has to do

For each text message the router has to judge in Auto mode, the app sends the message (up to 6,000 characters), prefixed with task: classification | query: , to the fleet gateway's /v1/embeddings endpoint. Ollama on the board returns 768 numbers. Two logistic-regression classifiers inside the app turn them into P(Gemma answers well) and P(needs the web), which takes microseconds.

So the board's extra job is one pass of a small model per message. We use ggml-org's Q8_0 GGUF of EmbeddingGemma 2, a file of about 310 MB.

The board has 8 GB of LPDDR5 shared by a 6-core Arm CPU and the GPU (NVIDIA's specs). The GPU has no memory of its own. Gemma 4 E2B already ran there under Ollama as ggml-org's 4-bit Q4_0 GGUF, fully on the GPU, at about 25 to 29 tokens per second (more on that setup).

Google's announcement says EmbeddingGemma 2 shares Gemma 4's text tokenizer and audio encoder, which gives a pipeline built on both "a lower combined total memory footprint". Ollama runs them as two separate models, so on our board each one needs its own memory.

Upgrading Ollama

The board ran Ollama 0.35.0. EmbeddingGemma 2 GGUF files need Ollama 0.40.4 or newer, the first release whose bundled llama.cpp includes the model. When we installed it, 0.40.4 was still a release candidate, so the board got 0.40.4-rc1. It has since become a stable release.

After the upgrade, Gemma kept running fully on the GPU.

The GPU attempt

Our first try loaded EmbeddingGemma 2 onto the GPU next to Gemma. Ollama's log shows only 864 MiB free at the moment it chose the GPU. Then the board dropped offline, and it took a power cycle to bring it back.

Memory pressure is our suspect. On this board, the free GPU memory Ollama measures comes out of the same 8 GB that the operating system and every other process use, and 864 MiB is a thin margin. The logs have no explicit out-of-memory line, though, so the cause is unproven.

A CPU-only copy of the model

After the power cycle we made a derived Ollama model from a Modelfile with these lines:

FROM hf.co/ggml-org/embeddinggemma-2-GGUF:Q8_0
PARAMETER num_gpu 0
PARAMETER num_ctx 512

num_gpu is the number of model layers Ollama offloads to the GPU, so 0 keeps the whole model on the CPU. The setting has dropped out of the parameter table in Ollama's Modelfile docs. Ollama still reads it, and ollama ps showed our copy at 100% CPU.

num_ctx 512 sets a small context window. Ollama's docs note that a larger context needs more memory.

We changed the processor and the context size together, so we can't say how much each one helped.

Memory and speed with both models loaded

With both models loaded, ollama ps showed:

Model	Size in ollama ps	Runs on
Gemma 4 E2B (Q4_0)	1.9 GB	100% GPU
EmbeddingGemma 2 (Q8_0, CPU-only copy)	640 MB	100% CPU

• About 700 MB of the board's memory stayed free.
• Gemma's speed held: 25.5 tokens per second before the embedding model loaded, 26.1 after.
• One embedding takes 36 ms (median) on the board's CPU. The same model takes about 20 ms on an Apple M4 Max laptop.
• The board's vectors match the Mac's at cosine similarity 0.9998 or higher, so the classifiers give practically the same scores on both.
• A full routing call, from the web app through the fleet gateway to the board and back, takes about 250 to 650 ms, mostly network.

The two rows add up to about 2.5 GB, yet only about 700 MB of the 8 GB was left. Per-model sizes leave out the rest of what runs on the board, so the number we watch is the board's total free memory. The fleet's agent reports it every two seconds.

An unload we can't explain yet

Ollama's FAQ says models stay in memory for 5 minutes by default before unloading. Our boards set keep-alive to -1, which keeps a model loaded indefinitely. Even so, the embedding model unloaded once. We haven't found the cause.

The first call after that took 3.3 seconds while Ollama loaded the model again. Gemma Gemini gives its router 2.5 seconds. After that, built-in keyword rules decide, and the route trace says EmbeddingGemma 2 didn't answer in time.

We added a warmup at boot, so the embedding model is loaded before the first message after a restart. The board already preloads Gemma at boot. An unload while the board is running would still send messages to the rules for the few seconds a reload takes, and the 2.5-second fallback keeps those replies coming.

Lessons for two models on unified memory

• Budget the whole board. On unified memory, the operating system, your services and every model draw from one pool. Watch total free memory and keep a margin.
• Choose placement yourself. Ollama's automatic choice is a guess on unified memory. PARAMETER num_gpu 0 settles it.
• Try the small model on the CPU. At 36 ms per embedding, a GPU could save at most 36 ms, small next to a few hundred milliseconds of network. The GPU stays with the model that generates text, and Gemma's speed did not drop.
• Check what the runtime did. ollama ps shows each model's size and its CPU and GPU split. Read it after every change.
• Expect cold loads. Warm models at boot, and give every caller a timeout and a fallback.
• Change one board at a time. Our fleet serves Gemma from two boards, and Gemma Gemini hands a turn to Gemini when Gemma fails before its first word. A board that goes down mid-change costs capacity while the site keeps answering.

You can see the result on gemmagemini.com: the chip under each reply opens a route trace with the router's verdict and decision time. For the fleet around this board, see Inside the Offline Base Jetson fleet, or watch live traffic on Offline Base's fleet page.