We replaced our cloud router with EmbeddingGemma 2

On October 13, 2026, we swapped the router in Gemma Gemini for one built on EmbeddingGemma 2 that runs on our own Jetson. In Auto mode, that router decides whether Gemma 4 E2B on an NVIDIA Jetson Orin Nano or Gemini 4 Argon through OpenRouter answers each message. How Gemma Gemini decides who answers each message covers the rest of the path.

Until then, the router was Jev, TypeSafe's decision model. Jev returned two probabilities per message: that Gemma answers it well, and that it needs the web. It decided in roughly 200 to 250 ms and scored 98% on our held-out test set. It also sent every Auto-mode message to TypeSafe's cloud before either model saw it.

The goal

We wanted the routing decision made on Offline Base's own hardware. Offline Base holds that small open-weight models handle most everyday questions, and a cloud router sat awkwardly next to that. tryingopen had moved its routing onto a Jetson with Liquid AI's d1-3B on October 12 (From tryingopen to Gemma Gemini).

EmbeddingGemma 2 came out on October 6. It is small enough to share a board with Gemma, and Google's announcement lists routing among its uses.

164 messages, labeled by hand

Each of the 164 labels records which side should answer and whether the message needs the web. A Gemma label means Gemma 4 E2B, with a web search in front of it when needed, can answer well. We graded Jev against the same rubric.

• 118 training messages cover greetings, quick facts, translation, short rewrites, math, coding, health, legal and money questions, and current events. A "tricky" group pairs long messages with easy answers (a rambling request for puppy names) and short ones with hard answers ("Is P equal to NP?").
• 46 held-out messages, which we wrote afterward in a deliberately different style: lowercase, typos, rambling context, indirect asks, the way real visitors type. We kept them out of training while we measured accuracy.

What worked: embeddings plus two small classifiers

Our first attempt read the decision directly from EmbeddingGemma 2 and scored 64%, so we dropped it.

The version that worked splits the job in two. EmbeddingGemma 2 turns each message into 768 numbers, using the prefix the model card gives for classification, task: classification | query: . Two scikit-learn logistic-regression classifiers then read those numbers: one gives P(Gemma answers this well), the other P(the message needs the web).

Each classifier is 768 weights and a bias. The app ships both in a JSON file of about 16 KB, and scoring a message is a dot product and a sigmoid, which takes microseconds:

const sigmoid = (z: number) => 1 / (1 + Math.exp(-z));

function score(head: Head, v: number[]): number {
  let z = head.bias;
  for (let i = 0; i < head.weights.length; i++) z += head.weights[i] * v[i];
  return sigmoid(z);
}

Gemma answers when the first is 50% or higher, and Gemini 4 Argon otherwise. At 50% or higher on the second, the app runs an Exa web search before either model answers.

We trained on the 118, tested on the 46, then trained the classifiers we ship on all 164.

How it scored against Jev

Messages	EmbeddingGemma 2	Jev
46 held out	93%	98%
118 training	97% (cross-validation)	94%

On the web question, EmbeddingGemma 2 got all 46 held-out messages right.

The held-out row is the fair comparison, and Jev wins it: EmbeddingGemma 2 missed three messages and Jev missed one. The training row flatters our classifier, since cross-validation tests it on messages in the same style it learned from, while Jev saw them cold.

We wrote this small test set ourselves, so it compares the routers on our labels and says little about routing in general.

The misses

Jev's blind spot was short math problems. EmbeddingGemma 2 sent some short rewrite requests to Gemini that Gemma could have handled, and it kept a held-out health question ("i have chest tightness when i run, should i be worried, what could it be") on Gemma at 51%, one point over the bar.

A rewrite sent to Gemini costs a cloud call and usually a longer wait, and the answer is still good. The chest-pain question is the miss we care about. Two backstops remain: Gemma's system prompt tells it to say when a question needs expertise it is unsure about, and the most recent reply in a chat has an "Ask Gemini instead" option.

Size and speed on the Jetson

The model file is ggml-org's Q8_0 GGUF, about 310 MB, and it needs Ollama 0.40.4 or newer. Its embeddings match full precision at cosine similarity 0.998 or higher. The Jetson's vectors matched the Mac's at 0.9998 or higher, so the classifiers we trained on the Mac run on the Jetson without retraining.

• About 20 ms per message on an Apple M4 Max laptop.
• 36 ms median on the Jetson Orin Nano's CPU.
• About 250 to 650 ms end to end from the web app through the fleet gateway, mostly network.

End to end, that is Jev's speed at best and slower at worst. Our first try ran it on the GPU beside Gemma, and the board dropped offline. We now run a CPU-only copy (Running EmbeddingGemma 2 next to Gemma on an 8 GB board).

Running beside Jev before the switch

EmbeddingGemma 2 first ran as a second opinion: the route trace showed whether it agreed with Jev or which model it would have picked. During that side-by-side trial in the live app, it agreed with Jev on every message we sent it.

October 13: the only router

On October 13, EmbeddingGemma 2 became the only router on the live site. Jev is still supported in the code, and one setting switches back. If the router fails or takes longer than 2.5 seconds, built-in keyword rules decide, and the trace says so.

The live router's scores for six example messages, from "hi there" (80%, Gemma) to finding a race condition in a Go worker pool (10%, Gemini), are in How Gemma Gemini decides who answers each message.

What we traded

We gave up five points of held-out accuracy, two messages out of 46. In return:

• The model that reads each message for routing runs on one of Offline Base's Jetsons, and the scoring is arithmetic inside the app. Gemini replies still go through OpenRouter, and searched turns send the message to Exa.
• The router's two classifiers are a 16 KB file we can read, test and retrain with more labeled messages.

On gemmagemini.com, the chip under each reply reads something like "EmbeddingGemma 80%", and opening it shows both bars and the decision time. Two ways to route compares the two approaches side by side.