Small models can handle most everyday questions

We labeled 164 messages by hand to train Gemma Gemini's router, and 88 of them (54%) went to Gemma 4 E2B, Google's small open-weight model, running on our own NVIDIA Jetson Orin Nano boards. The rest went to Gemini 4 Argon through OpenRouter. On gemmagemini.com, a router picks one of the two for every message.

We started Offline Base on one belief: small open-weight models are good enough for most everyday questions, and a frontier model is for the questions that need it. Why Offline Base built Gemma Gemini covers the company side.

The labeled set

Each label answers two questions: can Gemma 4 E2B, with a web search in front of it, answer this well, and does the answer need the web? We trained on 118 messages and tested on the other 46, which we wrote later in a looser style with lowercase, typos and rambling, then retrained the live router on all 164.

Set	Messages	Gemma	Gemini	Needs the web
Training	118	64	54	17
Held out	46	24	22	7
Total	164	88	76	24

We chose the categories and their sizes, so Gemma's 54% share describes our set, and real traffic could lean either way. We replaced our cloud router with EmbeddingGemma 2 explains the training.

What a small model answers well

Six of the 14 training categories went entirely to Gemma:

Category	Messages	Example
Casual chat	10	"hi there"
Short facts and definitions	10	"What does 'ephemeral' mean?"
Simple explanations	8	"Why is the sky blue?"
Rewrites and translations	8	"Fix the grammar: me and him goes to the store yesterday"
Short creative writing	8	"Write a haiku about rain"
Quick lookups, after a search	10	"Who won the Yankees game last night?"

Each asks for a short answer from common knowledge, the person's own text or fresh search results, which suits Gemma 4 E2B and its 2.3 billion effective parameters (model card).

We labeled by task. A long, rambling request for dachshund puppy names went to Gemma, and the five-word "Is P equal to NP?" went to Gemini. Nine messages in Spanish or French split six to Gemma and three to Gemini.

What a web search adds

Gemma 4's pre-training data has a cutoff of January 2025, per the same model card, so last night's score has to come from a search. Of our 164 messages, 24 need the web, and we labeled 16 of them for Gemma: weather, game results, prices, store hours and headlines. When the router's web score is 50% or higher, the app runs an Exa search first and hands the results to whichever model answers, with instructions to cite them.

Without that search, the app would send every message that needs the web to Gemini, and Gemma's share would fall from 88 messages to 72. With it, "What's the weather in NYC today?" scores 90% for needing the web and 94% for Gemma, so it gets a search and stays on a Jetson. On the 46 held-out messages, the router's web call matched our labels every time.

The other 8 web messages, across both sets, ask for research across many sources. They go to Gemini, which gets the search results too. How Gemma Gemini decides who answers each message covers the search settings.

What needs the big model

Six categories went entirely to Gemini: code (12 messages), math and logic (10), health, legal and money questions (8), long analysis and plans (8), research across many current sources (6), and complaints about a previous answer (4).

Code and multi-step math show the clearest gap. On Google's model card, Gemma 4 E2B scores 44.0% on LiveCodeBench v6 and 37.5% on AIME 2026 without tools. Gemma 4 31B, the largest model in the same family, scores 80.0% and 89.2%.

Health, legal and money questions go to Gemini because a wrong answer there costs more than a clumsy haiku. One training message reads "That's wrong. Try again and actually check your math this time." Complaints like that go to Gemini too, so a retry gets the larger model.

Gemma answers when the router gives it a 50% chance or higher. Scores from the live router:

Message	Chance Gemma answers well	Answered by
"hi there"	80%	Gemma
"Translate 'thank you very much' to Japanese"	90%	Gemma
"What's the weather in NYC today?"	94%	Gemma, after a search
"Prove that the square root of 2 is irrational"	20%	Gemini
"Is it safe to take ibuprofen with my blood pressure medication lisinopril?"	23%	Gemini
"Find the race condition in a Go worker pool using a shared map"	10%	Gemini

All six are training messages the router has seen, so the held-out results under Limits are the fairer test.

In Auto mode on gemmagemini.com, a reply the router decided carries a chip with its confidence (for example "EmbeddingGemma 80%"), and opening it shows Gemma's chance against the 50% bar, so you can test our labels against your own questions.

The speed difference

On a Jetson Orin Nano, Gemma 4 E2B writes about 25 to 29 tokens per second, and with the model warm its first word usually arrives in about 0.2 to 1.3 seconds. One live reply started at 1.1 seconds and ran at 29 tokens per second for 308 tokens. Gemma 4 E2B on a Jetson Orin Nano has the setup.

Gemini 4 Argon always reasons before it answers, and we run it at high effort, which Google's guide describes as "Most tokens, slowest first token." In our testing, Gemini took about 14 seconds end to end to write a full proof. A greeting doesn't need that reasoning, and on a Jetson Gemma starts replying in about a second. Routing adds about 250 to 650 ms, mostly network.

What a Stanford and Together AI study found

Our About page cites a larger study by researchers at Stanford University and Together AI. They ran more than 20 local models, each with at most 20 billion active parameters, on about a million single-turn queries, half of them real ChatGPT prompts from WildChat. When each query went to whichever local model got it right, local models answered 88.7% of them. The best single model, gpt-oss-120b, averaged 71.4%, and the share a local model could serve rose from 23.2% in 2023 to 71.3% in 2025.

The 88.7% assumes a perfect pick for every query, so it is a ceiling. Gemma 4 E2B sits far below the study's 20 billion limit, which is why Gemma Gemini keeps a frontier model one route away.

Limits

• We wrote the labels. All 164 labels reflect our judgment, and the router learns to copy them. A wrong label becomes a wrong route.
• The test set is small. EmbeddingGemma 2 routed 43 of the 46 held-out messages correctly (93%). Jev, the cloud router it replaced, missed one (98%). Each miss moves the score about 2 points. Two ways to route compares them.
• Misses go both ways. The router sent some short rewrites to Gemini that Gemma could have handled. It also kept a chest-pain question on Gemma at 51%, the miss that worries us. When Gemma has no search results, we instruct it to flag questions that need expertise it is unsure about, and "Ask Gemini instead" resends the latest message.
• Text only. Images and PDFs go to Gemini, because this Gemma setup reads only text.
• Facts age. If a search fails, Gemma answers from what it learned, and its model card warns that the models "may generate incorrect or outdated factual statements."