Gemma 4 E2B on a Jetson Orin Nano

When the router in Gemma Gemini keeps a message local, the reply comes from Gemma 4 E2B running on an NVIDIA Jetson Orin Nano in Offline Base's fleet. The chip under the reply starts with "Gemma 4 E2B · Jetson Orin Nano", followed by the router's verdict, such as "EmbeddingGemma 80%".

What Gemma 4 E2B is

Google released Gemma 4 on April 2, 2026, in four sizes: E2B, E4B, a 26B mixture-of-experts model and a 31B dense model (announcement). A 12B model followed on June 3. All of them are open-weight under Apache 2.0, and E2B is the smallest.

The E stands for effective. Google's model card says the small models use Per-Layer Embeddings (PLE), which give "each decoder layer its own small embedding for every token. These embedding tables are large but are only used for quick lookups, which is why the effective parameter count is much smaller than the total." E2B has 2.3 billion effective parameters, and 5.1 billion once the embedding tables are counted.

The card also lists 35 layers, a 128K-token context, and text, image and audio input.

Google's launch post says E2B and E4B "run completely offline with near-zero latency across edge devices like phones, Raspberry Pi, and NVIDIA Jetson Orin Nano."

The benchmark table in the model card lists the instruction-tuned E2B at 60.0% on MMLU Pro and 44.0% on LiveCodeBench v6.

The file: a 4-bit GGUF from ggml-org

We run ggml-org's Q4_0 GGUF, gemma-4-E2B-it-Q4_0.gguf, a 2.84 GB file. ggml-org builds it from the quantization-aware training (QAT) checkpoint Google published on June 5, 2026, so the model was trained to work with 4-bit weights. The conversion keeps the token embedding table at Q8_0 (8-bit values plus a scale per 32-weight block, 8.5 bits per weight) and packs the rest as Q4_0: blocks of 32 weights sharing one 16-bit scale, which works out to 4.5 bits per weight.

Google estimates the memory to load E2B's weights at 11.4 GB in BF16 and 2.9 GB in Q4_0, before any context. On an 8 GB board, the 4-bit build fits with room to spare.

Running on an 8 GB Jetson Orin Nano

Each Jetson Orin Nano has 8 GB of LPDDR5 shared by a 6-core Arm CPU and a 1,024-core Ampere GPU, and draws 7 to 25 watts (NVIDIA's specs). Offline Base runs five of them, and on October 14 two were serving Gemma 4 E2B (live fleet page).

Ollama serves the model on the GPU. A preload service loads Gemma at boot and Ollama's keep-alive holds it in memory, so the model is already loaded when the first message after a reboot arrives. That message can still hit the one-time compile described below.

One Gemma board also runs the EmbeddingGemma 2 router on its CPU. There, ollama ps lists Gemma at 1.9 GB on 100% GPU, and about 700 MB of memory stays free (field notes). The 1.9 GB is less than both the 2.84 GB file and Google's 2.9 GB estimate.

By default the app gives Gemma an 8,192-token context on the fleet and caps its replies at 2,048 tokens. In a long chat, it keeps the newest messages that fit. Gemma 4 thinks before it answers, and the app shows that reasoning in a collapsible section. We load the text model alone, so images and PDFs go to Gemini 4 Argon.

Measured speed

Measurement	Result
Generation speed, typical	25 to 29 tokens per second
First word, model warm	about 0.2 to 1.3 seconds
One live reply	first word at 1.1 s, 29 tokens per second, 403 tokens in, 308 out
Before and after adding EmbeddingGemma 2 to the board	25.5 and 26.1 tokens per second

By our arithmetic (1.1 s to the first word, then 308 tokens at 29 per second), that live reply took about 12 seconds from send to last word.

The first-word times above are for a warm board. Our boards run JetPack 7, and the CUDA build of Ollama we use ships without native code for the Orin's GPU. The first time a board sees a new prompt shape, it stops for 5 to 35 seconds to compile.

The app waits up to 45 seconds for Gemma's first word. If nothing arrives, or no online board has Gemma, Gemini 4 Argon answers and the route trace says so.

What it answers

In Auto mode, a router built on EmbeddingGemma 2 estimates the chance that Gemma answers each message well, and Gemma takes the message at 50% or higher. "hi there" scores 80% and a request to translate "thank you very much" into Japanese 90%, while a proof that the square root of 2 is irrational scores 20% and goes to Gemini 4 Argon. How Gemma Gemini decides who answers each message has the full table.

Gemma's share is greetings, quick facts and definitions, translations, short rewrites and current questions. When the router puts the chance a message needs the web at 50% or more, the app runs an Exa search first and adds the results, with their dates, to Gemma's system prompt, and Gemma cites them as [1], [2].

Gemini 4 Argon usually gets proofs, code, long analysis, and medical, legal and money questions. Without search results, Gemma's system prompt asks it to flag in one sentence any question that needs current information, careful multi-step work or expertise it is unsure about, so the person can resend it to Gemini 4 Argon.

The router misses sometimes: it sent some short rewrites to Gemini 4 Argon and kept a chest-pain question on Gemma at 51% (test results).

Why a small model is the default

Offline Base holds that small open-weight models answer most everyday questions well. E2B is sized for that job on this hardware.

• It fits. At about 2.9 GB to load, it leaves room on an 8 GB board for context, the operating system and, on one of our boards, a second model.
• Short answers arrive in seconds. At 25 tokens per second, a 100-token reply finishes four seconds after its first word.
• It runs on hardware we own. A Gemma answer costs us the board's power, 7 to 25 watts, while every Gemini 4 Argon turn is billed per token, thinking included (Argon's prices).
• Hard messages go to Gemini 4 Argon. The router usually sends proofs, code and medical questions there, so Gemma handles the everyday share.

Try it

On gemmagemini.com, pick Gemma 4 E2B in the "Who answers" menu to send every message to the Jetson, or leave it on Auto. The chip under each reply opens the route trace with that reply's time to first word and tokens per second. There is no account, and chats stay in your browser.