Why run AI on your own hardware

Since early October we have run a small model on hardware we own beside a large one in Google's cloud: Gemma 4 E2B on Offline Base's NVIDIA Jetson Orin Nano boards, and Gemini 3.8 Flash, then Gemini 4 Argon from October 13. Running both in Gemma Gemini has given us real numbers on what local hardware buys you and what it costs.

What you get

What you type stays on the machine

When the model runs on your own computer, the prompt and reply exist only in that computer's memory and on its disk. No provider receives the text, so none can log or keep it. Offline Base builds its products around that.

On October 13, Gemma Gemini moved its routing step onto local hardware. Auto mode used to ask Jev, TypeSafe's hosted decision model, to pick who answers. Now EmbeddingGemma 2 makes that call on one of our Jetson boards.

It works without a connection

Once the model file is on disk, the network is optional. Google's Gemma 4 announcement says E2B and E4B "run completely offline" on edge devices including the Jetson Orin Nano.

A one-time hardware cost

A cloud model bills by the token. On the Gemini API, Argon costs $2 per million input tokens and $10 per million output tokens, thinking included, through November 30, 2026, then $4 and $20. We reach Argon through OpenRouter, which lists the same $2 and $10 today, so we pay per token for every Gemini turn.

Local hardware is a one-time price plus power. NVIDIA lists the Jetson Orin Nano Super Developer Kit at $399, and its spec page gives 7 to 25 W. Offline Base's sustainable page puts a typical 200-token answer at around 0.03 Wh, by spec-sheet arithmetic for a 3B model. A Gemma turn has no per-token bill; its electricity is on the order of that rough estimate. Turns that need the web also call Exa, an outside search service.

Predictable speed on your own board

On a Jetson Orin Nano, Gemma 4 E2B writes about 25 to 29 tokens per second, and with the model warm the first word usually arrives in 0.2 to 1.3 seconds. On a machine only you use, those numbers depend on your hardware and setup, since you are the only one sending it work.

Cloud timing depends on the question and the provider. A full proof from Gemini 4 Argon at high reasoning took about 14 seconds in our testing, and Google's migration notes say hard prompts can take several minutes.

You decide when the model changes

Hosted models change on the provider's schedule. On October 8, 2026, Google deprecated gemini-3.7-flash and began sending its API requests to gemini-3.8-flash automatically (changelog).

A local model is a file you keep. Our Gemma is ggml-org's Q4_0 GGUF of Gemma 4 E2B, 2.84 GB. Our router is ggml-org's Q8_0 GGUF of EmbeddingGemma 2, about 310 MB, plus 16 KB of classifier weights. Both models use the Apache 2.0 license, and each file changes only when we replace it.

What it costs you

A smaller model

Gemma 4 E2B has 2.3 billion effective parameters, or 5.1 billion counting its per-layer embeddings (model card). The router aims to give it greetings, quick facts, translation and short rewrites, though it still sends some short rewrites to Gemini. It gives Gemma a 20% chance on "Prove that the square root of 2 is irrational" and 10% on finding a race condition in a Go worker pool, so those go to Gemini. Our Gemma setup is text-only, so images and PDFs go to Gemini too.

Offline Base's benchmarks page, built from Artificial Analysis screenshots dated May 30, 2026, puts small local models around 15 on the Intelligence Index and the best cloud system shown at 45.8. Those screenshots predate Argon; Artificial Analysis now scores Gemini 4 Argon (High) at 53.

Setup

Running a model yourself means choosing a file, a quantization, a runtime and settings, and keeping them compatible. EmbeddingGemma 2 needs Ollama 0.40.4, which was still a release candidate when we installed it, so we moved a board from Ollama 0.35.0 to 0.40.4-rc1.

Memory limits

A Jetson Orin Nano has 8 GB of memory shared by the CPU and GPU, and Gemma was already running fully on the GPU. Our first try loaded EmbeddingGemma 2 onto the GPU too. The logs show 864 MiB free when Ollama made that choice, then the board dropped offline. We suspect memory pressure, though no out-of-memory line in the logs proves it.

After a power cycle, a three-line Modelfile moved the embedding model to the CPU. About 700 MB of memory stayed free and Gemma held its speed (25.5 tokens per second before, 26.1 after); Running EmbeddingGemma 2 next to Gemma on an 8 GB board has the Modelfile and the numbers.

Keeping it running

On your own hardware, you are the on-call team. Some time after the CPU fix, the embedding model unloaded once even though keep-alive was set to forever, and we still don't know why. Reloading took 3.3 seconds, past the app's 2.5-second limit, so the built-in keyword rules routed those messages. We added a warmup at boot.

Where a hybrid fits

A hybrid keeps the local benefits for messages a small model handles and sends the rest to a large one. In Auto mode, Gemma answers when the router gives it a 50% or higher chance of answering well. "What's the weather in NYC today?" scores 0.94 with a 0.90 chance of needing the web, so the app searches with Exa first and Gemma answers from the results.

Each side covers for the other. If the Jetson is offline or fails before its first word, Gemini answers, and if the cloud can't be reached, Gemma answers (unless the message has an image or PDF). The reply notes the switch, and the route trace behind the chip says which model failed and why.

A public website gives you less privacy than a model on your own laptop. On gemmagemini.com your message goes to the app's server on Vercel and then across the internet to our boards, Gemini turns go through OpenRouter to Google, and messages that need the web go to Exa as a search query. Chats are saved only in your browser.

How this connects to Offline Base

Gemma Gemini is a spinoff of Offline Base, an early-stage company working to make local AI easy for everyone. Its bet is that small open-weight models are good enough for most everyday questions, with frontier models as something you call on for the hard ones.

Offline Base's products are meant to take on those setup, memory and upkeep costs. The company is building Base Drive, a plug-and-play kit that sets up a local AI on your own computer (targeted at $19.99 once, no subscription), and Base Router, the same private AI on an always-on box for the whole home. Both are in development and not yet available to order.

To see the split, open gemmagemini.com and tap the chip under any reply.