Inside the Offline Base Jetson fleet

When Gemma Gemini sends a message to Gemma, it goes to Offline Base's own rack of five NVIDIA Jetson Orin Nano boards. The same fleet runs EmbeddingGemma 2, which has decided who answers in Auto mode since October 13, and serves the edge-device models on tryingopen.

The boards

Each board has 8 GB of memory shared by its CPU and GPU. NVIDIA's spec page for the Jetson Orin Nano Super Developer Kit lists:

Part	Spec
GPU	NVIDIA Ampere, 1,024 CUDA cores, 32 Tensor Cores
CPU	6-core Arm Cortex-A78AE at 1.7 GHz
Memory	8 GB 128-bit LPDDR5 at 102 GB/s
AI performance	67 INT8 TOPS
Power	7 to 25 W

"Super" is the original Orin Nano hardware with higher clocks, enabled by a JetPack update in December 2024, which raised it from 40 to 67 sparse INT8 TOPS (33 dense) (NVIDIA's technical blog). Our boards run in MAXN SUPER mode, and under a full load test they peaked at 20 W and 56 to 62 °C.

Boards dial out

Each board runs a small Python agent that keeps one outbound WebSocket open to the control plane. That socket carries telemetry, logs, commands and model traffic, so the boards accept no incoming connections. Ollama listens only on each board's loopback interface, and every model request comes through the gateway, which checks the caller's key first.

The control plane is one FastAPI process with a SQLite database, running on one of the five boards behind an encrypted tunnel. If that board goes down, the gateway and console go with it. The other boards keep their models loaded and reconnect when it comes back. Until then, Gemma Gemini sends every message to Gemini 4 Argon. A watchdog restarts a stopped or hung control plane.

The console

Agents send tegrastats readings (CPU, GPU, memory, temperatures, power) every 2 seconds and a state report every 30 seconds. The control plane keeps a sample per board every 10 seconds for 7 days and streams each board's logs live.

Remote jobs run one action across many boards: reboots, power modes, Ollama model pulls, loads and unloads, Docker containers and shell commands. Admins also get a browser terminal, and the control plane writes every change to an audit log. Alerts fire for a board silent for 45 seconds, a board at 80 °C and a stopped Ollama.

The gateway

The gateway speaks the OpenAI API, so standard clients work with it.

Endpoint	What it does	Who uses it
GET /v1/models	Lists models on online boards, where each is loaded, and the combos	Gemma Gemini's health check
POST /v1/chat/completions	Sends a chat to one board, streamed or whole	Gemma Gemini, tryingopen
POST /v1/embeddings	Embeds text on a board	Gemma Gemini's router
POST /v1/route	Answers "local" or "cloud" with Liquid AI's d1-3B decision model on one of the boards	tryingopen

One key per app

Each app gets its own API key, shown once and stored only as a SHA-256 hash. A key can carry a requests-per-minute limit, a daily token limit that resets at 00:00 UTC, and a daily limit on cloud answers from combos. The gateway meters every request by key, model and board, and records its time to first token. Offline Base gives out free keys with a daily allowance; ask for one on the fleet page.

Picking a board

A board is eligible when it is online, out of maintenance, has the model and has fewer than 8 requests in flight. The lowest score wins:

score = 10 * in_flight + 6 * needs_load + gpu_percent / 20

needs_load is 1 when the model still has to load, so the least busy board wins, preferring a warm one. With no board holding the model, the gateway returns 503 no_capacity. When every board that has it is full, it returns 429 fleet_busy at once with Retry-After: 5, and Gemma Gemini hands that turn to Gemini 4 Argon.

A caller can pin a request to one board, or limit it to a pool of boards, and a maintenance flag drains a board from routing.

Ollama, warm at boot

One bootstrap command sets up a newly flashed board. It installs Ollama on loopback with keep-alive set to -1, so loaded models stay in memory. It also adds a preload service that loads the board's model at every boot, so the first request after a restart finds it warm. When we moved two boards to Gemma on October 8, we pointed their preload at Gemma so a reboot keeps the layout.

One quirk: the boards run JetPack 7, and Ollama's CUDA build lacks native code for the Orin GPU, so each new prompt shape costs a one-time compile of 5 to 35 seconds. Once warm, Gemma 4 E2B under Ollama writes about 25 to 29 tokens per second, with the first token usually arriving in 0.2 to 1.3 seconds (more on that setup).

Ollama ran Liquid's LFM2-family models one chat at a time, so since October 1 those chats go to llama-server, the llama.cpp server bundled with Ollama, which serves four at once.

What it serves

An 8 GB board holds one chat model. Since October 8, three boards have run Liquid AI's LFM2.5-VL-3B and two have run Gemma 4 E2B. At 21:39 UTC on October 14, four of the five boards were online and the public stats showed:

Model	Build	Boards	Used by
LFM2.5-VL-3B	Liquid AI's Q4_K_M GGUF	2 (third board offline)	tryingopen's default edge model
Gemma 4 E2B	ggml-org's Q4_0 GGUF	2	Gemma Gemini, tryingopen
EmbeddingGemma 2	ggml-org's Q8_0 GGUF, on the CPU	1 (shares a Gemma board)	Gemma Gemini's router

Gemma runs as Q4_0, a 2.84 GB file that ggml-org builds from Google's quantization-aware training checkpoint (why that matters); ggml-org publishes no Q4_K_M build of E2B. EmbeddingGemma 2 runs on the CPU of one Gemma board. Running EmbeddingGemma 2 next to Gemma on an 8 GB board explains why. Since September 30 the fleet had answered 6,448 requests and handled 4,170,806 tokens, all on the boards, and the four online boards were drawing 12.1 W together.

Combos

A combo puts a board model and a larger cloud model from the same family behind one model id. A board answers by default. The cloud model takes over when:

• the caller asks for it
• the board model can't read the chat's images
• the chat outgrows an 8K-token slot
• no board is free
• the d1-3B decision model judges the message too hard
• the board fails, or takes more than 45 seconds to start

OpenRouter then carries the chat to the cloud model, up to a fleet-wide default cap of 1,000 cloud answers a day.

combo/gemma-gemini pairs Gemma 4 E2B with Gemini 3.8 Flash, and two boards can serve its local side. combo/nemotron-lightning pairs NVIDIA's Nemotron 3 Nano 4B with Nemotron 3.5 Lightning, and no board carries its local model today. The public stats show zero cloud tokens since September 30.

Gemma Gemini does its own routing: EmbeddingGemma 2 picks the model, the app requests Gemma 4 E2B from the fleet, and the app calls Gemini 4 Argon through OpenRouter directly. How Gemma Gemini decides who answers each message follows one message through those steps.