Local models in 2026: what fits on phones, laptops, boards and GPUs

Every Gemma 4 E2B reply in Gemma Gemini comes from an NVIDIA Jetson Orin Nano, a palm-sized board with 8 GB of memory shared by its CPU and GPU. By Google's estimates, Gemma 4 at 4-bit needs 2.9 GB for E2B and 17.5 GB for 31B, a range that runs from our 8 GB boards to a 24 GB graphics card.

Memory sets the limit

A model needs room for its weights, its context (the KV cache, which grows with every token) and everything else on the machine. Google's load estimates for Gemma 4 include 20% overhead and leave out the context (Gemma 4 docs):

Gemma 4 size	BF16	8-bit (SFP8)	4-bit (Q4_0)
E2B	11.4 GB	5.7 GB	2.9 GB
E4B	17.9 GB	8.9 GB	4.5 GB
12B	26.7 GB	13.4 GB	6.7 GB
26B A4B	57.7 GB	28.8 GB	14.4 GB
31B	69.9 GB	34.9 GB	17.5 GB

The "E" in E2B means effective: 2.3 billion effective parameters, 5.1 billion counting the per-layer embedding tables used for lookups. The 26B A4B is a mixture-of-experts model with 3.8 billion parameters active per token, and all 25.2 billion sit in memory. All five sizes are Apache 2.0.

Four tiers of hardware

Phones

Google says E2B and E4B "run completely offline with near-zero latency across edge devices like phones, Raspberry Pi, and NVIDIA Jetson Orin Nano" (Gemma 4 announcement). Its mobile builds (LiteRT-LM) load E2B in 1.1 GB and E4B in 2.5 GB. Liquid AI reports 20 tokens per second for its LFM2.5-VL-3B vision-language model on a Galaxy S26 Ultra, a figure we have not checked.

Laptops with unified memory

On Apple Silicon the CPU and GPU share one pool of memory. Google says Gemma 4 12B runs "with just 16GB of VRAM or unified memory" (Gemma 4 12B announcement). Liquid AI's LFM2.5-24B-A2B, released October 13, has 24 billion parameters with 2 billion active per token. Its Q4_K_M file is 14.5 GB on Hugging Face.

Our Base Drive, a USB drive that sets up private AI on Apple Silicon Macs and Windows 11 laptops, targets this tier. It is in development, planned at $19.99, and not available to order yet.

Small boards

NVIDIA's Jetson Orin Nano Super Developer Kit has a 6-core Arm CPU, a 1,024-core Ampere GPU and 8 GB of LPDDR5 at 102 GB/s, and draws 7 to 25 watts. NVIDIA's Jetson FAQ lists it at $399. We run five Orin Nano boards (Inside the Offline Base Jetson fleet), and our planned Base Router home box uses the same board.

Our boards run JetPack 7, and Ollama's CUDA build there has no kernels compiled for the Orin GPU, so each new prompt shape triggers a one-time compile of 5 to 35 seconds.

Desktop GPUs

A 16 GB graphics card holds the 26B A4B at 4-bit (14.4 GB before context) with little to spare. The 31B needs 17.5 GB, which fits a 24 GB card. tryingopen.com runs Gemma 4 26B A4B, Gemma 4 31B and LFM2.5-24B-A2B on cloud providers, free in your browser with no account, so you can try them before buying a card.

Runtimes

llama.cpp (MIT, C and C++) loads GGUF files, supports 1.5-bit to 8-bit quantization, and runs on CUDA, Metal, Vulkan, HIP, SYCL or plain CPUs. llama serve -hf <repo> downloads a model from Hugging Face and starts an OpenAI-compatible server.

Ollama (MIT, Go) serves its own API plus a subset of the OpenAI API, by default on port 11434 of the local machine. Unless told otherwise, it unloads a model 5 minutes after the last request. Our boards set keep-alive to forever and preload the model at boot.

In the 0.40 releases, GGUF models run through llama.cpp's llama-server. The latest stable release is 0.40.4, from October 12. The model tag Gemma Gemini runs:

ollama run hf.co/ggml-org/gemma-4-E2B-it-GGUF:Q4_0

On one of our boards, moving LFM2.5-VL-3B from Ollama to llama-server with four slots raised total throughput from 25 to 66 tokens per second:

Setup	Whole board	Each chat	First word
Ollama	25 tokens/s	25 tokens/s, one chat at a time	up to 20 s of waiting
llama-server, 4 slots	66 tokens/s	about 19 tokens/s	0.1 to 0.3 s

On October 1, when every board ran LFM2.5-VL-3B, we moved that model's chat traffic to the llama-server binary that ships inside Ollama, with four 8K-token slots per board. Gemma 4 E2B still runs under Ollama.

GGUF files and quantization names

GGUF is the single-file format llama.cpp loads, and the one Ollama uses for models it runs through llama.cpp. A GGUF file holds the weights plus typed metadata such as the architecture and tokenizer. The suffix names the quantization:

• Q4_0: blocks of 32 weights in 18 bytes, so 4.5 bits per weight.
• Q8_0: 34 bytes per 32 weights, 8.5 bits per weight.
• Q4_K_M: mostly 4-bit k-quants with some tensors at 6 bits; llama.cpp measured 4.89 bits per weight on Llama 3.1 8B.
• BF16: plain 16-bit weights.

mmproj side files hold vision and audio encoders. ggml-org's Gemma 4 E2B repository has Q4_0 at 2.84 GB, Q8_0 at 4.97 GB and BF16 at 9.31 GB, and its Q4_0 is built from Google's quantization-aware training checkpoint. Quantization explained with two real models covers the trade-offs.

What fits in 8 GB

Each of our boards runs one chat model:

Model	Quantization	File	On a board
Gemma 4 E2B (Google)	Q4_0	2.84 GB	text only, fully on the GPU
LFM2.5-VL-3B (Liquid AI)	Q4_K_M	1.67 GB plus a vision projector	about 3 GB in memory with vision

Gemma 4 E2B runs at 25 to 29 tokens per second on a board, and the first word usually arrives in 0.2 to 1.3 seconds once the model is warm (Gemma 4 E2B on a Jetson Orin Nano). LFM2.5-VL-3B reads images and has a 32K context. Its LFM Open License v1.0 covers commercial use only for organizations under $10 million in annual revenue.

Two smaller models share boards with a chat model:

• EmbeddingGemma 2 is a 310 MB Q8_0 file. On our first try, Ollama put it on the GPU next to Gemma with only 864 MiB free, and the board dropped offline; we suspect memory pressure. A Modelfile with PARAMETER num_gpu 0 (zero layers on the GPU) moved it to the CPU. About 700 MB stayed free, and Gemma went from 25.5 to 26.1 tokens per second (Running EmbeddingGemma 2 next to Gemma on an 8 GB board).
• d1-3B, the Liquid AI decision model that picks which tryingopen messages stay on our boards, uses about 2.7 GB on a board that also serves LFM2.5-VL-3B.

At 20:09 UTC on October 14, the live fleet page showed four of five boards online: two serving LFM2.5-VL-3B and two serving Gemma 4 E2B, one of which also runs EmbeddingGemma 2.

Where to start

• 8 GB: Gemma 4 E2B at Q4_0 or LFM2.5-VL-3B at Q4_K_M, through Ollama or llama-server. Gemma 4 12B at 4-bit (6.7 GB) would leave about 1.3 GB for context and the system.
• 16 GB of unified memory: Gemma 4 12B, by Google's numbers.
• 24 GB of VRAM: the 26B A4B or the 31B at 4-bit.

To watch a small model work on our hardware, ask Gemma Gemini an everyday question and open the chip under the reply to see where it ran. Why run AI on your own hardware covers why we run models on hardware we own.