The number that actually predicts whether a model runs on your machine is VRAM (or unified memory on Apple Silicon), not GPU model name or marketing specs. Here’s the math in plain numbers, by parameter count and quantization.

The formula

A model’s file size at a given quantization is roughly:

size in GB ≈ (parameters in billions) × (bits per parameter / 8)

A 7B model at 8-bit (Q8) is roughly 7GB. The same 7B model at 4-bit (Q4) is roughly 3.5GB. Quantization is the single biggest lever you have — it trades a small amount of output quality for a large amount of memory, and for most self-hosted use cases (not research benchmarking), 4-bit is the sweet spot.

You need that much VRAM/RAM plus headroom for context (the conversation history and prompt) and OS overhead. As a working rule, add 10-20% on top of the raw model size for a few thousand tokens of context — more if you’re running long documents through it regularly.

Sizes by model class

These are Q4 quantization numbers, which is what most people actually run locally. Q8 roughly doubles the size; if you’re not sure why you’d want that over Q4, you probably don’t need it — the quality difference is small for most practical use, and the memory cost is not.

CPU vs GPU vs Apple Silicon unified memory

Dedicated GPU (NVIDIA/AMD): fastest by a wide margin when the model fits entirely in VRAM. The moment it doesn’t fit and layers spill over to system RAM, throughput drops sharply — a model that’s 90% in VRAM and 10% offloaded to CPU can run several times slower than one fully in VRAM, not proportionally slower. Either fit it or size down; partial offload is a last resort, not a sizing strategy.

Apple Silicon (M-series): unified memory means the GPU and CPU share the same RAM pool, so a Mac with 32GB unified memory can run models that would need a 24GB+ dedicated GPU on PC hardware, without VRAM being a separate constraint. The tradeoff is raw throughput — Apple Silicon inference is solid but generally slower token-per-second than an equivalent-tier NVIDIA GPU, even though it can fit larger models.

CPU only: works for smaller models (7B and under) if you have enough system RAM, but expect meaningfully slower generation — usable for background/batch tasks, painful for anything interactive where you’re waiting on a response in real time.

What actually limits you in practice

For most self-hosters, the honest starting point is: a 7B-13B model at Q4 on a GPU with 8-12GB VRAM, or an M-series Mac with 16GB+ unified memory. This covers general chat, coding assistance, and summarization competently, and it’s the tier where hardware most people already own is sufficient — you don’t need a new GPU purchase to start.

Jumping to 70B-class models is a different tier of investment (multi-GPU or a high-memory Mac Studio) and is worth it mainly if you’ve already hit the ceiling of what smaller models can do for your specific task, not as a default starting point.

Checking what you’ve got before you pull anything

# NVIDIA
nvidia-smi --query-gpu=memory.total --format=csv

# Apple Silicon
sysctl hw.memsize

Match the result against the table above before pulling a model, not after — pulling a 40GB file only to find it won’t load is a wasted download and, on a slow connection, a wasted afternoon. ollama pull doesn’t check your hardware for you before downloading; it downloads first, tries to load second, and fails with an out-of-memory error only at load time.