You don't need a giant AI model for dictation — your Mac is enough

There’s a reflex, when a product involves “AI,” to assume it needs the biggest, smartest model available — GPT-class, in the cloud, behind an API key. For a dictation app, that’s the wrong instinct. The job an AI model does here is small and well-defined, and a model small enough to live on your own machine handles it comfortably.

Dictation cleanup is a small job

Think about what the model is actually asked to do after your speech is transcribed:

• Fix capitalization and punctuation.
• Drop filler words — “um,” “uh,” “you know.”
• Tighten a rambling sentence, or reshape a note into a message.
• Follow a short instruction like “make it more formal” or “translate to Russian.”

None of that needs frontier reasoning. It needs a model that writes clean, grammatical text and follows a simple instruction — which even sub-1B models do well now. Reach for a 70B cloud model here and you’re paying (in latency, money, and privacy) for capability the task never uses. Worse, a round-trip to the cloud adds noticeable lag to something that should feel instant, and it sends your words off your machine.

Apple Silicon is quietly great at this

The reason a local model is not just tolerable but genuinely good on a Mac comes down to hardware. Every Apple Silicon Mac since the M1 uses unified memory: the CPU, GPU, and Neural Engine all share one fast pool of RAM. On a typical machine a model has to be shuffled across the (slow) boundary between system memory and a separate GPU’s VRAM. On a Mac there is no boundary — the model weights sit in memory the GPU can read directly.

That has two happy consequences for local AI. First, models load and run fast without a discrete GPU. Second, the amount of model you can run is bounded by your total RAM, not by a separate, much smaller pool of video memory — so a 32 GB or 64 GB Mac can hold models that would need an expensive dedicated GPU on a PC. Apple didn’t set out to build an AI workstation, but the unified-memory design turned every modern Mac into a capable one.

What OpenWhisp ships by default

Turn on AI cleanup and OpenWhisp downloads a small open-weights model that runs on your Mac through a bundled llama.cpp runtime — no Ollama, no server to start, no account. Three sizes are one click apart in Settings:

• Qwen2.5 0.5B (~491 MB) — the default. Fastest, lowest memory, and fine for cleanup.
• Qwen2.5 1.5B (~1.1 GB) — a step up in quality for a little more memory.
• SmolLM2 360M (~271 MB) — the smallest, for tight RAM.

All three are Apache-2.0, download once, and then work fully offline. For most people this is the whole story: it’s good, it’s private, and it costs nothing. More on the private, on-device setup here.

Hackers: bring your own, as big as your Mac allows

Here’s where being open source and built on llama.cpp pays off. The built-in model is just a default, not a ceiling. OpenWhisp’s AI cleanup can point at any OpenAI-compatible server, which means you can run a far larger model yourself and have dictation use it — no code changes, just a URL in Settings.

Say you have a Mac with plenty of unified memory (or a Linux box with a real GPU) and you want your voice edits handled by something much stronger — a Qwen3 32B (or the 30B-A3B mixture-of-experts variant, which is lighter to run for its quality). Start it behind an OpenAI-compatible endpoint:

# Option A — llama.cpp (the same engine OpenWhisp bundles)
llama-server -m qwen3-32b-Q4_K_M.gguf --host 0.0.0.0 --port 8080

# Option B — Ollama, which serves an OpenAI-compatible API
ollama serve            # listens on http://localhost:11434
ollama run qwen3:32b

Then in OpenWhisp choose the Local (private) AI provider and point it at your server:

• llama.cpp default: http://localhost:8080/v1
• Ollama: http://localhost:11434/v1

That’s it. Your dictation now gets rewritten by a 32B model that never leaves your network. Because OpenWhisp bundles the same llama.cpp under the hood, a self-hosted llama.cpp server behaves exactly like the built-in one, just bigger — the ceiling is your hardware, not the app.

The point

Good local AI isn’t about cramming the largest possible model onto your laptop. It’s about matching the model to the task. Dictation cleanup is a small task, so a small model on your Mac is the right, private, zero-cost default — and Apple Silicon makes it genuinely fast. And if you want more, the door is open: it’s open source, it speaks the standard API, and it’ll talk to whatever you can host.