Guide

What actually happens when your Mac transcribes audio on-device

On-device AI is a marketing phrase attached to a specific and quite interesting piece of engineering. Here is the whole pipeline, from file to text.

30 August 2026

Every app now says it does things "on-device". It is worth knowing what that actually involves, partly because it explains why local transcription got good so quickly, and partly because it is the difference between a claim you can verify and one you cannot.

This is the pipeline, in order.

Step 1: Everything becomes 16 kHz mono #

Whisper, the speech recognition model almost all of these apps use, takes exactly one kind of input: 16,000 samples per second, one channel, as floating point numbers between -1 and 1.

So the first thing any transcription tool does is throw away most of your audio. Your 48 kHz stereo recording is downmixed and downsampled. Video files have their audio track extracted and the picture discarded.

This sounds destructive and mostly is not: human speech lives below 8 kHz, and by the Nyquist limit a 16 kHz sample rate captures all of it. What you lose is music quality, not intelligibility.

The practical consequence: your expensive 96 kHz recording gains you nothing at this stage, and a badly compressed file cannot be rescued by it either.

Step 2: Voice activity detection decides what is worth transcribing #

A second, much smaller model runs first and marks which parts of the audio contain human speech.

The model in common use is Silero VAD — under a megabyte, and it answers one question: speech or not speech, on a fine time grid.

This exists for two reasons. It makes things faster, because decoding silence is wasted work. More importantly it prevents the failure mode where Whisper, handed audio containing no speech, generates text anyway and loops. VAD is why modern local transcription stopped producing pages of repeated phrases.

Step 3: Audio becomes a picture #

Whisper does not consume waveforms. It consumes a log-Mel spectrogram: a 2D image where the horizontal axis is time, the vertical axis is frequency on a scale that matches human hearing, and brightness is energy.

This is the actual conceptual leap in modern speech recognition. Once audio is an image, you can use the machinery built for computer vision on it — and that is exactly what happens next.

Step 4: The encoder runs, ideally on the Neural Engine #

Whisper is a transformer with two halves. The encoder reads the spectrogram and turns it into a dense numerical representation of what was said. This is the expensive half — heavy, regular matrix multiplication over a fixed-size input.

Which is precisely what Apple's Neural Engine is built for: a dedicated block on the chip that does one narrow kind of arithmetic extremely fast and at very low power. It is not general-purpose — you cannot run arbitrary code on it — but a transformer encoder is exactly its shape.

Getting the encoder onto it requires compiling the model into Core ML's format (.mlmodelc) ahead of time. Apps that do this ship a Core ML encoder alongside the model weights, and it is the single biggest speed difference between a fast local transcription app and a slow one on the same hardware.

Step 5: The decoder runs on the GPU #

The decoder produces the actual text, one token at a time, each one conditioned on everything before it.

This half is sequential, so it does not suit the Neural Engine's batch-oriented design. It runs on the GPU through Metal instead. On Apple Silicon that GPU shares memory with the CPU, so the model's weights do not have to be copied across a bus — a large part of why Apple Silicon punches above its weight for local inference.

Step 6: Timestamps come out of the same process #

Whisper emits special timestamp tokens interleaved with the words. That is how you get an SRT with sensible timings rather than a wall of text — the timing information falls out of the same decoding pass, not from a separate alignment step.

When VAD has skipped silence, the timings have to be mapped back onto the original recording, or every timestamp after the first gap would be wrong.

The model, and why it is smaller than it sounds #

Whisper's Base model is 74 million parameters. At full precision that is about 140 MB, most of which is redundant for this purpose.

Quantisation stores each weight with fewer bits. In a q5_1 model, weights are 5 bits plus a small per-block scale and offset — roughly 6 bits per weight instead of 16. The result is around 60 MB with accuracy differences that are hard to detect on ordinary speech.

That matters for a phone, where a 140 MB download and the memory to match is a real cost. It is also why the honest answer to "which model should I use" is usually a smaller one than people assume.

Long files, and why memory is the hard part #

Whisper's API takes one contiguous buffer of samples. Two hours at 16 kHz is about 470 MB of floats, plus a spectrogram of comparable size.

On a Mac, fine. On an iPhone, that is enough to get an app terminated by the system — which is why some mobile transcription apps quietly limit file length.

The fix is to transcribe in windows: keep the decoded audio on disk, map in one slice at a time, transcribe it, release it. Peak memory then depends on the window rather than the file. The subtlety is where to cut — split mid-word and you get a mangled sentence at every boundary, so a sensible implementation cuts at the quietest point within an overlap region, and shifts each window's timestamps back into the original timeline.

Why any of this matters to you #

Three things follow from the pipeline above that are not marketing claims:

It works with no network. There is no step above that requires one. This is checkable in thirty seconds — we wrote up how, including tests that do not require trusting us.

Your recording is not anyone's training data. It is not that a local app promises not to train on your audio. It is that your audio was never anywhere it could be.

Speed depends on your hardware, not someone's queue. A local transcription runs several times faster than real time on Apple Silicon, consistently, at 3am on a Sunday, without a rate limit.

The short version #

Audio is downsampled to 16 kHz mono, voice detection strips out the parts with no speech, the remainder becomes a spectrogram, a transformer encoder processes it on the Neural Engine, a decoder generates text on the GPU with timestamps interleaved, and quantisation makes the model small enough that all of this fits on a phone.

None of it needs a server, which is the whole point.