4-Bit vs 8-Bit Quantization for Local AI on Mac

Direct answer: start with a well-supported 4-bit model when memory capacity and download size decide whether the workflow runs at all. Test 8-bit when exact wording, structured output, multilingual speech, speaker similarity, or small quality differences affect the product. Bits per weight describe storage precision, not the complete runtime footprint or output quality. Token caches, activations, unquantized layers, audio codecs, and application headroom still consume memory. Compare quantizations from the same source checkpoint, conversion tool, group size, prompt, and stopping rule. Keep the smallest version that passes the real task, not the smallest file that produces a plausible demo.

Quantization decision table

Read the conversion metadata

Record source repository, revision, converter version, quantization mode, group size, per-layer exceptions, tokenizer, and any calibration dataset. A community filename is not enough provenance. Inspect the configuration stored with the model and save a checksum. MLX LM can quantize during conversion, while other ecosystems use different packed formats. Do not assume a GGUF label maps directly to an MLX-native artifact or that equal file sizes imply equal math.

Measure memory beyond the weight file

Load the longest realistic context or audio job and watch peak memory pressure, not only resident size after startup. Language models add a KV cache that grows with sequence length. Speech systems add mel features, acoustic state, waveform decoders, and output buffers. Some runtimes compile or stage extra copies. Use the Mac memory guide to reserve operating-system and companion-app headroom before deciding that a model fits.

Score the failure modes your users notice

For a language model, test schema validity, named entities, tool arguments, long-context recall, and refusal behavior. For TTS, test missing words, names, numbers, language switching, speaker similarity, noise, and long-form drift. Run blind comparisons where listening matters. Average preference can hide one critical regression. A 4-bit voice that sounds close but drops invoice numbers is not an acceptable production substitute.

Choose per model and per job

A product can offer a compact default and a higher-precision option with clear download and memory labels. Store the effective artifact in project metadata so reopening does not silently change results. Re-run the gate after converter, runtime, or model updates. The local model benchmark provides the disclosure format. Keep 8-bit only when it produces a measurable quality or reliability benefit that justifies its footprint.

Production acceptance record

Before acting on this 4-bit vs 8-bit quantization Mac recommendation, create an acceptance record for the exact feature and release. Name the owner, reviewer, user task, minimum Mac, operating system, application version, runtime, model revision, precision, input fixture, expected artifact, and pass threshold. Attach same source checkpoint and converter, quantization mode and group size, peak memory at longest task. Keep upstream documentation in one section and measurements from your own machine in another. Record every unsupported path, timeout, crash, incorrect output, and manual repair. A result counts as a pass only when the artifact is usable for the stated job. Fast but incomplete output, an unrecoverable cancellation, or a file that cannot be reopened remains a failure.

Run the record once on a clean launch, once after warmup, once with realistic companion applications open, and once after deliberately interrupting the workflow. Exercise the operating steps that matter most here: Pin one source checkpoint. Record the complete quantization recipe. Test the longest real input. Save raw timings and checks instead of copying only a winning screenshot. Include rejected options and the reason each lost, so a later framework or model update can reopen the decision without starting from memory. Assign an update trigger such as a new runtime release, model revision, minimum operating system, material support issue, or change in licensing. Review the record before changing defaults, removing a fallback, or publishing a broad performance claim.

Evidence to collect before publication

Convert one permissively licensed checkpoint to supported 4-bit and 8-bit MLX variants with the same tool and group size. Run at least 100 structured prompts or a fixed TTS script, plus 30 warm performance runs. Publish checksums, peak memory, load time, throughput, failures, and blind quality results. Keep any claimed winner limited to that model and task.

• Same source checkpoint and converter
• Quantization mode and group size
• Peak memory at longest task
• Task failures and blind quality score
• Thirty-run latency distribution

Reader worksheet

Create one row per candidate artifact with source revision, bits, mode, group size, file size, peak memory, longest tested input, quality pass rate, p50 latency, and license. Add a hard rejection reason for missing tokens or invalid output. Link the chosen speech artifact to the blind completion test.

Practical checklist

1. Pin one source checkpoint.
2. Record the complete quantization recipe.
3. Test the longest real input.
4. Score quality before speed.
5. Include application memory headroom.
6. Store the chosen artifact with project metadata.

Frequently asked questions