mlx-kquant

mlx-kquant is an MIT-licensed extension and command-line toolchain that brings GGUF-style K-quant codecs to MLX. It is aimed at developers who use Apple Silicon and want to turn an MLX or mlx-lm checkpoint into a K-quant safetensors checkpoint, run it locally, or train and merge a LoRA adapter without moving to a separate serving stack. The maintained reference is the official repository; v0.5.11 was released on 2026-11-08.

What it does

• K-quant MLX operations: Custom C++ and Metal operations expose K-quant encode, decode, matrix multiplication, and gathered MoE matrix multiplication to MLX.
• A practical CLI: mlx-kquant quantize, run, chat, lora, and fuse cover checkpoint conversion, local generation, adapter training, and merging.
• MLX-native checkpoints: The output is an MLX safetensors layout rather than a GGUF runtime replacement. The project also documents a load shim for stock mlx-lm.
• Codec coverage: The project documents K-quant and legacy codecs, plus selected IQ, ternary, and native floating-point wire formats. Support varies by operation and hardware.

When it fits

Choose mlx-kquant when you already have an MLX workflow and need an inspectable way to test lower-bit checkpoints, especially on Apple Silicon. It is not a one-click desktop model app and it does not make all quantized formats interchangeable. For general local-model management, compare Ollama; for a serving-oriented Apple Silicon option, compare mlx-serve.

Installation and workflow

The official quick start requires macOS 26.2 or later on Apple Silicon for its prebuilt Metal path. Install mlx-kquant[tools], convert a small supported Hugging Face or mlx-lm checkpoint with mlx-kquant quantize, then run a short prompt before attempting a larger model. The project documents Linux CPU paths too, but explicitly describes them as portability and CI paths rather than a throughput target.

Start with a throwaway model directory and preserve the original floating-point weights. Compare representative prompts before and after quantization, record memory use and latency on your own Mac, then use a LoRA only after that baseline is stable. Model licenses remain separate from the tool's MIT license.

Limits and safety

The project pins an exact MLX version because its extension depends on the MLX ABI and headers. Upgrades may require a rebuilt extension. Published quality and performance figures are project measurements, not independent guarantees, and codec support is not uniform. Keep API keys out of shell history, do not point experimental loaders at production model directories, and validate adapter output before using it in an agent workflow.

FAQ

Is mlx-kquant a general GGUF runner?

No. It is an MLX extension and toolchain that creates and loads MLX-native safetensors checkpoints.

Can it fine-tune a quantized base model?

The documented LoRA path keeps the quantized base frozen, trains an adapter, and can fuse the adapter back into a checkpoint.

Conclusion

mlx-kquant is a focused option for MLX developers who need K-quant experimentation, local inference, and LoRA work in one codebase. Read the official documentation, begin with a small checkpoint, and benchmark quality and resource use before relying on it.