ExLlamaV3

ExLlamaV3 is an inference library for running local LLMs on modern consumer GPUs. Its headline feature is the EXL3 quantization format built on QTIP, which lets big open weights fit on everyday hardware at surprisingly usable speed. It also adds flexible tensor-parallel and expert-parallel inference for consumer setups, continuous dynamic batching, speculative decoding, 2 to 8 bit cache quantization, multimodal and LoRA support, and a Hugging Face Transformers plugin. The recommended backend is TabbyAPI, which exposes an OpenAI-compatible server for local or remote inference.

Compare Ollama if you want a simpler scriptable local runtime, or a model like Llama if you are shopping for weights to run locally.

Key Features

• EXL3 quantization: a QTIP-based format that compresses weights so large models fit on consumer VRAM.
• Parallel inference: tensor-parallel and expert-parallel execution sized for consumer hardware setups.
• Dynamic batching: continuous batching keeps throughput high under changing request load.
• OpenAI-compatible server: TabbyAPI wraps it as a drop-in API for local or remote inference.
• Broad architecture support: runs DeepSeek, GLM, Gemma, Cohere Command, multiple MoE, and multimodal models.

Use Cases

Who Should Use This Tool?

• Local-model enthusiasts: run open weights on a single GPU without a cluster.
• Privacy-conscious teams: keep inference in-house behind an OpenAI-compatible endpoint.
• Power users of quantized models: largest practical models on the VRAM you already own.

Problems It Solves

1. VRAM ceiling: good quantization squeezes large models onto affordable GPUs.
2. Throughput under load: dynamic batching keeps a server responsive with many concurrent requests.
3. Vendor dependence: a local OpenAI-compatible server removes the need for a paid hosted model endpoint.

Pricing

ExLlamaV3 is MIT licensed and free. The only cost is your own GPU hardware and electricity, or the instance you pay for if you rent a GPU.

Advantages & Unique Selling Points

1. EXL3 as the default: quantization and inference are engineered together for speed per GB of VRAM.
2. Consumer-first parallelism: scaled for one or a few GPUs, not a data center.
3. API-compatible: TabbyAPI gives you a familiar OpenAI-shaped endpoint on top of the runtime.

Getting Started

1. Install the library and download an EXL3 quantized model.
2. Point a backend like TabbyAPI at the model to expose an OpenAI-compatible API.
3. Set the tensor and expert parallelism for your GPU count, then send requests.

Frequently Asked Questions

What is EXL3?

It is an EXL quantization format based on QTIP designed to let large models run on consumer GPUs at good speed.

Does it replace llama.cpp?

It is a different runtime with its own quantization and parallelism choices. Many users keep both and pick per model.

Can I use it as an API?

Yes. TabbyAPI is the recommended OpenAI-compatible server for ExLlamaV3.

Alternatives

• Ollama: simple local models plus a cloud option.
• Llama: Meta's open-weight family if you are choosing weights.
• OpenCode: an agent that can call a local API endpoint.

Tips

1. Set cache quantization to a low bit count to free VRAM for context.
2. Choose tensor parallelism to match your VRAM pool for large MoE models.
3. Use TabbyAPI to reuse existing client code with a local endpoint.

Conclusion

ExLlamaV3 makes large open models practical on hardware you already own. If you want a fast, EXL3-optimized local runtime behind an OpenAI-compatible API, start at github.com/turboderp-org/exllamav3.