Alternatives to halogen-flash-server

mlx-serve is a MIT Zig inference server for Apple Silicon that runs MLX and GGUF locally with OpenAI and Anthropic APIs, no Python.

An efficient local LLM inference library built around the EXL3 quantization format, running large open models on consumer GPUs with tensor and expert parallelism, and powering the TabbyAPI server.

Alibaba's open-weight architecture preview of Qwen4: 125B multimodal MoE with 6B active plus a 51B N-gram table, 262K native context, at $0.16/M input.