halogen-flash-server

halogen-flash-server (halogen) is an inference server written for one GPU and one model family: AMD Strix Halo (gfx1151) and Qwen3.8-Flash-Next. The repo peonist-ai/halogen-flash-server was created 2026-08-26. On 2026-09-20 GitHub listed 582 stars and 33 forks. 582 stars is a heat signal, not traffic. GitHub SPDX is Other. Read LICENSE.md before you redistribute. Weights are on Hugging Face. The image tag cited in the README quickstart on 2026-09-20 was 0.12.0. r/LocalLLaMA discussed 1M-context runs on Strix Halo with this stack.

It is not a general vLLM or llama.cpp replacement. Compare mlx-serve on Apple Silicon, ExLlamaV3 on NVIDIA consumer GPUs, or Ollama if you wanted a multi-OS runner.

Key Features

• One-silicon kernels: the README's claim is that every kernel is for this GPU and this model family, with no portable fallback. That is also why it will not start on other GPUs.
• OpenAI-compatible /v1: Chat Completions and Responses APIs, tool calls, Codex CLI wire_api = "responses", structured JSON schema, images when you set HALOGEN_VISION_TOWER.
• Speculation with a greedy identity check: at temperature 0, output is byte-identical to serial greedy decode. Draft head plus prompt lookup since 0.6.0.
• Bring your own GGUF (since 0.7.0): open a llama.cpp GGUF of this model (the README names unsloth UD-IQ4_XS) on the same kernels.
• Agent-harness notes: Pi, OpenCode, Codex CLI, hermes-agent, Cline, Roo Code, and aider send little or no thinking control to a custom OpenAI server. Server defaults are what those harnesses actually run.

Limitation: first start downloads about 118 GiB. The README wants a machine of its own on 128 GB unified memory. free over-reports because locked weights look like reclaimable cache. Wrong GPU, no start. Do not treat the 4x end-to-end table as a portable benchmark. Those rows are this author's measured prefill/decode on a stated 85 W envelope with IOMMU off.

Use Cases

• Strix Halo boxes (Ryzen AI Max+ 395 class) that should serve Flash-Next locally for Codex or another harness.
• People who already have a GGUF of this model and want this engine instead of a general runtime.
• Skip if you are on NVIDIA, Apple Silicon, or a GPU that is not gfx1151.

Pricing

Getting Started

1. On gfx1151 with Podman (Docker needs --group-add video --group-add render instead of keep-groups):

podman run --rm -p 8731:8731 \
  --device /dev/kfd --device /dev/dri --group-add keep-groups \
  --ipc=host --ulimit memlock=-1:-1 \
  -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
  -v ~/halogen-models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.12.0

2. Point a client at http://<host>:8731/v1. Model name halogen-qwen3.8-flash-next.
3. Read /health for what the running build actually supports.

First-party start: README.

Frequently Asked Questions

Will it run on a 5090 or an M5?

No. The README is explicit: this silicon only.

Why was my answer empty?

Thinking tokens count against max_tokens. Since 0.11.0 an answer room keeps a tail of the budget for the reply. Check finish_reason. "length" means the budget ran out.

Is the 4x claim audited?

It is the author's table against published EngramHalo.cpp, ROCmFP4, and CIRU-IU4 numbers, with stated power and IOMMU conditions. Reproduce on your box before you quote it.

Alternatives

• mlx-serve: Apple Silicon path for Flash-Next.
• ExLlamaV3: NVIDIA consumer GPUs.
• Qwen3.8-Flash-Next: the weights, independent of this engine.

Tips

1. Give it the host. Sharing memory with another fat process is how the README's hang shows up.
2. Set sampling defaults if your harness sends none. The model card's thinking settings are temperature 1.0, topp 0.95, topk 20.
3. Believe the startup line for host memory left, not MemAvailable.

Conclusion

halogen-flash-server is a Strix Halo specialist for Qwen3.8-Flash-Next, not a general local LLM runtime. Use it if that is your GPU. Use mlx-serve or ExLlamaV3 if it is not.