NInfer

NInfer is a from-scratch C++/CUDA inference engine for explicitly registered Qwen checkpoints on one NVIDIA GeForce RTX 5090. The repo Neroued/ninfer is Apache-2.0, created 2026-06-26. On 2026-09-06 GitHub listed about 1,446 stars. r/LocalLLaMA was comparing it with llama.cpp and vLLM on Qwen3.8-27B NVFP4. It is not a general model runner: one GPU, one resident model, startup-fixed capacity of one to eight active requests.

Compare Ollama if you want many models on ordinary hardware, or Qwen3.8 27B if you only needed the model card.

Key Features

• Registered artifacts only: Five identities in the README (Qwen3.6-27B groupwise-int / NVFP4, Qwen3.8-27B groupwise-int / NVFP4, Qwen3.6-35B-A3B groupwise-int). Each .ninfer file embeds tokenizer, chat template, and media frontend.
• Local HTTP APIs: OpenAI Chat Completions, OpenAI Responses Core, and Anthropic Messages, including streaming, tools (parsed, not executed), and usage accounting. Default example listens on 127.0.0.1:8080.
• Long context on one 5090: README examples use --max-context 240000 with a shared Device KV pool, Host State/KV tiers, and MTP speculative decoding.
• Multimodal path: Image, multi-image, video, and mixed messages when --vision is set at startup.
• Published 5090 numbers: README performance tables (RTX 5090, documented methodology) include Qwen3.8-27B NVFP4 concurrent MTP3 decode around 143.8 tok/s at C=1 and 766.6 tok/s at C=8. Treat those as first-party bench rows, not a third-party audit.

Limitation: Linux + RTX 5090 + CUDA 13.1 + sm_120a only. The build rejects other CUDA architectures. No multi-GPU, no runtime model discovery, no install target. Optional Ko-fi support is not a paid seat.

Use Cases

• People with a 5090 who want maximum single-GPU decode on the registered Qwen 27B/35B-A3B artifacts.
• Agent harnesses that already speak OpenAI or Anthropic HTTP and can point at localhost.
• People who should use Ollama instead if they do not have a 5090.

Pricing

GitHub on 2026-09-06.

Getting Started

1. Confirm you have a 64-bit Linux host with an RTX 5090 and CUDA Toolkit 13.1+.
2. Clone github.com/Neroued/ninfer, configure with CMake/Ninja, build ninfer and ninfer-serve.
3. Download a registered artifact, for example hf download neroued/Qwen3.8-27B-nvfp4-NInfer.
4. Serve with the README's ninfer-serve flags, then hit /v1/chat/completions.

First-party resource: NInfer README.

Frequently Asked Questions

Does it run Llama or Mixtral?

Not from the README. Only the listed Qwen artifacts.

Can I run it on a 4090 or Mac?

README says the build rejects CUDA architectures other than sm_120a (5090). Community forks exist; they are not this repo.

Same as Ollama?

No. Ollama is a general local runner. NInfer is a specialized 5090 engine.

Alternatives

• Ollama: Local runner for many models.
• Qwen3.8 27B: The model NInfer's hot path is built around.
• OpenRouter: Hosted multi-vendor API if you do not want a 5090.

Tips

1. Quote Apache-2.0, 1,446 stars, and RTX 5090-only from the README checked 2026-09-06.
2. Do not advertise llama.cpp-class portability.
3. --kv-capacity auto sizes the pool at startup; it stays fixed for the process.

Conclusion

NInfer is a 5090-only Apache-2.0 engine for a short list of Qwen checkpoints, with local OpenAI/Anthropic HTTP. Start at github.com/Neroued/ninfer, then decide whether Ollama already covers the hardware you actually own.