NInfer: pricing, features, alternatives

NInfer is a from-scratch C++/CUDA inference engine for explicitly registered Qwen checkpoints on one NVIDIA GeForce RTX 5090. The repo Neroued/ninfer is Apache-2.0, created 2026-06-26. On 2026-09-06 GitHub listed about 1,446 stars. r/LocalLLaMA was comparing it with llama.cpp and vLLM on Qwen3.8-27B NVFP4. It is not a general model runner: one GPU, one resident model, startup-fixed capacity of one to eight active requests.

Compare Ollama if you want many models on ordinary hardware, or Qwen3.8 27B if you only needed the model card.

Key Features

Registered artifacts only: Five identities in the README (Qwen3.6-27B groupwise-int / NVFP4, Qwen3.8-27B groupwise-int / NVFP4, Qwen3.6-35B-A3B groupwise-int). Each .ninfer file embeds tokenizer, chat template, and media frontend.
Local HTTP APIs: OpenAI Chat Completions, OpenAI Responses Core, and Anthropic Messages, including streaming, tools (parsed, not executed), and usage accounting. Default example listens on 127.0.0.1:8080.
Long context on one 5090: README examples use --max-context 240000 with a shared Device KV pool, Host State/KV tiers, and MTP speculative decoding.
Multimodal path: Image, multi-image, video, and mixed messages when --vision is set at startup.
Published 5090 numbers: README performance tables (RTX 5090, documented methodology) include Qwen3.8-27B NVFP4 concurrent MTP3 decode around 143.8 tok/s at C=1 and 766.6 tok/s at C=8. Treat those as first-party bench rows, not a third-party audit.

Limitation: Linux + RTX 5090 + CUDA 13.1 + sm_120a only. The build rejects other CUDA architectures. No multi-GPU, no runtime model discovery, no install target. Optional Ko-fi support is not a paid seat.

Use Cases

People with a 5090 who want maximum single-GPU decode on the registered Qwen 27B/35B-A3B artifacts.
Agent harnesses that already speak OpenAI or Anthropic HTTP and can point at localhost.
People who should use Ollama instead if they do not have a 5090.

Pricing

GitHub on 2026-09-06.

Piece	Price	Notes from first-party pages
ninfer	$0	Apache-2.0. About 1,446 stars.
Hardware	Your 5090	Required. Not a cloud API.
Ko-fi	Voluntary	README: not a purchase, not promised features.

Getting Started

Confirm you have a 64-bit Linux host with an RTX 5090 and CUDA Toolkit 13.1+.
Clone github.com/Neroued/ninfer, configure with CMake/Ninja, build ninfer and ninfer-serve.
Download a registered artifact, for example hf download neroued/Qwen3.8-27B-nvfp4-NInfer.
Serve with the README's ninfer-serve flags, then hit /v1/chat/completions.

First-party resource: NInfer README.

Frequently Asked Questions

Does it run Llama or Mixtral?

Not from the README. Only the listed Qwen artifacts.

Can I run it on a 4090 or Mac?

README says the build rejects CUDA architectures other than sm_120a (5090). Community forks exist; they are not this repo.

Same as Ollama?

No. Ollama is a general local runner. NInfer is a specialized 5090 engine.

Alternatives

Ollama: Local runner for many models.
Qwen3.8 27B: The model NInfer's hot path is built around.
OpenRouter: Hosted multi-vendor API if you do not want a 5090.

Tips

Quote Apache-2.0, 1,446 stars, and RTX 5090-only from the README checked 2026-09-06.
Do not advertise llama.cpp-class portability.
--kv-capacity auto sizes the pool at startup; it stays fixed for the process.

Conclusion

NInfer is a 5090-only Apache-2.0 engine for a short list of Qwen checkpoints, with local OpenAI/Anthropic HTTP. Start at github.com/Neroued/ninfer, then decide whether Ollama already covers the hardware you actually own.