DeepSeek open-sources DeepInfer: what it is and who should self-host

On 20 October 2026 DeepSeek published DeepInfer on GitHub (deepseek-ai/DeepInfer) under the Apache License 2.0. It is the inference framework for DeepSeek models. According to the README, it powers DeepSeek's large-scale production services. The release date comes from the @deepseek_ai post as relayed by a Telegram digest; the README itself has no date.

What DeepInfer is#

DeepInfer is code that runs a model on GPUs or NPUs fast. It is not a model and not a hosted service. The only supported model is DeepSeek-V4.1-Flash. By default it downloads the checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 into the Hugging Face cache on first run. If you already have the weights, set DEEPINFER_HF_MODEL_DIR to use a local copy.

What is in the box#

The README lists these features:

• optimized kernels;
• data, context and expert parallelism with load balancing;
• speculative decoding with confidence-based verification;
• bounded sliding-window replay for prefill;
• asynchronous scheduling;
• batch-invariant inference and deterministic sampling;
• structured outputs: regex, JSON schema and grammar.

Several of these features are NVIDIA-only.

What is not included#

This release provides offline inference only. The README says a full inference service also needs a KV cache system, prefill/decode disaggregation and load balancing, and these are not part of the release. So you get the engine, not the whole production stack. Putting it behind an API for users is work you would do yourself.

Hardware and setup#

The published benchmarks (prefill and decode Pareto charts) were measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain. The charts are images, so we do not quote throughput or latency numbers. We also have no data on whether DeepInfer changes anyone's API price, so we make no such claim.

Should you run it yourself?#

Self-hosting makes sense if you already operate SM100 or Ascend 950 hardware, need full control over the weights and the deterministic or batch-invariant behaviour, or want to build your own serving layer. The benchmark setup (dozens of GPUs for decode) shows what the framework is tuned for: large-scale serving.

If you just need DeepSeek-family output in your app, the framework is a lot of infrastructure to take on. Calling an API is simpler: no GPUs, no CUDA or CANN versions to match.

An API option for the same kind of task#

If you would rather not run your own GPU cluster, MiniMax-M2.7 on CheapTokens handles chat, coding and agent tasks over an API, with:

• one balance for all models and an OpenAI- and Anthropic-compatible API;
• up to 8,192 output tokens per answer and a 180K context;
• a fixed price per model with no peak hours;
• cheaper than OpenRouter in total, with input and output counted together.

Current prices per 1M tokens: input $0.05, output $0.15. An example workload of 2,000 input and 500 output tokens per request, 1,000 requests:

See the DeepSeek-V4.1-Flash-1017 article for what changed in the model itself.

Bottom line#

DeepInfer is a real open release of production inference code, but offline only and tied to specific hardware (NVIDIA SM100 or Ascend 950). Teams with that hardware and a serving team will find it interesting. Most developers will keep calling an API and compare costs for their own token mix.