The Chinese AI company has made public the inference engine powering its API, reversing an earlier decision to keep the technology proprietary. DeepInfer, released under the Apache 2.0 license, allows developers to deploy DeepSeek's V4.1 Flash model with many of the performance optimizations the company uses in production.

While DeepSeek previously made model weights available for download, the inference layer—the software managing request handling, chip orchestration, and response generation—remained closed. This layer significantly affects latency, throughput, and computational cost. DeepInfer addresses this gap by exposing DeepSeek's approach to these challenges.

We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.

DeepSeek

What's included in the release

DeepInfer bundles optimized computing kernels and communication libraries, including previously open-sourced components FlashMLA, DeepGEMM, and DeepEP. The framework adds multi-chip inference distribution, speculative decoding for faster token generation, asynchronous scheduling to minimize processor idle time, and structured output formatting.

https://x.com/deepseek_ai/status/2112476496283775135?ref_src=twsrc%5Etfw

Accompanying DeepInfer is deepseek-recipe, a library handling the conversion of API requests to model inputs and responses back to API format. Available in Rust and Python, it helps developers replicate DeepSeek's official API behavior in self-hosted deployments. The library was initially released in September with an updated version published alongside DeepInfer.

Why the company changed course

In April 2025, DeepSeek published a document explaining why it had decided against releasing its full inference engine. The company cited three main obstacles: the system had evolved into a highly customized fork of vLLM that its small research team could struggle to maintain as an open-source project; the software was tightly integrated with DeepSeek's internal infrastructure, including cluster management tools; and adapting it for external deployment would require significant modifications.

The engine is tightly coupled with our internal infrastructure, including cluster management tools, making it impractical for public deployment without significant modifications.

DeepSeek, April 2025

The current release appears to navigate these constraints by focusing on offline inference and core serving features while omitting several production components. Missing from the public version are KV cache management, infrastructure for distributing inference stages across machines, and load balancing across replicas. This narrower scope may explain how DeepSeek overcame the technical and organizational barriers it identified earlier.

Hardware support signals strategic shift

The most notable aspect of DeepInfer's launch is its hardware support matrix. Nvidia's SM100 architecture, used in Blackwell data center GPUs, receives support alongside Huawei's Ascend 950 NPU series. Older Nvidia architectures including Hopper are notably absent. This dual-platform approach from day one represents a departure from typical AI software releases.

The inclusion reflects DeepSeek's 2025 commitment to work with hardware manufacturers ahead of model launches, aiming for multi-architecture availability at release. US export controls limiting Chinese companies' access to Nvidia's most advanced processors have intensified DeepSeek's collaboration with Huawei. In September, Bloomberg reported DeepSeek planned to deploy at least 160,000 Huawei Ascend 950DT chips at an Inner Mongolia data center for model serving. The Information also reported in September that DeepSeek intends to increase Huawei chip usage for training, with deliveries expected in the fourth quarter.

Reuters reported in September that DeepSeek and Huawei were jointly developing software tools to simplify Ascend processor programming, further demonstrating the company's investment in Chinese hardware. The announcement has already generated discussion in developer communities, with some highlighting DeepSeek's quantitative finance origins as a potential source of its optimization expertise.

Feature parity and performance questions

Support levels differ between the two hardware platforms. Batch-invariant inference and deterministic sampling—features ensuring reproducible outputs under specified conditions—are available only on Nvidia hardware. DeepSeek's published performance benchmarks use Nvidia GPUs, with four SM100 units for prefill and 64 for decoding, leaving the relative performance on Huawei chips undocumented.

For developers without access to either chip family, the immediate utility lies in examining the released code and adapting its optimization techniques to other inference frameworks. DeepSeek has substantially expanded what the broader open-source community can access and experiment with from its inference technology.