How DeepInfer makes a rented GPU earn $38 an hour

Someone rents one GPU for $6.89 an hour. They run DeepSeek's newly open sourced inference engine on it and sell tokens at DeepSeek's off-peak prices. Going by DeepSeek's own numbers, that GPU brings in about $38 every hour. After paying the rent, $31.11 is left.

That's an 82% margin, on a GPU they don't even own.

At the same time, DeepSeek V4.1 Flash costs about 20x less than models from the frontier labs, like Claude Opus 5.5 or Claude Sonnet 5.5. Cheap for the buyer and very profitable for the seller usually don't go together. What makes it work here is the engine, DeepInfer, which DeepSeek open sourced this week. This post walks through how it works, using one request as the guide, and then shows where the money comes from. You don't need to know anything about LLM inference to follow along.

First, what is inference

Training is when a model learns. Inference is when you use it. Every time you send a message to a chatbot, a GPU somewhere runs the model on your message and writes the answer one token at a time (a token is a small piece of a word, stored as a number).

Most of the cost of inference is GPU time. So the whole game is easy to say and hard to do. Get as many tokens as you can out of every GPU-hour. One cluster running DeepInfer serves about 155 trillion tokens a day.

Here is the whole system in 3D. Press Next to follow one request through it. Black cubes are tokens, blue cubes are the model's notes (more on those in step 4), and every slab is one GPU. Each step says what you're looking at.

The same thing as a flat diagram. Follow the arrows left to right: your text becomes tokens, prefill reads them and writes notes, decode uses the notes to write the answer. The notes also go to storage so the next similar prompt can skip prefill.

Follow one request

Step 1: your message becomes tokens

The model can't read text, it reads tokens. The first stop is DeepSeek Recipe. You might send an OpenAI style Chat Completions request or an Anthropic style Messages request. Recipe turns either one into one list of tokens, plus your sampling settings like temperature.

It also blocks a sneaky trick. If someone types <｜User｜> inside their message to pretend to be a control token, Recipe encodes it as ordinary text. For long prompts it cuts the text at the special tokens and encodes the pieces in parallel, so this step stays fast.

Step 2: have we seen this before

Next is the Entrypoint. Before doing any heavy work, it asks the storage cluster if it already has the notes for the beginning of this prompt.

This matters a lot. Chat apps and agents keep sending the same long beginning (the system prompt and all the earlier turns) again and again. If the notes for that part already exist, nobody has to compute them again. The lookup is a hash chain over blocks of 128 tokens and only reads metadata, so it's quick.

You can see it in the price. Off-peak, an input token that misses the cache costs $0.15 per million. One that hits costs $0.003 per million. That's 50 times cheaper.

Step 3: prefill reads the prompt

Inference has two phases, and they behave very differently.

The first is prefill. All the new tokens of your prompt go through the model together, in one big pass. Lots of math at once, which is exactly what GPUs like. DeepInfer runs prefill on ordinary 8-GPU machines.

Each prefill step gets a time budget, and the scheduler works out how many tokens fit inside it (up to 32,768). Requests are also sorted into buckets by length: under 16K, 16K to 64K, 64K to 256K, and above. So a 300K-token prompt doesn't block a queue of short ones.

Step 4: the notes get saved

While prefill reads your prompt, every layer of the model writes notes about every token. These are the keys and values used by attention, which is why people call them the KV cache. Decode needs these notes to write the answer.

The notes go two ways. They go straight to the decode machines, and they also get written to a storage cluster running 3FS, DeepSeek's own distributed file system. It has about 100 PiB of SSDs. Notes stay for 3 days for agent traffic and 7 days for chat traffic.

Since the storage is shared, any prefill machine can read any note. The load balancer doesn't have to send you back to the same machine to get a cache hit, and that keeps the load balancing simple.

Step 5: decode writes the answer

The second phase is decode. It writes one token, adds it to the notes, and writes the next one. Every step has to read the model weights and the notes again, but it only does a little math for each request. With one request the GPU would mostly sit and wait for memory.

So you put many requests together. DeepInfer runs decode on supernodes of 64 to 128 GPUs that are joined by a fast network, and every GPU carries dozens of requests at the same time.

Prefill wants big math and decode wants big batches. That's why they run on separate machines. It also means DeepSeek can tune "time to first token" and "time between tokens" on their own.

Where the GPU-hours get saved

Everything so far is the shape of the system. The next few tricks are what squeeze more tokens out of each GPU.

384 experts spread over 64 GPUs

DeepSeek V4.1 Flash is a mixture of experts model. It has 384 expert blocks, and each token only visits a few of them. A small router picks which ones.

With 8 GPUs (EP8), each GPU holds 52 experts and has to read all of their weights every step. With 64 GPUs (EP64), each one holds only 7, one of which is a spare slot. Fewer bytes to read per step means each step is faster.

But now there's a new problem. Some experts get popular. If one expert gets too much traffic, its GPU becomes the slowest one, and all 64 GPUs wait for it every step. DeepInfer's expert load balancer (EPLB) looks at the last 1,000 steps and, every 10,000 steps, copies the hot experts into the spare slots. In production the slowest GPU ends up only about 9% slower than the average.

Try it below. Each orange slab is a GPU and the dots on it are the experts it holds. Switch between EP8 and EP64 and see how many experts each GPU carries. Then press "Make one expert hot" and watch its GPU turn red and the "slowest GPU" number go up. "Run EPLB" copies that expert into a spare slot on the other side of the ring, and the number drops.

From DeepSeek's own benchmark, at 64 requests per GPU, EP64 gives about 15,700 tokens per second per GPU at 246 tokens per second for each user. EP8 gives about 12,600 at 198. Faster for every user and more tokens per GPU, both at once.

Guess five, check them in one step

Writing one token per step is slow. So a small draft model guesses the next 5 tokens, and the big model checks all of them in a single step. If 3 of the guesses are right, you keep 3 tokens for the price of one step. This is called speculative decoding.

DeepInfer's version is called DSpark. It adds a confidence head that predicts how likely each guess is to pass. Guesses that will probably fail aren't worth the compute, so the scheduler only sends the promising ones to be checked. On average it checks about 3.6 guesses instead of 5 and still accepts almost the same number (3.21 vs 3.30). At 60 requests per GPU, that's 5.4% more tokens per second.

DEEPINFER_ENABLE_MTP_VARLEN=1
DEEPINFER_USE_NEXTN_DECODE=5   # max draft length

The GPU never waits for the CPU

Every decode step has four parts. Schedule, run the model, pick the token, clean up. The first and the last are CPU work. In a simple engine the GPU just sits idle while the CPU does them.

DeepInfer overlaps them. While the GPU runs step N, the CPU is already planning step N+1 and cleaning up step N−1. The GPU never has a gap. This is on by default.

Smaller numbers

The KV cache and the expert weights are stored in 4-bit floating point (FP4). Smaller numbers mean more requests fit in memory, and every step moves fewer bytes.

The money

Now put it together. DeepSeek says a GPU running DeepInfer brings in about $38 an hour, and that's at their off-peak (valley) prices, so it's the low end.

Pretend you don't own the GPU and rent one instead. In October 2026 an SM100 (B200-class) GPU on demand from Lambda, RunPod or Modal costs around $6.25 to $6.89 an hour. Take the expensive end.

revenue   $38.00 per GPU-hour
rent    − $6.89  per GPU-hour
kept    = $31.11 per GPU-hour   → 82% margin

This only counts the GPU rent. Storage servers and engineers cost extra on top, but the GPU is by far the biggest line. The stack below is one GPU-hour. Each coin is a dollar it earned, and the grey coins at the bottom are the rent you pay. Drag the rent slider and watch the margin. Even at $15 an hour you'd still keep more than half.

The bars on the right are the other half of the story, the price you pay to use these models.

Now look at it from the buyer's side. Here is what 1M input tokens plus 1M output tokens cost:

• Claude Opus 5.5: $24.00

• Claude Sonnet 5.5: $12.00

• DeepSeek V4.1 Flash: $0.75 off-peak, $1.50 at peak

That's 16 to 32 times cheaper off-peak, so about 20x on a normal mix of models. For agents that resend the same long context, the cache hit price ($0.003 per million) makes the gap even bigger.

So the low price and the fat margin come from the same place. Every step above, from the cache hits to the 5-token guesses, gets more tokens out of the same GPU-hour.

What's open, and how to try it

The release has the engine, the kernels (FlashMLA, DeepGEMM, DeepEP and some TileLang kernels), the parallelism, DSpark, async scheduling and structured outputs. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

What's not in it yet is the KV cache store, the full prefill/decode service and the load balancers. Those are tied to DeepSeek's internal setup. So you get the engine, and you build the rest of the car yourself.

export DEEPINFER_ACCELERATOR=nvidia   # or ascend
./build.sh
python examples/offline_inference.py

The repo is at github.com/deepseek-ai/DeepInfer, and the deployment Q&A in doc/inference_deployment_qa.md has a lot more detail.

Summary

DeepInfer runs prefill on 8-GPU machines and decode on 64-GPU rings, and keeps the model's notes on shared storage for days so repeated prompts are almost free. The experts are spread so thin that each GPU reads only 7 of them per step, and a draft model guesses tokens ahead so each step gives more than one. All of that adds up to about $38 for every GPU-hour. Rent that GPU for $6.89 and you keep 82%, while still charging about 20x less than the frontier labs.

Sources: DeepInfer README and deployment Q&A ($38/GPU-hour, 155T tokens/day, benchmarks); B200 rental prices, Oct 2026; DeepSeek V4.1 Flash API prices; Anthropic list prices for Claude.