H100 or H200 for DeepSeek V4 Flash? What we measured in production

We served the same model on both H100 and H200, under identical live traffic, for ten days. The results were not quite what the spec sheets would suggest, and the biggest factor turned out to be something neither datasheet mentions.

If you are choosing hardware to self-host deepseek-ai/DeepSeek-V4-Flash-0731, the obvious comparison is H100 versus H200: same Hopper compute die, but the H200 carries 141 GB of HBM3e at ~4.8 TB/s compared with the H100's 80 GB at ~2.0 TB/s.

You would expect the H200 to win on memory-bound decoding and the two to be close on compute-bound prefill. That is roughly what we found. But the number that actually decided our deployment was not bandwidth at all, it was how the GPUs are wired to each other.

The detail that decides your parallelism strategy

Not all 8-GPU boxes are the same. Before choosing a serving configuration, run this:

nvidia-smi topo -m

On our 8×H100 PCIe machine it returns:

      GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7
GPU0    X   NV12   PHB   PHB   PHB   PHB   PHB   PHB
GPU1  NV12    X    PHB   PHB   PHB   PHB   PHB   PHB
GPU2   PHB   PHB    X   NV12   PHB   PHB   PHB   PHB
GPU3   PHB   PHB  NV12    X    PHB   PHB   PHB   PHB
GPU4   PHB   PHB   PHB   PHB    X   NV12   PHB   PHB
GPU5   PHB   PHB   PHB   PHB  NV12    X    PHB   PHB
GPU6   PHB   PHB   PHB   PHB   PHB   PHB    X   NV12
GPU7   PHB   PHB   PHB   PHB   PHB   PHB  NV12    X

NVLink exists only in isolated pairs — 0↔1, 2↔3, 4↔5, 6↔7. Every other path is PHB: PCIe through a shared host bridge. There is no NVSwitch, so there is no full mesh.

This matters more than it sounds. Any peer-memory operation that crosses a pair boundary falls back to PCIe, and on this topology vLLM also drops to its slowest collective path. You can see it in the startup log:

Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0'
out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER',
'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']

Custom all-reduce and symmetric-memory paths need full P2P connectivity. On a paired-PCIe box they are unavailable, so you get generic NCCL over PCIe for every collective, twice per transformer layer, on every decode step.

Practical consequence

On an 8×H100 PCIe box, data-parallel and wide expert-parallel shapes do not work. They depend on all-to-all peer memory access that the topology cannot provide. Tensor parallelism across all 8 cards technically runs but needs --enforce-eager and NCCL_P2P_DISABLE=1, which costs you CUDA graphs.

Our H200 box has NVSwitch and full mesh connectivity, so it has none of these constraints.

Why this forces two different deployments

Because of the topology, the two machines cannot run the same configuration. This is the central practical finding:

The H100's split into two 4-GPU replicas is not a preference — it is the largest tensor-parallel group that behaves sanely on this interconnect, and running two of them uses the box fully.

H200 configuration — data parallel + expert parallel

python3 -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --tokenizer-mode deepseek_v4 \
  --data-parallel-size 8 \
  --enable-expert-parallel \
  --enable-ep-weight-filter \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice

# image: vllm/vllm-openai:v0.26.0
# env:   VLLM_USE_DEEP_GEMM=1  VLLM_RPC_TIMEOUT=600000
#        VLLM_ENGINE_READY_TIMEOUT_S=3600  TILELANG_CLEANUP_TEMP_FILES=1

H100 configuration — tensor parallel, two replicas

python3 -m vllm.entrypoints.openai.api_server \
  --model deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --tokenizer-mode deepseek_v4 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.88 \
  --max-num-seqs 128 \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --no-enable-flashinfer-autotune \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,
                         "draft_sample_method":"greedy"}'

# image: vllm/vllm-openai:v0.25.1   (run two of these, 4 GPUs each)
# env:   PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
#        VLLM_ALLREDUCE_USE_SYMM_MEM=0  VLLM_USE_DEEP_GEMM=1
#        VLLM_RPC_TIMEOUT=600000  VLLM_ENGINE_READY_TIMEOUT_S=7200

Three flags that are not obvious

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — at 0.88 utilisation the H100 box runs at roughly 99% of card memory. Without this the allocator fragments and OOMs under load; with it, the same configuration is stable.

--no-enable-flashinfer-autotune — runtime kernel autotuning was unstable for us on this model.

VLLM_ALLREDUCE_USE_SYMM_MEM=0 — symmetric-memory all-reduce cannot engage on a paired-PCIe topology anyway.

Measured side by side

Both machines served the same model from the same OpenRouter traffic pool, with load split by weighted routing. Average prompt size was ~11,900 tokens on both, so the workloads are directly comparable. Figures are a 24-hour window.

Two results are worth dwelling on.

First, median time-to-first-token is effectively identical — 0.50s vs 0.48s. Prefill is compute-bound, and these are the same compute die. If your workload is prompt-heavy and you care about first-token responsiveness at typical load, the H100 is not meaningfully behind.

Second, the ITL tails invert. The H100 looks better at p95 (71.9ms vs 260.9ms), but that is not a hardware win — it is a load artefact. The H200 was carrying 60.7 concurrent requests against the H100's 11.2. Under equal concurrency the ordering would reverse. This is a good reminder to check concurrency before reading percentile comparisons.

Where the H200 wins unambiguously is inter-token latency at the median: 18.3ms against 33.4ms, a 1.8× gap that tracks the ~2.4× memory-bandwidth ratio closely. Decoding is memory-bound, and no amount of configuration tuning changes it. We spent a week trying.

The tuning lesson that surprised us

On a memory-constrained box, the single most consequential knob was --max-num-batched-tokens — and it behaves more aggressively than we expected.

Measured on the same H100 replica, holding everything else constant:

Every doubling of the prefill budget halves the KV cache. The relationship is almost exactly inverse — 1,576,526 / 787,861 = 2.001, and 787,861 / 388,326 = 2.029.

This is not the gentle linear trade-off you might assume. Setting the batch budget too high can silently cost you most of your KV cache, which shows up later as preemptions and collapsing throughput at concurrency, not as an error at startup.

How to pick the number

Set --max-num-batched-tokens to the smallest power of two that exceeds your median prompt length. Ours averaged ~11,900 tokens, so 16,384 lets an average request prefill in a single scheduler step while keeping roughly 788k tokens of KV.

At 8,192 our requests needed two chunks each and concurrency stalled around 12 with capacity as the dominant wait reason. At 32,768 the KV cache halved again and preemptions appeared. The middle setting was worth roughly a 3× improvement in sustained concurrency.

Speculative decoding is not optional on the H100 path

DeepSeek V4 Flash uses sparse attention with a Lightning Indexer. On vLLM 0.25.1 we found the non-speculative decode path unstable on this model — it produced repeated Xid 31 MMU faults across four different GPUs, reproducibly, on hardware with zero NVLink or ECC errors. The speculative path (next_n=8) never faulted.

With dspark at 7 draft tokens we measured 62–66% draft acceptance, around 4.5–5.2 tokens emitted per verification step. It costs about 1.4 GiB more CUDA-graph memory and roughly 2% of KV cache. Worth it on both counts.

Calculating your own break-even point

Whether either machine pays for itself depends on your rental rate and what you charge. The arithmetic is simple enough to do yourself.

Current OpenRouter pricing for this model spans 26 providers. The largest cluster sits at $0.140 per million input tokens and $0.280 per million output, with cached input around $0.028. Blended against a prompt-heavy agentic mix (~93% input, ~20% of that cached) that works out near $0.126 per million tokens. A balanced chat workload lands closer to $0.21.

An 8-GPU box renting at R dollars per GPU-hour costs 192 × R per day. So:

Set that against the throughput column above — 0.74B/day on the H100 and 1.88B/day on the H200 under our live load. Note those are demand-limited, not capacity-limited: both boxes served whatever routing sent them. Neither was load-tested to its ceiling, so treat them as observed floors rather than maxima.

Two things follow. A prompt-heavy, cache-heavy workload earns markedly less per token than a balanced one, so your traffic shape matters as much as your hardware choice. And because the H200 served 2.5× the tokens for 33% more rent in our case, it delivered roughly 1.9× more tokens per dollar of rental despite the higher sticker price.

So which should you pick?

Choose the H200 if you are serving at scale, your workload generates long outputs, or inter-token latency is what your users feel. The bandwidth advantage is real, unavoidable and untunable, and NVSwitch unlocks data-parallel attention which is the right shape for MoE models like this one. On tokens per dollar of rental it was clearly ahead for us.

Choose the H100 if your workload is prompt-heavy with short outputs, you care about median first-token latency more than sustained decode speed, and you can accept a wider tail. Median TTFT was a dead heat. Just budget for two smaller replicas rather than one large one, and check nvidia-smi topo -m before you assume TP=8 is available.

Check your interconnect before your spec sheet. The difference between a paired-PCIe box and an NVSwitch box changed which parallelism strategies were available to us — and that decided more about our deployment than the memory bandwidth figures did.