LLM Speculative Decoding: Throughput Benchmarks, Draft Model Ratios, and GPU Memory Bandwidth

## Architecture of Speculative Inference Acceleration

Speculative decoding decouples token generation latency from large model parameter evaluation. By utilizing a compact, autoregressive draft model to predict a sequence of $K$ candidate tokens, the target model evaluates the entire draft verification block in a single forward pass through parallel matrix multiplication.

### 1. Memory Bandwidth Bound vs Compute Bound Transitions

Standard autoregressive token generation is bottlenecked by GPU High Bandwidth Memory (HBM3e) transfer rates:

$$\text{Time per Token} = \frac{\text{Model Weight Bytes}}{\text{Memory Bandwidth (GB/s)}} + \text{KV Cache Transfer Overhead}$$

When batch size is 1, floating-point units remain largely idle. Speculative decoding increases the arithmetic intensity of the verification pass, shifting execution toward compute-bound tensor core operations.

| Engine | Target Model | Draft Model | Acceptance Rate | Baseline (tok/s) | Speculative (tok/s) | Speedup |
|:-------|:------------|:------------|:---------------:|:----------------:|:-------------------:|:-------:|
| vLLM 0.6.4 | Llama-3.3-70B-Instruct | Llama-3.2-1B | 71.4% | 34.2 | 72.8 | 2.13x |
| TensorRT-LLM | Llama-3.3-70B-FP8 | Llama-3.2-1B-FP8 | 68.2% | 46.1 | 94.5 | 2.05x |
| SGLang 0.3 | Qwen-2.5-72B | Qwen-2.5-0.5B | 64.9% | 29.8 | 58.4 | 1.96x |

### 2. Optimal Lookahead Window Optimization

Selecting the speculative lookahead length $K$ requires balancing verification overhead against expected token acceptance:

```python
def compute_expected_speedup(acceptance_rate: float, draft_overhead_ms: float, target_step_ms: float, k: int) -> float:
# Expected accepted tokens per step
expected_accepted = (1.0 - (acceptance_rate ** (k + 1))) / (1.0 - acceptance_rate) - 1.0
total_step_time = (k * draft_overhead_ms) + target_step_ms
effective_tok_per_sec = (1.0 + expected_accepted) / (total_step_time / 1000.0)
baseline_tok_per_sec = 1.0 / (target_step_ms / 1000.0)
return effective_tok_per_sec / baseline_tok_per_sec
```