Inference's Techniques Age in Years. Its Economics Age in Weeks.

I read Philip Kiely’s Inference Engineering cover to cover last weekend. It’s a good book. The mechanics are clean, the diagrams are the right diagrams, and the author has clearly served real traffic. If a new grad joined my team tomorrow and asked what to read to stop embarrassing themselves in design reviews, I’d still hand them this book.

Half of it is aging badly anyway.

The math isn’t wrong. The problem is that the market the book prices against moves so fast that even a manuscript finished in January 2026 and launched in February sits in a different world by spring. The build-vs-buy arithmetic that held at press time now points somewhere different. The techniques chapters age like Knuth. The economics chapters age like a Best Buy circular.

This post is my attempt to separate the two. What in a recent inference book still holds? What has quietly inverted since the ink dried? And the part I care about most: what’s the replacement decision framework when the numbers shift again next quarter, which they will.

One caveat up front. I’m arguing with the shape of claims common across current inference writing, Kiely’s book included. Where I cite a specific chapter, I name it. Where I’m responding to a framing (“self-host to save money,” “speculative decoding is nearly free”), I’ll call that out as framing rather than as a direct quote.

What survives

Start with what ages well. The physics of decoder-only transformers hasn’t moved since 2023. Memory bandwidth still bounds decode. Compute still bounds prefill. The roofline still has two sides.

That means the following lessons will still be true when my kids are in high school.

The KV cache is the bottleneck, not the weights. Lilian Weng made this observation in 2023 and it has only gotten truer. A 7B Llama-class model burns roughly 0.5 MB of KV per token. On an A100-80GB, after weights and framework overhead, you have about 56 GB of KV budget, which works out to 14 concurrent sequences at 8K context. The crossover where one sequence’s KV cache exceeds the entire model weight footprint sits around 28K tokens. Teams routinely size clusters from parameter count and get blindsided at concurrency=14. It’s the “works at 1 QPS, dies at 10 QPS” regression, and it will still bite people in 2030.

Continuous batching is the first optimization that matters. Everything downstream (prefix caching, speculative decoding, tensor parallelism tuning) depends on you being batched first. Kiely gets this right, and so does every production team I’ve talked to.

Quantization is a quality signature, not a free win. Tim Dettmers’ LLM.int8 result was specifically about outlier feature handling at 175B. The field extrapolated it to 7B and got burned. Recent benchmarks show INT4 on a 7B dropping 8–10% on tool-use evals, which wrecks agentic pipelines. FP8 is the current sweet spot. The mechanism (outliers, vector-wise scales, calibration sets) is evergreen.

Roofline reasoning. Horace He’s “Making Deep Learning Go Brrrr” is cited across the inference literature for good reason. Identify which wall you’re hitting before you optimize. Kiely internalizes this.

All of it belongs on the shelf next to Hennessy & Patterson.

What inverted

The framing that stops aging well is the argument that you save money by self-hosting a 7B. That argument was priced against a market that’s already moved.

The arithmetic matters here, because this is the kind of thing you need to be able to re-run yourself.

A well-batched 7B on an H100 sustains roughly 3,000 tokens per second on vLLM under favorable conditions. That’s a public benchmark number on a single-node setup with saturated batching. AWS on-demand H100 pricing as of mid-2025 was $3.90/hr after the June price cut. GCP spot dipped to $2.25/hr. Do the division:

1,000,000 tokens ÷ 3,000 tok/s = 333 seconds = 0.0926 hours
On-demand: 0.0926 × $3.90 = $0.361 / M tokens
Spot:      0.0926 × $2.25 = $0.208 / M tokens

The bare hardware floor for a saturated single-tenant 7B is $0.21 to $0.36 per million tokens. That’s a lab number. Your production number will be worse, because real workloads don’t sustain saturated batching, latency SLOs force you below peak utilization, and you’re paying for salaries, on-call, and observability on top.

DeepSeek V3.2 is currently priced roughly $0.20–$0.30 per million blended, depending on how heavily you weight cache hits against cold input and output. Two honest caveats before I use this number. First, V3.2 is a sparse MoE (~671B total, ~37B active). It’s not a dense 7B, and its per-token compute cost is higher, not lower, than a 7B’s. Second, Chinese frontier labs have been pricing aggressively to win share. These prices are plausibly below the vendor’s marginal cost right now.

The conclusion I need doesn’t require “API beats your hardware floor.” It requires “API is close enough to your floor that the overhead eats the gap,” and that claim survives both caveats. You’re being offered a much stronger model, hosted, managed, with an SLA, at roughly the price of your unloaded GPU time. The math for single-workload self-hosting doesn’t pencil, subsidy or not.

You won’t beat that with a credit card and good intentions. You won’t beat it with better kernels either. The hosted vendor is multiplexing thousands of tenants onto shared capacity at utilizations you structurally cannot match as a single-workload shop. It’s the same pattern that beat enterprise colo in 2012 and self-run Hadoop in 2019.

When does self-host still win? I see two cases.

One is compliance. HIPAA, PCI, data residency. The traffic can’t leave your VPC, so cost stops being the decision variable.

The other is a platform amortizing across many workloads. This is the case most current inference framing misses, and it’s the one I want to spend real time on.

Workloads, not tokens

The standard rule of thumb is that self-hosting pays off above roughly 2M tokens per day. I’ve quoted this number in design docs. I was wrong to, because the 2M figure assumes a single workload holding one GPU at ~50% utilization.

Production platforms don’t look like that. Atlassian publicly reports greater than 60% cost reduction on an internal serving platform carrying LLMs, search encoders, and content moderation. The important detail is that those workloads are partially uncorrelated. LLM traffic peaks during US business hours. Moderation runs on a different diurnal curve. Search encoders fire on a third schedule.

The math changes when you amortize:

Single workload @ 2M tok/day:            50% util → break-even
Three uncorrelated workloads @ ~700K ea: ~50% combined util → break-even

Stated as a heuristic:

Multi-workload break-even (tok/day) ≈ (single-workload break-even) ÷ N, where N is the number of uncorrelated workloads sharing capacity.

The correlation caveat is load-bearing. If all your AI features are user-facing chat that fires during the same business hours, N≈1 no matter how many features you ship. The win comes from mixing user-facing LLMs with batch moderation, offline search encoders, nightly summarization jobs, and so on.

When that mix exists and N is 5–10, per-workload break-even drops from 2M to somewhere in the 200K–400K tokens/day range. That’s well within reach of companies I previously would have told to stay on an API.

The counterintuitive consequence: a 500-person SaaS with eight uncorrelated AI features has better self-host economics than a 5,000-person SaaS with one heavily-loaded AI feature. The correct framing isn’t “are we big enough?” It’s “do we have enough inference-shaped workloads with different traffic shapes to share a GPU?”

Hazy Research and Character.AI have gestured at workload shape as the limiting variable. I haven’t seen the generalization to a per-workload division rule stated crisply anywhere, and I’d be happy to be pointed at prior art if it exists.

Why Stripe’s 73% isn’t your 73%

The other production number everyone quotes is Stripe’s 73% inference cost reduction on 50M daily calls, down to a third of the prior GPU fleet after migrating from Hugging Face Transformers to vLLM. I’ve seen this cited in three vendor pitches in the last six months, usually to justify adopting whatever engine Stripe adopted.

That’s the wrong takeaway, because the number isn’t portable.

The total effective throughput gain works out to roughly 11×. Three mechanisms contribute, using rough public benchmarks.

Continuous batching over a naive HF Transformers baseline is about 4×. vLLM’s original paper claimed 2.7×; it has improved since.

Prefix caching on templated prompts adds another 2–3×. Character.AI reports 95% prefix hit rates on conversational traffic, and separately has driven KV-cache size down by more than 20× via MQA, hybrid attention horizons, and cross-layer KV sharing. Stripe’s workload (fraud classification, merchant categorization) is heavily templated in the same way.

FP8/INT8 numeric format contributes 1.5–2× on top of that.

Product: 4 × 2.5 × 1.7 ≈ 17× theoretical; 11× realized; the delta is interaction losses. That checks out qualitatively.

What I want to resist is pretending I can cleanly split 73% into decimal contributions. Taking logs, roughly: batching is a bit under half, workload shape a third, numeric format the rest. That’s handwave-calibrated, not derived.

The qualitative claim is the load-bearing one. Most of the Stripe win is workload-shaped, not engine-shaped. Your open-ended chat workload, your RAG pipeline with fresh retrieval context, your code generation: none of those get a 95% prefix hit. Strip out the prefix contribution and what’s left is roughly half of Stripe’s number. Still great. But if you promised 73% in a design doc and delivered 40–50%, your credibility takes the hit.

This pattern keeps showing up. Production wins get credited to the engine when the bulk of the win was the shape of the workload the team built around that engine. Character.AI’s 20× KV-cache reduction is a workload story (long dialogues plus custom MQA variants). Stripe’s 73% is a workload story (templated classification plus prefix caching). Atlassian’s 60% is a workload story (multi-tenant amortization). In every headline case, the engine did its job. But the win was shaped, not selected.

If an honest chapter on this existed, it would carry a title like “Your workload is the optimization, not the engine.” The existing inference literature is scattered on this point. Hazy Research and Character.AI have the right instinct, but nobody has put it in a serving book as the central thesis.

The retrofit tax

One more piece before the take-home. The “advanced” optimizations chapter, speculative decoding in particular, inherits Karpathy’s framing: at batch=1, decode is memory-bandwidth-bound, so you have spare compute to verify K tokens in parallel. It’s essentially free.

Karpathy is describing a mechanism. The usual inference-book treatment implicitly extends that into a project plan. Those are different things, and the gap is expensive.

Price it out. Retrofitting speculative decoding into an existing serving stack touches continuous batching, KV allocation, tokenizer alignment, observability, and the eval harness. Call it nine engineer-months, which is on the low end of what I’d budget for a serious integration. At a US-coastal fully-loaded $29K per engineer-month (this number roughly halves in Eastern Europe or India, so adjust), that’s $261K of sunk cost.

Now the break-even. Here’s where I have to correct the first draft of my own reasoning. The relevant opportunity cost isn’t the API price. It’s the self-host blended cost of the team considering the retrofit. By this post’s own argument, a team at the scale where spec decoding retrofit is on the table is already self-hosting at multi-workload amortized rates, plausibly $0.05–$0.10/M. Using the API price here would understate the token volume required.

Recent production writeups suggest realized speedups come in well below the theoretical 2–3×. I’ll use 25% as a midpoint for planning purposes; treat that as my assumption, not a cited number.

Break-even tokens:
$261K ÷ ($0.08/M × 0.25 savings) ≈ 13 trillion tokens

Thirteen trillion tokens a year is ~36 billion tokens a day. That’s well above Stripe’s published volume. For the handful of operators globally at that scale (a few dozen, not fifty), spec decoding is a rational retrofit. For everyone else, it’s an NPV-negative project dressed up as a feature. Couple that with ZenML corpus data showing 9–18 month production model lifetimes, and the retrofit doesn’t finish paying back before the model itself gets deprecated.

If you’re starting fresh, bake spec decoding in from day one. If you’re retrofitting, run the arithmetic against your own blended cost, not mine. Most readers will find the number is “don’t.”

The take-home rule

Two rules survive the numbers drifting.

First, read inference books for the physics and re-run the economics quarterly. Epoch AI puts the frontier inference API price decline at roughly 5–10× per year at constant capability. That doesn’t translate one-to-one to build-vs-buy math, because GPU rental rates have fallen more like 2× over the same window, so the ratio that drives the flip has moved a few-fold over the last cycle. Not enough to invalidate the techniques, but enough that any specific dollar figure in even a recent manuscript is probably pointing at a drifting conclusion. Techniques age well because they’re grounded in hardware ratios that move slowly. Economics age fast because they’re priced off a commodity on a learning curve. Don’t conflate the two.

Second, the right break-even question is “how many workloads?”, not “how many tokens?”. Self-hosting economics are governed by utilization, and utilization is governed by uncorrelated workload count. Everything else is secondary. If your AI features all fire at the same time on the same traffic shape, you don’t have five workloads. You have one, five times over.

Kiely’s book will still be on my shelf next year. The KV cache chapter, the batching chapter, the roofline reasoning: those are the things you can’t Google your way to in a hurry. But the chapters that quote prices, the ones that argue build-vs-buy from specific current numbers, those I mark with a post-it that says check the market before you cite this. Every quarter. Because the market keeps moving, and the book, being a book, does not.

The half-life of an inference technique is years. The half-life of an inference dollar figure is weeks. Books can’t really help with the second problem. Spreadsheets can.

The views and opinions expressed in this blog are my own and do not represent those of my employer.