Decode and Training Now Ship on Separate Silicon

Serving a model and training one stopped being the same job long before anyone built separate silicon for them. On 22 April at Google Cloud Next, Google announced its eighth-generation TPUs as two parts rather than one: TPU 8t for training, TPU 8i for inference. The announcement is the least interesting thing about it. What is worth reading is which resource each part was given more of, because that is a vendor confirming in silicon which resource each of the two jobs actually runs out of.

Decode has terrible arithmetic intensity. To emit one token you stream the weights it touches out of memory and multiply them against a single activation vector: a few FLOPs per byte moved, so the part sits on memory bandwidth and interconnect while its matrix units wait. Training inverts every term. Large batches amortise each weight read across thousands of examples, gradients tolerate a deep pipeline, and the limit moves back onto raw arithmetic and the network between chips. Mixture-of-Experts widens the gap again: routing turns one forward pass into an all-to-all exchange, which is a latency problem when a user is waiting on a token and a bandwidth problem when nobody is. These were never one workload. They shared a part because the part was there.

Google's 22 April post, "Our eighth generation TPUs: two chips for the agentic era", turns that division into a product line, and the technical deep dive published the same day gives the numbers to read it by. TPU 8t is the arithmetic part: a 9,600-chip superpod, 121 ExaFlops, native FP4 in the matrix units, 216 GB of HBM. TPU 8i is the memory part: 288 GB of HBM at 8,601 GB/s — roughly 1.3x the training part's bandwidth — and 384 MB of on-chip SRAM, three times the previous generation, which Google describes as enough to keep a long-context KV cache resident on silicon instead of re-reading it during decode. The serving part was given more memory and less peak arithmetic than the training part. That is the arithmetic-intensity argument arriving as a spec sheet.

The headline gains are Google's own, taken on Google's benchmarks, and they are not even the same measurement. The announcement claims nearly 3x the compute performance per pod for 8t against the previous generation; the deep dive claims an up to 2.7x performance-per-dollar improvement for large-scale training, and an up to 80% performance-per-dollar improvement for 8i over the previous Ironwood part, qualified as holding particularly at low-latency targets for large MoE models. Read all three as vendor figures. The qualification is the useful part, because it names a regime rather than a number: the serving gain is claimed where latency is tight and the model is sparse, which is precisely where decode's arithmetic intensity is worst and where a part built around bandwidth and SRAM has the most room to help. A claim with a regime attached is the only kind that survives contact with someone else's traffic.

The cost of a turn is your hourly rate times the output tokens in it, divided by what that hour actually delivers: tokens per second on one stream, multiplied by the streams the part holds concurrently at your latency target. Procurement negotiates the numerator and reports the win in percent. The denominator is where a part built for decode shows up, and it moves in multiples — the same p99 held at twice the concurrency halves the bill with nobody renegotiating anything. That is why the SRAM figure matters more here than the ExaFlops one. A KV cache that stays resident does not make one stream cheaper per hour; it raises how many streams fit under the same latency ceiling, which is a denominator effect and shows up nowhere on a rate card. Cost per solved task is still the metric. What the split changes is that its denominator now sits on a specific part, and the part with the good denominator will not always be the one your cluster already runs.

The decision that binds you is architectural, taken months before finance sees a bill, by people who are not in the pricing conversation. A hand-written kernel with no portable fallback; a runtime pinned to one vendor's release; a quantisation format frozen because it was what the cluster ran last year — on a line whose training half now takes FP4 natively in its matrix units. Each of those is a defensible local choice. Together they are the reason cheap capacity on a different part arrives as a migration project rather than a configuration change. None of that is a bet on Google. It is a bet that decode's arithmetic intensity will come to resemble training's, and it will not: streaming a weight out of memory to produce one token is what the algorithm does, not what this generation of hardware happens to make it do. So the question the 8i asks is the question the next serving part will ask again, and the one after that. The serving path answers it with whatever is in the repository on the day it is asked.