Fifteen Minutes of Cache, Then Three Hours of 404

A resilience review that counts providers is measuring the wrong plane. The useful number is how long your traffic can still find your servers after the thing that answers that question stops answering — and the useful decision is what the proxy says once that number runs out.

Railway runs customer workloads on three substrates: Google Cloud, AWS, and its own hardware. At 22:20 UTC on 19 May 2026 an automated Google Cloud enforcement action suspended its production account. Railway's own incident report, and the independent write-ups at InfoQ and ilert, all describe that action as incorrect and delivered with no prior notice. The dashboard went dark, the API went dark, and so did every customer service, including the ones running on machines Google does not own. The AWS and bare-metal compute never stopped. It stopped being addressable. Three substrates in the data plane bought nothing here, because the authority that made the data plane addressable lived in one account, and the suspension took that account whole. More regions inside it would have changed nothing either.

For the first fifteen minutes the platform looked like a multi-cloud system absorbing a hit, and the cache is the reason. Railway's edge proxies hold routing tables fetched from that control plane, so traffic kept moving over tables already in memory. The report timestamps the turn to the minute: at 22:35 the cached routes began expiring, and workloads on AWS and Railway Metal began returning 404. Fifteen minutes is the honest measure of the dependency, and the only one that describes it: how long the data plane could still answer with the control plane gone. Every cached read from an authority has a number like that. It is usually a TTL somebody picked for freshness rather than for survival, and it stays invisible for exactly as long as the authority keeps answering.

What happens when that number runs out is set by one behaviour in the proxy: what it does with a cache it can no longer refresh. Fail-empty treats expiry as knowledge. No entry, no route, 404 — an answer that is true about the cache and false about the world, terminal for the client, and cacheable downstream. Fail-static keeps serving the last good table and marks the response stale. Railway's edge could not resolve routes from 22:35 until 01:38, just over three hours, while the services those routes pointed at were up the whole time. Fail-static would have carried that particular window at no cost, because on the substrates outside Google Cloud the last good table was still correct, and where it had gone wrong — instances Google had stopped — the client was getting an error either way. The general price of fail-static is staleness: a deploy that lands nowhere, a drained instance still taking traffic, a scaled-down replica still in rotation.

Google handed the account back at 22:29, seven minutes after Railway filed a P0, nine minutes after the suspension, and six minutes before the first 404. The authority returned before the data plane went blind, and the platform was still not whole hours later; Railway puts the event at roughly eight hours and timestamps its final recovery update at 07:58. Having the account back restored nothing by itself. Instances were stopped and disks were unreachable, and the rebuild ran in the only sequence available to it, the one its own dependencies imposed: first persistent disk at 23:09, all disks by 23:54, compute and networking at 01:30, edge traffic at 01:38, the dashboard at 02:55. Which is why sizing the cache to cover the outage is the wrong budget. It has to cover the recovery, and recovery only begins when the control plane comes back.

One boundary matters more than any tuning of that number, and it is not a matter of degree. Fail-static belongs to facts about where things are: route resolution, service discovery, quota lookups. Those facts do not become false because the authority went quiet — a route to a running instance is as true in the sixteenth minute as it was in the first. Authorization is the opposite case. There the cached entry is a permission, and the event that falsifies it is precisely the event you would be caching through: a credential revoked, a role removed, an employee gone. Serving the last good answer there makes the TTL a security parameter, the exact window in which a revoked credential still works, and the availability it buys is paid for by whoever the credential was taken from, in revocation lag they never agreed to. The test is a single question: can this fact be falsified by something you would want to act on immediately? Routes go stale slowly and visibly. Permissions go stale on purpose.

Neither of the two numbers in this incident was created by it. Railway's edge had fifteen minutes of memory on 18 May, and its proxies were already written to treat an expired cache as an answer; the suspension only said both facts out loud. That is what makes a control-plane dependency hard to govern: it is a measurable quantity that nothing routinely measures. Uptime graphs cannot show it, because it exists only while the control plane is down. A provider count cannot show it, because it is not a property of the providers. It lives in one TTL and one fallback branch, and it stays a private number until an outage publishes it. The eight hours belong to Google. The fifteen minutes belonged to Railway, and they were fifteen minutes long the day before as well.