Five days after the room cooled down, someone is still deciding which features to shed at 02:00 — and the ranking they need was due before the outage
Ihor K
CEO
Thermal excursions do not fail cleanly. They kill some hardware outright, wound the rest invisibly, and hand the on-call rotation a week of work that no runbook covers.
Day five. The status page is green, the incident channel is archived, the postmortem has a calendar slot. Your fleet is three nodes short, the replacements are somewhere in logistics, and traffic is back to an ordinary Tuesday. Someone on call has twenty minutes to decide which features shed load tonight, with no product owner awake and no runbook that covers it. The failure that put them there was short. Proton's incident report puts the loss of cooling in the main room of its Frankfurt datacenter just after 23:00 CEST on 26 August, dates the start of the user-facing incident to around midnight, and has most services back for most users by 01:30 CEST and the last of them around 02:00. The company's status page draws the same window with harder edges: the global-outage entry opens at 00:09 CEST and resolves at 02:27 — about two hours and twenty minutes. Five days later, on 1 September, its status page was still reporting residual hardware failures from the overheating and services at reduced capacity while engineers brought additional infrastructure online.
Nothing paged, because in the terms the alerts are written in nothing was wrong. Take a ten-node tier at 60% utilisation: under an M/M/1 approximation the mean queueing delay runs at ρ/(1−ρ) service times, which is 1.5. Lose three nodes and the surviving seven carry the same offered load at about 86%, where that factor is roughly 6. Four times the queueing on unchanged traffic, with no error budget spent and not one failed request. A threshold written against errors cannot see a tail made only of latency, and that tail is what the customer lives with for the rest of the week.
Heat also leaves latent faults behind, which is where the rest of the tail comes from. Proton's report has the main room going from about 21.8 °C to 51.9 °C in under half an hour, some probes reading 60 °C, and network cards across the Frankfurt infrastructure reaching 105 °C against a 45 °C normal, at which point they disable themselves until a cold reset. Some servers, the report says, suffered heat death; whether the excursion shortened the lifespan of the survivors it does not yet know. That last sentence is the tail in one line. The equipment that came back passes every health check and will fail on its own schedule, and the size of the fleet you are planning capacity against is not a number anyone can state yet.
Two details in that report belong together. Cooling was lost because an air-filter replacement was performed on both redundant air compressors — at night, unannounced, and, Proton says, without the operator passing on the resulting failure, which cut the time available to respond. Around midnight the primary and the backup network switch on one rack failed together, and that rack held several primary database copies. Redundancy is a claim about independence, and neither pair was independent: one shared a volume of hot air, the other shared a maintenance procedure. The second common element never appears on the architecture diagram, so counting redundant units returns the wrong answer.
The repair then turns out to be more dangerous than the failure. Proton's status page for 1 September records exceptional database maintenance following the overheating: the switchover exposed additional instability under high load, traffic had to be limited, outgoing mail was re-enabled at 19:26 CEST and the incident closed at 20:49. That is the ordinary shape of stateful failover — rehearsed at low load with warm caches, then run for real exactly once, at peak, on a shrunken fleet, with writes still arriving. What it does at load can only be learned at load: on a scheduled weekday, in working hours, with an abort that a specific person is authorised to call.
The instinct is to buy a shorter MTTR, and the arithmetic does not support it: that budget buys back the two hours and twenty minutes and none of the five days. What the money should buy instead is a definition of degraded written per surface, in daylight, before anyone is under load — which queues drain first, which features shed load, which writes are refused outright rather than accepted into a backlog no one will drain, and who may make each of those calls at 02:00 without waking anyone. Per surface, because search degrading to stale results is survivable and a security notification degrading to three hours late is not; they fail in different currencies, and one ranking cannot hold both. Keep the two clocks apart while writing it. Inside the thermal deadline — the ride-through window between losing cooling and the first thermal cutoff, a budget set by physics rather than by policy — there is no time to ask anyone, so the trigger fires and the on-call is told what it did. The week that follows is the opposite case: there is time, the choice is a real judgment, and what is missing at 02:00 on day five is not information but standing authority to act on it.
Which leaves the green light. It measures reachability, and on day six reachability is true and beside the point. Proton's status page for that same day notes suspicious-login push notifications delivered several hours later than expected: a control that returned no error, raised no alert, and was counted as delivered while failing at the one thing it exists to do. An incident wants an end condition stated in the same per-surface units as the degradation ranking — the fleet back to N, no surface still running in a reduced mode, every latency-defined promise back inside its stated bound — and it should be closed against that rather than against a colour. Until it is, the incident is still running, and the person it will wake has a right to know that it is.