ClickHouse 26.3 LTS batches small inserts inside the server, and two identical-looking OKs can now mean disk or memory
Ihor K
CEO
A default changed three layers below the ingest tier, and the release filed it under performance. What it altered is the meaning of a successful write.
ClickHouse 26.3, the LTS line tagged v26.3.2.3-lts on 27 March, turns asynchronous inserts on by default; the commit that flipped it merged on 4 March and describes the effect in one sentence — small inserts are batched from now on. Read as a release note, that is a performance item. Read as an interface change, it is the relocation of a job: accumulating small writes into batches large enough for a columnar store now happens inside the server. That part is defensible on its own terms. What travelled with it, unannounced, is the answer to a different question — what a write that returned successfully is entitled to promise. Teams built, staffed and paged an entire tier to do the first job. The second one they inherited.
The tier existed for a mechanical reason. A columnar engine writes each INSERT as an immutable part and merges parts in the background, so cost tracks part count, not row count. Take 2,000 events a second at one row per insert: 2,000 parts a second for the merge scheduler to chase. Buffer the same rows and flush once a second and you get one part — identical data, two thousand times less bookkeeping. That is why the engine's complaint about small writes arrives as a rejected insert rather than a slow one, and why the documented advice has always been to batch before the write reaches the table. Doing it in the server is the right place for it: the server is the only party that sees every client's writes at once, which no client-side batcher can.
Then the part that matters. A synchronous INSERT that returns cleanly is a right to act on: delete the source message, advance the offset, tell the caller the data is stored. Each of those is irreversible, and each reads the return as durability rather than as receipt. Asynchronous insert keeps that reading available. A second setting, wait_for_async_insert, is on by default, and while it is on the server answers only after the buffer has been through a flush and the rows are on a replica. Upgrade and touch nothing and your acknowledgement means in April what it meant in February.
What changed is the kind of thing that guarantee is. It used to be a property of the write path: one behaviour, the same for every client, alterable only by changing the code that writes. It is now a value, and the merged change enumerates where the value can be set — the users profile in the server config, the session, the individual query, the MergeTree table. Any client that wants a lower p99 can attach wait_for_async_insert=0 to its statement and be told OK the moment its rows are copied into memory. Nothing marks that response. Same status, same shape, and under load roughly the same latency as the strong one. So the acknowledgement no longer carries the strength of its own promise; that has to be known out of band, from whoever set it, at whichever of the four levels they set it.
The cost lands in one specific place: a path that destroys its source on success. A consumer reads a message, inserts, sees OK, commits the offset — or deletes the SQS message, or acks the delivery — and the only copy of that event is now the buffered one. With the wait off, that OK described a memory write, and the commit rested on a promise nobody made. Anything that fails after it fails at flush time, in a server log, in a process with no client left to return an error to. The loss is quiet by construction: nothing raises, nothing retries, and the shortfall appears later as a count that will not reconcile, past the point where the source could still be replayed. That is where the real case for a broker in front of the warehouse now lives, and it is the one part of that case the new default does not weaken. Not batching — the server does that. Replay: a retained log lets you rewind an hour of wrong rows and rebuild the table, and a flushed buffer keeps no history to rewind. Per-key ordering and backpressure while the warehouse is rejecting come along with it, but replay is the property that decides whether a weak acknowledgement is survivable or terminal.
Both settings should be stated explicitly, in both directions, in the same migration as the table they govern, rather than inherited from whichever version a cluster happens to be running. But a stated setting is a claim about behaviour, and this claim is cheap to falsify. On a staging cluster: insert a row, wait for the OK, then take the server down hard before the flush interval elapses. Look twice — once at the table, once at the source. If the row is absent and the message is gone as well, you have just watched an acknowledgement mean less than the code that consumed it assumed. Any ingest path that has never been through that ten-minute exercise is running on a guarantee that has only ever been asserted.