Slow the Water, Not the Boats: Lock-Flight Discipline and the Case for Bounded Concurrency

A flight of canal locks is a queueing system with a maintenance regime. Water arrives at the top of the flight at whatever rate the feeder supplies it, descends through a series of chambers, and leaves at the bottom having lifted or lowered a boat at each stage. Nothing about the arrangement makes the water move faster. Everything about it makes the movement orderly, measurable and survivable under load.

That distinction — slowing the input rather than accelerating the output — is the whole of backpressure, and British canal engineers had settled it before the invention of the telegraph.

The oldest queueing theory in Britain

The canal age produced structures of considerable ambition: the Bingley Five Rise, opened in 1774, lifts boats some sixty feet across five connected chambers in a single operation, and remains in daily use. A flight is not merely a sequence of locks; it is a coordinated system in which each chamber's state constrains its neighbours. Empty the wrong pound at the wrong moment and the flight stops.

Operationally, the flight is governed by a keeper whose authority is absolute in the narrow sense that matters: they decide what enters, in which order, and how quickly. Boats do not negotiate with the water. They wait at the top of the flight until the keeper judges that the system can accept them.

What a lock actually does to a stream

A single lock imposes a hard admission limit: one or two narrowboats per chamber, no matter how many are queuing behind. This is a remarkably honest form of capacity control. There is no pretence that additional arrivals increase throughput, because the chamber holds what it holds.

Distributed services routinely abandon this honesty. Thread pools are enlarged, connection limits raised, autoscalers given more headroom, and timeouts extended — all in the belief that the constraint is the chamber rather than the water supply. The result is familiar: latency climbs, retries multiply, queues deepen, and the system spends its remaining capacity on work that will never be useful.

The lock keeper's first principle is the one most engineering teams resist longest. Throughput is set by the slowest stage, and the correct response to overload is to reduce arrivals, not to enlarge the chamber.

Side ponds and the price of throughput

Lock flights consume water, and water is expensive to supply at the summit of a canal. Eighteenth-century engineers responded with the side pond: a small reservoir alongside the chamber that captures part of the discharge from each emptying and returns it to the next filling, cutting consumption substantially without altering the flight's nominal capacity.

This is precisely the logic of an efficient queueing design that preserves work rather than discarding it — a bounded buffer with controlled reuse rather than an unbounded one that simply grows. The side pond does not increase the number of boats a lock can pass in an hour. It ensures that the water required to pass them is not thrown away.

The keeper's discretion: admission control

Lock keepers exercise judgement that no timetable captures. Hire boats, inexperienced crews and vessels drawing too much water are handled differently from a working pair that has run the flight a hundred times. Priority is applied transparently, and it is applied at the point of entry rather than somewhere in the middle of the chamber.

In services, this maps onto the admission-control layer: token buckets for steady-state shaping, concurrency limits per tenant or per client, separate pools for interactive and batch traffic, and rejection performed early enough that the work has not already consumed downstream capacity. A request refused at the edge costs microseconds. The same request refused after it has fanned out across four services costs an order of magnitude more and pollutes every queue it touched.

Fairness, priority and the problem of the wide beam

Fairness in a flight is negotiated in the open. A wide-beam boat occupies more of the chamber, and everyone on the lock side can see it. The negotiation is not always equitable, but it is legible, which is more than can be said for most multi-tenant systems.

Designing fairness into a service means deciding, deliberately, what a unit of work costs. Per-client rate limits, weighted fair queueing and request-cost accounting all depend on an answer to a question that teams often leave implicit. Without it, the heaviest consumer sets the effective service level for everyone else, and the remaining tenants experience degradation they cannot diagnose.

When the pound runs dry

The characteristic failure mode of a canal is not a broken lock but a drained pound — a reach between locks that loses water faster than the feeder can replace it, leaving boats aground and the flight halted. Recovery requires refilling, which takes time that no amount of urgency can compress.

Software systems have pounds of their own: connection pools, thread pools, in-memory caches and message broker storage. Each can be drained by work that is admitted faster than it can be completed, and each recovers slowly once exhausted. The architectural response is the bulkhead — separating pools so that one saturated reach does not strand the entire network — and the operational response is a queue-depth alert that fires well before the bed is exposed.

Instrumentation the keeper would recognise

A lock keeper works from paddles, gauges and the visible state of the water. Nothing is inferred; everything is observed. For a rate-limited service the equivalent instrumentation is arrival rate against completion rate, queue depth and age, rejection counts by reason, and saturation of each downstream dependency — exposed in a form that an on-call engineer can read at three in the morning.

The discipline of slowing down

The enduring lesson of the lock flight is that a system under pressure is not improved by enthusiasm. It is improved by refusing work it cannot complete, in a place and at a time where refusal is cheap, and by conserving the resources that make recovery possible. The boats still get through. They simply get through in an order that the water can support.