Back in the 90s. I was part of a team working R&D project on flexible manufacturing. One of the central concepts was the use of buffers (also called decouplers) between manufacturing stations. These buffers were the space needed to place not finished products while they waited for the next station. Typically a conveyor belt. The main service is that both stations could run at slightly different rhythm, and they would not be causing problems on the other one. Either starving it or overwhelming it. Queues are pretty similar. You can have a consumer and a producer working and the queue in between prevents them from being blocked by each other. Thus I now see the main role of queues as decouplers.
Hum... The biggest hype on process engineering by the 00s was buffer size reduction. Because those buffers interfere with each other in chaotic ways, and they tend to turn "small problems that blow up soon, with small consequences" into "huge accumulated problem, that blows up hours after it appeared, with business-risking consequences".
You can also observe this in games like Dyson Sphere Program, (which is all workers and queues and buffers) where adding a buffer storage section of a belt only hides the fact that you are under-producing one of the components required.
The buffer smooths out bursty flow but you don't want that in the middle of the pipeline, as it actually represents mid-pipeline inefficiency. You should actually be fixing the upstream or downstream problem.
[1] or other automation games like Factorio, Mindustry
I'll note that speedrunners absolutely buffer mid-pipeline in Factorio, and not just for hand-crafting purpouses. Sometimes you're waiting for R&D, sometimes you're just running half the machines for twice as long, giving you the same output while saving on build costs. The actual bottlenecks are constantly shifting. "I'm not speedrunning!" you might say, but every regular game could've started as a speedrun that could've gotten you to where you are faster.
Understanding the tendency of mid-pipeline buffers to hide problems is useful, but scorning them entirely is also suboptimal.
Reducing buffer size puts back pressure on the whole system, which can be valuable to manage load (but often throttles faster stages and that throttling makes people uncomfortable). A meaningful metric is how much of the buffer is used at any given time and the throughout. If the buffer is backed up, that says there's a bottle neck on the consumption side of the buffer and more bandwidth is needed there. For whatever reason, adjusting buffer sizes is the more common action taken. A buffer provides throughput management but it also provides info/metrics about the operation of the system.
Indeed. You want to have small buffers, because big buffers increase WIP (Work In Progress), product gets damaged, problems are hidden, etc. That follows the JIT (Just In Time) ideas. But, you still want to have buffers to decouple both operations. You also want to constraint the size of the buffer, where the producer stops when the buffer is full.
In the case of queues (As in AWS SQS), I am not aware of any limits where you can also stop the producer when the queue reaches a size. I believe it is desirable just from my experience on Manufacturing systems back in the day.
That's a very good analogy. Queues are not there to solve overload (and they never were), they are there as an *architecture tool* that allows decoupling and *can* (not always) ease the scaling of the queue process (workers).
I think the back-pressure should always be implemented from the very beginning, as it also helps with defining the requirements of what the service should be able to handle
You're essentially describing what a silicon engineer would call independent "clock domains" (the stations) and "clock-domain-crossing signals" (the workpieces.) And, indeed, you would also tend to handle clock-domain-crossing signals by sticking an async FIFO between the two clock domains.
In manufacturing you have mass. Stuff has weight to it and sometimes I think it would be best to imagine data has mass.
In the widget factory there is the option to put stuff in the warehouse until you need it. Great in principle, but if you are the guy having to do the heavy lifting to get stuff crammed into the warehouse, and retrieved, then you can end up wondering why you are in the job, which promised so much more than spending all day in the warehouse rather than making stuff.
With web applications we will gladly get gigabytes of stuff from the other side of the world, just in case we need it. If all of that data weighed grams or even tonnes, then we would do things very differently, to be more like the Toyota Way, with just-in-time and the rest of it.
Hence my suggestion when building for the web, imagine every byte has mass. Design accordingly.
Queues fix throughput volatility (not throughput mismatch) at the cost of added latency. If your widget producer is producing 1000 widgets every half-hour and 0 every other half-hour, and your widget consumer needs to consume 100 widgets every six minutes, a 1000-widget queue solves the problem, in exchange for a half-hour increase in end-to-end processing time. But, as the title and article allude, your widget producer and widget consumer still need to process widgets at the same rate on average. The longer the time window needed for those averages to match, the larger your queue needs to be (and the higher the latency).
That's also why queues are inappropriate for addressing traffic spikes caused by market dynamics (celebrity news, sales events etc.). Those dynamics typically occur over the course of several hours, whereas a web request's latency SLA is on the order of several seconds.
I'd like to note that the only latency that gets increased is latency of the backpressure "signal". Shorter queues can never result in items being processed earlier, I don't think so. It will only increase the time they have to wait until they can "enter the system". Or more of those items will be rejected ("lost event").
Once you've read this, pick up Harchol–Balter's book Performance Modeling and Design of Computer Systems: Queueing Theory in Action. It's a really good introduction to the depth of this topic, and you'll come out with superpowers you didn't have before.
I'm not sure how you came away with that impression. Three out of three reviewers say they overall enjoyed the book. The complaints fall mostly into four buckets:
- "I wish the book was simpler" (Jesse)
- "I wish the book was more advanced" (Murat)
- "I wish software engineering was more advanced" (Andrew)
- "I didn't understand the arguments the author made for why studying single-server exponential response time systems helps with drawing conclusions for time-sharing, heavy-tailed response time systems" (Jesse)
None of these paint the book in a bad colour, as far as I can tell. They say more about the reader's expectations than the book itself.
> you'll come out with superpowers you didn't have before.
with the impressions from the reviewers.
I don't think they got super powers from the book. In fact their outcomes mirrors my own outcomes when going deep into some math topics and then bringing them to work.
This book is one of the few books I own and I couldn't agree more with these reviews. I read maybe 20 to 50 pages of it, and didn't and up with a lot of practically relevant insights. I couldn't say it's a bad book, maybe it's even a good one.
But theory about computer science is always waaaay to removed from practical reality. Only a tiny bit of basic theory is applicable in reality, and from then on, in practice we're just busy fighting with practical problems, we're hardly getting to the theoretic ones.
I don't think this is a good resource for an intro tbh. Unless you are interested in proofs and have some probability basics covered, it feels quite dense.
I liked Principles of Product Development Flow a lot more because it was easier to digest, although it's a different application of queuing theory.
That is also a good book containing a few practical applications of queueing theory, but it won't do anything to help you analyse your own systems on a more fundamental level.
Author clearly has a wealth of real experience, but I have trouble reconciling some of it to the “real world.”
Supposing that you have “too many” messages in your queue, commanding your frontend client to retry its transaction that would’ve added one more, instead of accepting and enqueuing one additional job, doesn’t seem to me to change much. Instead of creating a mess for whoever is in charge of those servers, the mess is created directly in view of the end user, who sees whatever you show them when their transaction is being retried.
Their point about the bottleneck being the real problem that must be addressed if loads are going to be sustained at such a high level is indisputable, though.
I think I would define the necessary rule as: the queue’s maximum size just needs to be greater than the spikes you expect, but that’s of course no insight, just a definition.
I have found queues to be incredibly valuable at solving situations where load has occasional spikes, but urgency of the jobs being done is low. For instance, every time a user views a piece of content you want to make sure that you increment a counter of how many times the content has been viewed, and you also want to touch the timestamp of when that user last did a thing. If that happens even two hours late, it’s probably gonna be fine. The thing that the queue pattern excels at in the realm of Web applications, especially, is allowing you to have an HTTP GET which can be served entirely by a Web worker that is only allowed to talk to a read replica, which allows extensive horizontal scale. Analytics and other incidentals can be handled async in background jobs (and indeed, in emergencies, load-shedding those ancillary things has barely any impact).
I recognize that all of this probably sounds “obvious” - but I have seen enough codebases that do synchronous writes during GET transactions that I would stop short of calling this “common knowledge.”
> the queue’s maximum size just needs to be greater than the spikes you expect
There is one truth I have come to know, said by someone far wiser tha me: A queue is either empty or full. Which is to say a queue can either handle all the data coming in, or it can’t. When it can’t it will fill to capacity. This is a probabilistic thing, and you can only decide how many nines to plan for. And it’s worse than it looks at first, because queuing theory is very non intuitive with non linearities that make it very hard to reason about wothout having your nose rubbed in it.
So that means, that yes, you can keep doubling the size of your queue. And no, you can’t ever make it big enough to deal with a poisson distribution. And while you’re at it you will likely need to add workers. And you’re still back to capacity planning and deciding how much money to throw at the problem.
What you may be getting at, and what the article sort of failed at, is that queues are still super valuable for smoothing small spikes, or even large predictable ones. But a queue alone, without backpressure, or overflow will likely cause systems to fall over. Sometimes in ways that are hard to recover from, especially if you have some kind of microservices inspired architecture, where one thing going offline causes another queue elsewhere to fill. Or worse, bringing a failed service back online stresses another system causing it to fall offline. (not meant to be a dig on microservices by the way)
A request that typically completes in a second or two being queued for an hour is absolutely a mess in view of the end-user. All such a queue is doing is hiding the user's exposure to the mess from the admin.
Similar things apply to buffered channels in Go: buffering frequently hides deadlocks, etc, but you don't observe them until the buffer is full. So buffers generally should be kept either as zero or very small to be able to catch synchronisation issues early
"People misuse queues all the time. The most egregious case being to fix issues with slow apps, and consequently, with overload. But to say why, I'll need to take bits of talks and texts I have around the place, and content that I have written in more details about in Erlang in Anger."
This feels like an unfinished gripe—summarize the key point now rather than promising a future deep dive. Backing claims with a concise example would make the argument useful instead of vague.
Back in the 90s. I was part of a team working R&D project on flexible manufacturing. One of the central concepts was the use of buffers (also called decouplers) between manufacturing stations. These buffers were the space needed to place not finished products while they waited for the next station. Typically a conveyor belt. The main service is that both stations could run at slightly different rhythm, and they would not be causing problems on the other one. Either starving it or overwhelming it. Queues are pretty similar. You can have a consumer and a producer working and the queue in between prevents them from being blocked by each other. Thus I now see the main role of queues as decouplers.
Hum... The biggest hype on process engineering by the 00s was buffer size reduction. Because those buffers interfere with each other in chaotic ways, and they tend to turn "small problems that blow up soon, with small consequences" into "huge accumulated problem, that blows up hours after it appeared, with business-risking consequences".
Queues are pretty similar.
You can also observe this in games like Dyson Sphere Program, (which is all workers and queues and buffers) where adding a buffer storage section of a belt only hides the fact that you are under-producing one of the components required.
The buffer smooths out bursty flow but you don't want that in the middle of the pipeline, as it actually represents mid-pipeline inefficiency. You should actually be fixing the upstream or downstream problem.
[1] or other automation games like Factorio, Mindustry
I'll note that speedrunners absolutely buffer mid-pipeline in Factorio, and not just for hand-crafting purpouses. Sometimes you're waiting for R&D, sometimes you're just running half the machines for twice as long, giving you the same output while saving on build costs. The actual bottlenecks are constantly shifting. "I'm not speedrunning!" you might say, but every regular game could've started as a speedrun that could've gotten you to where you are faster.
Understanding the tendency of mid-pipeline buffers to hide problems is useful, but scorning them entirely is also suboptimal.
Reducing buffer size puts back pressure on the whole system, which can be valuable to manage load (but often throttles faster stages and that throttling makes people uncomfortable). A meaningful metric is how much of the buffer is used at any given time and the throughout. If the buffer is backed up, that says there's a bottle neck on the consumption side of the buffer and more bandwidth is needed there. For whatever reason, adjusting buffer sizes is the more common action taken. A buffer provides throughput management but it also provides info/metrics about the operation of the system.
Indeed. You want to have small buffers, because big buffers increase WIP (Work In Progress), product gets damaged, problems are hidden, etc. That follows the JIT (Just In Time) ideas. But, you still want to have buffers to decouple both operations. You also want to constraint the size of the buffer, where the producer stops when the buffer is full. In the case of queues (As in AWS SQS), I am not aware of any limits where you can also stop the producer when the queue reaches a size. I believe it is desirable just from my experience on Manufacturing systems back in the day.
That's a very good analogy. Queues are not there to solve overload (and they never were), they are there as an *architecture tool* that allows decoupling and *can* (not always) ease the scaling of the queue process (workers).
I think the back-pressure should always be implemented from the very beginning, as it also helps with defining the requirements of what the service should be able to handle
You're essentially describing what a silicon engineer would call independent "clock domains" (the stations) and "clock-domain-crossing signals" (the workpieces.) And, indeed, you would also tend to handle clock-domain-crossing signals by sticking an async FIFO between the two clock domains.
One point of TFA is that when you decouple, you reduce feedback.
In manufacturing you have mass. Stuff has weight to it and sometimes I think it would be best to imagine data has mass.
In the widget factory there is the option to put stuff in the warehouse until you need it. Great in principle, but if you are the guy having to do the heavy lifting to get stuff crammed into the warehouse, and retrieved, then you can end up wondering why you are in the job, which promised so much more than spending all day in the warehouse rather than making stuff.
With web applications we will gladly get gigabytes of stuff from the other side of the world, just in case we need it. If all of that data weighed grams or even tonnes, then we would do things very differently, to be more like the Toyota Way, with just-in-time and the rest of it.
Hence my suggestion when building for the web, imagine every byte has mass. Design accordingly.
Queues fix throughput volatility (not throughput mismatch) at the cost of added latency. If your widget producer is producing 1000 widgets every half-hour and 0 every other half-hour, and your widget consumer needs to consume 100 widgets every six minutes, a 1000-widget queue solves the problem, in exchange for a half-hour increase in end-to-end processing time. But, as the title and article allude, your widget producer and widget consumer still need to process widgets at the same rate on average. The longer the time window needed for those averages to match, the larger your queue needs to be (and the higher the latency).
That's also why queues are inappropriate for addressing traffic spikes caused by market dynamics (celebrity news, sales events etc.). Those dynamics typically occur over the course of several hours, whereas a web request's latency SLA is on the order of several seconds.
I'd like to note that the only latency that gets increased is latency of the backpressure "signal". Shorter queues can never result in items being processed earlier, I don't think so. It will only increase the time they have to wait until they can "enter the system". Or more of those items will be rejected ("lost event").
Once you've read this, pick up Harchol–Balter's book Performance Modeling and Design of Computer Systems: Queueing Theory in Action. It's a really good introduction to the depth of this topic, and you'll come out with superpowers you didn't have before.
This review is not so nice: https://emptysqua.re/blog/review-queue-theory-book/
I'm not sure how you came away with that impression. Three out of three reviewers say they overall enjoyed the book. The complaints fall mostly into four buckets:
- "I wish the book was simpler" (Jesse)
- "I wish the book was more advanced" (Murat)
- "I wish software engineering was more advanced" (Andrew)
- "I didn't understand the arguments the author made for why studying single-server exponential response time systems helps with drawing conclusions for time-sharing, heavy-tailed response time systems" (Jesse)
None of these paint the book in a bad colour, as far as I can tell. They say more about the reader's expectations than the book itself.
Sure. But I'm trying to connect what you said:
> you'll come out with superpowers you didn't have before.
with the impressions from the reviewers.
I don't think they got super powers from the book. In fact their outcomes mirrors my own outcomes when going deep into some math topics and then bringing them to work.
This book is one of the few books I own and I couldn't agree more with these reviews. I read maybe 20 to 50 pages of it, and didn't and up with a lot of practically relevant insights. I couldn't say it's a bad book, maybe it's even a good one.
But theory about computer science is always waaaay to removed from practical reality. Only a tiny bit of basic theory is applicable in reality, and from then on, in practice we're just busy fighting with practical problems, we're hardly getting to the theoretic ones.
I don't think this is a good resource for an intro tbh. Unless you are interested in proofs and have some probability basics covered, it feels quite dense.
I liked Principles of Product Development Flow a lot more because it was easier to digest, although it's a different application of queuing theory.
That is also a good book containing a few practical applications of queueing theory, but it won't do anything to help you analyse your own systems on a more fundamental level.
Yeah I totally agree. My point was that it can be intimidating as an intro book to the topic. Excellent material otherwise
Even if you just read the first few chapters of this you will not come out unchanged.
Author clearly has a wealth of real experience, but I have trouble reconciling some of it to the “real world.”
Supposing that you have “too many” messages in your queue, commanding your frontend client to retry its transaction that would’ve added one more, instead of accepting and enqueuing one additional job, doesn’t seem to me to change much. Instead of creating a mess for whoever is in charge of those servers, the mess is created directly in view of the end user, who sees whatever you show them when their transaction is being retried.
Their point about the bottleneck being the real problem that must be addressed if loads are going to be sustained at such a high level is indisputable, though.
I think I would define the necessary rule as: the queue’s maximum size just needs to be greater than the spikes you expect, but that’s of course no insight, just a definition.
I have found queues to be incredibly valuable at solving situations where load has occasional spikes, but urgency of the jobs being done is low. For instance, every time a user views a piece of content you want to make sure that you increment a counter of how many times the content has been viewed, and you also want to touch the timestamp of when that user last did a thing. If that happens even two hours late, it’s probably gonna be fine. The thing that the queue pattern excels at in the realm of Web applications, especially, is allowing you to have an HTTP GET which can be served entirely by a Web worker that is only allowed to talk to a read replica, which allows extensive horizontal scale. Analytics and other incidentals can be handled async in background jobs (and indeed, in emergencies, load-shedding those ancillary things has barely any impact).
I recognize that all of this probably sounds “obvious” - but I have seen enough codebases that do synchronous writes during GET transactions that I would stop short of calling this “common knowledge.”
> the queue’s maximum size just needs to be greater than the spikes you expect
There is one truth I have come to know, said by someone far wiser tha me: A queue is either empty or full. Which is to say a queue can either handle all the data coming in, or it can’t. When it can’t it will fill to capacity. This is a probabilistic thing, and you can only decide how many nines to plan for. And it’s worse than it looks at first, because queuing theory is very non intuitive with non linearities that make it very hard to reason about wothout having your nose rubbed in it.
So that means, that yes, you can keep doubling the size of your queue. And no, you can’t ever make it big enough to deal with a poisson distribution. And while you’re at it you will likely need to add workers. And you’re still back to capacity planning and deciding how much money to throw at the problem.
What you may be getting at, and what the article sort of failed at, is that queues are still super valuable for smoothing small spikes, or even large predictable ones. But a queue alone, without backpressure, or overflow will likely cause systems to fall over. Sometimes in ways that are hard to recover from, especially if you have some kind of microservices inspired architecture, where one thing going offline causes another queue elsewhere to fill. Or worse, bringing a failed service back online stresses another system causing it to fall offline. (not meant to be a dig on microservices by the way)
A request that typically completes in a second or two being queued for an hour is absolutely a mess in view of the end-user. All such a queue is doing is hiding the user's exposure to the mess from the admin.
I agree with this - queues may not be the end all solution but it is a valuable tool in our kit.
And in the right situations, it can be enough.
Two past submissions with notable discussions:
https://news.ycombinator.com/item?id=39041477 - 18 Jan 2024, 153 comments
https://news.ycombinator.com/item?id=8632043 - 19 Nov 2014, 60 comments
Or you could just play Factorio.
Similar things apply to buffered channels in Go: buffering frequently hides deadlocks, etc, but you don't observe them until the buffer is full. So buffers generally should be kept either as zero or very small to be able to catch synchronisation issues early
Reminds me of this other queuing theory blogpost classic https://medium.com/swlh/fifo-considered-harmful-793b76f98374
TLDR LIFO (stack, not queue) is often a better choice for many workloads, despite violating our sense of fairness.
Huh. That's so obvious but I never would have thought of it.
Mind blown.
"People misuse queues all the time. The most egregious case being to fix issues with slow apps, and consequently, with overload. But to say why, I'll need to take bits of talks and texts I have around the place, and content that I have written in more details about in Erlang in Anger."
This feels like an unfinished gripe—summarize the key point now rather than promising a future deep dive. Backing claims with a concise example would make the argument useful instead of vague.
[flagged]