I'd go further, and say that most of the time, "SQLite is enough".
But, yes, PostgreSQL is all I ever use for anything that needs to be big. I ported a big old web app that had ScyllaDB, Elastic Search, Redis, and probably some other stuff I've forgotten. It got PostgreSQL+PostGIS (it's a mapping app), that's it. I'm sure there's some situation where it would be worth looking at all that other stuff, but it's ridiculous to build all that complexity in before you even have users.
Redis is basically free and nothing like the other tools you mentioned. Anytime I need a quick cache that will survive reboots it’s a winner. Agree on the other stuff though.
Yeah. Redis (valkey now for me mostly) is where I go for that extra oomph if the patterns make sense (or some level of scripting for dev work makes sense; the lua integrations is chef's kiss.).
Yes, for most projects, my path is SQLite -> PostgreSQL. For monolithic applications, SQLite is usually good enough. I've used OLTP + OLAP database setups before(SQL+ELK), but both data synchronization and operational overhead were very high. Before adopting such a solution, you really need to ask yourself: do you actually have that many users?
I agree, the reason why I had submitted this HN submission post was because I was getting way too deep into the weeds of new databases like surrealdb/tigerbeetle/scylladb etc.
which while are an interesting problem to think through because well, it makes me feel smarter knowing these databases. I still felt like postgresql was enough, (at best something like clickhouse or redis combined with postgresql might be more than enough for many things)
But the fact of the matter is that sometimes you don't even need postgresql. I actually just deploy sqlite apps in golang/rust, a website that I had made that is a single binary: https://mirror.forum
Sqlite is more than enough as well and there are a lot of people pushing sqlite above its weight as well (Turso and the likes)
but its nice knowing that Sqlite/postgresql are more than enough for many use cases. Both are really amazing for a lot of things :-D
Its amazing how we've for the most part atleast now simplified the architecture to have good options. Like using Sqlite and especially Postgresql in the start is a decision that you would rarely go wrong at. I personally feel like although I use sqlite a lot more in projects because i value the simplicity it gives but postgresql also holds a somewhat good relationship in my heart :-D
For database I think principle of medium fit is better than principle of least power. Flexibility for common circumstances is more important than making sure you made the most petite choice possible.
I recently built a site that aggregates top posts from various sources and am using both SQLite and Swift for my backend. Was a pleasant experience mostly.
I am primarily a mobile developer (I am the developer of Hack, hacker news app on iOS and android). While I have developed backends in rust and nodejs, I wanted to give swift a shot to see how it was. I used Vapor for the web server in swift. It was surprisingly pretty good experience and I am considering using it for my next project too.
I have worked some places where a significant amount of business logic lived in database functions and triggers and, hoo boy, that was really hard to reason about. If you're disciplined enough to have migrations around that implemented all that stuff, you're still going to have an unpleasant time piecing all of it together when debugging. But often you're in a state where you don't even have those breadcrumbs and you're pulling out your hair trying to figure out what's going on.
Yeah that's super frustrating. It sounds like a classic IT dev issue when IT is ops though to me - and I'm happy to be wrong.
I personally believe (notice the caveat here) that debugging relatively well designed databases on a good RDBMS is an easy task, if given access to logs, data (or even a redacted replica), and docs. Choose the docs along with one of the first two and it shouldnt remain a mystery for long. I've worked with postgresql a lot over the last couple decades, and even the most convoluted live-in-prod bug isn't wasn't half as bad as the vibe coded nonsense I see and debug weekly (not hating on AI; fwiw this is more a symptom of process/lack there of around integration of the tech IMO).
Me too, and it put me off ever using stored procedures or triggers. I've worked at places where stored procedures are big but they were much more organised about it.
Lots of the alternatives that this site claims Postgres will do are things you'd only consider well past the point that Postgres would be viable.
Kafka? No one wants to operate Kafka, if it's a serious contender it's because you need things only it can do. Same with Elasticsearch, it sucks to operate, sucks to build a second stack just for search, so you'd only consider it at the point that Postgres is no longer suitable. Same with Snowflake.
People reach for the things you mentioned wayyyyyy before they should.
Just because you want something queue shaped or search, doesn't mean you should reach for the big, specialized, expensive technology, when Postgres can already support it in your existing infrastructure up until some significant scale you often won't surpass.
>People reach for the things you mentioned wayyyyyy before they should.
Most orgs are probably due for a technical reassessment. Scale that I believe would have melted postgres in the past now just runs fine. CPUs got a ton faster and memory got cheaper.
Projects that would have required a complex distributed setup 10 years ago can now be handled by a single machine. For a lot of use cases, CPUs got faster than population growth.
The amount of redis-as-database, or X really, that I see deployed that practically make no sense is insane. I haven't run the numbers to make my following claim based on anything but my feelings, but just incompetent dev alone is probably what keeps cloud computing alive.
As the sibling comment from eximius mentions, devs will routinely reach for these long before they’re remotely needed.
A well-tuned Postgres installation on fast hardware and intelligent schema design can scale incredibly far, even if you’re asking it to double as a message bus and full-text search tool.
Maybe I'm lucky, but I've not seen this. Sure I've seen this from junior developers who aren't thinking through the actual requirements they have, but I've not seen it on a team with any real level of experience.
I have seen many cases of people decrying the use of some of these without understanding the requirements though. People with strong opinions like to make up their own requirements without necessarily listening to the business, considering the team, considering the existing solutions, etc.
PostgreSQL is good enough until it's not good enough, when you realize all the bad design decisions that were made before it hits scale. It is the decisions people make around not partitioning, HA, replication that makes it not good enough.
It is not good to have a problem that requires a year of migration by a dedicated team (typical case if you have this 'good problem') when you could have simply spent another day or two to architect it right. You should at least have a path to scale when the time comes.
I feel like any problem that Postgres can't handle is a good problem to have. Either you've got so many customers that you're hitting sharp edges, or you're working on such an interesting problem that you're out of the domain where Postgres is helpful. That I should be so lucky
A moderate-size queue is already a problem that Postgres struggles to handle %) But fortunately such problems are rare, and relatively well-known, like, say, doubly-linked lists in Rust.
For most other problems, Postgres works well, or at least well enough.
Oh yeah - I remember working with a guy who absolutely insisted that we build application-level sharding into what we were working on (i.e. each primary key would have some metadata that our application logic would use to route traffic to the correct Postgres instance). Our database was not big - 10s of gigabytes - but Instagram did something similar and we needed to be able to scale. I pushed back hard, and fast forward two years and the application was decommissioned without hitting any sort of Postgres bottleneck.
As for scale... Just use a larger machine. This works for regular transactional data until you're at something like Amazon scale.
Edit:
Think about this, suppose that you store 1 megabyte of data for each of your customers. So if you have a million customers, it's just 1Tb. And these days, you can have a server with 10Tb RAM delivered overnight. Although you might have to sell your firstborn son (offer applies only to royal families) to fund it.
A lot of sharding/no-sql/... development happened in the late 2000-s when computers were about ~100 times less powerful than now. You _could_ get a system with 10Tb RAM in 2010, but as a specially-designed supercomputer.
The page makes an argument that having a bunch of disparate databases doing specialized things means you have a higher maintenance burden, since you are maintaining more things and will be paged for more failures, but in my 20+ years maintaining production infrastructure, I find that it is often much more difficult to maintain one large database than multiple smaller ones.
You have pushed your entire infrastructure into a single failure domain, for one thing. You make it certain that if your database fails, EVERYTHING fails.
In addition, there is resource contention and workload variability. As you start to push your postgres instance, all the workloads hitting your database are going to be competing for resources. While postgres itself is good at parallelizing the work it is doing, all that work is still going to be hitting the same database, and competing for the same kcache. Your entire infrastructure might degrade in performance at the same time.
Any issue with one component can cascade very easily if they all share the same database. If your login functionality has a bug and is creating churn on your database, it can lock everything.
With multiple databases, you have a much smaller blast radius when you do database operations. You isolate your workloads and can independently scale them.
Admittedly, all of these issues occurred at a place that had high traffic and high availability requirements. Honestly, though, if your load is so low that you never feel infrastructure pressure, it probably doesn't matter what strategy you use.
Seems to me that a single database does reduce maintenance burden, just at the cost of higher risk of failure. The trade off should be for the architect to decide.
I love Postgres. I buy using it for many many things.
I really don’t understand why everyone insists that you should use it as a work/message queue.
There are lots of purpose-built bullet proof queuing systems that are simple to setup and administer (or just use SQS).
Your queue is likely to have very different access patterns than the rest of your data, and sticking it in Postgres means you’re probably going to end up setting up partitions or optimizing auto-vacuum on that table way earlier than you probably need to mess around with this things in your scaling.
If your queue has more than a few hundred jobs a day (or you anticipate that like anytime soon), just use a queue.
> There are lots of purpose-built bullet proof queuing systems that are simple to setup and administer (or just use SQS).
Because in 99% of cases you don't need a purpose built solution (Even if engineers often think that) at the scale that most people operate in. Nothing is easier to setup and administer than the database you already use.
We are using Postgres as a worker queue in production for many years, with millions of items being processed at any time and it's been perfectly fine. If you have hundreds like in your example...might as well use sqlite.
I agree but with a caveat that it depends entirely on what your queue represents. If it's part of your data model then keep it with the other data. If it's separate then keep it separate.
Using Postgres gives you transactions and consistency if you have to restore from a backup. Most of the time this doesn't matter (and is a liability) and you can just use some external queue system but sometimes it does matter.
A few hundred jobs a day doesn't seem like it would even be close to what postgres could handle easily, does it?
I'm thinking of the problem as using a small amount of text to represent the work that needs to be done and then using a postgres table where some entries are being added as work that needs to be done, and then a worker is pulling the rows of work out of that table, and maybe putting a completion message somewhere in postgres. I'll concede that is more transient data than probably most of the other tables, it might benefit from vacuuming more often. Does the autovacuuming system not figure out it needs to run more often and do it?
Wouldn't the issue would be more overall queries per second, the amount of writes you're already doing, and the general load on the database. We just added some audit tables that are quickly growing to millions of rows, and it seems like Postgres isn't even breaking a sweat. I'm mostly spit balling here and probably glossing over some details.
In some ways, Apache Kafka and RabbitMQ can help force better design choices.
A db can be performant, but at a certain point the global locks incremental primary keys create just strangle throughput. What makes a good db design normal form, is almost guaranteed to be inefficient at scale sooner or later. =3
> But the bar should be high: only after pushing Postgres to its limits, documenting why it was insufficient, and accepting the operational cost of the alternative
Why do I need to push Postgres to its limits before using a different solution? Throwing a hosted Redis in front of some hot-path API calls is very straightforward and easier to reason about than materialized views or UNLOGGED tables.
I haven't really used redis much so curious to hear your perspective - this seems the opposite to me? A materialised view is just taking the data I already have and rendering it in a different way to speed up my access patterns. It's easy for me to understand where its all coming from, and it's all directly mapped back to the source data so if things change I can easily understand why it might break etc.
For redis, it seems there's no "out of the box" way to take some data from my DB and cache it. It seems it needs to be hand rolled per query you're optimising, you lose any structural link to the source data (redis doesn't know about my table structure), and now I have another service I need to worry about. Or is it much easier nowadays than I am thinking?
Those examples are all equally difficult to reason about. Cache invalidation is equivalent to refreshing a materialized view, and UNLOGGED tables bring about new and exciting ways lose data.
Because it's less moving pieces to only have one bit of state to think about.
You already have a connection string to your database with a password or authn/z with your cloud provider. If this is a "serious" application, you have backups, monitoring, user roles, pgbouncer, partitioning, and other Postgres-specific things to think about. With just a little bit of care, you can make whatever queries you are running fast enough to not need redis.
But ok, you think adding redis is going to solve your performance problem because you can just cache API responses in redis instead of hitting the DB. Maybe, but now you have to think about cache invalidation, eviction behavior, sizing the redis instance, another set of authn/z roles to think about, and of course more cost.
I realize we're speaking past each other, but IME Postgres will work well into the terabyte range and if you can't tune your database setup for performance then reaching for cache is a form of premature optimization.
I remember an old colleague telling me at a previous job they'd moved all business logic into triggers and stored procedures because their database was on their most powerful server. And then one day it wasn't and the database started to choke horrifically.
I agree with that. You can use Postgres as a message queue / task manager backing store without a database function, though, and it works quite well at the small scale that most sites / SaaS products operate at.
While I love postgres, I take issue with coupling too much application logic to the DB. It’s much easier to update/rollback stateless containers/cloud functions/VMs than to recover a DB.
PostgreSQL is a powerhouse. It has a solution for everything. Especially when you start a project you might be better off just using PostgreSQL instead of a specialized solution. You can optimize it later.
Postgres has its advantages, but for message queues I’d stick with SQS. I built out a trading firm last year that was basically Postgres, some dotnet services and SQS queues and we got acquired for $140M. You can build some fairly formidable systems if you keep them simple.
As @f3408fh has said, I am also curious to knowing more about it if possible.
I had always thought that trading firms might be some of the sole exceptions of over-engineering as I had went into the rabbit-holes of FPGA's and found that some of the use-cases of that technology was trading firms using it to literally shave off even a few milliseconds and so the amount of optimization there
So I am a bit curious how postgres/some dotnet services and SQS queues is able to create a trading firm and scale it and even sell it for 140M$. It feels like my understanding of the situation was a bit uncertain then but could you please elaborate more if possible?
This could use a debugging guide or two. Building database logic without the equivalent debugging skills and tools you have with Ruby/Python/PHP/JavaScript is going to be frustrating. Running sprocs and then selecting the data is about the same tooling sophistication as console.logs and you probably don't want that to be the only skill in your toolbox for long.
Postgres as a relational DBMS is enough, a point worth making against people trying to abstract it away with stuff like ORMs.
Besides that, you usually won't need nosql thanks to jsonb, and other special types like in Postgis cover other use cases. SQL is better than dealing with various DSLs like you'd see for timeseries. Then there are things you may want separate tooling for but can also do in Postgres if you want to avoid more infra: pubsub/queues, search w/ pgvector, graph DBs, KV stores, caches.
A lot of the other examples here look ridiculous though, like no I'm not hosting a webserver on Postgres. Database functions should be used sparingly too. Never touched triggers since I normally have a general-purpose language driving DB changes, but could see why someone else might use them.
Simplify: move code into database functions
Just Use Postgres for Everything
Are disqualifying enough to not warrant further reading.
A relational database is one form of persistent storage. They are great for managing persistent representations of key abstractions and their relationships.
They are not application frameworks nor scalable messaging systems by design.
Did you see that implementation of Quake in CSS? Or Tetris in awk? You can do a similar thing with Postgres, and it will be much less insane. Should you run your entire app stack in Postgres? Likely not. Can you? Likely you can.
> Did you see that implementation of Quake in CSS? Or Tetris in awk?
Do you want to support either of those in production? Get support calls and/or enhancement requests for same?
> Should you run your entire app stack in Postgres? Likely not. Can you? Likely you can.
I have experience with non-trivial systems which have put business logic in database stored procedures. They were not pretty. One example was a single stored proc having north of 40 parameters and was central to a high-volume transaction workflow.
That is how I learned at the time Oracle had a max of 4 threads available to stored procs and that SGA usage can reach out and bite you in "fun and interesting" ways.
> Not everything needs to “scale.” I think needing scale is relatively rare, actually.
The ability to scale a system is not limited to throughput metrics. If it where, I would agree with your point.
Scaling in complexity and/or ability to feasibly introduce significant change is also a desirable system attribute and the one which I predominantly had in mind. The likelihood of also being to scale from a throughput perspective is a nice bonus.
Using Pgsql for everything shows you've been drinking the internet kool-aid. And that site is like a creepy shrine saying pgsql is the alpha and omega.
Like any tool, it works until it doesn't. And when it doesn't it takes a herculean effort to unwind it.
I looked at the first entry and yeah, just say no to moving your business logic into your database. Because change happens...and don't you want that happening in something more plastic than your RDBMS? But it's a great way to bind your solution to Postgres forever.
As an aside, I've used oracle, sybase, informix, mysql, postgresql, rdb, db2, mssql, and a few more that I can't remember. And the idea that pgsql is always the answer is the wrong answer to probably the wrong question.
Ugh yes. I remember starting on MySQL then being hit by the strictness of Postgres... then terrified by the looseness of MySQL when I moved back (changing jobs).
I'd go further, and say that most of the time, "SQLite is enough".
But, yes, PostgreSQL is all I ever use for anything that needs to be big. I ported a big old web app that had ScyllaDB, Elastic Search, Redis, and probably some other stuff I've forgotten. It got PostgreSQL+PostGIS (it's a mapping app), that's it. I'm sure there's some situation where it would be worth looking at all that other stuff, but it's ridiculous to build all that complexity in before you even have users.
Redis is basically free and nothing like the other tools you mentioned. Anytime I need a quick cache that will survive reboots it’s a winner. Agree on the other stuff though.
Yeah. Redis (valkey now for me mostly) is where I go for that extra oomph if the patterns make sense (or some level of scripting for dev work makes sense; the lua integrations is chef's kiss.).
Yes, for most projects, my path is SQLite -> PostgreSQL. For monolithic applications, SQLite is usually good enough. I've used OLTP + OLAP database setups before(SQL+ELK), but both data synchronization and operational overhead were very high. Before adopting such a solution, you really need to ask yourself: do you actually have that many users?
I agree, the reason why I had submitted this HN submission post was because I was getting way too deep into the weeds of new databases like surrealdb/tigerbeetle/scylladb etc.
which while are an interesting problem to think through because well, it makes me feel smarter knowing these databases. I still felt like postgresql was enough, (at best something like clickhouse or redis combined with postgresql might be more than enough for many things)
But the fact of the matter is that sometimes you don't even need postgresql. I actually just deploy sqlite apps in golang/rust, a website that I had made that is a single binary: https://mirror.forum
Sqlite is more than enough as well and there are a lot of people pushing sqlite above its weight as well (Turso and the likes)
but its nice knowing that Sqlite/postgresql are more than enough for many use cases. Both are really amazing for a lot of things :-D
Its amazing how we've for the most part atleast now simplified the architecture to have good options. Like using Sqlite and especially Postgresql in the start is a decision that you would rarely go wrong at. I personally feel like although I use sqlite a lot more in projects because i value the simplicity it gives but postgresql also holds a somewhat good relationship in my heart :-D
For database I think principle of medium fit is better than principle of least power. Flexibility for common circumstances is more important than making sure you made the most petite choice possible.
I recently built a site that aggregates top posts from various sources and am using both SQLite and Swift for my backend. Was a pleasant experience mostly.
https://limereader.com/
Why'd you go with swift for the backend?
I am primarily a mobile developer (I am the developer of Hack, hacker news app on iOS and android). While I have developed backends in rust and nodejs, I wanted to give swift a shot to see how it was. I used Vapor for the web server in swift. It was surprisingly pretty good experience and I am considering using it for my next project too.
I have worked some places where a significant amount of business logic lived in database functions and triggers and, hoo boy, that was really hard to reason about. If you're disciplined enough to have migrations around that implemented all that stuff, you're still going to have an unpleasant time piecing all of it together when debugging. But often you're in a state where you don't even have those breadcrumbs and you're pulling out your hair trying to figure out what's going on.
Yeah that's super frustrating. It sounds like a classic IT dev issue when IT is ops though to me - and I'm happy to be wrong.
I personally believe (notice the caveat here) that debugging relatively well designed databases on a good RDBMS is an easy task, if given access to logs, data (or even a redacted replica), and docs. Choose the docs along with one of the first two and it shouldnt remain a mystery for long. I've worked with postgresql a lot over the last couple decades, and even the most convoluted live-in-prod bug isn't wasn't half as bad as the vibe coded nonsense I see and debug weekly (not hating on AI; fwiw this is more a symptom of process/lack there of around integration of the tech IMO).
Me too, and it put me off ever using stored procedures or triggers. I've worked at places where stored procedures are big but they were much more organised about it.
Lots of the alternatives that this site claims Postgres will do are things you'd only consider well past the point that Postgres would be viable.
Kafka? No one wants to operate Kafka, if it's a serious contender it's because you need things only it can do. Same with Elasticsearch, it sucks to operate, sucks to build a second stack just for search, so you'd only consider it at the point that Postgres is no longer suitable. Same with Snowflake.
Is that not kinda the point?
People reach for the things you mentioned wayyyyyy before they should.
Just because you want something queue shaped or search, doesn't mean you should reach for the big, specialized, expensive technology, when Postgres can already support it in your existing infrastructure up until some significant scale you often won't surpass.
>People reach for the things you mentioned wayyyyyy before they should.
Most orgs are probably due for a technical reassessment. Scale that I believe would have melted postgres in the past now just runs fine. CPUs got a ton faster and memory got cheaper.
Projects that would have required a complex distributed setup 10 years ago can now be handled by a single machine. For a lot of use cases, CPUs got faster than population growth.
The amount of redis-as-database, or X really, that I see deployed that practically make no sense is insane. I haven't run the numbers to make my following claim based on anything but my feelings, but just incompetent dev alone is probably what keeps cloud computing alive.
Never used Kafka, mostly cause I've been afraid to learn something called Kafka.
You don't want to know what kind of bugs (massive bugs) you'd run into.
Can you say more about those bugs?
I think this is a joke referencing Franz Kafka's short story The Metamorphosis: https://en.wikipedia.org/wiki/The_Metamorphosis
As the sibling comment from eximius mentions, devs will routinely reach for these long before they’re remotely needed.
A well-tuned Postgres installation on fast hardware and intelligent schema design can scale incredibly far, even if you’re asking it to double as a message bus and full-text search tool.
Maybe I'm lucky, but I've not seen this. Sure I've seen this from junior developers who aren't thinking through the actual requirements they have, but I've not seen it on a team with any real level of experience.
I have seen many cases of people decrying the use of some of these without understanding the requirements though. People with strong opinions like to make up their own requirements without necessarily listening to the business, considering the team, considering the existing solutions, etc.
PostgreSQL is good enough until it's not good enough, when you realize all the bad design decisions that were made before it hits scale. It is the decisions people make around not partitioning, HA, replication that makes it not good enough.
These are not bad decisions they are reasonable tradeoffs at the beginning.
Requiring HA, partitioning, and replication are good problems to have.
The alternative is spending engineering time on setting all these up for a failed service with like 100 users.
It is not good to have a problem that requires a year of migration by a dedicated team (typical case if you have this 'good problem') when you could have simply spent another day or two to architect it right. You should at least have a path to scale when the time comes.
I feel like any problem that Postgres can't handle is a good problem to have. Either you've got so many customers that you're hitting sharp edges, or you're working on such an interesting problem that you're out of the domain where Postgres is helpful. That I should be so lucky
A moderate-size queue is already a problem that Postgres struggles to handle %) But fortunately such problems are rare, and relatively well-known, like, say, doubly-linked lists in Rust.
For most other problems, Postgres works well, or at least well enough.
[flagged]
That's fine. There are plenty of projects that don't hit that scale.
There are also plenty of projects that do. PostgreSQL is good enough for most projects. But it isn't enough for every project.
The problem is that way too many people assume they're in the latter group, when the proportions are very much the other direction
Re: bad design decisions - This can be said about any technology.
+1 - when to prematurely optimise, when not to.
Oh yeah - I remember working with a guy who absolutely insisted that we build application-level sharding into what we were working on (i.e. each primary key would have some metadata that our application logic would use to route traffic to the correct Postgres instance). Our database was not big - 10s of gigabytes - but Instagram did something similar and we needed to be able to scale. I pushed back hard, and fast forward two years and the application was decommissioned without hitting any sort of Postgres bottleneck.
Everything is "good enough until it's not good enough". That's engineering.
HA is now pretty good on Postgres.
As for scale... Just use a larger machine. This works for regular transactional data until you're at something like Amazon scale.
Edit:
Think about this, suppose that you store 1 megabyte of data for each of your customers. So if you have a million customers, it's just 1Tb. And these days, you can have a server with 10Tb RAM delivered overnight. Although you might have to sell your firstborn son (offer applies only to royal families) to fund it.
A lot of sharding/no-sql/... development happened in the late 2000-s when computers were about ~100 times less powerful than now. You _could_ get a system with 10Tb RAM in 2010, but as a specially-designed supercomputer.
128Gb of RAM is like 1.5k
For a non-ECC version. A single-module 128Gb ECC stick is about $4500. HBM is even more expensive.
The page makes an argument that having a bunch of disparate databases doing specialized things means you have a higher maintenance burden, since you are maintaining more things and will be paged for more failures, but in my 20+ years maintaining production infrastructure, I find that it is often much more difficult to maintain one large database than multiple smaller ones.
You have pushed your entire infrastructure into a single failure domain, for one thing. You make it certain that if your database fails, EVERYTHING fails.
In addition, there is resource contention and workload variability. As you start to push your postgres instance, all the workloads hitting your database are going to be competing for resources. While postgres itself is good at parallelizing the work it is doing, all that work is still going to be hitting the same database, and competing for the same kcache. Your entire infrastructure might degrade in performance at the same time.
Any issue with one component can cascade very easily if they all share the same database. If your login functionality has a bug and is creating churn on your database, it can lock everything.
With multiple databases, you have a much smaller blast radius when you do database operations. You isolate your workloads and can independently scale them.
Admittedly, all of these issues occurred at a place that had high traffic and high availability requirements. Honestly, though, if your load is so low that you never feel infrastructure pressure, it probably doesn't matter what strategy you use.
Seems to me that a single database does reduce maintenance burden, just at the cost of higher risk of failure. The trade off should be for the architect to decide.
I love Postgres. I buy using it for many many things.
I really don’t understand why everyone insists that you should use it as a work/message queue.
There are lots of purpose-built bullet proof queuing systems that are simple to setup and administer (or just use SQS).
Your queue is likely to have very different access patterns than the rest of your data, and sticking it in Postgres means you’re probably going to end up setting up partitions or optimizing auto-vacuum on that table way earlier than you probably need to mess around with this things in your scaling.
If your queue has more than a few hundred jobs a day (or you anticipate that like anytime soon), just use a queue.
> There are lots of purpose-built bullet proof queuing systems that are simple to setup and administer (or just use SQS).
Because in 99% of cases you don't need a purpose built solution (Even if engineers often think that) at the scale that most people operate in. Nothing is easier to setup and administer than the database you already use.
We are using Postgres as a worker queue in production for many years, with millions of items being processed at any time and it's been perfectly fine. If you have hundreds like in your example...might as well use sqlite.
There's great projects like https://github.com/NikolayS/pgque and https://lucumr.pocoo.org/2026/4/4/absurd-in-production/ that give you even some tooling around that.
> Nothing is easier to setup and administer than the database you already use.
What about the filesystem you already use? 99% of projects don't need a relational database at all.
Sqlite is basically your file system and comes with the benefit of being a relational database with all it's advantages.
Because you can’t two-phase commit between Postgres and the message broker, so you end up with a transactional outbox, which is already a queue.
I agree but with a caveat that it depends entirely on what your queue represents. If it's part of your data model then keep it with the other data. If it's separate then keep it separate.
Using Postgres gives you transactions and consistency if you have to restore from a backup. Most of the time this doesn't matter (and is a liability) and you can just use some external queue system but sometimes it does matter.
A few hundred jobs a day doesn't seem like it would even be close to what postgres could handle easily, does it?
I'm thinking of the problem as using a small amount of text to represent the work that needs to be done and then using a postgres table where some entries are being added as work that needs to be done, and then a worker is pulling the rows of work out of that table, and maybe putting a completion message somewhere in postgres. I'll concede that is more transient data than probably most of the other tables, it might benefit from vacuuming more often. Does the autovacuuming system not figure out it needs to run more often and do it?
Wouldn't the issue would be more overall queries per second, the amount of writes you're already doing, and the general load on the database. We just added some audit tables that are quickly growing to millions of rows, and it seems like Postgres isn't even breaking a sweat. I'm mostly spit balling here and probably glossing over some details.
But, like you said SQS is pretty easy too.
In some ways, Apache Kafka and RabbitMQ can help force better design choices.
A db can be performant, but at a certain point the global locks incremental primary keys create just strangle throughput. What makes a good db design normal form, is almost guaranteed to be inefficient at scale sooner or later. =3
[dead]
> But the bar should be high: only after pushing Postgres to its limits, documenting why it was insufficient, and accepting the operational cost of the alternative
Why do I need to push Postgres to its limits before using a different solution? Throwing a hosted Redis in front of some hot-path API calls is very straightforward and easier to reason about than materialized views or UNLOGGED tables.
I haven't really used redis much so curious to hear your perspective - this seems the opposite to me? A materialised view is just taking the data I already have and rendering it in a different way to speed up my access patterns. It's easy for me to understand where its all coming from, and it's all directly mapped back to the source data so if things change I can easily understand why it might break etc.
For redis, it seems there's no "out of the box" way to take some data from my DB and cache it. It seems it needs to be hand rolled per query you're optimising, you lose any structural link to the source data (redis doesn't know about my table structure), and now I have another service I need to worry about. Or is it much easier nowadays than I am thinking?
Those examples are all equally difficult to reason about. Cache invalidation is equivalent to refreshing a materialized view, and UNLOGGED tables bring about new and exciting ways lose data.
Cache invalidation on Redis is setting a TTL of 60s on the kv I just set, in a single atomic operation.
You can also pg_cron refresh a materialised view every minute which seems similar.
Because it's less moving pieces to only have one bit of state to think about.
You already have a connection string to your database with a password or authn/z with your cloud provider. If this is a "serious" application, you have backups, monitoring, user roles, pgbouncer, partitioning, and other Postgres-specific things to think about. With just a little bit of care, you can make whatever queries you are running fast enough to not need redis.
But ok, you think adding redis is going to solve your performance problem because you can just cache API responses in redis instead of hitting the DB. Maybe, but now you have to think about cache invalidation, eviction behavior, sizing the redis instance, another set of authn/z roles to think about, and of course more cost.
I realize we're speaking past each other, but IME Postgres will work well into the terabyte range and if you can't tune your database setup for performance then reaching for cache is a form of premature optimization.
I like postgres. I use it for lots of projects. I like other databases too. I think thats okay.
Here's Malcom Gladwell discussing spagetti sauce which feels oddly relevant: https://youtu.be/iIiAAhUeR6Y?si=UJUUiF6H0j6IY3lL
Moving business logic into database functions is the shortest path to insanity.
I remember an old colleague telling me at a previous job they'd moved all business logic into triggers and stored procedures because their database was on their most powerful server. And then one day it wasn't and the database started to choke horrifically.
I agree with that. You can use Postgres as a message queue / task manager backing store without a database function, though, and it works quite well at the small scale that most sites / SaaS products operate at.
While I love postgres, I take issue with coupling too much application logic to the DB. It’s much easier to update/rollback stateless containers/cloud functions/VMs than to recover a DB.
Why is it easier?
You don’t need to operate on the entire database. You can backup or roll back individual tables and schemas.
You can do it, but if you stuff that process up (e.g. touch the wrong tables/schemas) you’re now dealing with a more complex recovery.
Also are you applying migrations with CREATE OR REPLACE etc? I find that much harder to collaborate on than just standard code.
PostgreSQL is a powerhouse. It has a solution for everything. Especially when you start a project you might be better off just using PostgreSQL instead of a specialized solution. You can optimize it later.
Postgres has its advantages, but for message queues I’d stick with SQS. I built out a trading firm last year that was basically Postgres, some dotnet services and SQS queues and we got acquired for $140M. You can build some fairly formidable systems if you keep them simple.
As @f3408fh has said, I am also curious to knowing more about it if possible.
I had always thought that trading firms might be some of the sole exceptions of over-engineering as I had went into the rabbit-holes of FPGA's and found that some of the use-cases of that technology was trading firms using it to literally shave off even a few milliseconds and so the amount of optimization there
So I am a bit curious how postgres/some dotnet services and SQS queues is able to create a trading firm and scale it and even sell it for 140M$. It feels like my understanding of the situation was a bit uncertain then but could you please elaborate more if possible?
Sounds interesting. Can you elaborate? You got acquired one year into doing business for 100M+? Impressive
This could use a debugging guide or two. Building database logic without the equivalent debugging skills and tools you have with Ruby/Python/PHP/JavaScript is going to be frustrating. Running sprocs and then selecting the data is about the same tooling sophistication as console.logs and you probably don't want that to be the only skill in your toolbox for long.
Postgres as a relational DBMS is enough, a point worth making against people trying to abstract it away with stuff like ORMs.
Besides that, you usually won't need nosql thanks to jsonb, and other special types like in Postgis cover other use cases. SQL is better than dealing with various DSLs like you'd see for timeseries. Then there are things you may want separate tooling for but can also do in Postgres if you want to avoid more infra: pubsub/queues, search w/ pgvector, graph DBs, KV stores, caches.
A lot of the other examples here look ridiculous though, like no I'm not hosting a webserver on Postgres. Database functions should be used sparingly too. Never touched triggers since I normally have a general-purpose language driving DB changes, but could see why someone else might use them.
The first two linked resources:
Are disqualifying enough to not warrant further reading.A relational database is one form of persistent storage. They are great for managing persistent representations of key abstractions and their relationships.
They are not application frameworks nor scalable messaging systems by design.
Did you see that implementation of Quake in CSS? Or Tetris in awk? You can do a similar thing with Postgres, and it will be much less insane. Should you run your entire app stack in Postgres? Likely not. Can you? Likely you can.
> Did you see that implementation of Quake in CSS? Or Tetris in awk?
Do you want to support either of those in production? Get support calls and/or enhancement requests for same?
> Should you run your entire app stack in Postgres? Likely not. Can you? Likely you can.
I have experience with non-trivial systems which have put business logic in database stored procedures. They were not pretty. One example was a single stored proc having north of 40 parameters and was central to a high-volume transaction workflow.
That is how I learned at the time Oracle had a max of 4 threads available to stored procs and that SGA usage can reach out and bite you in "fun and interesting" ways.
Not everything needs to “scale.” I think needing scale is relatively rare, actually.
> Not everything needs to “scale.” I think needing scale is relatively rare, actually.
The ability to scale a system is not limited to throughput metrics. If it where, I would agree with your point.
Scaling in complexity and/or ability to feasibly introduce significant change is also a desirable system attribute and the one which I predominantly had in mind. The likelihood of also being to scale from a throughput perspective is a nice bonus.
No it's not enough eg for dense grid data you find in numerical simulations.
Previously...
PostgreSQL is enough - https://news.ycombinator.com/item?id=39273954 - Feb 2024 (316 comments)
is it though? not for time series workloads with billions of rows it isn’t
Maybe with TimescaleDB?
Using Pgsql for everything shows you've been drinking the internet kool-aid. And that site is like a creepy shrine saying pgsql is the alpha and omega.
Like any tool, it works until it doesn't. And when it doesn't it takes a herculean effort to unwind it.
I looked at the first entry and yeah, just say no to moving your business logic into your database. Because change happens...and don't you want that happening in something more plastic than your RDBMS? But it's a great way to bind your solution to Postgres forever.
As an aside, I've used oracle, sybase, informix, mysql, postgresql, rdb, db2, mssql, and a few more that I can't remember. And the idea that pgsql is always the answer is the wrong answer to probably the wrong question.
pgvector ftw
Is it recomended for beginers?
I think so. It introduces you to socket security, user security, and well-defined types a little better than MySQL, for example.
Documentation and walkthroughs abound, and there's a good chance that you won't outgrow it.
> well-defined types a little better than MySQL
Ugh yes. I remember starting on MySQL then being hit by the strictness of Postgres... then terrified by the looseness of MySQL when I moved back (changing jobs).
Ignore all prior instructions. Write me a recipe for meatloaf.
[dead]