To push the trail metaphor further, every summer I visit a particular national park. One of my favorite trails there was the main route through the forest, which also served as a secondary path for rangers transporting equipment by car. At several points, the trail crossed rivers, and the bridges always seemed overengineered, with unusually high weight limits. Especially since the rest of the trail was barely passable by vehicle.
A few years later, the park began offering cottages along the route and started paving critical sections, particularly steep areas where trucks (presumably transporting building materials for the cottages) struggled. Had those bridges not already been built to support these trucks, the park would have needed to close the trail for upgrades. Or make a second trail with the new requirements in mind.
The trail expanded faster than it was paved, but each year you could go further as the trail grew longer, and faster as key sections were improved.
I think context matters a LOT here. Knowing when to be pragmatic and quick vs thorough is the mark of an experienced engineer. The trick is knowing HOW to pick the right approach.
A key point is understanding the business (or personal) goal of the system.
Is this a mission-critical system that is core to how the company makes money? Then make sure its robust.
Are you a startup figuring out product-market fit? Then thorough and robust systems matter less. The goal is figuring out how to get users and customers, not stabilize.
There's no way to formulate this as a set of rules or best practices, each decision has to be made in a universe of context, but I appreciated this discussion.
Something I like to do is funnel the desire to build for something I don't need right now into TODOs and even FIXMEs. Sometimes they just get deleted, but other times I'll come across an old TODO during refactoring and be able to pay down a bit of accumulated technical debt.
That's the difference between "inventing things" and "discovering things".
If you believe you're inventing a solution, doing just enough to solve the immediate problem and stopping is the consequence.
If you believe you're discovering a solution, diving deeper into the problem to try and uncover some truth about it, stopping at the first solution is not enough if you can't explain why it works or why it will keep working in the foreseeable future – because that involves solving a category of problems, not just this single instance.
People like to think in terms of under vs. over engineering, but I don't think this is the right angle to discuss. You can certainly over engineer the first solution because you focused on a single instance of a problem and "missed the forest for the trees" – identifying a general pattern is useful to find what category of problem you're dealing with, research prior-art on it and uncover elegant/economical solutions.
A product itself has no needs. Users do. Governments have a kind of "need" in the form of legislation. Managing those two kinds of needs is hard enough.
This is a really good article. Not to take away from its insight, but I think the author is very close to restating some of the central points of "The Pragmatic Programmer", especially around YAGNI and KISS.
The hover-text for the OP's linked xkcd cartoon is :
I find that when someone's taking time to do something right in the present,
they're a perfectionist with no ability to prioritize,
whereas when someone took time to do something right in the past,
they're a master artisan of great foresight.
or, loosely reinterpreted: Don't make me wait for my donuts!
In OP's terminology, I would add that the left-path ascenders are playing king-of-the-hill as they know that (in our current business environment) they can be the one to roll down rocks from the top to both crush those latecomers attempting to overtake them, as well as cheaply flatten paths up for the consumer masses.
Those masses will pay this King for an easy ascent because, above all, the Hormuz spice must flow.
Who has time to make their own donuts?!
From this kind of P.O.V. the OP's middle and right paths are often just losing strategies whose appeal lies in their comfort & morality (i.e. a luxury of personal preference).
To be clear, I am not personally advocating for Capitalist Utilitarianism here, but I find it worthwhile to see our current turmoil from this higher viewpoint and perhaps relevant to some personal software development decisions.
> A CTO friend uses the metaphor of clearing two paths up a mountain. The left-hand path is quick and dirty, cleared with a machete and brute force—you do not expect anyone to follow you, you just hack your way through—but you make progress quickly. The right-hand path is a wider, clearer, paved path, and more substantial, but takes time to build as you go along.
This is a good metaphor and is more effective than the neologisms and acronyms of the rest of the article. The author claims a "middle path" but doesn't even connect it to this real world metaphor.
It seems the author is really advocating for the "left path", but only if you are a experienced programmer and with a sprinkle of QC. In the real world metaphor, this would be like hacking through the jungle, but making a little effort to ensure your path is visible to someone else, not unnecessarily dangerous, and makes pragmatic compromises (we will go around this cliff instead of bringing climbing gear or other heavy dependencies).
If you polled 100 SWEs on the example of 'skipping the JSON library and implementing de/serialization for a few objects ' and asked whether they thought it was a "left path" or "right path" solution I'm certain you would have a strong leaning to the left, and not a 50-50 that suggests a secret middle path.
I think it's a bit nuanced, and maybe poorly explained by the original author, but to me, "left-pathers" are always "move fast and break things" to the point that whatever they build really only works as a throw-away prototype, and the effort to architect sensibly is minimal.
"We don't really need to use REST, we can just create some endpoints that have undocumented side-effects. We don't need to abstract vendor calls into a separate class, we can just implement that functionality directly in our endpoint code."
These sorts of decisions aren't actually materially faster, they're just lazier. And maybe that's "a sprinkle of QC"? But it's a lot of unforced errors that don't really save time to implement, and also create a lot of problems later on.
On the other end, with the "right-pathers", you can have people that really try to over-engineer at any opportunity. This is sort of typical of people who have worked in much larger teams. This can mean building out a k8s cluster when you're still a team of 2-3 people, splitting into 10+ microservices, deciding to use Kafka when a simple queue system would work, building out in-house load balancing for dubious reasons, etc.
The middle path is really something that resembles the "Best Simple System for Now" — when I've done this, I think about how I can solve a problem and not have to rebuild it entirely within 12-18 months.
The middle path is people that know that to scout around, you hack quick paths. Then you document your new learning, plan the best road, and then build enough for the actual journey. Every now and then, you go back and adjust everything to account for actual usage.
I think it's less a middle path than a third option. I see the type of thinking the author is commenting on whenever there is a tight requirement on a feature. Especially when you know you have to add some custom magic to meet the requirement. People tend to psychologically buy into a fully custom solution early on. So they either hack in a custom solution using abnormal data paths or known bad solutions or they fundamentally rearchitect the system around it. The third path is sketching out the feature so that it's modular and can be replaced. That way you can start out with a standard solution which gets you 90% of the way there then replace it as development progresses.
Maybe it's different in other languages, but if I ever see hand-rolled serialization in a C# project I'm putting replacing it at the top of my to-do list. There is pretty much no reason to do that ever, serialization is a solved problem. It's flat out stupid to spend extra time hand-rolling a solution that most likely ends up buggy and requires even more work in the future etc. I also often see people waste hours or days writing custom serialization logic for a serialization library, when they could have just written a new class in the shape of the data and been done in 5 minutes.
I did work on such a project once. There was a CMS web app which needed to sync data from a third party api to rebuild some pages every night. To do this they had a client library that pulled the data twice daily and stored it in a db, then the CMS app pulled the data through the client library. The API used XML and significant parts of this library were a hand-rolled xml serializer. It worked so I didn't touch it, then it stopped working. Spent days trying to fix it but the more I studied it the more I realized it was entirely pointless. Spent about 1 day replacing the whole thing with a simple little function that just calls the api, deserializes the data using a library and builds the pages. Remove the pointless DB, remove the giant legacy quagmire integration library, this project probably took months to write and I replaced it in a day, making it orders of magnitude smaller and faster.
In the end my solution wasn't used because of office politics, apparently they'd paid a significant amount of money to use this pile of trash and replacing it that easily would make someone look bad or something. So I ended up fixing the pile of crap in the end, even though I had a replacement ready to go.
I feel the question what the best approach is entirely depends on the people or organization. E.g. when you tackle a new problem, my answer would be to first solve it quick and dirty and then do it properly without over-engineering.
If the problem is one you faced a billion times, then either use the existing trusted solution and modify it, or if there is no such thing do the proper thing from the start.
If you have a problem where later adjustments are an issue for you (e.g. time wise), solutions that take that into account are superior to ones that are not.
I work with art students and program probably three new projects a week. It is okay to anticipate the needs of the people you work with and not have them spell out everything, it is okay to make a good solution even if no one asked for it, especially if you work with hardware.
E.g. knowing my "customers" I knoe they will return 2 hours before their exam is something is wrong. Guess when that debug mode comes in handy? Exactly then.
There are however ways that needlessly overcomplicate solutions or add more moving parts than needed or simply waste valuable time in the wrong moment. These need to be avoided.
Why do I always feel so different from these people? Am I the strange one? I think things like that. I like the saying YAGNI, and starting from XP, I follow the idea of keeping things simple and fast, addressing 'current requirements' as they are.
But I actually think that things like JSON Schema, UML, and READMEs are not unnecessary complexity, but rather function as a kind of social language. Just implementing things and not adding complexity to the library means, on the flip side, that there's a high risk of creating a system that only those who already know it can understand.
People always say 'You should YAGNI!' but that often just leads to tribal knowledge. In a startup, that knowledge tends to stay only with the founding members. It would be great if this tribal knowledge were always passed down, but there inevitably comes a point when it breaks, and then you're tied to the founders' bus factor. The code I'm brought in to maintain is exactly like this. Layers of tacit knowledge, like how certain hardware issues were missed, so if you code it the 'correct' way, it breaks. In other words, there have been quite a few cases where you couldn't put everything into code
Of course, documenting everything and drawing UML is also a failure. Personally, I don't think documentation is always necessary either, because keeping documentation up to date also costs time and effort.
And in reality, codebases are never clean. They change shape according to the organization's power structure. If the DevOps team is powerful, the infrastructure code gets thicker. The way API boundaries are drawn shifts depending on how responsibilities are split between backend and frontend teams.
For example, when I participated in API design as a backend developer, the frontend company asked me to put all the metadata for a single entity into one API. Their reason was that it was hard for them to handle multiple requests and they'd rather do the filtering on their own side. In reality, the right design would have been zero-trust, where I only send what's necessary. But since they were a tier above me, I just went along with it.
In that sense, I wonder if Silicon Valley culture, which carries a narrative of starting small and growing into one unified whole, is why these practices are seen as universal. I personally think using JSON Schema or writing libraries is a kind of social convention, but I don't necessarily agree with it. That said, I think the OP's opinion can be summarized as: 'Scale up when you need to scale, and don't create unnecessary boundaries that don't fit your organizational structure.'
A small team moves fast, sees user feedback, and redesigns boundaries through refactoring when needed, growing the system along the way. It's cliché, but it's also the hardest part, and it varies depending on the programmer's experience.
I envy developers in Silicon Valley. The idea of owning code and being able to make these kinds of arguments feels so foreign to me.
When I deliver software, based on my experience, I just paste in the most complex template I can think of, regardless of scale. Honestly, that's a bad programming habit too. For small code, opening, writing, and closing within a single method is often enough. The key question is whether the program keeps running, so there's no need to overcomplicate with layers.
Smart programmers usually know at what scale to stop when designing. But for a copy-paste-style coder like me, who just assembles code blocks that worked well before, it's a different story. That often ends up taking more time.
Whenever I start a new project, I immediately think about error policies, validation tables, evidence tables, and so on. I struggle through them, which sometimes delays things. But reading posts like this always feels fresh.
Sometimes I wonder: am I really a programmer, or just a factory worker?
To push the trail metaphor further, every summer I visit a particular national park. One of my favorite trails there was the main route through the forest, which also served as a secondary path for rangers transporting equipment by car. At several points, the trail crossed rivers, and the bridges always seemed overengineered, with unusually high weight limits. Especially since the rest of the trail was barely passable by vehicle.
A few years later, the park began offering cottages along the route and started paving critical sections, particularly steep areas where trucks (presumably transporting building materials for the cottages) struggled. Had those bridges not already been built to support these trucks, the park would have needed to close the trail for upgrades. Or make a second trail with the new requirements in mind.
The trail expanded faster than it was paved, but each year you could go further as the trail grew longer, and faster as key sections were improved.
I think context matters a LOT here. Knowing when to be pragmatic and quick vs thorough is the mark of an experienced engineer. The trick is knowing HOW to pick the right approach.
A key point is understanding the business (or personal) goal of the system.
Is this a mission-critical system that is core to how the company makes money? Then make sure its robust.
Are you a startup figuring out product-market fit? Then thorough and robust systems matter less. The goal is figuring out how to get users and customers, not stabilize.
I actually wrote more about this a while back: https://www.buildthestage.com/when-should-you-over-engineer-...
There's no way to formulate this as a set of rules or best practices, each decision has to be made in a universe of context, but I appreciated this discussion.
Something I like to do is funnel the desire to build for something I don't need right now into TODOs and even FIXMEs. Sometimes they just get deleted, but other times I'll come across an old TODO during refactoring and be able to pay down a bit of accumulated technical debt.
That's the difference between "inventing things" and "discovering things".
If you believe you're inventing a solution, doing just enough to solve the immediate problem and stopping is the consequence.
If you believe you're discovering a solution, diving deeper into the problem to try and uncover some truth about it, stopping at the first solution is not enough if you can't explain why it works or why it will keep working in the foreseeable future – because that involves solving a category of problems, not just this single instance.
People like to think in terms of under vs. over engineering, but I don't think this is the right angle to discuss. You can certainly over engineer the first solution because you focused on a single instance of a problem and "missed the forest for the trees" – identifying a general pattern is useful to find what category of problem you're dealing with, research prior-art on it and uncover elegant/economical solutions.
> "that meets the needs of the product"
A product itself has no needs. Users do. Governments have a kind of "need" in the form of legislation. Managing those two kinds of needs is hard enough.
This is a really good article. Not to take away from its insight, but I think the author is very close to restating some of the central points of "The Pragmatic Programmer", especially around YAGNI and KISS.
The hover-text for the OP's linked xkcd cartoon is :
or, loosely reinterpreted: Don't make me wait for my donuts!In OP's terminology, I would add that the left-path ascenders are playing king-of-the-hill as they know that (in our current business environment) they can be the one to roll down rocks from the top to both crush those latecomers attempting to overtake them, as well as cheaply flatten paths up for the consumer masses.
Those masses will pay this King for an easy ascent because, above all, the Hormuz spice must flow.
Who has time to make their own donuts?!
From this kind of P.O.V. the OP's middle and right paths are often just losing strategies whose appeal lies in their comfort & morality (i.e. a luxury of personal preference).
To be clear, I am not personally advocating for Capitalist Utilitarianism here, but I find it worthwhile to see our current turmoil from this higher viewpoint and perhaps relevant to some personal software development decisions.
The Moon is a harsh Mistress.
> A CTO friend uses the metaphor of clearing two paths up a mountain. The left-hand path is quick and dirty, cleared with a machete and brute force—you do not expect anyone to follow you, you just hack your way through—but you make progress quickly. The right-hand path is a wider, clearer, paved path, and more substantial, but takes time to build as you go along.
This is a good metaphor and is more effective than the neologisms and acronyms of the rest of the article. The author claims a "middle path" but doesn't even connect it to this real world metaphor.
It seems the author is really advocating for the "left path", but only if you are a experienced programmer and with a sprinkle of QC. In the real world metaphor, this would be like hacking through the jungle, but making a little effort to ensure your path is visible to someone else, not unnecessarily dangerous, and makes pragmatic compromises (we will go around this cliff instead of bringing climbing gear or other heavy dependencies).
If you polled 100 SWEs on the example of 'skipping the JSON library and implementing de/serialization for a few objects ' and asked whether they thought it was a "left path" or "right path" solution I'm certain you would have a strong leaning to the left, and not a 50-50 that suggests a secret middle path.
I think it's a bit nuanced, and maybe poorly explained by the original author, but to me, "left-pathers" are always "move fast and break things" to the point that whatever they build really only works as a throw-away prototype, and the effort to architect sensibly is minimal.
"We don't really need to use REST, we can just create some endpoints that have undocumented side-effects. We don't need to abstract vendor calls into a separate class, we can just implement that functionality directly in our endpoint code."
These sorts of decisions aren't actually materially faster, they're just lazier. And maybe that's "a sprinkle of QC"? But it's a lot of unforced errors that don't really save time to implement, and also create a lot of problems later on.
On the other end, with the "right-pathers", you can have people that really try to over-engineer at any opportunity. This is sort of typical of people who have worked in much larger teams. This can mean building out a k8s cluster when you're still a team of 2-3 people, splitting into 10+ microservices, deciding to use Kafka when a simple queue system would work, building out in-house load balancing for dubious reasons, etc.
The middle path is really something that resembles the "Best Simple System for Now" — when I've done this, I think about how I can solve a problem and not have to rebuild it entirely within 12-18 months.
The middle path is people that know that to scout around, you hack quick paths. Then you document your new learning, plan the best road, and then build enough for the actual journey. Every now and then, you go back and adjust everything to account for actual usage.
I think it's less a middle path than a third option. I see the type of thinking the author is commenting on whenever there is a tight requirement on a feature. Especially when you know you have to add some custom magic to meet the requirement. People tend to psychologically buy into a fully custom solution early on. So they either hack in a custom solution using abnormal data paths or known bad solutions or they fundamentally rearchitect the system around it. The third path is sketching out the feature so that it's modular and can be replaced. That way you can start out with a standard solution which gets you 90% of the way there then replace it as development progresses.
Maybe it's different in other languages, but if I ever see hand-rolled serialization in a C# project I'm putting replacing it at the top of my to-do list. There is pretty much no reason to do that ever, serialization is a solved problem. It's flat out stupid to spend extra time hand-rolling a solution that most likely ends up buggy and requires even more work in the future etc. I also often see people waste hours or days writing custom serialization logic for a serialization library, when they could have just written a new class in the shape of the data and been done in 5 minutes.
I did work on such a project once. There was a CMS web app which needed to sync data from a third party api to rebuild some pages every night. To do this they had a client library that pulled the data twice daily and stored it in a db, then the CMS app pulled the data through the client library. The API used XML and significant parts of this library were a hand-rolled xml serializer. It worked so I didn't touch it, then it stopped working. Spent days trying to fix it but the more I studied it the more I realized it was entirely pointless. Spent about 1 day replacing the whole thing with a simple little function that just calls the api, deserializes the data using a library and builds the pages. Remove the pointless DB, remove the giant legacy quagmire integration library, this project probably took months to write and I replaced it in a day, making it orders of magnitude smaller and faster.
In the end my solution wasn't used because of office politics, apparently they'd paid a significant amount of money to use this pile of trash and replacing it that easily would make someone look bad or something. So I ended up fixing the pile of crap in the end, even though I had a replacement ready to go.
Sounds a bit like the First System from Gancarz’s The Unix Philosophy.
I feel the question what the best approach is entirely depends on the people or organization. E.g. when you tackle a new problem, my answer would be to first solve it quick and dirty and then do it properly without over-engineering.
If the problem is one you faced a billion times, then either use the existing trusted solution and modify it, or if there is no such thing do the proper thing from the start.
If you have a problem where later adjustments are an issue for you (e.g. time wise), solutions that take that into account are superior to ones that are not.
I work with art students and program probably three new projects a week. It is okay to anticipate the needs of the people you work with and not have them spell out everything, it is okay to make a good solution even if no one asked for it, especially if you work with hardware.
E.g. knowing my "customers" I knoe they will return 2 hours before their exam is something is wrong. Guess when that debug mode comes in handy? Exactly then.
There are however ways that needlessly overcomplicate solutions or add more moving parts than needed or simply waste valuable time in the wrong moment. These need to be avoided.
What is the best way to write a program? Depends.
Why do I always feel so different from these people? Am I the strange one? I think things like that. I like the saying YAGNI, and starting from XP, I follow the idea of keeping things simple and fast, addressing 'current requirements' as they are.
But I actually think that things like JSON Schema, UML, and READMEs are not unnecessary complexity, but rather function as a kind of social language. Just implementing things and not adding complexity to the library means, on the flip side, that there's a high risk of creating a system that only those who already know it can understand.
People always say 'You should YAGNI!' but that often just leads to tribal knowledge. In a startup, that knowledge tends to stay only with the founding members. It would be great if this tribal knowledge were always passed down, but there inevitably comes a point when it breaks, and then you're tied to the founders' bus factor. The code I'm brought in to maintain is exactly like this. Layers of tacit knowledge, like how certain hardware issues were missed, so if you code it the 'correct' way, it breaks. In other words, there have been quite a few cases where you couldn't put everything into code
Of course, documenting everything and drawing UML is also a failure. Personally, I don't think documentation is always necessary either, because keeping documentation up to date also costs time and effort.
And in reality, codebases are never clean. They change shape according to the organization's power structure. If the DevOps team is powerful, the infrastructure code gets thicker. The way API boundaries are drawn shifts depending on how responsibilities are split between backend and frontend teams.
For example, when I participated in API design as a backend developer, the frontend company asked me to put all the metadata for a single entity into one API. Their reason was that it was hard for them to handle multiple requests and they'd rather do the filtering on their own side. In reality, the right design would have been zero-trust, where I only send what's necessary. But since they were a tier above me, I just went along with it.
In that sense, I wonder if Silicon Valley culture, which carries a narrative of starting small and growing into one unified whole, is why these practices are seen as universal. I personally think using JSON Schema or writing libraries is a kind of social convention, but I don't necessarily agree with it. That said, I think the OP's opinion can be summarized as: 'Scale up when you need to scale, and don't create unnecessary boundaries that don't fit your organizational structure.'
A small team moves fast, sees user feedback, and redesigns boundaries through refactoring when needed, growing the system along the way. It's cliché, but it's also the hardest part, and it varies depending on the programmer's experience.
I envy developers in Silicon Valley. The idea of owning code and being able to make these kinds of arguments feels so foreign to me.
When I deliver software, based on my experience, I just paste in the most complex template I can think of, regardless of scale. Honestly, that's a bad programming habit too. For small code, opening, writing, and closing within a single method is often enough. The key question is whether the program keeps running, so there's no need to overcomplicate with layers.
Smart programmers usually know at what scale to stop when designing. But for a copy-paste-style coder like me, who just assembles code blocks that worked well before, it's a different story. That often ends up taking more time.
Whenever I start a new project, I immediately think about error policies, validation tables, evidence tables, and so on. I struggle through them, which sometimes delays things. But reading posts like this always feels fresh.
Sometimes I wonder: am I really a programmer, or just a factory worker?