The biggest issue with DNS is not the protocol, or even the reference implementation. It's the people who think they are clever and try to make things better by making them worse.
The most egregious of course is ISPs rewriting TTLs (or resolvers that just ignore them). But there are other implementation issues too, like caching things that shouldn't be or doing it wrong. I've seen resolvers that cache a CNAME and the A record it resolves to with the TTL of the CNAME (which is wrong).
I'm also very concerned about the "WHY DNS MATTERS FOR SYSTEM DESIGN" section. While everything there is correct enough, it doesn't dive into the implication of each and how things go wrong.
For example, using DNS for round robin balancing is an awful idea in practice. Because Comcast will cache one IP of three, and all of a sudden 60% of your traffic is going to one IP. Similar issue with regional IPs. There are so many ways for the wrong IP to get into a cache.
ISP DNS servers really ought to be banned, they are always so bad. I've seen traffic days later on a record with 1 hour TTL. In general I see like 50% traffic move after the initial 1-2x TTL interval, another 40-45% over next several hours up to one day, and then the last 5-1% can take forever.
For round-robin, I've actually had it work reasonably well for API usage. Of course it's not ideal, but when I wanted to roll out new things slowly over several days and could not use a load balancer or reverse proxy, it kind of worked. I think most API users are just running with a reasonable resolver and not residential ISP ones.
When I moved reddit from one datacenter to another, about 70% of the traffic shifted within the TTL. Another 20% moved within a week. Took till the end of a month after the change to get to about 98%
But after two months, about 1% was still going to the old server (I had set it up as a proxy for the cutover). Most of that traffic looked like crawlers that were written in things like Python or Ruby and had probably hard coded the IP or done something where it just didn't know what a TTL was.
So at that point I just shut down the old server.
You're probably right about API clients using better resolvers though. I was talking about consumer facing things where a lot of people would be on ISP DNS.
I have to admit - I still grind my teeth every time I see "dns propagation" used without a direct follow-up that it's a myth, you're looking at cascading cache expiry.
Propagation might be a useful way to visualise it, but doesn't match reality unless every cache is a warm cache.
DNS changes propagate. They just do-so in a pull, not push, way.
It’s accurate to say that a user is waiting for the change to propagate if they are sitting there clicking re-try as they wait for the cascading cache expirations to do their thing.
Yes! The idea of DNS records "propagating" gave me entirely the wrong mental model of DNS very early in my career. Granted, the confusion didn't last long because I read the cricket book soon after, but it was still pretty jarring.
> Without DNS, you'd need to memorize IP addresses for every website.
This used to be true until virtual hosting came along, allowing for several domains to point to the same IP address, but only for non-HTTPS traffic. Then a bit later we got SNI (Server Name Indication) that did the same thing for HTTPS.
I remember having web servers with 10-12 public IP adresses when I started working. The number of IPv4 addresses needed has been greatly reduced since.
It might be worth mentioning the concept of "stub resolver" and clarifying a bit that a nameserver is a resolver. That might be being pedantic, but thought it might be worth clarifying that the difference conceptually may just be what the particular dns server answering the query is authoritative for, if anything.
One other thing that might be worth a mention is the concept of the OS' resolver and "suffix search order", with an example of connecting (https, ping, ssh, whatever protocol) to a host using just the hostname, and the aforementioned mechanism that (probably) allows this to connect to the FQDN you want. (Also, now that I type that, do you mention "FQDN" at all? If not, maybe should.)
On that note one final thought that occurs to me is the error/confound that may occur if a hostname is entered and is not resolved, but does resolve with one of the domain suffixes attached on a retry (particularly can be confusing with a typo coupled with a wildcard A record in a domain, for example.) I recognize that the lines that look like DNS records are not explicitly stated to be in a format for any particular dns server software, and even if they were, they're snippets without larger context so we don't know what the $ORIGIN for the zone might be, an adjacent concept you might want to explore, even if just for your own edification is that of the effect of a terminating "." at the end of a hostname, either at resolution or configuration time.
Just offering feedback that might help you add to the article.
> You can change which resolver you use in your network settings. I switched to 1.1.1.1 on my machines - it's noticeably faster than my ISP's default resolver.
Noticeably faster as in just loading a website? Or in some script where small differences add up? I thought typical DNS lookup was sub 100ms, but I've never tried switching my resolver so I'm curious
It's the most baroque protocol that is still somehow surviving from the initial Internet. There are so many weird limitations, like not being able to use CNAME for apex zones. Or the entire DNSSEC fiasco.
I think CNAMES were a mistake, think of them as lightweight NS records. It is a simple protocol solving a rather complex system. I have respect for simple things I have tried implementing my own server, did not go well.
You have to go back five decades if you want to use a host file. My point is that cache and making are the three hard things in computer science DNS has had both problems several times. I choose DNS over hosts files every time.
Only oddity was the reference to the "router cache". I agree if your browser tried to lookup example.com the local cache would be used, but then it would be the system's configured DNS server - and that would most likely be an ISP, rather than your local router.
(Assuming a typical home connection, your router is _probably_ not a DNS server with local cache, it probably is a DHCP server which will hand out the upstream/ISPs' nameservers.)
I think this is probably quite dependent on what’s normal for ISPs in the region. In the UK for example, every ISP router I’ve had runs a DNS server and it’s that which is given out via DHCP. It then forwards onto the ISPs DNS platform.
American here, most of ISPs here do it as well. With modern router hardware, there is plenty of hardware available to run tiny DNS server that caches and forwards all requests to ISP upstream. Memory overhead is probably about 50MB and CPU overhead is trivial, probably .1% or less.
I would argue the contrary - most home routers are running a DNS server of some kind. They forward to upstream, but will resolve local names like your printer and whatnot.
dnsmasq is the defacto tool on these embedded devices for dhcp+dns. probably a billion deployments. it's up there with sqlite for most used tech.
IIRC a resolver is what people would think of as a DNS server only it's not an authority for any domains. Like you said, they're used to get load off of authoritative servers and are very common. I think dnsmasq is mentioned explicitly in the O'Reilly locust book but it's been a while.
My parents are with Bell (the biggest ISP in Canada) and use the Bell Gigahub (Router/AP/Switch in one). It does have a DNS cache and the its set as the DNS resolver in their DHCP configuration.
Cool, but the macOS commands being wrong irks me. You could at least check them before publishing. It seems like you used an LLM to write that part. And the table "quick reference" at the end is also a LLM tell.
This might be the easiest-to-understand breakdown of DNS that I've seen to date. I've owned a domain since the late 90s, but never really understood everything the acronyms or concepts involved in making it work. Well done!
Honestly, I guess it's a fine article for someone who isn't very technical, but it provides very little real detail, and this wasn't an example of DNS breaking anything - it worked as designed.
The biggest pain of DNS for most people is if someone has set the TTL to an absurdly large number, or if a resolver isn't respecting TTL. And once you get into advanced configurations, SOAs and delegation certainly create their own headaches!
This is not a bad overview of DNS from a theoretical perspective. It’s also pretty well written and has good examples and figures.
What I think is missing is a bit more of the “in practice” side. If the author was surprised about TTL values, I doubt they have much experience with some of the other pitfalls, so I’m not surprised (not a knock on the author). But there is a reason why the phrase “It’s always DNS” exists.
As an example, it could be helpful to mention that ISP DNS resolvers (or any caching resolver in the path) could decide to ignore the TTL. In this case, your 360 sec TTL might not get updated for an hour or a day or longer. This can be infuriating to troubleshoot.
A section on troubleshooting might also be beneficial. But this mainly consists of checking results from different resolvers in your path - does it work with a local resolver? Your ISPs DNS? The authoritative server?
> it could be helpful to mention that ISP DNS resolvers (or any caching resolver in the path) could decide to ignore the TTL
not exactly the same but it reminds me of MTU. You can set it to whatever you want but it only takes one router that you don't control in the path to undo what you're trying to accomplish.
The biggest issue with DNS is not the protocol, or even the reference implementation. It's the people who think they are clever and try to make things better by making them worse.
The most egregious of course is ISPs rewriting TTLs (or resolvers that just ignore them). But there are other implementation issues too, like caching things that shouldn't be or doing it wrong. I've seen resolvers that cache a CNAME and the A record it resolves to with the TTL of the CNAME (which is wrong).
I'm also very concerned about the "WHY DNS MATTERS FOR SYSTEM DESIGN" section. While everything there is correct enough, it doesn't dive into the implication of each and how things go wrong.
For example, using DNS for round robin balancing is an awful idea in practice. Because Comcast will cache one IP of three, and all of a sudden 60% of your traffic is going to one IP. Similar issue with regional IPs. There are so many ways for the wrong IP to get into a cache.
There is a reason we say "it's always DNS".
ISP DNS servers really ought to be banned, they are always so bad. I've seen traffic days later on a record with 1 hour TTL. In general I see like 50% traffic move after the initial 1-2x TTL interval, another 40-45% over next several hours up to one day, and then the last 5-1% can take forever.
For round-robin, I've actually had it work reasonably well for API usage. Of course it's not ideal, but when I wanted to roll out new things slowly over several days and could not use a load balancer or reverse proxy, it kind of worked. I think most API users are just running with a reasonable resolver and not residential ISP ones.
When I moved reddit from one datacenter to another, about 70% of the traffic shifted within the TTL. Another 20% moved within a week. Took till the end of a month after the change to get to about 98%
But after two months, about 1% was still going to the old server (I had set it up as a proxy for the cutover). Most of that traffic looked like crawlers that were written in things like Python or Ruby and had probably hard coded the IP or done something where it just didn't know what a TTL was.
So at that point I just shut down the old server.
You're probably right about API clients using better resolvers though. I was talking about consumer facing things where a lot of people would be on ISP DNS.
> It's the people who think they are clever and try to make things better by making them worse.
you mean DNSSEC, right? RIGHT?
I didn't actually. :). I meant people who change their DNS server or client (usually server) to do "clever" or "more efficient" things.
DNSSEC at least has its own RFC and uses the TXT record, which was added to DNS specifically for this kind of use case.
How should we make DNS secure, and why aren't you tptacek?
I have to admit - I still grind my teeth every time I see "dns propagation" used without a direct follow-up that it's a myth, you're looking at cascading cache expiry.
Propagation might be a useful way to visualise it, but doesn't match reality unless every cache is a warm cache.
DNS changes propagate. They just do-so in a pull, not push, way.
It’s accurate to say that a user is waiting for the change to propagate if they are sitting there clicking re-try as they wait for the cascading cache expirations to do their thing.
https://jvns.ca/blog/2021/12/06/dns-doesn-t-propagate/
And checkout their Mess with DNS playgound!
Yes! The idea of DNS records "propagating" gave me entirely the wrong mental model of DNS very early in my career. Granted, the confusion didn't last long because I read the cricket book soon after, but it was still pretty jarring.
I grind my teeth every time I hear "I need an urgent DNS change" :/
> Without DNS, you'd need to memorize IP addresses for every website.
This used to be true until virtual hosting came along, allowing for several domains to point to the same IP address, but only for non-HTTPS traffic. Then a bit later we got SNI (Server Name Indication) that did the same thing for HTTPS.
I remember having web servers with 10-12 public IP adresses when I started working. The number of IPv4 addresses needed has been greatly reduced since.
You'd still need to memorize the IP address without DNS, otherwise how would you know to which server to connect?
The fact that a server can serve multiple vhosts and do TLS cert selection via SNI is not related to the lookup of what server to connect to.
You'd still need to send a Host header for the website you want and you can't do that in the address bar.
Just connect to cloudflare /s
It might be worth mentioning the concept of "stub resolver" and clarifying a bit that a nameserver is a resolver. That might be being pedantic, but thought it might be worth clarifying that the difference conceptually may just be what the particular dns server answering the query is authoritative for, if anything.
One other thing that might be worth a mention is the concept of the OS' resolver and "suffix search order", with an example of connecting (https, ping, ssh, whatever protocol) to a host using just the hostname, and the aforementioned mechanism that (probably) allows this to connect to the FQDN you want. (Also, now that I type that, do you mention "FQDN" at all? If not, maybe should.)
On that note one final thought that occurs to me is the error/confound that may occur if a hostname is entered and is not resolved, but does resolve with one of the domain suffixes attached on a retry (particularly can be confusing with a typo coupled with a wildcard A record in a domain, for example.) I recognize that the lines that look like DNS records are not explicitly stated to be in a format for any particular dns server software, and even if they were, they're snippets without larger context so we don't know what the $ORIGIN for the zone might be, an adjacent concept you might want to explore, even if just for your own edification is that of the effect of a terminating "." at the end of a hostname, either at resolution or configuration time.
Just offering feedback that might help you add to the article.
I really like Julia Evan's explainer on this as well: https://wizardzines.com/zines/dns/
> You can change which resolver you use in your network settings. I switched to 1.1.1.1 on my machines - it's noticeably faster than my ISP's default resolver.
Noticeably faster as in just loading a website? Or in some script where small differences add up? I thought typical DNS lookup was sub 100ms, but I've never tried switching my resolver so I'm curious
To avoid centralisation, I suggest running your own resolver. 1.1.1.1 censors a lot of places.
Curious how you’d measure this
https://www.dnsperf.com/#!dns-resolvers
https://github.com/cleanbrowsing/dnsperftest
> DNS broke my site for three hours. But now I actually understand it
I have been broken for three decades and I still don't understand DNS. It is a simple protocol but people use it in complicated manners.
Simple? Oh no. Simple it is not.
It's the most baroque protocol that is still somehow surviving from the initial Internet. There are so many weird limitations, like not being able to use CNAME for apex zones. Or the entire DNSSEC fiasco.
I think CNAMES were a mistake, think of them as lightweight NS records. It is a simple protocol solving a rather complex system. I have respect for simple things I have tried implementing my own server, did not go well.
why you need DNS for at server? just use hosts file. why your server would need to resolve domains on the internet? client yeah, server no.
You have to go back five decades if you want to use a host file. My point is that cache and making are the three hard things in computer science DNS has had both problems several times. I choose DNS over hosts files every time.
EDIT I might be off by one here.
> why you need DNS for at server? just use hosts file.
IP's can change without warning.
interesting.
But it's not an issue at all, and it provides a convenience that can be depended on by a lot of your dependencies.
Code may use domains instead of ip addresses (which provides resiliency), package managers like apt depend on domains. And so on.
clusters and other load-balanced workloads. who wants to maintain hosts files across a fleet of containers or multiregion load-balanced situations?
Only oddity was the reference to the "router cache". I agree if your browser tried to lookup example.com the local cache would be used, but then it would be the system's configured DNS server - and that would most likely be an ISP, rather than your local router.
(Assuming a typical home connection, your router is _probably_ not a DNS server with local cache, it probably is a DHCP server which will hand out the upstream/ISPs' nameservers.)
I think this is probably quite dependent on what’s normal for ISPs in the region. In the UK for example, every ISP router I’ve had runs a DNS server and it’s that which is given out via DHCP. It then forwards onto the ISPs DNS platform.
In Scotland I was with Telewest, then Virgin, and my memory is always that the DHCP pushed out the external IP of the ISP's DNS servers.
Nowadays I'm in Finland and definitely the router runs no DNS service, the DHCP service advertises the ISP resolvers.
Probably depends on the region/ISP I guess, but I had no expectation that it would be the more common option.
American here, most of ISPs here do it as well. With modern router hardware, there is plenty of hardware available to run tiny DNS server that caches and forwards all requests to ISP upstream. Memory overhead is probably about 50MB and CPU overhead is trivial, probably .1% or less.
I would argue the contrary - most home routers are running a DNS server of some kind. They forward to upstream, but will resolve local names like your printer and whatnot.
dnsmasq is the defacto tool on these embedded devices for dhcp+dns. probably a billion deployments. it's up there with sqlite for most used tech.
IIRC a resolver is what people would think of as a DNS server only it's not an authority for any domains. Like you said, they're used to get load off of authoritative servers and are very common. I think dnsmasq is mentioned explicitly in the O'Reilly locust book but it's been a while.
My parents are with Bell (the biggest ISP in Canada) and use the Bell Gigahub (Router/AP/Switch in one). It does have a DNS cache and the its set as the DNS resolver in their DHCP configuration.
The system's configured DNS resolver is usually your router.
Cool, but the macOS commands being wrong irks me. You could at least check them before publishing. It seems like you used an LLM to write that part. And the table "quick reference" at the end is also a LLM tell.
and the formatting, and "the cool part? the weird part? the catch?", and "engaging" questions at the end of not just this but other posts on that blog
unfortunately the writing is getting praised even here so it will continue
This might be the easiest-to-understand breakdown of DNS that I've seen to date. I've owned a domain since the late 90s, but never really understood everything the acronyms or concepts involved in making it work. Well done!
Thanks! That was exactly what I was going for - making DNS approachable for everyone, not just sysadmins. Glad it helped!
Namasté. First blog post I’ve read from Nepal, very well done.
One nitpick I had was that the subdomain examples are more likely to be hosts. They could be either, but was not mentioned. Perhaps I missed.
I love DNS!
https://www.instagram.com/p/DUTSLcjkfJn/
Honestly, I guess it's a fine article for someone who isn't very technical, but it provides very little real detail, and this wasn't an example of DNS breaking anything - it worked as designed.
The biggest pain of DNS for most people is if someone has set the TTL to an absurdly large number, or if a resolver isn't respecting TTL. And once you get into advanced configurations, SOAs and delegation certainly create their own headaches!
This is not a bad overview of DNS from a theoretical perspective. It’s also pretty well written and has good examples and figures.
What I think is missing is a bit more of the “in practice” side. If the author was surprised about TTL values, I doubt they have much experience with some of the other pitfalls, so I’m not surprised (not a knock on the author). But there is a reason why the phrase “It’s always DNS” exists.
As an example, it could be helpful to mention that ISP DNS resolvers (or any caching resolver in the path) could decide to ignore the TTL. In this case, your 360 sec TTL might not get updated for an hour or a day or longer. This can be infuriating to troubleshoot.
A section on troubleshooting might also be beneficial. But this mainly consists of checking results from different resolvers in your path - does it work with a local resolver? Your ISPs DNS? The authoritative server?
> it could be helpful to mention that ISP DNS resolvers (or any caching resolver in the path) could decide to ignore the TTL
not exactly the same but it reminds me of MTU. You can set it to whatever you want but it only takes one router that you don't control in the path to undo what you're trying to accomplish.
Well written! Is so easy to understand and read that I can easily share it even with non-tech people c:
Appreciate it! That's the goal - tech concepts shouldn't need a CS degree to understand.
well written
would really be happy to have had these explanations before I had to figure it out for myself.
then you have these guys who reached the next level
https://www.dns.toys/
DNS is an example of a globally distributed DB
Well written article!
Glad you liked it. Thank you