Hey there and welcome to the third part of this blog series! You may find the previous parts below:
- First part – Troubleshooting connectivity (Layer 4 TCP/IP)
- Second part – Troubleshooting latency, performance, and throughput
In this third episode, we will be talking about:
It’s worth mentioning that I will be mostly focusing on troubleshooting DNS from a client perspective, rather than DNS server troubleshooting. So open your favorite command line utility or terminal and follow me while we explore some of these helpful tools for DNS troubleshooting!
It is always DNS! Isn’t it? I bet that you heard this phrase before, it may be true to a certain degree. DNS may be the underlying cause of many different types of issues which at first sight may look like a “non-working” application, SQL database, website, and a long list of, etc.
Let’s do a quick review of how DNS works. Domain Name System (DNS) may be one of the most important network protocols out there, the reason is simple, the internet runs over IP addresses, all the private and public network devices from the routers that interconnect us all together, to the web servers running your favorite websites, to all our smartphones and smart devices communicate using IP addresses. As humans, most of us are not great at remembering long strings of numbers (especially true for IPv6… :p), It is easier to remember “facebook.com” than “157.240.19.35”.
So the solution for this problem is DNS! DNS allows us to “translate” names to IP addresses and vice-versa, we are all using DNS servers or forwarders. We can find our DNS servers by running a quick command such as:
# Windows
# Open a command prompt or PowerShell session and run:
ipconfig /all
# Then look for your network card "DNS Servers"
#In my case:
DNS Servers . . . . . . . . . . . : 10.0.0.8
75.75.75.75
75.75.76.76
1.1.1.1
# Linux
# There's a few ways to check it from here, some of the ones I know:
cat /etc/resolv.conf
nameserver 192.168.144.1
resolvectl
#or
systemd-resolve --status
Current DNS Server: 10.0.0.8
DNS Servers: 10.0.0.8
75.75.75.75
1.1.1.1So these folks provide us with name resolution, now how do we actually know, or how we can test our DNS server’s resolutions? And what kind of problems can we find in the cases it doesn’t work as expected?
NSLookup: Name System lookup, is a simple and effective tool that we can use to troubleshoot and diagnose DNS issues. It’s by default available in most operating systems. Let’s imagine this scenario, we have a user working in a remote office that calls us and lets us know that they are not able to access any websites on their laptop.
Whenever they try searching a website, they get the following error:
Since the office is interconnected to our own office through VPN, the user gives us its IP so we can get remote access to its computer.
Once we are in, we can see that there’s local connectivity since we can reach the computer remotely, we check if our routing is good, by pinging a public IP address from the user’s machine, and we can reach them with no issues.
So we move our troubleshooting to DNS, and we ask our users to provide us with the URL they are not able to access, in their case “cats.com”. Let’s start our testing with NSLoookup.
With NSLookup, we are able to test if our client machine can resolve a DNS record, we can use either PowerShell or Command prompt.
# Running nslookup, we can see the DNS server for our
# client (address highlighted below) is not returning
# any records for cats.com
PS C:\Users\Administrator> nslookup cats.com
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 10.0.0.159
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
# The good thing with NSLookup is that we can use
# different DNS servers, so if one of not working, we
# can compare the results with a different one, for
# instance, let's try looking up the same record, but
# this time using google.dns server (8.8.8.8).
Using google DNS we are able to resolve the A record for cats.com. This indicates an issue with the current’s client DNS (10.0.0.159). Upon further investigation, we find this is a legacy DNS server that should not be used, and the user had it configured statically in its computer, so we proceed to change it.
Now, we do our testing again:
So now that we are using the current company’s DNS server and that we can resolve the record for cats.com, let’s try browsing back to it.
Yei! Our user is now able to access cats.com and we both can continue our utterly important businesses.
DIG: This is another awesome DNS troubleshooting tool, it will be installed in most of your Linux distributions by default and will provide us with great details when obtaining DNS information for troubleshooting. Let’s check some of its options and results:
In red we have our DIG command, the query sent to the DNS server, the answers/address received, and the server itself that responded to our query.
As we can see from the example above, we get a lot of useful information from using DIG, including even more, advanced details such as flags, query time, status, and more.
Honorific mention: https://www.digwebinterface.com/
This website allows us to query multiple DNS servers globally for the same record, these can be helpful when we encounter situations where one location may be having DNS issues, while the other is working correctly, this will help us to identify if the DNS record may be behind a CDN with multiple public IPs. For instance:
One thing worth noting whenever working with DNS, sometimes you may find occasions when you have an incorrect or old record that is still cached on the client’s machine. You can get a list of the records saved on a computer by using:
# Windows
ipconfig /displaydns
# In order to delete these cached record you may run from an
# elevated command prompt window:
ipconfig /flushdns
Additionally, all the OS’es have a “local DNS” also known as the host file, you may find occasions where you encounter records with an excessive time-to-live that won’t clear after a Flushdns, or that will continue to resolve even after we checked our DNS server resolves to a different address. The host file takes resolution preference over DNS. For instance:
Whenever we see these traits, we are probably dealing with a record saved in the host file of our computer, we can check the host file by navigating to it and opening it with a text editor.
For Windows: c:\windows\system32\drivers\hosts
For Linux /etc/hosts
Contents of a host file:
I wanted to mention, in the case that you may want to check for an specify DNS record type and check if it exits, you can do so from both NSLookup and Dig just like this:
# For NSlookup, notice the -type flag, you may specify this as #TXT, CNAME, A, PTR, MX, etc.
PS C:\tools\arm-templates> nslookup -type=CNAME google.com
google.com
primary name server = ns1.google.com
responsible mail addr = dns-admin.google.com
serial = 522566291
refresh = 900 (15 mins)
retry = 900 (15 mins)
expire = 1800 (30 mins)
default TTL = 60 (1 min)
PS C:\tools\arm-templates> nslookup -type=A google.com
Non-authoritative answer:
Name: google.com
Addresses: 142.250.138.113
142.250.138.101
142.250.138.139
142.250.138.102
142.250.138.100
142.250.138.138
PS C:\tools\arm-templates> nslookup -type=MX google.com
Non-authoritative answer:
google.com MX preference = 10, mail exchanger = smtp.google.com
smtp.google.com internet address = 142.250.114.27
smtp.google.com internet address = 142.250.113.27
smtp.google.com internet address = 142.251.116.26
smtp.google.com internet address = 142.250.138.26
smtp.google.com internet address = 142.250.138.27
smtp.google.com AAAA IPv6 address = 2607:f8b0:4023:1002::1a
smtp.google.com AAAA IPv6 address = 2607:f8b0:4023:1000::1b
smtp.google.com AAAA IPv6 address = 2607:f8b0:4023:1009::1b
smtp.google.com AAAA IPv6 address = 2607:f8b0:4023:1006::1b
# in the case of Dig, you can do the same thing with the -t
# flag:
$ dig -t A google.com
; <<>> DiG 9.16.1-Ubuntu <<>> -t A google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25494
;; flags: qr rd ad; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;google.com. IN A
;; ANSWER SECTION:
google.com. 0 IN A 142.250.138.113
google.com. 0 IN A 142.250.138.102
google.com. 0 IN A 142.250.138.100
google.com. 0 IN A 142.250.138.139
google.com. 0 IN A 142.250.138.101
google.com. 0 IN A 142.250.138.138
;; Query time: 0 msec
;; SERVER: 172.20.160.1#53(172.20.160.1)
;; WHEN: Fri Apr 07 17:54:36 CDT 2023
;; MSG SIZE rcvd: 134
# If you're only interested in the records themselves, you may
# use the +short option as shown below:
$ dig -t A +short google.com
142.250.138.113
142.250.138.102
142.250.138.100
142.250.138.139
142.250.138.101
142.250.138.138
There’s a LOT more to talk about DNS, including its record types, DNS hierarchical’s architecture, DNS Secure (DNS SEC), statuses, and much much more. I may explore these topics in a future post, but hopefully, this will get your curiosity itching to investigate more about these and other tools to understand and troubleshoot old good DNS.
And that’s it for now, as I said there’s a LOT more to learn here, but wanted to give a quick overview of these tools and how you can use them for troubleshooting.
I will see you next time for the second and following parts of this series where we will be talking about:
- Packet analysis
- Wireshark and TCPDump
- Troubleshooting the application layer (Layer 7 – HTTP/HTTPS)
- CURL and OpenSSL
- Dev-tools and Fiddler
Thanks for reading! And wish you lots of success in your DNS troubleshooting!
