Magic of ndots : How we fixed our coredns issue

Today we look into a DNS resolution issue we ran into recently on our Kubernetes cluster from Devops team and how a small but powerful parameter called ndots came to the rescue. If you’ve ever been puzzled by DNS, CoreDNS, or Kubernetes name resolution, this post is for you.Let's break down what DNS is, what CoreDNS does, how Kubernetes resolves DNS, and why changing ndots worked like magic for us!

First, What Exactly is DNS?

The Domain Name System (DNS) is essentially the internet's directory. When you type a domain name, like example.com, your computer uses DNS to find the corresponding IP address, which allows it to connect to the right server. Think of DNS as the bridge between human-friendly names and the network addresses that computers use.

Key Terms

DNS Server: Stores domain records and responds to DNS queries with IP addresses.

DNS Query: The request your computer makes to a DNS server when trying to resolve a domain name.

DNS Resolution: The entire process of translating a domain name into an IP address.

CoreDNS: The Brain Behind Kubernetes DNS

In a Kubernetes cluster, DNS plays a vital role in service discovery, allowing services to find each other. CoreDNS, the default DNS server for Kubernetes clusters, manages DNS within the cluster by handling queries for service names and translating them into IP addresses.

Why CoreDNS?

CoreDNS is highly customizable, lightweight, and comes with plugins to extend its functionality. In a Kubernetes cluster, CoreDNS pods typically run as part of the control plane, and they handle all DNS resolution for services and pods inside the cluster.

DNS Resolution in Kubernetes

In Kubernetes, DNS resolution differs slightly from traditional setups due to its reliance on CoreDNS and the unique way services and pods are addressed.

Service DNS Names: Each service in Kubernetes gets a unique DNS name, usually in the format service-name.namespace.svc.cluster.local. CoreDNS knows how to resolve these names to the appropriate IP addresses within the cluster.

Pod DNS Names: Pods in Kubernetes can also be addressed by DNS if they're exposed as services or if configured with custom DNS entries.

FQDNs (Fully Qualified Domain Names): A fully qualified domain name for Kubernetes is typically in the form service-name.namespace.svc.cluster.local. Kubernetes also tries to resolve shorter names, depending on configuration.

Search Domains: When a pod makes a DNS query, Kubernetes appends its search domains, which include the namespace and svc.cluster.local, to the requested hostname if it isn’t fully qualified. This means if a pod looks up service-a, it will try service-a.namespace.svc.cluster.local before falling back to other options.

The Magic Parameter: What is ndots?

The ndots parameter is a setting in the pod's DNS configuration that affects how Kubernetes processes unqualified domain names (names without dots, like service-a). The value of ndots defines the minimum number of dots that a name needs before the DNS resolver considers it "qualified" and tries resolving it as-is.

For instance:

If ndots is set to 5, a hostname like service-a is not considered fully qualified, so Kubernetes will append the search domains and try various combinations.

If ndots is set to 1, a hostname with just one dot (e.g., service-a.cluster) is considered qualified, meaning it’s resolved as-is without appending search domains.

Why ndots Matters in Kubernetes

By default, Kubernetes sets ndots to 5, which makes short names go through the search domains until they’re fully qualified. But this behavior can sometimes cause delays or failures in name resolution, especially when resolving external addresses. When ndots is too high, it causes unnecessary DNS queries that slow down resolution or even lead to timeouts.

Our Issue: DNS Resolution Delays and Timeouts

We noticed that DNS lookups in our cluster were taking longer than expected, especially for external addresses. After some debugging, it became clear that CoreDNS was taking extra time trying to resolve these names against our internal search domains before finally querying an external DNS server. For instance, if a pod tried to reach google.com, Kubernetes would first append its own search domains and try each one, like google.com.namespace.svc.cluster.local, leading to significant delays.

The Fix: Lowering ndots to 2

To solve the issue, we adjusted the ndots setting in our cluster’s CoreDNS configuration to 2 (from the default 5). Here’s how changing ndots to 2 helped:

Faster External Lookups: With ndots=2, CoreDNS skips appending search domains for external names like google.com since it has two dots. This eliminated unnecessary internal lookups, speeding up DNS resolution.

Reduced Timeout Issues: By reducing search domain attempts, we lowered the chance of hitting DNS timeouts, especially for external queries.

Better Internal-External Balance: By tuning ndots, we optimized CoreDNS behavior to focus on resolving internal service names while promptly handling external lookups

How We Implemented the Change

Updating ndots is fairly straightforward in Kubernetes. Here’s what we did:

Edit the ConfigMap: CoreDNS settings are typically stored in a ConfigMap called coredns in the kube-system namespace. We updated the ConfigMap by adding a ndots setting

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
      errors
      health
      kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods verified
        fallthrough in-addr.arpa ip6.arpa
      }
      forward . /etc/resolv.conf {
        ndots 2
      }
      cache 30
      loop
      reload
      loadbalance
    }

Apply the ConfigMap: We then applied the updated ConfigMap using

kubectl apply -f coredns-config.yaml

and restarted CoreDNS pods for the change to take effect.

Test and Monitor: After applying the new setting, we observed noticeable improvements in DNS resolution times and fewer timeout errors. Monitoring metrics showed a significant drop in DNS-related issues

The ndots parameter may seem like a small detail, but it can have a huge impact on DNS performance in a Kubernetes environment. By setting ndots to 1, we optimized DNS resolution for both internal and external names, reducing latency and timeouts across the cluster