Kubernetes makes it easy to distribute container replicas (Pods) across several compute Nodes, improving redundancy and performance. However, this also results in networking challenges: How can your apps reliably communicate when you’re constantly adding and removing Pod replicas and Services?
The Kubernetes DNS service is the answer. This core cluster component implements name resolution and service discovery for your Pods and Services. It lets you connect to predictable hostnames instead of unreliable IP addresses that could change at any time.
In this guide, we’ll learn how Kubernetes DNS is implemented and explore how you can use it to connect your Pods and Services. We’ll also share some troubleshooting tips to help you debug Kubernetes DNS issues.
What we’ll cover:
The Kubernetes DNS service (kube-dns) is a built-in Kubernetes Service that exposes a DNS server. By default, when a Service is created, Kubernetes automatically assigns it a DNS name in the format <service-name>.<namespace>.svc.cluster.local. This internal DNS resolution is managed by CoreDNS, which watches the Kubernetes API for new services and updates DNS records accordingly.
DNS (Domain Name System) is the mechanism that converts hostnames like cluster.local to IP addresses, such as 192.168.49.2. It’s a crucial part of internet infrastructure and plays a key role in Kubernetes networking.
Connecting to Pods and Services using DNS names ensures the connection can be made reliably, even after Pods are scaled or Services are recreated. Conversely, using IP addresses is a Kubernetes anti-pattern that can lead to several problems:
- Pods cannot be assigned static IP addresses. Deleting a Pod and recreating it will assign a new random IP address from the available pool, breaking apps that were configured to use the old IP.
- Services do support static IP addresses, but this can make them harder to use in multiple environments. Hardcoding a specific IP in a Service’s manifest creates issues if you then deploy the manifest to another cluster that uses a different IP address range, such as a local development cluster. It also risks IP address conflicts if the cluster already includes another Service that’s assigned the same IP.
- Configuring apps to connect to specific IP addresses makes it harder to check they’re using the correct address. IP addresses can mask frustrating errors and cause developers confusion. It’s not obvious what you’re connecting to or whether the right instance has been selected.
The in-cluster Kubernetes DNS service addresses these problems by letting Pods resolve names like demo-service.demo-namespace.svc.cluster.local to the correct in-cluster IP address. Pods are automatically configured to use the Kubernetes DNS service for their name lookups, enabling you to use these names without changing any settings first.
Benefits of Kubernetes DNS
Kubernetes DNS provides several benefits that enhance the management and scalability of applications within a Kubernetes environment:
- Automatic service discovery: Kubernetes DNS allows services to be discovered automatically, making it easier for applications to locate and communicate with each other.
- Simplified networking: DNS abstracts service addresses, removing the need to hardcode IP addresses, which can change dynamically.
Scalability: As services scale up or down, Kubernetes DNS ensures that all pods and services are properly mapped, keeping network communication seamless. - Health monitoring: DNS can provide health checks for services, ensuring that only healthy services are resolvable.
- Ease of configuration: Kubernetes automatically manages DNS entries for services, reducing configuration complexity and manual errors.
Kubernetes clusters run a DNS server that resolves the name lookup requests made by processes running in Pods. When a DNS request references a Kubernetes Service, the correct Service Cluster IP address is returned automatically. This allows the Pod that made the request to connect to the target Service.
If Kubernetes does not know the requested name, the request will be forwarded up through the networking stack so external DNS servers can attempt a resolution.
Kubernetes uses CoreDNS as its default DNS server. Kubernetes releases older than v1.13 used kube-dns, a wrapper around Dnsmasq. CoreDNS is written as a multithreaded Go application, improving performance and memory use at scale. Only one container is needed per instance (compared with three containers for kube-dns).
Your cluster’s CoreDNS Pods will be visible in the kube-system namespace. Depending on your environment, you may have a single CoreDNS Pod, or multiple Pods distributed across different Nodes, as shown below:
$ kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-c5c6457c-rtd4v 1/1 Running 0 42d 10.244.3.35 do-copt-4vcpu-8gb-gn9zo <none> <none>
coredns-c5c6457c-zqvqz 1/1 Running 0 5d11h 10.244.3.223 do-copt-4vcpu-8gb-gn9zx <none> <none>Running multiple CoreDNS Pods on different Nodes improves redundancy by ensuring your cluster can still resolve DNS lookups if one Node fails. The option may need to be enabled via your Kubernetes provider’s control plane high availability option.
Your cluster will also have a kube-dns Service in the kube-system namespace:
$ kubectl get services -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 76mThe Service is called kube-dns due to the historical use of the kube-dns DNS server. The Service was not renamed when CoreDNS was introduced. This ensures backwards compatibility with tools that depend on the kube-dns name.
The kube-dns Service exposes the CoreDNS Pods running in your cluster. The Kubelet process on your Nodes automatically configures new Pods to use the Service as their default DNS nameserver. You can prove this with a simple Kubectl test:
# Note the ClusterIP address assigned to the kube-dns Service
$ kubectl get service kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}'
10.96.0.10
# Start a simple Pod
$ kubectl run nginx --image nginx:latest
pod/nginx created
# Check the contents of the Pod's /etc/resolv.conf file
$ kubectl exec nginx -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5The resolv.conf file configures the Pod’s DNS lookup process. The nameserver setting specifies the IP address of the cluster’s DNS service, which means that the Pod’s DNS lookups will be resolved by Kubernetes DNS.
The search and ndots lines are also important. They configure the Pod to automatically resolve unqualified names such as demo-pod by first trying demo-pod.default.svc.cluster.local, then demo-pod.svc.cluster.local, then demo-pod.cluster.local.
The resolution will only occur if there are less than five dots (. characters) in the name, so demo-pod will be resolved in this way, but a.b.c.d.e.f will not. Now let’s look at these search paths and their features in more detail.
What is the difference between KubeDNS and CoreDNS?
CoreDNS is a more flexible and extensible DNS server that has replaced KubeDNS as the default DNS service in Kubernetes. It is written in Go and designed with a modular plugin-based architecture, allowing custom DNS behavior and better performance.
KubeDNS, in contrast, is a simpler and older solution built specifically for Kubernetes service discovery. It lacks the flexibility and extensibility of CoreDNS and has been deprecated in favor of the latter.
CoreDNS supports a broader range of DNS features, has a lighter memory footprint, and integrates more cleanly with modern Kubernetes environments—making it the preferred choice today.
The Kubernetes DNS system enables seamless service discovery for the workloads running in your cluster. Kubernetes assigns each Pod and Service a predictable DNS name that can be resolved using the kube-dns Service seen above.
All Kubernetes DNS names are subdomains of your cluster’s domain name. This is usually cluster.local unless you customized it when you started your cluster.
We’ll show cluster.local as the cluster domain in the following examples, but you should replace this value with your own cluster domain if you’ve made changes.
Kubernetes Service discovery
Kubernetes Services are automatically assigned a DNS name in the format <service-name>.<namespace-name>.svc.cluster.local. For example, if you create a service called mongodb in the demo-app namespace, then your application’s Pods can connect to the service using mongodb.demo-app.svc.cluster.local. This name will resolve to the Cluster IP address assigned to the Service.
Headless Services behave a little differently. They’re still assigned a DNS name in the same format, but DNS lookups will resolve to the set of IP addresses assigned to the Pods selected by the Service. Clients need to implement their own selection process to decide which of the Pods to connect to.
Connecting to Services within the same Kubernetes namespace
Pods can connect to Services in the same namespace using unqualified names. For example, if a Pod and a Service called mongodb share a namespace, then the Pod can connect to the Service using the short name mongodb. The domain search paths that we saw in /etc/resolv.conf above will automatically resolve the unqualified name to the Service’s full DNS name.
Connecting to Services in a different namespace
The target namespace must be specified when a Pod needs to connect to a Service in a different namespace. You can still use short names via the <service>.<namespace> syntax. For instance, a Pod in the frontend namespace could connect to mongodb.backend to use the MongoDB Service in the backend namespace.
Kubernetes DNS records for Pods
It’s not possible to directly assign DNS names to individual Kubernetes Pods. You should use a Service to expose any Pods that other Pods in your cluster need access to. You can then use the Service’s DNS name to connect to the Pods reliably.
If you don’t want to expose a Service in your cluster, you can create a Headless Service instead. Headless Services are specifically designed to enable name discovery for Pods without the traffic proxying and load balancing capabilities of standard Services:
apiVersion: v1
kind: Deployment
metadata:
name: demo-deployment
namespace: demo-app
spec:
replicas: 3
selector:
matchLabels:
app: demo-app
template:
metadata:
labels:
app: demo-app
spec:
containers:
- name: nginx
image: nginx:latest
---
apiVersion: v1
kind: Service
metadata:
name: headless-service
namespace: demo-app
spec:
# Specifies the service is headless
clusterIP: None
selector:
app: demo-appYou can then perform a DNS lookup for the service’s name (headless-service.demo-app.svc.cluster.local) to discover the individual IP addresses of the Pods that the Service selects.
$ host headless-service.demo-app.svc.cluster.local
headless-service.demo-app.svc.cluster.local has address 10.244.0.4
headless-service.demo-app.svc.cluster.local has address 10.244.0.5
headless-service.demo-app.svc.cluster.local has address 10.244.0.6Note that the DNS query returns the IP addresses of the Pods behind the Services, not the IP address of the Service itself. Headless services don’t have their own IP addresses as they do not handle any traffic.
The default Kubernetes DNS behavior is appropriate for most Pods. However, you may sometimes need to customize how a specific Pod resolves its DNS lookups. The dnsPolicy and dnsConfig Pod spec fields let you control what happens when a container in the Pod makes a DNS lookup.
1. Using dnsPolicy
The dnsPolicy field affects how the Pod interacts with the default Kubernetes DNS configuration. It has four possible values:
Default— The standard behavior is used, based on the DNS configuration applied to the Node that the Pod is scheduled to.ClusterFirst— When a name can’t be internally resolved to a Kubernetes Service, it will first be forwarded to upstream nameservers set in the cluster’s configuration.ClusterFirstWithHostNet— A variant ofClusterFirstthat also considers the DNS configuration of the Node’s host network for Pods that have thehostNetworkfield defined.None— The Kubernetes DNS system will not apply to the Pod. Alternative DNS configuration must be specified using thednsConfigfield (see below).
These options provide basic control over how the Pod’s DNS requests mesh with the built-in Kubernetes features.
2. Using dnsConfig
The dnsConfig Pod setting lets you manually specify extra DNS servers for the Pod. Those servers will be consulted when the Pod makes a DNS lookup request. You can also add additional search domains to change how unqualified names are handled.
The field supports three different properties:
nameservers— A list of DNS server IP addresses that will be added to the Pod’s DNS configuration. E.g.192.168.0.1searches— A list of DNS search domains to use during unqualified name lookups. E.g.public-services.example.comoptions— A list of objects withname/valueproperties that will be passed as options to the Pod’sresolv.confDNS configuration file.
Your settings will be merged with the defaults that Kubernetes provides, unless you also set dnsPolicy: None.
Example: Setting Kubernetes Pod DNS Settings With dnsPolicy and dnsConfig
Here’s an example manifest for a Pod that opts out of using the default Kubernetes DNS resolution behavior. It applies its own nameservers and search domains instead:
apiVersion: v1
kind: Pod
metadata:
name: demo-pod
spec:
containers:
- name: demo-app
image: demo-app:latest
dnsPolicy: None
dnsConfig:
nameservers:
- 192.168.0.1
- 192.168.0.2
searches:
- public-services.example.comThis Pod won’t be able to resolve names using *.cluster.local because the dnsPolicy is set to None. All DNS requests will be submitted first to the nameserver at 192.168.0.1, then 192.168.0.2 if resolution fails. Unqualified names like demo-service will be automatically transformed to demo-service.public-services.example.com before the DNS lookup occurs.
Kubernetes DNS is usually simple and reliable, but it’s still possible you’ll encounter problems with name resolution or Pod configuration. Here are a few quick troubleshooting tips to help you out with common Kubernetes DNS issues.
1. Check that CoreDNS is running
Kubernetes DNS resolution will fail if the kube-dns service is unavailable or your CoreDNS Pods enter an unhealthy state. Use Kubectl to check that the Service exists and Pods is running:
$ kubectl get service kube-dns -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 5h
$ kubectl get pods -l k8s-app=kube-dns -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5dd5756b68-snkb2 1/1 Running 1 (4h10m ago) 5hIf there’s no Service or the Pods aren’t shown as Running, then the Kubernetes DNS service has failed and DNS resolution won’t be working. You’re probably facing a control plane outage, so you may need to use your Kubernetes distribution management tools to attempt a restart.
2. Browse the CoreDNS Pod logs for errors
DNS resolution errors may be explained by log messages written by the CoreDNS Pods. You can access the logs using Kubectl:
$ kubectl logs -l k8s-app=kube-dns -n kube-systemA healthy log should show only INFO messages. Look for warnings and errors that could provide clues to what’s gone wrong in your configuration.
3. Check the CoreDNS ClusterRole is correctly permissioned
CoreDNS must be assigned an RBAC ClusterRole that grants it permission to monitor changes to Pods, Services, Endpoints, and Namespaces in your cluster. If the role is misconfigured, the DNS server won’t be able to detect new Pods and Services, so DNS names will stop registering.
Use the following Kubectl command to check the policies included in the system:coredns ClusterRole:
$ kubectl describe clusterrole system:coredns -n kube-system
Name: system:coredns
Labels: <none>
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
endpoints [] [] [list watch]
namespaces [] [] [list watch]
pods [] [] [list watch]
services [] [] [list watch]
endpointslices.discovery.k8s.io [] [] [list watch]If your role doesn’t include the same policies, then the missing permissions could be the cause of your DNS problems. You should edit the ClusterRole to attach the list and watch policies for each monitored object.
3. Use a test container to check the DNS Service is working
Starting a test container lets you quickly check whether the Kubernetes DNS service is responding to queries correctly.
Use Kubectl to interactively start a Pod with the busybox image:
$ kubectl run -it busybox --image busybox:latest
/ #Busybox is a lightweight image that includes the nslookup DNS resolution tool. Try using nslookup to query the kubernetes.default hostname. It should successfully resolve to the Cluster IP address of the kube-dns Service.
/ # nslookup kubernetes.default
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: kubernetes.default
Address: 10.96.0.1If you see an error message (such as server can't find <name>), then this signals a problem with your DNS configuration. Check the kube-dns Service is running and ensure you’ve not customized the networking configuration or Kubernetes DNS settings on the Node that runs the Pod.
4. Debug Pod issues by checking the resolv.conf file
Finally, checking a Pod’s resolv.conf file is an easy way to spot accidental misconfigurations caused by incorrect dnsConfig entries.
The kubectl exec command lets you connect to a named Pod and use cat to read the contents of /etc/resolv.conf:
$ kubectl exec -it demo-pod -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5This allows you to inspect the DNS configuration that the Pod will use for name lookups. If you see any unexpected nameservers or search domains, you should inspect the Pod’s spec for wrong dnsConfig values. They could interfere with the name resolution process.
If you need assistance managing your Kubernetes projects, look at Spacelift. It brings with it a GitOps flow, so your Kubernetes Deployments are synced with your Kubernetes Stacks, and pull requests show you a preview of what they’re planning to change.
You can also use Spacelift to mix and match Terraform, Pulumi, AWS CloudFormation, and Kubernetes Stacks and have them talk to one another.
To take this one step further, you could add custom policies to reinforce the security and reliability of your configurations and deployments. Spacelift provides different types of policies and workflows that are easily customizable to fit every use case. For instance, you could add plan policies to restrict or warn about security or compliance violations or approval policies to add an approval step during deployments.
You can try Spacelift for free by creating a trial account or booking a demo with one of our engineers.
Kubernetes DNS allows your Pods to communicate with Services using predictable domain names instead of numerical IP addresses. Kubernetes automatically assigns each Service a DNS name and configures Pods to query the in-cluster DNS server. This allows you to build more reliable applications that are easier to operate at scale.
DNS is a key component of the Kubernetes Service architecture. Check out our detailed guides to using Services and the broader Kubernetes networking model to learn more about exposing Pods within your cluster and externally.
Don’t forget to give Spacelift a try to automate Kubernetes provisioning and governance at scale, alongside your other IaC resources.
Manage Kubernetes easier and faster
Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.
