How to Set Up Prometheus Monitoring on Kubernetes [Tutorial]

Kubernetes is the most popular orchestrator for running containerized workloads in production. It gives you a complete set of tools for deploying, scaling, and administering your containers.

Kubernetes alone isn’t enough to successfully operate apps, though. You also need visibility into cluster utilization, performance, and any errors that occur. Prometheus is an open-source monitoring system that collects metrics in a time series database, allowing you to answer these questions.

In this article, you’ll learn how to set up and use Prometheus with your Kubernetes cluster. We’ll cover the basics of installing Prometheus, querying data, setting up visual dashboards, and managing alerting rules. 

You’ll need kubectl, Helm 3.14 or later, and a Kubernetes cluster before you begin. The commands below also work on Helm 4.

We will cover:

1. What is Prometheus?
2. Why use Prometheus for Kubernetes monitoring?
3. What is kube-prometheus-stack?
4. How to set up Prometheus monitoring on a Kubernetes cluster
5. Kubernetes Prometheus best practices

Prometheus is an open-source monitoring and alerting toolkit and a graduated CNCF project. It stores metrics in a purpose-built time series database tuned for fast writes and range queries.

Prometheus collects metrics by scraping. It pulls data from application endpoints and servers at regular intervals rather than waiting for them to push. Prometheus 3 also accepts OpenTelemetry (OTLP) ingestion and remote write, but scraping remains the default model and the one this tutorial uses.

There are many reasons why you would use Prometheus for Kubernetes monitoring:

• Built-in support for K8s service discovery – automatically discover and monitor new services and pods as they are deployed and scaled up within the K8s cluster
• Rich data model – granular categorization and querying of metrics based on various attributes such as pod labels, namespace, service name, etc
• Integration with visualization tools – ships with Grafana dashboards preconfigured against the Prometheus data source, so you can build custom Kubernetes dashboards without wiring up the connection yourself
• Scalability and performance – Prometheus handles large volumes of data with minimal resource overhead, making it ideal for Kubernetes
• Proven reliability – a graduated CNCF project, and the de facto metrics backend for Kubernetes, with service discovery that keeps up as Pods are rescheduled
• Community support – Prometheus, as well as Kubernetes, are part of CNCF, which has a large community and powerful documentation and tutorials
• Open-source – Prometheus is open-source, making it a flexible choice for monitoring Kubernetes

See also: Prometheus with Helm Chart on Kubernetes Setup

The kube-prometheus-stack Helm chart is the simplest way to bring up a complete Prometheus stack inside your Kubernetes cluster. It bundles several different components in one automated deployment:

• Prometheus – Prometheus is the time series database that scrapes, stores, and exposes the metrics from your Kubernetes environment and its applications.
• Node-Exporter – Prometheus works by scraping data from a variety of configurable sources called exporters. Node-Exporter is an exporter which collects resource utilization data from the Nodes in your Kubernetes cluster. The kube-prometheus-stack chart automatically deploys this exporter and configures your Prometheus instance to scrape it.
• Kube-State-Metrics – Kube-State-Metrics is another exporter that supplies data to Prometheus. It exposes information about the API objects in your Kubernetes cluster, such as Pods and containers.
• Grafana – Although you can directly query Prometheus, this is often tedious and repetitive. Grafana is an observability platform that works with several data sources, including Prometheus databases. You can use it to create dashboards that surface your Prometheus data.
• Alertmanager – Alertmanager is a standalone Prometheus component that provides notifications when metrics change. You can use it to get an email when CPU utilization spikes or a Slack notification if a Pod is evicted, for example.

Deploying, configuring, and maintaining all these components individually can be burdensome for administrators. Kube-Prometheus-Stack provides an automated solution that performs all the hard work for you.

Let’s see how to set up and use Prometheus with your Kubernetes cluster in practice.

1. Install kube-prometheus-stack with Helm

First, register the chart’s repository in your Helm client:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

Next, update your repository lists to discover the chart:

$ helm repo update

Now you can run the following command to deploy the chart into a new namespace in your cluster:

$ helm install kube-prometheus-stack \
  --create-namespace \
  --namespace kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack
NAME: kube-prometheus-stack
LAST DEPLOYED: Tue Jan  3 14:26:18 2023
NAMESPACE: kube-prometheus-stack
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace kube-prometheus-stack get pods -l "release=kube-prometheus-stack"

It can take a couple of minutes for the chart’s components to start. Run the following command to check how they’re progressing:

$ kubectl -n kube-prometheus-stack get pods
NAME                                                       READY   STATUS    RESTARTS      AGE
alertmanager-kube-prometheus-stack-alertmanager-0          2/2     Running   1 (66s ago)   83s
kube-prometheus-stack-grafana-5cd658f9b4-cln2c             3/3     Running   0             99s
kube-prometheus-stack-kube-state-metrics-b64cf5876-52j8l   1/1     Running   0             99s
kube-prometheus-stack-operator-754ff78899-669k6            1/1     Running   0             99s
kube-prometheus-stack-prometheus-node-exporter-vdgrg       1/1     Running   0             99s
prometheus-kube-prometheus-stack-prometheus-0              2/2     Running   0             83s

Once all the Pods show as Running, your monitoring stack is ready to use. The data exposed by the exporters will be automatically scraped by Prometheus.

Now you can start querying your metrics.

2. Query Kubernetes metrics with PromQL

Prometheus includes a web UI that you can use to query your data. This is not exposed automatically. You can access it by using Kubectl port forwarding to redirect local traffic to the service in your cluster:

$ kubectl port-forward -n kube-prometheus-stack svc/kube-prometheus-stack-prometheus 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

This command redirects traffic to localhost:9090 to the Prometheus service. Visiting this URL in your web browser will reveal the Prometheus UI:

The expression input at the top of the screen is where you enter your queries as PromQL expressions. Start typing to reveal autocomplete suggestions for the available metrics, then press Execute or hit Enter.

Try selecting the node_memory_Active_bytes metric, which surfaces the memory consumption of each of the Nodes in your cluster. Press the “Execute” button to run your query. The results will be displayed in a table that provides the query’s raw output:

Most metrics are easier to interpret as graphs.

Switch to the “Graph” tab at the top of the screen to see a visualization of the metric over time. You can use the controls above the graph to change the time period that’s displayed.

PromQL queries allow detailed interrogation of your data. Manually running individual queries in the Prometheus UI is an inefficient form of monitoring, however.

Next, let’s use Grafana to visualize metrics conveniently on live dashboards.

3. Access Grafana and get the admin password

Start a new Kubectl port forwarding session to access the Grafana UI. Use port 80 as the target because this is what the Grafana service binds to.

You can map it to a different local port, such as 8080, in this example:

$ kubectl port-forward -n kube-prometheus-stack svc/kube-prometheus-stack-grafana 8080:80
Forwarding from 127.0.0.1:8080 -> 3000
Forwarding from [::1]:8080 -> 3000

Next, visit http://localhost:8080 in your browser to reach the Grafana login page. The user is admin.

Chart version 79.0.0 removed the fixed default password. Quoting the chart’s own UPGRADE.md: “This version removes the default password for the grafana administrator, that in previous versions was set as prom-operator. For users that do not explicitly set an admin password, the value will now be generated randomly.”

So read the generated password out of the Kubernetes Secret:

$ kubectl get secret -n kube-prometheus-stack kube-prometheus-stack-grafana \
  -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The chart generates a 40-character random string and stores it under the admin-password key, alongside admin-user. It looks up the existing Secret on each upgrade, so the password survives helm upgrade rather than being regenerated.

On charts older than 79.0.0 the password is prom-operator. To set your own instead, pass --set grafana.adminPassword=<value> at install time, or point grafana.admin.existingSecret at a Secret you manage yourself.

After you’ve logged in, you’ll initially reach the Grafana welcome screen:

Use the sidebar to switch to the Dashboards screen. This is where all your saved dashboards can be found, including the prebuilt ones that come with Kube-Prometheus-Stack deployments.

4. Explore the prebuilt Kubernetes dashboards

There are several included dashboards that contain the metrics scraped from Node-Exporter, Kube-State-Metrics, and various Kubernetes and Prometheus components. Here are a few notable ones:

Monitoring cluster utilization with “Kubernetes / Compute Resources / Cluster”

This dashboard provides an overview of the resource utilization for your entire cluster. Headline statistics are displayed at the top, with more detailed information presented in panels below.

Viewing a node’s resource consumption with “Node Exporter / Nodes”

Data collected by Node-Exporter is provided by this dashboard. It shows detailed resource utilization information on a per-Node basis. You can change the selected Node using the “instance” dropdown at the top of the dashboard.

Viewing the resource consumption of individual pods with “Kubernetes / Compute Resources / Pod”

This dashboard shows the resource requests, limits, quotas, and utilization for individual Pods. You can select the namespace and Pod to view from the dropdowns at the top of the screen.

The time frame can be customized on all Grafana dashboards using the controls in the top-right corner of the screen. You can refresh the data or change the auto-refresh interval with the button next to the time frame selector.

5. Scrape your own application with a ServiceMonitor

The prebuilt dashboards cover the cluster itself. To monitor your own workloads, create a ServiceMonitor pointing at your Service.

One detail traps almost everyone. The chart sets serviceMonitorSelectorNilUsesHelmValues: true, which makes Prometheus select only ServiceMonitors labeled with the Helm release name. A ServiceMonitor without that label is ignored silently, with no error in any log:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  labels:
    release: kube-prometheus-stack   # required, or Prometheus ignores this
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics

The label value is your Helm release name, which is kube-prometheus-stack if you followed step 1. To drop the requirement entirely, set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues: false. Namespace is not a constraint: the chart leaves serviceMonitorNamespaceSelector empty, so Prometheus looks across all namespaces.

Use a PodMonitor for Pods with no Service in front of them, and a ScrapeConfig for targets outside the cluster.

6. Send Kubernetes alerts to Slack or a webhook with Alertmanager

Monitoring must be automated to be effective. You need to receive alerts when important metric stops meeting expectations, such as when a spike in memory consumption occurs. Otherwise, you have to continually check your dashboards or run queries to determine whether you need to take action.

Alertmanager is a separate service in the Prometheus project that receives alerts from Prometheus and delivers the notifications. kube-prometheus-stack deploys it for you.

Kube-Prometheus-Stack’s bundled Alertmanager is configured by merging in custom chart values when you deploy the stack with Helm. First, prepare a YAML file that nests your Alertmanager settings under the top-level alertmanager key. Here’s an example that sends all alerts to a webhook URL:

alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      receiver: demo-webhook
      group_wait: 5s
      group_interval: 10s
      repeat_interval: 1h
    receivers:
      - name: "null"
      - name: demo-webhook
        webhook_configs:
          - url: http://example.com/webhook
            send_resolved: true

The route section specifies that alerts should be directed to the demo-webhook receiver. This is configured to send a POST request to http://example.com/webhook each time an alert is triggered or resolved. The request’s payload is described in the Alertmanager documentation.

The extra "null" receiver is not a workaround for a bug. The chart’s default route includes a child route that sends the Watchdog alert to a receiver named null:

route:
  receiver: 'null'
  routes:
    - receiver: 'null'
      matchers:
        - alertname = "Watchdog"
receivers:
  - name: 'null'

Helm merges maps but replaces lists. Your route override merges with the chart’s, so that Watchdog child route survives and still points at null. Your receivers list, being a list, replaces the chart’s entirely, so null no longer exists and Alertmanager refuses to load the config with undefined receiver "null" used in route. Keep the null receiver, or override route.routes yourself.

Save your YAML file to alertmanager-config.yaml in your working directory. Next run the following command to redeploy the Prometheus stack and apply your Alertmanager settings:

$ helm upgrade kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  -n kube-prometheus-stack \
  --reset-then-reuse-values \
  -f alertmanager-config.yaml

--reset-then-reuse-values needs Helm 3.14 or later. Reach for it instead of --reuse-values, which the Helm documentation describes as reusing the last release’s values without picking up values that are new in the chart you are upgrading to. 

Because kube-prometheus-stack adds values on almost every release, –reuse-values is a common cause of failed upgrades on this chart. –reset-then-reuse-values resets to the chart’s built-in values, reapplies your previous release values, then merges your -f overrides on top.

Don’t worry – you won’t lose any of your existing data. The command performs an in-place upgrade of your deployment.

It could take a few minutes for Alertmanager to reload its configuration after the deployment completes. You’ll then begin to receive requests to your webhook URL, as alerts are triggered.

To send a test alert, first start a port forwarding session to your Alertmanager instance:

$ kubectl port-forward -n kube-prometheus-stack svc/kube-prometheus-stack-alertmanager 9093:9093

Next run the following command to simulate triggering a basic alert from a Kubernetes service in a specific namespace:

$ curl -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"alert-demo","namespace":"demo","service":"demo"}}]' \
  http://127.0.0.1:9093/api/v2/alerts

Use api/v2. The Alertmanager v1 API was deprecated in 0.16.0 and removed in 0.27.0; the v1 endpoints now return 410 Gone with a deprecation message. The chart currently ships Alertmanager 0.33.1, so a request to api/v1/alerts always fails. 

Prometheus 3 dropped support for talking to Alertmanager over v1 as well, so api_version: v1 is no longer valid in the alerting config. The chart already defaults alertmanager.apiVersion to v2.

After a few moments, you should receive a request to your webhook URL. The request’s body will describe the alert’s details.

You should consider the following when using Prometheus with Kubernetes:

1. Use the Prometheus operator for K8s
2. Configure service monitors
3. Leverage K8s labels and annotations
4. Take advantage of persistent storage for Prometheus
5. Setup Alertmanager
6. Monitor Prometheus performance
7. Secure your Prometheus instance
8. Do regular updates

1. Use the Prometheus operator for K8s

Deploying Prometheus using the Prometheus operator, will help with managing Prometheus instances and their configurations automatically.

This will allow you to define also your monitoring requirements declaratively using K8s CRDs.

2. Configure ServiceMonitors, PodMonitors, and ScrapeConfigs

A ServiceMonitor tells Prometheus which Kubernetes Services to scrape, selected by label. Use a PodMonitor for Pods with no Service, and a ScrapeConfig for targets outside the cluster. All three are Prometheus Operator custom resources, so you declare scrape targets as Kubernetes objects instead of editing Prometheus configuration files. Remember the release label covered above, or Prometheus will ignore what you create.

3. Use Kubernetes labels and annotations consistently

Labels are how ServiceMonitors find their targets, so a consistent labeling scheme across your workloads is what makes selector-based discovery work. Settle on a convention early. Retrofitting labels across a running cluster means editing every Service and every ServiceMonitor that selects it.

4. Take advantage of persistent storage for Prometheus

The chart ships storageSpec: {}, which means a default install has no persistence at all. Prometheus writes to an emptyDir and you lose every metric the moment the Pod restarts. Retention also defaults to 10d, which is shorter than most teams assume.

Set both explicitly:

prometheus:
  prometheusSpec:
    retention: 30d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

5. Setup Alertmanager routing

Alertmanager deduplicates, groups, and routes the alerts Prometheus sends it. Configure routing so alerts reach the right destination: email, Slack, PagerDuty, or a webhook, matched on severity or any other label you attach.

6. Monitor Prometheus performance

Even if Prometheus is used for monitoring aspects related to your K8s cluster in this context, it is crucial to monitor its performance as well, to ensure it doesn’t become a bottleneck.

Watch metric cardinality in particular. A single label with unbounded values, a user ID or a request path, multiplies your series count and is the usual cause of Prometheus running out of memory. Adjust CPU and memory requests as your target count grows.

7. Secure your Prometheus instance

Prometheus is no exception to the rest of your infrastructure. Use Kubernetes RBAC to control who can reach the monitoring stack, and put TLS in front of the Prometheus and Grafana endpoints.

Note that the kubectl port-forward commands in this tutorial are for local exploration only. In production, expose Grafana through an Ingress with authentication in front of it and leave the Prometheus and Alertmanager UIs unexposed.

8. Upgrade the chart and its CRDs deliberately

Helm does not upgrade CRDs. A helm upgrade across a chart major version updates every workload but leaves the Prometheus Operator custom resource definitions at their old schema, which surfaces later as fields that silently do nothing.

The chart’s UPGRADE.md lists the exact kubectl apply --server-side commands for each version. For chart 88.x, which ships Prometheus Operator v0.93.0, apply the CRDs before upgrading:

$ kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.93.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml

Repeat for the other CRDs in that directory: alertmanagers, alertmanagerconfigs, podmonitors, probes, prometheusagents, prometheusrules, scrapeconfigs, servicemonitors, and thanosrulers.

Since chart 68.4.0 there is also a crds.upgradeJob.enabled option that applies the CRDs through Helm hooks before the operator deploys. It is off by default and the chart still marks it as preview, so test it before relying on it.

Good observability is essential for Kubernetes clusters running production workloads. You need to understand resource utilization, see where Pods are being scheduled, and track the errors and logs emitted by your applications.

Kube-Prometheus-Stack is a convenient route to setting up monitoring for your cluster. It configures Prometheus, Grafana, Alertmanager, and vital metrics exporters for you, reducing maintenance overheads. The basic installation comes with useful prebuilt dashboards that you can extend with custom queries and metrics scraped from your own applications. Instrumenting a system for Prometheus is a complex topic, but you can get started by exploring the official client libraries for exporting metrics from your code.

Once your Kubernetes metrics land in Grafana, your infrastructure automation is the next obvious gap on the same dashboard. Spacelift exposes a Prometheus exporter, so run counts, failures, and queue times from your Spacelift account sit next to your cluster metrics in the same Grafana instance. Our tutorial on monitoring your Spacelift account via Prometheus walks through it.