Configuring Prometheus with Helm Chart on Kubernetes

In this blog post, we will install Prometheus on K8s using Helm charts. We will also customize the Prometheus configuration via Helm charts to connect various node exporter instances to expose their metrics on Prometheus.

1. What is Helm?
2. What is Prometheus?
3. What is a node exporter?
4. How to set up Prometheus on K8s using Helm charts

Various components that make an application run on Kubernetes are defined in the declarative format in configuration files. Depending on the complexity of applications being deployed, the number of configuration files/manifests may pose some management difficulties.

Helm is a package manager for Kubernetes that simplifies application deployment and management by providing a higher-level abstraction for manifest files. It streamlines deployment by packaging the Kubernetes manifests into a single Helm chart, which can be installed and uninstalled consistently any number of times.

The Helm architecture is based on three big concepts: chart, repository, and release. A Helm chart is a compilation of Kubernetes manifest files, a repository is a place where all the charts are versioned and maintained, and an instance is a single installation of a chart. In this blog post, we will use the community-developed Helm chart for Prometheus.

See also Kustomize vs Helm comparison.

Prometheus is an open-source monitoring and alerting tool designed for monitoring of applications orchestrated by platforms like Kubernetes. Its architecture is based on a time-series database and a robust alerting system that enables querying and visualization using tools like Grafana.

The diagram below represents Prometheus components.

Source

The primary component of Prometheus is the server, which is responsible for collecting and storing time-series data. The server periodically scrapes metrics from targets such as applications, services, or infrastructure components and stores them in a time-series database. 

The Prometheus Query Language (PromQL) enables us to retrieve and manipulate this data for analysis and visualization. The data model is based on key-value pairs and timestamps. It excels at providing real-time insights into system performance, resource utilization, and app health. 

Then, Prometheus’s alerting system allows us to define rules based on metrics and trigger alerts when specific conditions are met to proactively address the issues.

A node exporter is an exporter designed to export system metrics to Prometheus’s Push Gateway. Exporters are agents that expose metrics from various systems in a format Prometheus can understand. They act as connectors between Prometheus and different system components.

Additionally, Grafana — a popular open-source dashboard and visualization platform — is often used in conjunction with Prometheus to create visually appealing and insightful representations of the monitored data.

There are three ways to install the node exporters and connect Prometheus to collect metrics from them:

1. On the K8s as a pod
2. As an independent Docker container
3. As non-containerized instance

The diagram below shows the final result of the steps we will follow in this article:

Step 1: Install Prometheus using Helm chart

Download the appropriate files based on the OS and processor architecture and run the executable to install the Prometheus locally or on a VM instance. 

The download comes with a sample configuration file named “prometheus.yml” to further configure the targets to listen to. Use the same to define targets, rules, and other configuration options, and run Prometheus again. These steps are usually automated by using Docker or by implementing configuration management tools.

There are a couple of main ways to install Prometheus in a Kubernetes environment. We can do it manually or by using Helm charts. 

As we have seen in previous sections, Prometheus installation depends on multiple components, such as the alert manager and push gateway. To achieve this manually, we would need to create Kubernetes configuration files for deployment, services, config maps, etc.

Kubernetes environments typically support microservice-based architecture, which involves deploying multiple instances of software components that support various services. Managing configuration files for all of these can get tedious. This is where Helm charts are used. Like Prometheus, there could be other applications that are not the services themselves, but they do perform important secondary operations.

Using Helm charts is a better way to deploy Prometheus on Kubernetes. 

To install Prometheus using Helm, we first add and update the prometheus-community Helm repository by running the commands below:

The steps above add the repository and the charts for Prometheus deployment. 

Install the charts by providing a release name as shown below:

The steps above add the repository and the charts for Prometheus deployment. 

Install the charts by providing a release name as shown below:

In the example included in this blog post, we will call this release “prometheus”. 

The updated command looks like this:

The output below indicates that the Helm chart for Prometheus is successfully installed.

To view all the Kubernetes components installed, run kubectl get all command. It should present you with an output similar to below.

As we can see, the Helm chart has deployed multiple pods and services for Prometheus to work.

Additionally, we can verify which Helm release is currently deployed.

Step 2: Analyze the Prometheus dashboard

Note that all the services are assigned a ClusterIP with the port number. To access the Prometheus dashboard, we are interested in the “prometheus-server” service. However, we cannot access it since it is not forwarded on localhost. 

To access the Prometheus dashboard, start by running the command below. This is also indicated in the output of the “helm install” command.

Read also: How to Use Kubectl Port-forward in Kubernetes Applications

The output:

Now, open http://localhost:9090/graph in the browser. It should show the Prometheus dashboard, as shown below.

Navigate to Status > Targets, and we should see below targets registered:

When Prometheus is installed using the prometheus-community Helm chart, it deploys Prometheus components and registers the same as targets.

As we can see from the screenshot above, it is monitoring K8s API server, Nodes, Push Gateway, and Service Endpoints. Changing the configuration file can modify this target registration, but we will do the same in the next step.

For now, we are interested in the “prometheus” target, which is currently monitoring the URL endpoint where we have the Prometheus dashboard running. The state is marked as “UP” and it also provides a few labels using which we can query the logs related to this.

To double-check, open “http://localhost:9090/metrics” in a different browser. It will show all the events created by the service, as seen below:

Navigate to the homepage and click on the “Table” tab. This view provides a search-type field where we are expected to input a PromQL expression that helps query the logs stored in time series format.

Prometheus captures many attributes in a log entry. This also depends on the type of “exporter” and the service exposing these logs. There is no definite answer to what attribute expressions are available for query, but if we start typing into the expression box, it should provide us with the list as seen below.

Let’s assume that we are interested in knowing more about the HTTP requests. 

Select the “prometheus_http_requests_total” attribute for the query and click on execute. It will return all the logs that include this information.

Notice how the searchable attributes are mentioned in the curly braces. We can use it to filter the results. 

For example, to apply a query on “handler” attribute, we can choose to write the query expression as below.

This is a very basic form of PromQL query, but it is possible to perform aggregate operations.

Step 3: Configure Prometheus deployment

In the previous step, we did not make use of the prometheus.yml file to customize or configure the Prometheus deployment on Kubernetes. To configure various Prometheus settings, we can serve the “value.yml” file to Helm while installing the prometheus-community chart. The best way to go forward in this case is to get the default values for the chart from the corresponding repository.

In the root directory of your project, create a file named “values.yml” or “prometheus.yml”. Paste and save the default values copied from the artifact repository. This makes it easy to change/remove/add the configuration. For example, if we don’t want to see all the targets monitored by default configuration, as shown in the second step, we can remove the same.

Note that this configuration YAML file contains a path that describes the contents of prometheus.yml in the same way we would write it otherwise without Helm. However, it also contains values for other Helm configurations.

To change the Prometheus configuration, scroll down to the serverFiles: path in this YAML file. The contents described in the prometheus.yml: path describe the contents of the YAML file fed to Prometheus.

We have a node_exporter running as a separate container outside of Kubernetes. The node_exporter Docker container services are exposed on port 8081. Verify this by checking the metrics from the browser, as seen below.

Let’s assume we now want to monitor the same using Prometheus deployed using Helm. 

Add the following target in this path: serverFiles: prometheus.yml: > scrape_configs: > - job_name: prometheus

The first target ‘localhost:9090’ is the default value provided. This is what was currently being registered as a “prometheus” target as we’ve seen in Step 2. We have added the new target “docker.for.mac.localhost:8081”, to monitor the independently running Docker container. 

Check this document for more information if you are running on a different system.

To put it into action, uninstall the Helm chart:

Install it again. This time, provide the configuration file as shown below.

List the pods:

Forward the server on port 9090:

Open the Prometheus dashboard and make sure the new target is being registered, as shown below.

Thus, we are now able to query the metrics produced by independently running the node exporter Docker container.

Step 4: Collect logs from non-containerized services

Although Prometheus is running in the Kubernetes environment, its Push gateway is also capable of capturing metrics from non-containerized services. In this step, we will introduce a new instance of node exporter, this time running an executable file locally and exporting the metrics on a different port.

The procedure is the same. Update the prometheus.yml configuration by adding a new target as shown below.

Reinstall the Helm chart and forward the port so it is accessible in the browser. This time, when we check the targets on the Prometheus dashboard, we should be able to see the new target, as seen in the screenshot below.

Prometheus can provide critical and accurate insights about the applications and services deployed on K8s. We then installed Prometheus on the K8s environment using the Helm chart. We can also add a custom configuration to the prometheus.yml file via Helm charts to listen to various node exporter instances.

Check out also how to set up Prometheus monitoring for the Kubernetes cluster.

Do you need an even simpler way to manage CI/CD pipelines on Kubernetes? Check out how Spacelift can help you cut down complexity and automate your infrastructure. It’s even got a Prometheus exporter ready to deliver metrics from your Spacelift account to your Grafana dashboards and other tools! Learn more with our tutorial on Monitoring your Spacelift account via Prometheus. 

Get started on your journey by creating a free trial account or booking a demo with our engineering team.