Prometheus On Kubernetes

Kube Prometheus Stack has 8 components by default, and we’ll learn what each one does and how it's useful.

Node-exporter

Node exporter is a Prometheus exporter that provides data about the underlying OS and hardware of your cluster. For example, it exports metrics about CPU usage, memory utilization, disk I/O, and network statistics. However, since it was built before the container era, it is limited to data from Linux hosts and does not extend to container-specific metrics.

Kube-State-Metrics

Kube State Metrics is a Prometheus exporter that generates metrics about the state of Kubernetes objects. It collects data from the Kubernetes API server and converts it into Prometheus metrics, enabling you to monitor the health and performance of your Kubernetes cluster.

Why do you need it? The Kubernetes APIServer was not built to efficiently query the history of Kubernetes objects. But Prometheus was built for this type of historical time-series data. So, using Kube State Metrics, you have valuable data in a usable format.

cAdvisor

The Kubernetes APIServer was not built to store rapidly changing data (e.g. hundreds of writes per second) so data like container CPU usage is not available in a pod’s YAML.

cAdvisor gathers everything about containers running in your cluster. This includes historic resource and network usage of the containers. It natively supports most container types out of the box.

Metrics Server and kubectl top

Metrics Server is a Kubernetes component used to expose the current CPU and memory usage of nodes and pods via Kubernetes API. When you run kubectl top, the data it shows is from Metrics Server. It is primarily used for Horizontal Pod Autoscaling and to work with Kubectl top command.

The Metrics Server only stores current CPU and memory usage of nodes and pods, not historical data. So it cannot expose metrics over time like cAdvisor.

Prometheus

Prometheus collects and stores metrics in a time series. Meaning, it gathers data of all the endpoints it is connected to at every x amount of time. What’s unique about this is that Prometheus pulls this data from all its endpoints, instead of the traditional push model. 

All the stored data can be queried using a language called PromQL. One of the main features of Prometheus is to create rules to provide alerting, historical data analysis, better monitoring etc. Kube Prometheus Stack provides you with 100+ alerting rules by default. Creating new rules or customizing existing ones can be done using the PrometheusRule custom resource.

By exposing the Prometheus application on port 9090 you can view the web UI to run PromQL queries and see all the rules. You can also create graphs, but they are not persistent and efficient compared to tools like a visualization tool like Grafana. 

Prometheus Operator

The Prometheus Operator simplifies the deployment and management of Prometheus in Kubernetes using Kubernetes Custom Resource Definitions(CRD’s). It enables users to create, configure, and manage Prometheus instances and related resources, such as Alertmanager and Grafana, through declarative configuration.

AlertManager

The Prometheus AlertManager helps you with sending alerts, grouping alerts and creating silences. Once an alerting rule fires in Prometheus, the data is sent to Alertmanager. Here based on your configuration an Alert is sent to one or more of your chosen destinations, example Email, Slack, MS Teams.

AlertManager also provides you with a simple UI where you can view details about the firing alerts and your AlertManager configuration. It is hosted on port 9093 of the AlertManager pod.

However these alerts can be difficult to respond to as Prometheus (technically AlertManager) only sends notifications about metrics that meet an alert condition. It doesn't correlate those alerts with the data you need to troubleshoot them like pod logs. This is where an alert enrichment engine like the Robusta open source becomes necessary.

Grafana

Grafana is a data visualization tool, in this case we use Prometheus time series data as our input. It is highly customizable and provides you with many default dashboards with elaborate graphs. Kube Prometheus Stack provides you with many out of the box dashboards using the Mixins Project.

Closing Notes

The Kube Prometheus Stack offers a comprehensive monitoring solution integrating essential components like Prometheus, Prometheus Operator, AlertManager, and Grafana into a Helm chart. You can leverage this chart to setup Prometheus based monitoring on Kubernetes in minutes.

Did you know Robusta installs the Kube Prometheus Stack and also offers a robust alerting mechanism out of the box? Try out Robusta today.