Monitoring Servers and Infrastructure with Prometheus
In today’s dynamic and ever-evolving IT landscape, effective monitoring is crucial for ensuring the reliability and performance of servers and infrastructure. Prometheus, an open-source systems monitoring and alerting toolkit originally developed at SoundCloud, has emerged as a popular choice for this purpose. This blog delves into how Prometheus works, its key features, and best practices for monitoring servers and infrastructure.
What is Prometheus?
Prometheus is a powerful open-source monitoring solution that collects and stores metrics as time series data, i.e., metrics information is stored with the timestamp at which it was recorded, along with optional key-value pairs called labels. Prometheus was built with a focus on reliability and scalability, making it an ideal choice for monitoring complex cloud-native environments.
Key Features of Prometheus
- Multidimensional Data Model: Prometheus's data model is based on time series identified by metric names and key/value pairs, allowing for a high degree of flexibility.
- Powerful Query Language: PromQL, Prometheus's query language, enables complex queries to aggregate and analyze metrics data.
- Pull-Based Architecture: Prometheus collects metrics by scraping HTTP endpoints on monitored targets.
- Service Discovery: Prometheus can discover targets dynamically, making it well-suited for dynamic cloud environments.
- Alerting: Prometheus has a built-in Alertmanager for handling alerts, including deduplication, grouping, and routing.
Prometheus Architecture
Prometheus's architecture is designed for reliability and scalability. The core components include:
- Prometheus Server: Responsible for scraping and storing data.
- Client Libraries: Libraries for instrumenting application code.For example if you have your backend code in node, you will write client code , i.e. add a metric endpoint which will enable prometheus to scrape data from your backend server (in this case the NodeJS server)
- Pushgateway: For ephemeral or batch jobs to push metrics to Prometheus.
- Alertmanager: Handles alerts generated by Prometheus.
- Exporters: Bridge third-party data into Prometheus metrics format.
Although prometheus displays all its metrics and data via graphs through its prometheus web ui which is fairly good ,many developers prefer to integrate the data with grafana which is a better ui library , it dispalys the metrics in a better manner when compared to prometheus web ui .
![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F36b9d781-f19d-4955-b176-0da879e134b8_1351x811.png)
Setting Up Prometheus
1. Installation
Prometheus can be installed on various platforms. Here, we demonstrate installation on a Linux server.
wget https://github.com/prometheus/prometheus/releases/download/v2.29.1/prometheus-2.29.1.linux-amd64.tar.gz
tar xvfz prometheus-2.29.1.linux-amd64.tar.gz
sudo mv prometheus-2.29.1.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.29.1.linux-amd64/promtool /usr/local/bin/
2. Configuration
Prometheus is configured via a YAML file. Here’s a basic example configuration (`prometheus.yml`):
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node_exporter'
static_configs:
- targets: ['localhost:9100']
3. Running Prometheus
Start Prometheus with the following command:
prometheus --config.file=prometheus.yml
Prometheus will now be running on `http://localhost:9090`.
Monitoring with Exporters
Node Exporter
Node Exporter is a popular exporter for hardware and OS metrics exposed by *NIX kernels.
Installation
# Download Node Exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
# Extract the package
tar xvfz node_exporter-1.2.2.linux-amd64.tar.gz
# Move Node Exporter binary to /usr/local/bin
sudo mv node_exporter-1.2.2.linux-amd64/node_exporter /usr/local/bin/
Running Node Exporter
Start Node Exporter with:
node_exporter
Node Exporter will run on `http://localhost:9100/metrics`.
Configuring Prometheus to Scrape Node Exporter
Add Node Exporter to your `prometheus.yml` configuration:
scrape_configs:
- job_name: 'node_exporter'
static_configs:
- targets: ['localhost:9100']
In case you want to monitor your database you run an docker container on your database which will pull monitoring metrics from your database , here is an example using mongodb as an database .
docker run -d -p 9216:9216 -p 17001:17001 percona/mongodb_exporter:0.40 --mongodb.uri=mongodb://127.0.0.1:17001
Reload the Prometheus configuration to apply the changes.
Promql queries will be discussed in the next blog .
Best Practices for Using Prometheus
1. High Availability: Deploy Prometheus in a highly available configuration to avoid single points of failure.
2. Efficient Querying: Optimize PromQL queries to prevent performance issues.
3. Retention Policies: Configure appropriate retention policies to manage storage use.
4. Security: Secure Prometheus endpoints and use TLS for communication.
5. Dashboards: Use Grafana for creating rich, visual dashboards with Prometheus as the data source.
Conclusion
Prometheus offers a robust, flexible, and scalable solution for monitoring servers and infrastructure. With its powerful querying capabilities, dynamic service discovery, and integrated alerting, Prometheus stands out as a preferred choice for modern IT environments. By following best practices and leveraging its ecosystem, you can achieve comprehensive and reliable monitoring tailored to your infrastructure’s needs.