Prometheus Monitoring

Prometheus Monitoring

Prometheus is a time-series metrics database and monitoring system built for cloud-native and microservices environments. It collects metrics by scraping HTTP endpoints (pull model), stores them with labels for multi-dimensional querying, and evaluates alert rules. Paired with Grafana for visualization and Alertmanager for routing alerts, it is the standard monitoring stack for production infrastructure.

Prometheus architecture

Prometheus pull model:
  Prometheus server (every 15 seconds):
    Scrapes: http://server1:9100/metrics  ← Node Exporter
    Scrapes: http://server2:9100/metrics  ← Node Exporter
    Scrapes: http://appserver:8080/metrics ← App metrics endpoint
         ↓
    Stores time-series in TSDB (local)
    Evaluates alert rules
         ↓                     ↓
  Grafana (dashboards)    Alertmanager (routes alerts to Slack/PagerDuty)

Installing Prometheus

# Download Prometheus from GitHub releases:
PROM_VERSION="2.52.0"
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar xvf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
sudo mv prometheus-${PROM_VERSION}.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/promtool /usr/local/bin/

# Create prometheus user and directories:
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

# Create systemd service:
sudo nano /etc/systemd/system/prometheus.service

/etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus   --config.file /etc/prometheus/prometheus.yml   --storage.tsdb.path /var/lib/prometheus   --storage.tsdb.retention.time=30d
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
# Access at http://localhost:9090

Node Exporter for server metrics

# Node Exporter exposes Linux server metrics to Prometheus
# Install on each server to monitor:
NODE_VERSION="1.8.1"
wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_VERSION}/node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
tar xvf node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
sudo mv node_exporter-${NODE_VERSION}.linux-amd64/node_exporter /usr/local/bin/

sudo useradd --no-create-home --shell /bin/false node_exporter
sudo systemctl enable --now node_exporter
# Metrics available at http://localhost:9100/metrics

# Add to prometheus.yml:
# scrape_configs:
#   - job_name: 'linux-servers'
#     static_configs:
#       - targets: ['server1:9100', 'server2:9100']

Alerting rules

# Create alert rules in /etc/prometheus/alerts.yml:
sudo nano /etc/prometheus/alerts.yml

/etc/prometheus/alerts.yml — common server alerts

groups:
- name: server-alerts
  rules:
  - alert: HighCPUUsage
    expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "CPU > 80% on {{ $labels.instance }}"

  - alert: DiskSpaceLow
    expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 15
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Disk < 15% free on {{ $labels.instance }}"

  - alert: MemoryPressure
    expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
    for: 2m
    labels:
      severity: critical

Conclusion

Prometheus is the right choice when you need programmatic alerting on any metric combination (not just preset checks), long-term metric storage for capacity planning, and integration with a wide ecosystem of exporters (MySQL, Redis, Docker, Kubernetes). The learning curve is the PromQL query language — start with the alert rules above and the community-contributed Grafana dashboards (grafana.com/grafana/dashboards) to get value quickly before writing custom queries.