MONITORING
PROMETHEUS CHEATSHEET.

Prometheus Basics

Installation

# macOS
brew install prometheus

# Linux (download)
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvf prometheus-2.45.0.linux-amd64.tar.gz
cd prometheus-2.45.0.linux-amd64

# Run
./prometheus --config.file=prometheus.yml

# Docker
docker run -d --name prometheus -p 9090:9090 prom/prometheus

# Kubernetes (kube-prometheus-stack)
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack

Prometheus Configuration

prometheus.yml

global:
  scrape_interval: 15s      # How often to scrape targets
  evaluation_interval: 15s  # How often to evaluate rules

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - "alerts/*.yml"
  - "recording_rules/*.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node_exporter:9100']

  - job_name: 'kubernetes-nodes'
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        replacement: ${1}:10250

Metrics Types

http_requests_total{method="GET", status="200"} 12345

cpu_usage{core="0"} 0.75

http_request_duration_seconds_bucket{le="0.5"} 100
http_request_duration_seconds_sum 50
http_request_duration_seconds_count 200

http_request_duration_seconds{quantile="0.5"} 0.25
http_request_duration_seconds{quantile="0.9"} 0.5
http_request_duration_seconds_sum 50
http_request_duration_seconds_count 200

Metric Naming Conventions

# Format
<namespace>_<subsystem>_<name>_<unit>

# Examples
prometheus_http_requests_total
http_request_duration_seconds
node_cpu_seconds_total
container_memory_usage_bytes

PromQL (Prometheus Query Language)

Basic Queries

# Instant vector (current value)
http_requests_total
http_requests_total{method="GET"}

# Range vector (last 5 minutes)
http_requests_total[5m]

# Offset (5 minutes ago)
http_requests_total offset 5m

# Aggregations
sum(http_requests_total)
avg(http_requests_total)
max(http_requests_total)
min(http_requests_total)

# Aggregation with grouping
sum(http_requests_total) by (method)
sum(http_requests_total) without (instance)

# Rate functions
rate(http_requests_total[5m])    # per‑second average
irate(http_requests_total[5m])   # instant per‑second
increase(http_requests_total[1h]) # increase over 1 hour

# Count
count(http_requests_total)
count(http_requests_total) by (status)

Common PromQL Functions

Operators

# Arithmetic
http_requests_total + 100
http_requests_total - 100
http_requests_total * 2
http_requests_total / 2
http_requests_total % 2

# Comparison
http_requests_total > 1000
http_requests_total < 1000
http_requests_total >= 1000
http_requests_total <= 1000
http_requests_total == 1000
http_requests_total != 1000

# Logical
http_requests_total > 1000 and cpu_usage > 0.8
http_requests_total > 1000 or cpu_usage > 0.8
http_requests_total unless cpu_usage > 0.8

Recording Rules

# recording_rules.yml
groups:
  - name: recording_rules
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_request_duration_seconds:95
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))

      - record: node_memory_usage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

      - record: pod:container_cpu_usage:sum
        expr: sum(container_cpu_usage_seconds_total) by (pod, namespace)

Alerting Rules

# alerts.yml
groups:
  - name: instance
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

      - alert: HighCPUUsage
        expr: (100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% for more than 10 minutes."

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is above 90% for more than 10 minutes."

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk space is below 10% on {{ $labels.mountpoint }}."

      - alert: HighRequestLatency
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High request latency on {{ $labels.job }}"
          description: "95th percentile latency is above 2 seconds for more than 10 minutes."

Alertmanager Configuration

# alertmanager.yml
route:
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'

  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
      continue: true
    - match:
        severity: warning
      receiver: 'slack'
      continue: true

receivers:
  - name: 'default'
    email_configs:
      - to: 'team@example.com'

  - name: 'slack'
    slack_configs:
      - channel: '#alerts'
        api_url: 'https://hooks.slack.com/services/...'
        icon_emoji: ':warning:'
        title: '{{ .GroupLabels.alertname }}'

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '...'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster']

Exporters

docker run -d --name node_exporter -p 9100:9100 prom/node-exporter

docker run -d --name blackbox_exporter -p 9115:9115 prom/blackbox-exporter

docker run -d --name mysql_exporter -p 9104:9104 prom/mysqld-exporter --config.my-cnf=/path/to/.my.cnf

docker run -d --name postgres_exporter -p 9187:9187 prometheuscommunity/postgres-exporter --config.my-cnf=/path/to/.my.cnf

docker run -d --name redis_exporter -p 9121:9121 oliver006/redis_exporter

docker run -d --name mongodb_exporter -p 9216:9216 percona/mongodb_exporter

java -javaagent:jmx_prometheus_javaagent-0.19.0.jar=9404:config.yml -jar app.jar

docker run -d --name pushgateway -p 9091:9091 prom/pushgateway

# Push metrics
echo "some_metric 42" | curl --data-binary @- http://localhost:9091/metrics/job/my_job

Service Discovery

Kubernetes

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+):(?:\d+);(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (.+)

EC2 (AWS)

scrape_configs:
  - job_name: 'ec2'
    ec2_sd_configs:
      - region: us-east-1
        port: 9100
    relabel_configs:
      - source_labels: [__meta_ec2_instance_id]
        target_label: instance
      - source_labels: [__meta_ec2_instance_state]
        action: keep
        regex: running

Grafana Dashboard

# Install Grafana
brew install grafana
# or
docker run -d --name grafana -p 3000:3000 grafana/grafana

# Default login
username: admin
password: admin

# Add Prometheus data source
URL: http://prometheus:9090

# Popular dashboards (IDs)
1860 – Node Exporter Full
11074 – Kubernetes Cluster
10413 – Spring Boot
15760 – PostgreSQL
15915 – MySQL

PromQL Query Examples

# CPU usage (percentage)
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage (percentage)
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Disk usage (percentage)
(1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100

# Request rate (per second)
sum(rate(http_requests_total[5m])) by (method, status)

# 95th percentile latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# Pod memory usage
sum(container_memory_usage_bytes) by (pod, namespace)

# Pod CPU usage
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod, namespace)

# Top 5 pods by CPU
topk(5, sum(rate(container_cpu_usage_seconds_total[5m])) by (pod))

Best Practices

• Use labels wisely – avoid high‑cardinality labels (e.g., user_id, request_id)
• Keep metrics simple – don't add too many dimensions
• Use rate() for counters – shows meaningful per‑second rates
• Use increase() for totals – over a time range
• Use recording rules – for expensive queries and pre‑aggregation
• Set retention period – configure --storage.tsdb.retention.time
• Monitor Prometheus itself – scrape http://localhost:9090/metrics
• Use federation – for hierarchical scraping
• Use service discovery – dynamic targets (K8s, EC2, Consul)
• Use Alertmanager – for notifications (email, Slack, PagerDuty)
• Use silences – temporarily silence alerts during maintenance
• Use inhibition – suppress less important alerts during outages
• Use Blackbox Exporter – for endpoint monitoring (HTTP, TCP, ICMP)
• Use Pushgateway – for batch jobs (not for long‑running services)
• Version your alert rules – store in Git with your code
• Test alerts – use promtool test rules
• Use Grafana for dashboards – visualise metrics effectively

Promtool Commands

# Check configuration
promtool check config prometheus.yml

# Check rules
promtool check rules alerts.yml

# Test rules
promtool test rules test.yml

# Query metrics (like curl for Prometheus)
promtool query instant http://localhost:9090 "up"

# Query range
promtool query range http://localhost:9090 "up" --start=2024-01-01T00:00:00Z --end=2024-01-01T01:00:00Z --step=15s