Prometheus

Production Prometheus setup covering scrape configuration, service discovery,

recording rules, alert rules, and operational best practices for infrastructure

and application monitoring.

When to Use

| Scenario | Example |

|----------|---------|

| Set up metrics collection | New service needs Prometheus scraping |

| Configure service discovery | K8s pods, file-based, or static targets |

| Create recording rules | Pre-compute expensive PromQL queries |

| Design alert rules | SLO-based alerts for availability and latency |

| Production deployment | HA setup with retention and storage planning |

| Troubleshoot scraping | Targets down, metrics missing, relabeling issues |

Architecture

Applications ──(/metrics)──→ Prometheus Server ──→ AlertManager → Slack/PD
      ↑                           │
  client libraries          ├──→ Grafana (dashboards)
  (prom client)             └──→ Thanos/Cortex (long-term storage)

Installation

Kubernetes (Helm)

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageVolumeSize=50Gi

Core Configuration

prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: production
    region: us-west-2

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  # Self-monitoring
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  # Node exporters
  - job_name: node-exporter
    static_configs:
      - targets: ["node1:9100", "node2:9100", "node3:9100"]
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: "([^:]+)(:[0-9]+)?"
        replacement: "${1}"

  # Application metrics (TLS)
  - job_name: my-app
    scheme: https
    metrics_path: /metrics
    tls_config:
      ca_file: /etc/prometheus/ca.crt
    static_configs:
      - targets: ["app1:9090", "app2:9090"]

Service Discovery

Kubernetes Pods (Annotation-Based)

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels:
          [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels:
          [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

Pod annotations to enable scraping:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    prometheus.io/path: "/metrics"

File-Based Discovery

scrape_configs:
  - job_name: file-sd
    file_sd_configs:
      - files: ["/etc/prometheus/targets/*.json"]
        refresh_interval: 5m

targets/production.json:

[{
  "targets": ["app1:9090", "app2:9090"],
  "labels": { "env": "production", "service": "api" }
}]

Discovery Method Comparison

| Method | Best For | Dynamic |

|--------|----------|---------|

| static_configs | Fixed infrastructure, dev | No |

| file_sd_configs | CM-managed inventories | Yes (file watch) |

| kubernetes_sd_configs | K8s workloads | Yes (API watch) |

| consul_sd_configs | Consul service mesh | Yes (Consul watch) |

| ec2_sd_configs | AWS EC2 instances | Yes (API poll) |

Recording Rules

Pre-compute expensive queries for dashboard and alert performance:

# /etc/prometheus/rules/recording_rules.yml
groups:
  - name: api_metrics
    interval: 15s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))

      - record: job:http_error_rate:ratio
        expr: job:http_errors:rate5m / job:http_requests:rate5m

      - record: job:http_duration:p95
        expr: >
          histogram_quantile(0.95,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
          )

  - name: resource_metrics
    interval: 30s
    rules:
      - record: instance:node_cpu:utilization
        expr: >
          100 - (avg by (instance)
            (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

      - record: instance:node_memory:utilization
        expr: >
          100 - ((node_memory_MemAvailable_bytes
            / node_memory_MemTotal_bytes) * 100)

      - record: instance:node_disk:utilization
        expr: >
          100 - ((node_filesystem_avail_bytes
            / node_filesystem_size_bytes) * 100)

Naming Convention

level:metric_name:operations

| Part | Example | Meaning |

|------|---------|---------|

| level | job:, instance: | Aggregation level |

| metric_name | http_requests | Base metric |

| operations | :rate5m, :ratio | Applied functions |

Alert Rules

# /etc/prometheus/rules/alert_rules.yml
groups:
  - name: availability
    rules:
      - alert: ServiceDown
        expr: up{job="my-app"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is down"
          description: "{{ $labels.job }} down for >1 minute"

      - alert: HighErrorRate
        expr: job:http_error_rate:ratio > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Error rate {{ $value | humanizePercentage }} for {{ $labels.job }}"

      - alert: HighP95Latency
        expr: job:http_duration:p95 > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency {{ $value }}s for {{ $labels.job }}"

  - name: resources
    rules:
      - alert: HighCPU
        expr: instance:node_cpu:utilization > 80
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "CPU {{ $value }}% on {{ $labels.instance }}"

      - alert: HighMemory
        expr: instance:node_memory:utilization > 85
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "Memory {{ $value }}% on {{ $labels.instance }}"

      - alert: DiskSpaceLow
        expr: instance:node_disk:utilization > 90
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: "Disk {{ $value }}% on {{ $labels.instance }}"

Alert Severity Guide

| Severity | Threshold | Response |

|----------|-----------|----------|

| critical | Service down, data loss risk | Page on-call immediately |

| warning | Degraded, approaching limit | Investigate within hours |

| info | Notable but not urgent | Review in next business day |

Validation

# Validate config syntax
promtool check config prometheus.yml

# Validate rule files
promtool check rules /etc/prometheus/rules/*.yml

# Test a query
promtool query instant http://localhost:9090 'up'

# Reload config without restart
curl -X POST http://localhost:9090/-/reload

Best Practices

| Practice | Detail |

|----------|--------|

| Naming: prefix_name_unit | Snake_case, _total for counters, _seconds/_bytes for units |

| Scrape intervals 15–60s | Shorter wastes resources and storage |

| Recording rules for dashboards | Pre-compute anything queried repeatedly |

| Monitor Prometheus itself | prometheus_tsdb_*, scrape_duration_seconds |

| HA deployment | 2+ instances scraping same targets |

| Retention planning | Match --storage.tsdb.retention.time to disk capacity |

| Federation for scale | Global Prometheus aggregates from regional instances |

| Long-term storage | Thanos or Cortex for >30d retention |

Troubleshooting Quick Reference

| Problem | Diagnosis | Fix |

|---------|-----------|-----|

| Target shows DOWN | Check /targets page for error | Fix firewall, verify endpoint, check TLS |

| Metrics missing | Query up{job="x"} | Verify scrape config, check /metrics endpoint |

| High cardinality | prometheus_tsdb_head_series growing | Drop high-cardinality labels with metric_relabel_configs |

| Storage filling up | Check prometheus_tsdb_storage_* | Reduce retention, add disk, enable compaction |

| Slow queries | Check prometheus_engine_query_duration_seconds | Add recording rules, reduce range, limit series |

| Config not applied | Check prometheus_config_last_reload_successful | Fix syntax, POST /-/reload |

NEVER Do

| Anti-Pattern | Why | Do Instead |

|-------------|-----|------------|

| Scrape interval < 5s | Overwhelms targets and storage | Use 15–60s intervals |

| High-cardinality labels (user ID, request ID) | Explodes TSDB series count | Use logs for high-cardinality data |

| Alert without for duration | Fires on transient spikes | Always set for: 1m minimum |

| Skip recording rules | Dashboards compute expensive queries every load | Pre-compute with recording rules |

| Store secrets in prometheus.yml | Config often in Git | Use file-based secrets or env substitution |

| Ignore up metric | Miss targets silently going down | Alert on up == 0 for all jobs |

| Single Prometheus instance in prod | Single point of failure | Run 2+ replicas with shared targets |

| Unbounded retention | Disk fills, Prometheus crashes | Set explicit --storage.tsdb.retention.time |

Templates

| Template | Description |

|----------|-------------|

| templates/prometheus.yml | Full config with static, file-based, and K8s discovery |

| templates/alert-rules.yml | 25+ alert rules by category |

| templates/recording-rules.yml | Pre-computed metrics for HTTP, latency, resources, SLOs |