Design a Metrics Monitoring and Alerting System

An inside look at how massive cloud infrastructures monitor health, ingest time-series data, and trigger real-time alerts without collapsing under massive write loads.

System Requirements

We are designing an internal infrastructure monitoring system (similar to Datadog or Prometheus). It will monitor server metrics (CPU, Memory, Disk Space, Request Rates) and trigger alerts.

• Scale: 1,000 server pools × 100 machines × 100 metrics = 10 million metrics constantly reporting.
• Retention: Keep raw data for 7 days, 1-minute resolution for 30 days, and 1-hour resolution for 1 year.
• Latency: Low query latency for dashboarding and near real-time alerting.

Note: Log aggregation (ELK stack) and distributed tracing (Jaeger) are out of scope for this specific design.

High-Level Architecture

A monitoring system is composed of five distinct pillars.

flowchart TD
    Sources[Metrics Sources] --> Collect[Metrics Collector]
    Collect --> MQ[Kafka Queue]
    MQ --> DB[(Time-Series DB)]
    
    DB --> Query[Query Service]
    Query --> Dash[Visualization (Grafana)]
    Query --> Alert[Alert Manager]
    Alert --> Notify[PagerDuty / Email]
      

1. Data Collection: Extracting data from application servers.
2. Data Transmission: Buffering the data safely via a message queue (Kafka).
3. Data Storage: Specialized database optimized for time-series data (e.g., InfluxDB).
4. Alerting: Rule engine that checks data against thresholds.
5. Visualization: Plotting the time-series data on charts.

Deep Dive: Data Collection (Push vs. Pull)

How do we get metrics from 100,000 servers into our central system? There is a fierce industry debate between the "Pull" model (used by Prometheus) and the "Push" model (used by Datadog).

The Pull Model

A centralized Metrics Collector periodically pings every server's /metrics endpoint and scrapes the data.

• Pros: Easy to tell if a server is down (it stops responding to the pull). Easy to debug locally (just visit the /metrics URL in your browser).
• Cons: The Collector needs to know the IP address of every single server, requiring complex integration with Service Discovery (Zookeeper/etcd) and tricky firewall configurations.

The Push Model

A lightweight Agent is installed on every server. It actively pushes its metrics to the central Collector.

• Pros: No need for complex Service Discovery. The Collector sits behind a Load Balancer and accepts traffic from anywhere. Great for short-lived batch jobs that might die before a Pull can happen.
• Cons: If the Collector goes down, metrics are lost. Must be heavily rate-limited to prevent agents from DDoS-ing the central server.

Conclusion: Large organizations generally support both, using Push for distributed microservices and Pull for internal legacy infrastructure.

Deep Dive: Data Storage (Time-Series Databases)

Metrics data is uniquely structured. It is essentially an infinite stream of (metric_name, labels, timestamp, value). This results in a massive volume of continuous writes, but reads are usually spiky (only happening when a dashboard is opened during an incident).

Relational databases (MySQL) will collapse under this write load. We must use a Time-Series Database (TSDB) like InfluxDB or Prometheus.

TSDB Optimizations

1. Double-Delta Encoding: Timestamps take up 8 bytes. Since metrics are reported at regular intervals (e.g., exactly every 10 seconds), the DB doesn't store the full timestamp. It stores the delta (difference) between timestamps. It then calculates the delta of the delta, which is usually 0. This compresses data by over 90%.
2. Downsampling: Querying a year's worth of data at a 10-second resolution would return billions of rows and crash the dashboard. The TSDB runs background jobs to "roll up" data. Raw 10-second data older than 7 days is averaged into 1-minute chunks, saving massive amounts of disk space and radically speeding up long-term queries.

Deep Dive: Alerting System

The Alert Manager is an independent service that continuously evaluates data against predefined YAML rules.

- alert: HighCPUUsage
  expr: cpu_usage > 90
  for: 5m
  labels:
    severity: page
      

If CPU usage exceeds 90% for a sustained 5 minutes, an alert event is generated. Crucially, the Alert Manager must perform Deduplication. If a cluster of 50 servers goes down, you don't want the on-call engineer to receive 50 text messages. The Alert Manager groups those 50 alerts into a single "Cluster Offline" notification before pushing it to PagerDuty or Slack.