Monitoring Plus

增强型监控功能，支持 Prometheus、Grafana、告警规则和 SLO 跟踪。

特性

• Prometheus：指标收集与查询
• Grafana：可视化与仪表盘
• Loki：日志聚合
• 告警：规则、路由与升级机制
• SLO/SLI：服务等级跟踪

快速参考

Prometheus

配置文件

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert_rules.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: 'app'
    static_configs:
      - targets: ['app:8080']
    metrics_path: /metrics

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

指标类型

# 计数器（Counter）
http_requests_total{method="GET", status="200"}

# 仪表值（Gauge）
node_memory_MemAvailable_bytes

# 直方图（Histogram）
http_request_duration_seconds_bucket{le="0.5"}

# 汇总值（Summary）
http_request_duration_seconds{quantile="0.99"}

常用查询语句

# 请求速率
rate(http_requests_total[5m])

# 错误率
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

# 延迟 P95
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# 内存使用量
process_resident_memory_bytes / 1024 / 1024

# CPU 使用率
rate(process_cpu_seconds_total[5m])

Grafana

仪表盘 JSON 配置

{
  "dashboard": {
    "title": "Application Metrics",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "legendFormat": "{{method}} {{status}}"
          }
        ]
      },
      {
        "title": "Error Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(http_requests_total{status=~'5..'}[5m]) / rate(http_requests_total[5m])",
            "legendFormat": "Error %"
          }
        ]
      },
      {
        "title": "Latency P95",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
            "legendFormat": "P95"
          }
        ]
      }
    ]
  }
}

告警规则

Prometheus 告警规则

# alert_rules.yml
groups:
  - name: app_alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency detected"
          description: "P95 latency is {{ $value }}s"

      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service is down"
          description: "{{ $labels.instance }} has been down for more than 1 minute"

      - alert: HighMemoryUsage
        expr: process_resident_memory_bytes / 1024 / 1024 > 512
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage"
          description: "Memory usage is {{ $value }}MB"

Alertmanager 配置

# alertmanager.yml
global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alerts@example.com'
  smtp_auth_username: 'alerts@example.com'
  smtp_auth_password: 'password'

route:
  group_by: ['alertname', 'severity']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pager'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'default'
    email_configs:
      - to: 'team@example.com'

  - name: 'pager'
    pagerduty_configs:
      - service_key: 'your-pagerduty-key'

  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx'
        channel: '#alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.description }}'

Loki

配置文件

# loki-config.yml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

LogQL 查询语句

# 按标签过滤
{job="app"}

# 按关键词过滤
{job="app"} |= "error"

# 正则表达式过滤
{job="app"} |= `error|warn`

# 指标查询
rate({job="app"} |= "error" [5m])

# 直方图统计
histogram_quantile(0.99, sum(rate({job="app"} |= "error" [5m])) by (le))

SLO/SLI 跟踪

SLI 定义

# sli-config.yml
slis:
  - name: availability
    description: "成功请求的百分比"
    sli:
      type: "success_rate"
      good_query: "sum(rate(http_requests_total{status!~'5..'}[5m]))"
      total_query: "sum(rate(http_requests_total[5m]))"
    slos:
      - target: 0.99
        window: "30d"
      - target: 0.999
        window: "7d"

  - name: latency
    description: "在 500ms 内完成的请求占比"
    sli:
      type: "latency"
      query: "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))"
    slos:
      - target: 0.95
        threshold: 0.5
        window: "30d"

错误预算

# 剩余错误预算
slo:availability:sli{job="app"} - 0.99

# 消耗速率
(1 - slo:availability:sli{job="app"}) / (1 - 0.99)

仪表板模板

应用程序仪表板

{
  "panels": [
    {"title": "请求速率", "type": "graph", "expr": "rate(http_requests_total[5m])"},
    {"title": "错误率", "type": "graph", "expr": "rate(http_requests_total{status=~'5..'}[5m]) / rate(http_requests_total[5m])"},
    {"title": "延迟 P50/P95/P99", "type": "graph", "expr": "histogram_quantile(0.5/0.95/0.99, rate(http_request_duration_seconds_bucket[5m]))"},
    {"title": "CPU 使用率", "type": "gauge", "expr": "rate(process_cpu_seconds_total[5m])"},
    {"title": "内存使用率", "type": "gauge", "expr": "process_resident_memory_bytes / 1024 / 1024"}
  ]
}

最佳实践

1. 关注症状而非原因 - 以用户影响为依据，而非故障源头
2. 附带操作手册 - 告警触发时应有明确应对流程
3. 合理设置严重等级 - 并非所有问题都需标记为 P1
4. 使用记录规则 - 预计算高开销查询以提升性能
5. 从外部监控 - 通过外部合成监控验证服务可用性
6. 设定 SLO - 明确系统的可靠性目标
7. 追踪错误预算 - 在可靠性与发布速度之间取得平衡
8. 结构化日志输出 - 使用 JSON 格式日志便于解析和分析