可观测性六维信号深度实战

从 “告警响了” 到 “我知道问题出在哪，而且知道怎么修”的完整信号链。每维信号给出 Go 代码示例 + PromQL/LokiQL/TraceQL 查询模板 + K8s 配置。

一、Metrics 指标：从 RED/USE 到 PromQL 实战

1.1 Go 服务埋点——Prometheus Client

package metrics

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
    "time"
)

// RED 信号定义
var (
    // Rate — 以 Histogram 同时捕获计数和分布
    RequestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request latency distributions.",
        Buckets: prometheus.DefBuckets, // [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
    }, []string{"method", "endpoint", "status"})

    // Errors 通过 status 标签区分，不单独设 Counter
    // Duration 从 Histogram 计算 P50/P95/P99

    // 业务指标
    OrderCreated = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "orders_created_total",
        Help: "Total number of orders created.",
    }, []string{"source", "status"})

    // Go runtime 指标（进程级 USE）
    Goroutines = promauto.NewGauge(prometheus.GaugeOpts{
        Name: "go_goroutines_current",
        Help: "Current number of goroutines.",
    })

    GCSummary = promauto.NewSummary(prometheus.SummaryOpts{
        Name: "go_gc_duration_seconds",
        Help: "GC pause duration summary.",
    })
)

// Middleware 方式自动记录
func MetricsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        wrapped := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
        next.ServeHTTP(wrapped, r)
        duration := time.Since(start).Seconds()

        RequestDuration.WithLabelValues(
            r.Method,
            r.URL.Path,
            http.StatusText(wrapped.statusCode),
        ).Observe(duration)
    })
}

1.2 PromQL 十大实战查询

# 1. 服务可用性（SLI）
sum(rate(http_requests_total{status!~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# 2. P99 延迟
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, endpoint))

# 3. 错误率突增检测（同比 30m 前）
rate(http_requests_total{status=~"5.."}[5m])
  / rate(http_requests_total{status=~"5.."}[5m] offset 30m) > 2

# 4. CPU 饱和度（容器级）
rate(container_cpu_cfs_throttled_seconds_total[5m]) > 0

# 5. Pod 内存 OOM Kill 风险
container_memory_working_set_bytes{container!=""}
  / container_spec_memory_limit_bytes > 0.85

# 6. 慢查询 Top 10（按 P99，取 1h 窗口）
topk(10,
  histogram_quantile(0.99,
    sum(rate(http_request_duration_seconds_bucket[1h])) by (le, endpoint)))

# 7. Error Budget 剩余（月窗口）
(1 - (
  sum(increase(http_requests_total{status=~"5.."}[30d]))
    / sum(increase(http_requests_total[30d]))
)) / 0.999  # SLO=99.9%

# 8. 烧钱率（Burn Rate，6h 窗口 × 14.4x 阈值）
sum(rate(http_requests_total{status=~"5.."}[6h]))
  / sum(rate(http_requests_total[6h]))
  / (1 - 0.999) > 14.4

# 9. 服务间依赖调用量
sum(rate(http_request_duration_seconds_count[5m])) by (source, target)

# 10. Pod 重启频率（Rolling 窗口）
rate(kube_pod_container_status_restarts_total[1h]) > 0

1.3 Recording Rules——预计算降基数

# rules/availability.yml
groups:
  - name: slo_rules
    interval: 30s
    rules:
      # 先预聚合，再对外暴露。降低 Grafana 查询基数
      - record: job:http_requests_total:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_errors_total:rate5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)

      - record: job:availability:ratio5m
        expr: |
          (sum(rate(http_requests_total{status!~"5.."}[5m])) by (job))
            / sum(rate(http_requests_total[5m])) by (job)

      # P50/P90/P99 延迟预聚合
      - record: job:latency:p50
        expr: |
          histogram_quantile(0.50,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))

      - record: job:latency:p99
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))

1.4 告警规则——Google SRE 多窗口 Burn Rate

# rules/alerts.yml
groups:
  - name: slo_alerts
    rules:
      # 严重：1h 窗口烧钱率 > 14.4x（等价于 2% 错误预算 1h 烧完）
      - alert: HighErrorRate_Critical
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
              / sum(rate(http_requests_total[1h]))
          ) / (1 - 0.999) > 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "错误预算烧钱率严重（1h > 14.4x）"
          description: "{{ $labels.job }} 1h 内烧掉了年预算的 2%"

      # 警告：6h 窗口烧钱率 > 6x
      - alert: HighErrorRate_Warning
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
              / sum(rate(http_requests_total[6h]))
          ) / (1 - 0.999) > 6
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "错误预算烧钱率警告（6h > 6x）"

  - name: infrastructure
    rules:
      # CPU 被 Throttle 超过 5% 时间
      - alert: CPUThrottlingHigh
        expr: |
          rate(container_cpu_cfs_throttled_seconds_total[5m])
            / rate(container_cpu_usage_seconds_total[5m]) > 0.05
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Pod {{ $labels.pod }} CPU throttling > 5%"

      # 内存使用超过 limit 90%（OOM 临近）
      - alert: MemoryNearLimit
        expr: |
          container_memory_working_set_bytes{container!=""}
            / container_spec_memory_limit_bytes > 0.90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Pod {{ $labels.pod }} memory usage > 90% of limit"

[!tip] 告警原则

• P0 (page at 3am)：直接影响 SLO，必须人工介入。如 HighErrorRate_Critical
• P1 (page during work hours)：SLO 有风险但非紧急
• P2 (ticket)：趋势异常但不影响可用性
• 告警必须可行动——如果凌晨接到告警不知道该做什么，这条告警就不该存在

1.5 Kube-Prometheus-Stack 部署

# kube-prometheus-stack values 关键配置
prometheus:
  prometheusSpec:
    # 外部标签——跨集群标识
    externalLabels:
      cluster: prod-k8s-01
      region: ap-guangzhou
    # 保留时间
    retention: 30d
    retentionSize: 80GB
    # 采集间隔
    scrapeInterval: 30s
    evaluationInterval: 30s
    # 远程写入——长期存储
    remoteWrite:
      - url: "http://mimir-distributor.mimir:9009/api/v1/push"
    # 高基数保护
    enforcedLabelLimit: 20
    enforcedLabelNameLengthLimit: 50
    enforcedLabelValueLengthLimit: 200

  # 额外的 ServiceMonitor
  additionalServiceMonitors:
    - name: care-mate-api
      selector:
        matchLabels:
          app: care-mate-api
      endpoints:
        - port: metrics
          interval: 30s
          path: /metrics

grafana:
  # 预置 Dashboard 数据源
  additionalDataSources:
    - name: Loki
      type: loki
      url: http://loki-gateway.loki:80
    - name: Tempo
      type: tempo
      url: http://tempo-query-frontend.tempo:3100
      jsonData:
        tracesToLogsV2:
          datasourceUid: loki
          tags: ['job', 'instance', 'pod', 'namespace']

---

二、Logs 日志：结构化规范 + Go 最佳实践

2.1 Go 结构化日志——Zap 封装

package logger

import (
    "context"
    "go.opentelemetry.io/otel/trace"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
    "os"
)

var Log *zap.Logger

func Init(env string) {
    config := zap.NewProductionEncoderConfig()
    config.TimeKey = "timestamp"
    config.EncodeTime = zapcore.ISO8601TimeEncoder
    config.EncodeLevel = zapcore.CapitalLevelEncoder

    var core zapcore.Core
    if env == "development" {
        // 本地开发：console 编码，debug 级别
        encoder := zapcore.NewConsoleEncoder(config)
        core = zapcore.NewCore(encoder, zapcore.AddSync(os.Stdout), zapcore.DebugLevel)
    } else {
        // 生产：JSON 编码，info 级别
        encoder := zapcore.NewJSONEncoder(config)
        core = zapcore.NewCore(encoder, zapcore.AddSync(os.Stdout), zapcore.InfoLevel)
    }

    Log = zap.New(core,
        zap.AddCaller(),
        zap.AddCallerSkip(1),       // 跳过 wrapper 层
        zap.AddStacktrace(zapcore.ErrorLevel),
        zap.Fields(
            zap.String("service", os.Getenv("SERVICE_NAME")),
            zap.String("version", os.Getenv("APP_VERSION")),
        ),
    )
}

// Ctx 从 context 提取 trace_id 自动注入日志
func Ctx(ctx context.Context) *zap.Logger {
    span := trace.SpanFromContext(ctx)
    if !span.SpanContext().IsValid() {
        return Log
    }
    return Log.With(
        zap.String("trace_id", span.SpanContext().TraceID().String()),
        zap.String("span_id", span.SpanContext().SpanID().String()),
    )
}

// 中间件自动注入 trace_id 到所有请求日志
func LoggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        logger := Ctx(ctx)

        logger.Info("request",
            zap.String("method", r.Method),
            zap.String("path", r.URL.Path),
            zap.String("user_agent", r.UserAgent()),
            zap.String("client_ip", r.RemoteAddr),
        )

        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

2.2 日志规范——必须包含的字段

[!important] 每条生产日志必备

字段	说明	示例
timestamp	ISO 8601 格式	2026-08-04T12:00:00Z
level	DEBUG/INFO/WARN/ERROR/FATAL	ERROR
service	服务名	care-mate-api
trace_id	OTel TraceID	0af7651916...
message	人类可读描述	order creation failed
error	error.Error() 原文（仅 ERROR）	dial tcp 10.0.1.5:3306: timeout

反模式：

• log.Printf("user %d created order") ——无结构、无 trace_id
• 到处打 Info("processing...") ——信息量为零
• 错误日志不带上下文（哪个用户、哪个请求）

2.3 Loki 部署与查询

# Loki single-binary 模式（< 10GB/day 适用）
loki:
  auth_enabled: false
  storage:
    type: s3
    s3:
      endpoint: oss-cn-guangzhou.aliyuncs.com
      bucketnames: loki-data
      region: cn-guangzhou
  limits_config:
    # 限制单条日志大小
    max_entries_limit_per_query: 5000
    # 限制查询时间范围
    max_query_length: 721h      # 30d
    max_query_parallelism: 32

# Promtail 采集配置
promtail:
  config:
    clients:
      - url: http://loki-gateway.loki/loki/api/v1/push
    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        pipeline_stages:
          # 从 Pod 注解提取 pipeline
          - cri: {}
          # 添加 k8s 元数据标签
          - labels:
              namespace: ""
              pod: ""
              container: ""

# LogQL 十大查询

# 1. 按 trace_id 查找全链路日志
{service="care-mate-api"} |= "0af7651916cd43dd8448eb211c80319c"

# 2. ERROR 级别日志（最近 1h）
{service=~"care-mate-.*"} | level="ERROR" | line_format "{{.timestamp}} {{.message}}"

# 3. 慢请求日志（> 3s）
{service="care-mate-api"} | json | duration > 3000

# 4. 统计每分钟 ERROR 数
sum(count_over_time({service="care-mate-api"} | level="ERROR" [1m]))

# 5. 某 Pod 的错误日志聚合（按错误信息分组）
sum by (error) (
    count_over_time({pod="care-mate-api-7d4f-abc"} | level="ERROR" | json error="error" [15m])
)

# 6. 某个端点的请求量趋势
sum(count_over_time({service="care-mate-api"} |= "/api/orders" [5m]))

# 7. 搜索特定错误消息
{service=~"care-mate-.*"} |~ "(?i)connection.*(refused|timeout)"

# 8. 去掉健康检查噪音
{service="care-mate-api"} != "/health" != "/metrics"

# 9. 只看数据库相关日志
{namespace="production"} |= "mysql" or "postgres"

# 10. 关联 Metrics：按 trace_id 定位到 Tempo trace
# 在 Grafana 中日志面板自动提供 "Tempo" 链接按钮

---

三、Traces 链路追踪：OTel Go SDK 完整接入

3.1 Go 服务端初始化

package telemetry

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
    "time"
)

func InitTracer(ctx context.Context, serviceName, otelEndpoint string) (*sdktrace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint(otelEndpoint),
        otlptracegrpc.WithInsecure(), // 内网用 insecure
    )
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter,
            sdktrace.WithMaxExportBatchSize(512),
            sdktrace.WithBatchTimeout(5*time.Second),
        ),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName(serviceName),
            semconv.ServiceVersion("1.0.0"),
            semconv.DeploymentEnvironment("production"),
            semconv.K8SNamespaceName("production"),
        )),
        // 采样策略——见 3.3 节
        sdktrace.WithSampler(sdktrace.ParentBased(
            sdktrace.TraceIDRatioBased(0.10), // 10% 采样
        )),
    )

    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    return tp, nil
}

3.2 HTTP/gRPC 自动埋点

import (
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
    "net/http"
)

// HTTP 服务端：包裹 Handler
mux := http.NewServeMux()
mux.Handle("/api/orders", otelhttp.NewHandler(orderHandler, "CreateOrder"))

// HTTP 客户端：自动传播 trace context
client := &http.Client{
    Transport: otelhttp.NewTransport(http.DefaultTransport),
}

// gRPC 服务端
s := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

// gRPC 客户端
conn, _ := grpc.Dial(target,
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)

// MySQL/Redis 自动埋点
import (
    "github.com/uptrace/opentelemetry-go-extra/otelgorm"
    "github.com/go-redis/redis/extra/redisotel/v9"
)
// GORM: db.Use(otelgorm.NewPlugin())
// Redis: rdb.AddHook(redisotel.NewTracingHook())

3.3 采样策略——四层决策

请求到达 → AlwaysSample（如：带 "debug=true" header）
          → NeverSample（如：/health /metrics）
          → Head-based：TraceIDRatioBased(10%)
          → Tail-based：OTel Collector 端判断（慢/错 → 保留）

# OTel Collector tail-based sampling
processors:
  tail_sampling:
    decision_wait: 30s          # 等 30s 让所有 span 到齐
    num_traces: 50000           # 内存中保持的 trace 数量
    policies:
      # 策略 1：所有错误的 trace 保留
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      # 策略 2：延迟 > 1s 的 trace 保留
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}
      # 策略 3：其余按 10% 采样
      - name: default
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

3.4 自定义 Span——业务关键路径

func (s *OrderService) Create(ctx context.Context, req *CreateOrderReq) (*Order, error) {
    // Top-level span
    ctx, span := otel.Tracer("order-service").Start(ctx, "OrderService.Create")
    defer span.End()

    span.SetAttributes(
        attribute.String("order.source", req.Source),
        attribute.Int("order.item_count", len(req.Items)),
    )

    // 校验库存——子 Span
    if err := s.checkInventory(ctx, req.Items); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "inventory check failed")
        return nil, err
    }

    // 扣减库存
    ctx, deductSpan := otel.Tracer("order-service").Start(ctx, "InventoryService.Deduct")
    defer deductSpan.End()

    if err := s.inventory.Deduct(ctx, req.Items); err != nil {
        deductSpan.RecordError(err)
        deductSpan.SetStatus(codes.Error, "deduct failed")
        return nil, err
    }
    deductSpan.SetAttributes(attribute.Int("deduct.duration_ms", 45))

    // 创建订单——Event 记录关键节点
    order, err := s.repo.Save(ctx, req)
    if err != nil {
        span.RecordError(err)
        return nil, err
    }
    span.AddEvent("order.created", trace.WithAttributes(
        attribute.String("order.id", order.ID),
    ))

    return order, nil
}

3.5 Tempo 查询——TraceQL

# TraceQL 查询模板

# 1. 按服务名 + 错误状态查找慢 trace
{ resource.service.name = "care-mate-api" && status = error }

# 2. 查找延迟 > 2s 的 trace
{ span.duration > 2s && name =~ "OrderService.*" }

# 3. 查找调用某个下游的所有 trace
{ resource.service.name = "care-mate-api" && span.http.target = "/api/orders" }

# 4. 查找包含特定属性的 trace
{ span.db.system = "mysql" && span.db.operation = "select" }

# 5. 链路结构查询——找出调用链含 4 层以上的
{ span.service.name = "care-mate-api" } | select( count() > 4 )

# 6. 比对两次发布之间的 P99 延迟变化
{ span.http.route = "/api/orders" && span.startTime > "2026-08-04T10:00:00Z" }
  | rate() by (span.http.status_code)

---

四、Profiles 持续剖析

4.1 Go pprof 生产环境接入

import (
    "net/http"
    _ "net/http/pprof"
    "runtime"
    "github.com/felixge/fgprof"
)

// 启动 pprof HTTP 端点（建议内网端口）
go func() {
    http.DefaultServeMux.Handle("/debug/fgprof", fgprof.Handler())
    // runtime.SetMutexProfileFraction(1)  // 开启 mutex profiling
    // runtime.SetBlockProfileRate(1)       // 开启 block profiling
    http.ListenAndServe(":6060", nil)
}()

// 手动采集 heap profile
func dumpHeap() {
    f, _ := os.Create(fmt.Sprintf("heap_%d.pprof", time.Now().Unix()))
    runtime.GC()
    pprof.WriteHeapProfile(f)
    f.Close()
}

4.2 Pyroscope eBPF 零侵入（推荐）

# Pyroscope agent DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: pyroscope-agent
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: pyroscope-agent
  template:
    metadata:
      labels:
        app: pyroscope-agent
    spec:
      hostPID: true
      containers:
        - name: agent
          image: grafana/pyroscope:latest
          args:
            - "ebpf"
            - "--application-cache-size=500"
          env:
            - name: PYROSCOPE_SERVER_ADDRESS
              value: "http://pyroscope-server:4040"
          securityContext:
            privileged: true
          volumeMounts:
            - name: sys
              mountPath: /sys
      volumes:
        - name: sys
          hostPath:
            path: /sys

[!tip] 何时用 Profiles

• CPU 飙升但 metrics 看不出具体哪里 → Flamegraph
• 内存持续增长→ heap profile 找泄漏对象
• Goroutine 数量异常 → goroutine profile 找阻塞/泄漏
• GC 频繁 → alloc profile 找高频分配点

---

五、Events 变更事件

5.1 K8s Events Watcher（Go）

import (
    "context"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
)

func WatchEvents(clientset *kubernetes.Clientset) {
    watcher, _ := clientset.CoreV1().Events("").Watch(context.Background(), metav1.ListOptions{})

    for event := range watcher.ResultChan() {
        ev := event.Object.(*corev1.Event)

        // 只关注 Warning 级别的事件
        if ev.Type != "Warning" {
            continue
        }

        logger.Log.Error("k8s event",
            zap.String("namespace", ev.Namespace),
            zap.String("name", ev.InvolvedObject.Name),
            zap.String("kind", ev.InvolvedObject.Kind),
            zap.String("reason", ev.Reason),
            zap.String("message", ev.Message),
            zap.Time("first_timestamp", ev.FirstTimestamp.Time),
        )
    }
}

5.2 变更事件采集矩阵

事件类型	采集方式	字段
代码部署	CI/CD webhook → Loki	deploy_id, git_commit, service, timestamp
配置变更	ConfigMap/Secret 变更事件	configmap_name, old_hash, new_hash, timestamp
扩缩容	HPA/KEDA 事件	deployment, old_replicas, new_replicas, reason
基础设施变更	Terraform state / IaC	resource, action(create/update/delete), timestamp
依赖变更	服务发现变更（Nacos/K8s Service）	service, old_endpoints, new_endpoints

[!important] 变更关联排障 70% 的故障由变更引入。Grafana 中用 Annotations 功能把部署事件标记在监控面板上——看到延迟尖峰对应的一条 deploy 线，问题就定位了一半。

---

六、RUM 真实用户监控

6.1 前端接入 Grafana Faro

<script src="https://cdn.jsdelivr.net/npm/@grafana/faro-web-sdk/dist/faro-web-sdk.iife.js"></script>
<script>
  GrafanaFaroWebSdk.initializeFaro({
    url: 'https://faro-collector.example.com/collect',
    app: {
      name: 'care-mate-web',
      version: '1.2.3',
      environment: 'production',
    },
    sessionTracking: {
      enabled: true,
      samplingRate: 0.1,
    },
    instrumentations: [
      new GrafanaFaroWebSdk.WebVitalsInstrumentation(),
      new GrafanaFaroWebSdk.ErrorsInstrumentation(),
      new GrafanaFaroWebSdk.ConsoleInstrumentation({ disabledLevels: ['debug'] }),
    ],
    // 把前端 trace 和后端 trace 关联
    propagateTraceHeaderCorsUrls: [/.*/],
  });
</script>

6.2 RUM 核心指标

指标	含义	SLO 建议
LCP (Largest Contentful Paint)	最大内容渲染	< 2.5s
FID/INP (Interaction to Next Paint)	交互延迟	< 200ms
CLS (Cumulative Layout Shift)	布局偏移	< 0.1
TTFB (Time to First Byte)	首字节时间	< 800ms
JS Error Rate	JS 报错率	< 0.1%
API Error Rate	API 调用错误率	< 0.5%

---

相关笔记

• 可观测性知识总览 — 理论框架总览
• Metrics 与 Prometheus 实战 — Prometheus 专项
• 分布式追踪 OpenTelemetry — OTel 专项
• 日志体系 Loki 与 ELK 对比 — 日志存储选型
• SLO SLI 错误预算与智能告警 — SLO 专项
• K8s 可观测性实战 — K8s 场景专项
• SRE-SLI-SLO-ErrorBudget实战手册 — Error Budget 深度
• SRE-从零到一实施路线图 — 阶段实施指南