monitoring-setup-guide

Write a monitoring setup guide for a service — defining what to measure, how to alert on it, and how to build the observability stack covering the f…

不碰外部（只输出文字）无严重或高危命中mohitagw15856/pm-claude-skills

它会碰到什么

扫了多少1 个文本文件，21 KB

它会碰到什么不碰外部（只输出文字）

命中总数0 处

命中统计严重 0 · 高 0 · 中 0 · 低 0

这一栏是扫描器报的事实，不是结论。命中多不等于有毒（安全工具、规则库、示例脚本本来就会包含危险写法），命中少也不等于干净。它和你手上的凭据、文件、网络有什么关系，需要你自己看。

技能内容

Monitoring Setup Guide Skill

Produce a complete monitoring setup guide for a service — defining exactly what to measure, how to structure logs, how to configure alerts with actionable thresholds, and how to build dashboards that answer real operational questions. A good monitoring guide eliminates "we don't know what's happening in production" as a root cause category, and gives on-call engineers a single source of truth for what healthy looks like.

Required Inputs

Ask for these if not already provided:

Service name and description — what the service does and its role in the system
Tech stack — language, framework, and infrastructure (e.g. Go/gRPC on Kubernetes, Python/FastAPI on ECS)
Current monitoring tooling — Datadog, Prometheus + Grafana, CloudWatch, New Relic, Honeycomb, or none yet
Key user journeys — the 2–4 most important things a user or consumer does with the service (these drive what to alert on)
Existing alerts — paste any existing alert configurations or describe what's currently monitored

Output Format

Monitoring Setup Guide: [Service Name]

Team: [Team name] | Tech lead: [Name]

Stack: [Language/Framework] on [Infrastructure]

Monitoring platform: [Datadog / Prometheus+Grafana / CloudWatch / etc.]

Date: [Date] | Review cycle: Quarterly

1. Monitoring Philosophy

Good monitoring answers three questions:

Is the service healthy right now? (alerting)
Was it healthy in the past, and is it trending worse? (dashboards + SLO tracking)
Why did something fail? (logs + traces)

This guide defines the answers for [Service Name]. Every alert must be actionable — if an on-call engineer cannot take a specific action in response to the alert, the alert should not exist.

Key user journeys monitored:

Journey 1: [e.g. "User submits a payment — POST /charges, receives confirmation"]
Journey 2: [e.g. "User views transaction history — GET /transactions"]
Journey 3: [e.g. "Subscription renewal job runs — background worker processes billing events"]

2. The Four Golden Signals

Apply the four golden signals specifically to [Service Name]:

Latency

Latency measures how long requests take to complete. Track it separately for successful and failed requests — slow failures hide behind fast errors if you only measure aggregate latency.

|---|---|---|---|

Latency SLO targets:

|---|---|---|---|

| GET /api/v1/[resource] | < [50] ms | < [200] ms | < [500] ms |

| POST /api/v1/[resource] | < [100] ms | < [400] ms | < [1000] ms |

| GET /health | < [10] ms | < [20] ms | < [50] ms |

| [Background job name] | < [5] sec | < [15] sec | < [60] sec |

Traffic

Traffic measures demand on the system. Use it to detect unexpected spikes, traffic drops (which can indicate upstream failures), and to capacity-plan.

| Metric | Description | Source |

|---|---|---|

| [service].request.count | Requests per second | Application / load balancer |

| [service].request.count_by_endpoint | RPS broken down by endpoint | Application |

| [service].queue.messages_consumed_per_second | Consumer throughput | Queue consumer |

| [service].queue.depth | Messages waiting in queue | Queue metrics |

Traffic baselines (update after observing production for 2+ weeks):

|---|---|---|---|

Errors

Errors measure the fraction of requests that fail. Distinguish between client errors (4xx — caller is doing something wrong) and server errors (5xx — the service is broken).

| Metric | Description | Alert on? |

|---|---|---|

| [service].request.error_rate | 5xx errors / total requests | Yes — see alert rules |

| [service].request.client_error_rate | 4xx errors / total requests | Threshold alert — sudden spike may indicate API misuse |

| [service].dependency.error_rate | Errors calling downstream dependencies | Yes — upstream health signal |

| [service].queue.dlq_depth | Messages in dead-letter queue | Yes — indicates processing failures |

Saturation

Saturation measures how "full" the service is — how close to maximum capacity are the constrained resources.

|---|---|---|---|

3. Business Metrics

Beyond the golden signals, track metrics that measure whether the service is delivering business value. These matter for SLO reporting and product dashboards.

|---|---|---|---|

4. Log Strategy

Structured Logging Schema

All logs must be structured JSON. Do not emit unstructured text logs in production. Every log line must include the mandatory fields.

Mandatory fields (every log line):

{
  "timestamp": "2024-01-15T10:23:45.123Z",
  "level": "info",
  "service": "[service-name]",
  "version": "[git-sha-short]",
  "trace_id": "[uuid-from-request-context]",
  "span_id": "[span-uuid]",
  "request_id": "[uuid-per-request]",
  "message": "[human readable description]"
}

Request log (emit for every HTTP request):

{
  "timestamp": "...",
  "level": "info",
  "service": "[service-name]",
  "event": "http_request",
  "method": "POST",
  "path": "/api/v1/[resource]",
  "status_code": 201,
  "duration_ms": 45,
  "user_id": "[uuid — DO NOT log PII directly]",
  "request_id": "[uuid]",
  "trace_id": "[uuid]"
}

Error log (emit for every error with context):

{
  "timestamp": "...",
  "level": "error",
  "service": "[service-name]",
  "event": "error",
  "error_code": "[application-error-code]",
  "error_message": "[description — no sensitive data]",
  "stack_trace": "[stack trace]",
  "request_id": "[uuid]",
  "trace_id": "[uuid]",
  "context": {
    "[key]": "[relevant context without PII]"
  }
}

Log Levels — When to Use Each

| Level | Use when | Example |

|---|---|---|

| error | Something failed that requires attention — this should page on-call eventually | Database query failed, external API returned 5xx, required config missing |

| warn | Something unexpected happened but service is still functioning | Retry succeeded after failure, cache miss on expected hit, rate limit approaching |

| info | Significant business events and request lifecycle | Request received, payment processed, user authenticated, job started/completed |

| debug | Detailed diagnostic information — off in production by default | Query parameters, intermediate computation results, cache key lookups |

What NOT to Log

Never log:

Passwords, tokens, API keys, or secrets (even hashed)
Full credit card numbers or PAN data
Social security numbers or government IDs
Full names + dates of birth + contact info in the same log line (PII aggregation)
Request/response bodies in full (use field-level extraction instead)
Health check requests (too noisy — exclude GET /health from access logs)

5. Distributed Tracing Setup

Distributed tracing is mandatory for any service that calls other services. It enables root-cause analysis across service boundaries.

Instrumentation Checklist

[ ] Tracing library installed:
    - Go: go.opentelemetry.io/otel
    - Python: opentelemetry-sdk, opentelemetry-instrumentation
    - Node: @opentelemetry/sdk-node
    - Java: opentelemetry-java-instrumentation

[ ] Tracer initialized at service startup with service name and version

[ ] Trace context propagated via W3C Trace Context headers:
    traceparent: 00-[trace-id]-[span-id]-01
    tracestate: [optional vendor-specific]

[ ] Automatic instrumentation enabled for:
    [ ] Inbound HTTP/gRPC requests (creates root span)
    [ ] Outbound HTTP/gRPC calls (creates child spans)
    [ ] Database queries (creates child spans with sanitized query)
    [ ] Cache operations (Redis, Memcached)
    [ ] Message queue produce/consume

[ ] Custom spans added for:
    [ ] Key business operations ([e.g. payment processing, user lookup])
    [ ] Background jobs (each job execution = root span)
    [ ] Third-party API calls with custom attributes

[ ] Span attributes to capture on all spans:
    - user.id (if authenticated — no PII)
    - deployment.environment (production/staging)
    - service.version (git SHA)
    - [service-specific key attributes]

[ ] Trace exporter configured to: [Datadog / Jaeger / Tempo / OTLP endpoint]

[ ] Sampling rate configured:
    - Production: [1–10]% of requests (adjust based on volume and cost)
    - Always sample: errors, slow requests (>p99 threshold), and 100% of [critical endpoint]

Trace Instrumentation Examples

# Python — OpenTelemetry example
from opentelemetry import trace

tracer = trace.get_tracer("[service-name]")

def process_payment(payment_data):
    with tracer.start_as_current_span("process_payment") as span:
        span.set_attribute("payment.amount_cents", payment_data["amount"])
        span.set_attribute("payment.currency", payment_data["currency"])
        # Never: span.set_attribute("payment.card_number", ...)
        try:
            result = _do_process(payment_data)
            span.set_status(trace.StatusCode.OK)
            return result
        except PaymentError as e:
            span.set_status(trace.StatusCode.ERROR, str(e))
            span.record_exception(e)
            raise

6. Alert Rules Specification

Every alert must have: a name, a condition, a threshold, a severity, and a clear on-call action. Alerts without a clear action should not exist.

Alert Definitions

|---|---|---|---|---|

Alert Configuration Examples

# Prometheus / Grafana alerting rules (adapt for your platform)
groups:
  - name: [service-name]-alerts
    rules:

      - alert: [Service]HighErrorRate
        expr: |
          (
            sum(rate([service]_http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate([service]_http_requests_total[5m]))
          ) > 0.01
        for: 2m
        labels:
          severity: critical
          team: [team-name]
        annotations:
          summary: "High error rate on [Service Name]"
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 1%)"
          runbook_url: "[runbook link]"

      - alert: [Service]HighP99Latency
        expr: |
          histogram_quantile(0.99,
            sum(rate([service]_http_request_duration_seconds_bucket[5m])) by (le, endpoint)
          ) > [0.5]
        for: 3m
        labels:
          severity: warning
          team: [team-name]
        annotations:
          summary: "p99 latency elevated on [Service Name]"
          description: "p99 latency on {{ $labels.endpoint }} is {{ $value | humanizeDuration }}"
          runbook_url: "[runbook link]"

# Datadog monitor configuration (Python SDK or Terraform)
import datadog

datadog.initialize(api_key="[key]", app_key="[key]")

datadog.api.Monitor.create(
    type="metric alert",
    query=f"sum(last_5m):sum:{{service}}.http.errors{{service:[service-name]}} / sum:{{service}}.http.requests{{service:[service-name]}} > 0.01",
    name="[Service] High Error Rate",
    message="Error rate exceeded 1%. @pagerduty-[service-oncall]\n\nRunbook: [link]",
    tags=["service:[service-name]", "team:[team-name]"],
    options={
        "thresholds": {"critical": 0.01, "warning": 0.005},
        "notify_no_data": False,
        "evaluation_delay": 60,
    }
)

7. Dashboard Layout Specification

The primary service dashboard must answer "is the service healthy right now?" at a glance. Use this layout:

┌─────────────────────────────────────────────────────────────────────┐
│  [SERVICE NAME] — Service Health Dashboard           [Time range ▼] │
├───────────────┬───────────────┬───────────────┬─────────────────────┤
│  Error rate   │  p99 Latency  │  RPS (current)│  SLO budget remaining│
│  [BIG NUMBER] │  [BIG NUMBER] │  [BIG NUMBER] │  [BIG NUMBER / days] │
│  vs SLO: 0.1% │  vs SLO: 500ms│  vs avg: [N]  │  [Error budget gauge]│
├───────────────┴───────────────┴───────────────┴─────────────────────┤
│                   Error rate over time (24h)                        │
│  [Time series: 5xx rate line, SLO threshold line]                   │
├─────────────────────────────────┬───────────────────────────────────┤
│  Latency percentiles over time  │  Request throughput over time     │
│  [Lines: p50, p95, p99, p999]   │  [Bars: RPS by endpoint]          │
│  [SLO threshold horizontal line]│                                   │
├─────────────────────────────────┴───────────────────────────────────┤
│  Latency heatmap (all requests — shows distribution shape)          │
├─────────────────────────────────┬───────────────────────────────────┤
│  CPU utilisation over time      │  Memory utilisation over time     │
│  [All instances/pods — lines]   │  [All instances/pods — lines]     │
│  [Alert threshold: 80%]         │  [Alert threshold: 85%]           │
├─────────────────────────────────┴───────────────────────────────────┤
│  DB: connection pool utilisation│  DB: query latency (p99 per query)│
├─────────────────────────────────┴───────────────────────────────────┤
│  [Business metric 1 over time]  │  [Business metric 2 over time]    │
│  e.g. Payment success rate      │  e.g. Orders created/min          │
└─────────────────────────────────┴───────────────────────────────────┘

Second dashboard — Dependency Health:

┌─────────────────────────────────────────────────────────────────────┐
│  [SERVICE NAME] — Dependency Health                                 │
├─────────────────────────────────────────────────────────────────────┤
│  For each dependency: error rate | latency | current status         │
│  [Database]    [N]% errors | [N]ms p99 | ● Healthy / ⚠ Degraded    │
│  [Redis]       [N]% errors | [N]ms p99 | ● Healthy                 │
│  [External API][N]% errors | [N]ms p99 | ● Healthy                 │
├─────────────────────────────────────────────────────────────────────┤
│  Outbound call latency over time (one line per dependency)          │
├─────────────────────────────────────────────────────────────────────┤
│  Circuit breaker / fallback state (if implemented)                  │
└─────────────────────────────────────────────────────────────────────┘

8. Observability Debt Analysis

Honest assessment of what is missing today and what the priority to add it is:

|---|---|---|---|---|---|

Total observability debt: [N] items | Estimated effort: [N days]

Quality Checks

[ ] Every alert has a named on-call action — no alert says "investigate" without specifying what to investigate first
[ ] Alert thresholds are calibrated against production baselines, not set to default values from a template
[ ] Structured logging is implemented — no unstructured text log lines in production
[ ] PII is explicitly excluded from logs — a named engineer has verified this
[ ] Distributed tracing is propagating trace IDs across all service boundaries (verify with a test request)
[ ] The primary dashboard answers "is the service healthy?" in under 10 seconds — no hunting for the right panel
[ ] Business metrics are tracked alongside infrastructure metrics — not just four golden signals
[ ] Observability debt items have owners and dates — not just "would be nice to have"

Anti-Patterns

[ ] Do not create alerts without a specific on-call action — an alert that just says "investigate" trains engineers to ignore it
[ ] Do not set alert thresholds from a template without calibrating against production baselines — uncalibrated thresholds cause either alert fatigue or missed incidents
[ ] Do not log PII, tokens, or secrets — a logging standard is incomplete without an explicit list of what must never be logged
[ ] Do not measure only the four golden signals without adding at least one business metric alert — infrastructure health can be green while the business-critical path is silently failing
[ ] Do not deploy distributed tracing without verifying that trace IDs propagate across all service boundaries — partial tracing is worse than no tracing because it produces misleading incomplete traces

想直接用这个技能？

本站把开放许可（MIT / Apache 等）的技能按仓库打包整理到网盘，点一下转存到你自己的网盘，不用一个个从 GitHub 拉。许可未声明的技能只给原始仓库链接，不打包。

去原始仓库

它属于哪个仓库

仓库mohitagw15856/pm-claude-skills

星标★ 1,367

本站分层T1

该仓技能数3611

原文件路径plugins/pm-engineering/skills/monitoring-setup-guide/SKILL.md

同一个仓库里的其他技能

看这个仓库的全部 3611 个技能

同名技能的其他版本

有 3 个不同仓库或目录里都有叫 monitoring-setup-guide 的技能。它们内容并不相同，别混用：

mohitagw15856/pm-claude-skills — Write a monitoring setup guide for a service — defining what to measure, how to alert on i
mohitagw15856/pm-claude-skills — Write a monitoring setup guide for a service — defining what to measure, how to alert on i