load-testing-plan

Load Testing Plan Skill

Produce a complete load and performance testing plan for a service — covering test objectives, scenario definitions, tooling configuration, success thresholds, and CI integration. A good load testing plan eliminates ambiguity about what "performance is acceptable" means, so engineers can run tests and get a pass/fail answer without having to interpret raw numbers themselves.

Required Inputs

Ask for these if not already provided:

• Service name and key endpoints — which endpoints are under test (path, method, typical request/response shape)
• Current traffic baseline — current requests/sec, p50/p99 latency, error rate under normal load
• Peak traffic expectations — expected peak RPS (e.g. 10× baseline for flash sales, or seasonality peak)
• SLO targets — latency SLOs (p99 < X ms), error rate SLO (< Y%), availability target
• Preferred testing tool — k6, Locust, JMeter, Gatling, or no preference
• Test environment availability — dedicated load test environment, staging, or production (with traffic shaping)

Output Format

---

Load Testing Plan: [Service Name]

Author: [Name] | Team: [Team name]

Date: [Date] | Review cycle: Before each major release and quarterly

Testing tool: [k6 / Locust / JMeter / Gatling]

Test environment: [Environment name and URL]

---

1. Objectives and Scope

What we are testing: [Service name] handles [describe function — e.g. "user authentication requests from the mobile and web clients"]. This plan validates that the service meets its SLOs under expected and elevated traffic conditions.

In scope:

• [Endpoint 1: METHOD /path — description]
• [Endpoint 2: METHOD /path — description]
• [Endpoint 3: METHOD /path — description]

Out of scope:

• [Any endpoints explicitly excluded and why — e.g. "admin APIs — low traffic, excluded from load test"]
• [Third-party integrations that cannot be load-tested — mock them instead]

---

2. Performance Targets (Success Criteria)

Every scenario has explicit pass/fail thresholds. A test run FAILS if any threshold is breached.

| Metric | Baseline scenario | Stress scenario | Spike scenario | Soak scenario |

|---|---|---|---|---|

| p50 latency | < [X] ms | < [X × 1.5] ms | < [X × 2] ms | < [X] ms |

| p95 latency | < [Y] ms | < [Y × 1.5] ms | < [Y × 2] ms | < [Y] ms |

| p99 latency | < [Z] ms | < [Z × 2] ms | < [Z × 3] ms | < [Z] ms |

| Error rate | < [0.1]% | < [1]% | < [2]% | < [0.1]% |

| Throughput | ≥ [N] RPS | ≥ [N × 3] RPS | N/A | ≥ [N] RPS |

| Failed requests | 0 (5xx) | < [threshold] | < [threshold] | 0 (5xx) |

SLO reference: These thresholds are derived from the service SLOs — p99 < [Z ms], error rate < [0.1]%, availability [99.9]%.

---

3. Traffic Model

Baseline traffic (current production):

• Average RPS: [N] req/sec
• Peak RPS (observed): [N] req/sec
• Request distribution by endpoint:
• [Endpoint 1]: [X]% of traffic
• [Endpoint 2]: [Y]% of traffic
• [Endpoint 3]: [Z]% of traffic

Simulated user behaviour:

• Think time between requests: [X–Y] seconds (randomised)
• Session duration: [N] minutes average
• Authenticated vs anonymous ratio: [X]%/[Y]%
• Geographic distribution: [Region 1 X]%, [Region 2 Y]%

---

4. Test Scenarios

Scenario 1: Baseline (Steady-State)

Purpose: Confirm the service performs acceptably under normal production load.

Duration: 10 minutes

Load profile: Ramp to [N] RPS over 2 minutes, hold for 8 minutes.

Concurrency: [N] virtual users

Pass criteria: All thresholds in the Baseline column of the targets table above.

---

Scenario 2: Stress Test

Purpose: Find the breaking point — how much load can the service handle before SLOs are breached?

Duration: 20–30 minutes

Load profile: Ramp from [N] RPS (baseline) to [N × 5] RPS in 5-minute steps. Hold each step for 5 minutes. Stop at first SLO breach.

Concurrency: Scales with RPS target

What to record:

• RPS at which p99 latency first exceeds SLO
• RPS at which error rate first exceeds SLO
• Whether the service recovers when load drops back to baseline

---

Scenario 3: Spike Test

Purpose: Simulate a sudden traffic surge (flash sale, viral event, bot attack).

Duration: 15 minutes

Load profile: Hold at [N] RPS (baseline) for 3 minutes, spike to [N × 10] RPS instantly, hold for 5 minutes, drop back to baseline for 7 minutes.

What to record:

• Latency during spike and recovery
• Whether the service sheds load gracefully (rate limiting, queue depth)
• Time to recover to baseline latency after spike ends

---

Scenario 4: Soak / Endurance Test

Purpose: Detect memory leaks, connection pool exhaustion, and slow degradation over time.

Duration: 4–8 hours (run overnight)

Load profile: Steady [N × 1.5] RPS (50% above baseline) for entire duration.

What to watch:

• Memory usage trend over time (should not grow unboundedly)
• Error rate trend (should be flat, not creeping up)
• GC pause frequency (JVM/Go services)
• Database connection pool utilisation
• p99 latency trend (should not creep up over hours)

---

5. Test Environment Requirements

Infrastructure

| Component | Requirement | Notes |

|---|---|---|

| Service under test | Isolated from production | [N] replicas, matching prod resource limits |

| Database | Separate instance with production-scale data | Seed script in section 7 |

| Cache (Redis/Memcached) | Empty at test start | Ensures cold-start conditions are tested |

| Load generator | Separate from service under test | [N] vCPUs, [N] GB RAM minimum |

| Network | Low-latency path to service | Do not run generator on same host |

Data Seeding

Before every test run, ensure the environment has:

# Seed test users (needed for authenticated endpoint tests)
[seed command or script path — e.g. python scripts/seed_load_test_users.py --count 10000]

# Seed test data for read endpoints
[seed command — e.g. ./scripts/seed_products.sh --count 50000]

# Verify seed completed
[verification command — e.g. psql $DB_URL -c "SELECT COUNT(*) FROM users WHERE load_test=true"]

Test data rules:

• Never use real production user data in load tests
• Tag all test-generated records with load_test=true for easy cleanup
• Run cleanup after each test: [cleanup command]

---

6. Tooling Setup

k6 Script Skeleton

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

// Custom metrics
const errorRate = new Rate('error_rate');
const endpointLatency = new Trend('endpoint_latency', true);

// Test configuration — override per scenario
export const options = {
  scenarios: {
    baseline: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '2m', target: [BASELINE_VUS] },
        { duration: '8m', target: [BASELINE_VUS] },
        { duration: '1m', target: 0 },
      ],
    },
  },
  thresholds: {
    http_req_duration: [
      'p(95)<[Y_MS]',
      'p(99)<[Z_MS]',
    ],
    error_rate: ['rate<0.01'],
    http_req_failed: ['rate<0.01'],
  },
};

// Auth helper — get token once per VU
export function setup() {
  const loginRes = http.post('[BASE_URL]/auth/login', JSON.stringify({
    username: `load_test_user_${Math.floor(Math.random() * 10000)}@example.com`,
    password: __ENV.LOAD_TEST_PASSWORD,
  }), { headers: { 'Content-Type': 'application/json' } });

  check(loginRes, { 'login ok': (r) => r.status === 200 });
  return { token: loginRes.json('access_token') };
}

export default function (data) {
  const headers = {
    Authorization: `Bearer ${data.token}`,
    'Content-Type': 'application/json',
  };

  // Endpoint 1: [Description]
  const res1 = http.get('[BASE_URL]/[endpoint-1]', { headers });
  check(res1, {
    '[endpoint-1] status 200': (r) => r.status === 200,
    '[endpoint-1] latency < [X]ms': (r) => r.timings.duration < [X],
  });
  errorRate.add(res1.status >= 400);
  endpointLatency.add(res1.timings.duration, { endpoint: '[endpoint-1]' });

  sleep(Math.random() * [THINK_TIME_MAX] + [THINK_TIME_MIN]);

  // Endpoint 2: [Description]
  const res2 = http.post('[BASE_URL]/[endpoint-2]',
    JSON.stringify({ [key]: '[value]' }),
    { headers }
  );
  check(res2, {
    '[endpoint-2] status 201': (r) => r.status === 201,
  });
  errorRate.add(res2.status >= 400);
}

Locust Script Skeleton (alternative)

from locust import HttpUser, task, between
import random

class [ServiceName]User(HttpUser):
    wait_time = between([THINK_TIME_MIN], [THINK_TIME_MAX])
    token = None

    def on_start(self):
        """Called once per simulated user — authenticate."""
        user_id = random.randint(1, 10000)
        response = self.client.post("/auth/login", json={
            "username": f"load_test_user_{user_id}@example.com",
            "password": "[LOAD_TEST_PASSWORD]",
        })
        self.token = response.json()["access_token"]
        self.headers = {"Authorization": f"Bearer {self.token}"}

    @task([WEIGHT_1])  # Weight = relative frequency
    def [endpoint_1_task](self):
        """[Endpoint 1 description]"""
        with self.client.get(
            "/[endpoint-1]",
            headers=self.headers,
            catch_response=True
        ) as response:
            if response.elapsed.total_seconds() > [LATENCY_THRESHOLD]:
                response.failure(f"Too slow: {response.elapsed.total_seconds()}s")

    @task([WEIGHT_2])
    def [endpoint_2_task](self):
        """[Endpoint 2 description]"""
        self.client.post(
            "/[endpoint-2]",
            json={"[key]": "[value]"},
            headers=self.headers,
        )

Running Tests

# k6 — run baseline scenario
k6 run --env BASE_URL=https://[test-env-url] scripts/load_test.js

# k6 — run stress scenario with output to InfluxDB
k6 run --out influxdb=http://[influxdb-host]:8086/k6 \
  --env SCENARIO=stress \
  scripts/load_test.js

# Locust — headless run
locust -f locustfile.py \
  --headless \
  --users [N] \
  --spawn-rate [N] \
  --run-time 10m \
  --host https://[test-env-url] \
  --csv=results/[run-id]

# Locust — web UI (interactive)
locust -f locustfile.py --host https://[test-env-url]

---

7. Metrics to Capture

Capture all of the following during every test run. Missing any of these makes result comparison unreliable.

| Metric | Source | Why it matters |

|---|---|---|

| p50, p95, p99, p999 latency per endpoint | Load tool | SLO validation |

| Error rate (4xx, 5xx) per endpoint | Load tool | SLO validation |

| Requests/sec (throughput) | Load tool | Capacity baseline |

| CPU utilisation (%) | Infra monitoring | Saturation signal |

| Memory utilisation (%) | Infra monitoring | Leak detection |

| GC pause time / frequency | JVM/Go metrics | Latency spike root cause |

| DB connection pool: active/idle/waiting | DB metrics | Pool exhaustion detection |

| DB query latency (p99) | DB metrics | Downstream bottleneck |

| Cache hit rate | Cache metrics | Miss storm detection |

| Pod/instance count (if autoscaling) | Infra | Scaling behaviour |

| Network in/out bytes | Infra | Bandwidth saturation |

---

8. Result Analysis Framework

After each test run, work through this analysis in order:

Step 1 — Pass/fail check

Compare all captured metrics against the thresholds in Section 2. Record pass/fail per scenario.

Step 2 — Latency distribution

Plot the full latency histogram, not just percentiles. A bimodal distribution (two humps) indicates two distinct code paths — investigate the slow hump.

Step 3 — Error correlation

If errors occurred, correlate them with:

• Time of occurrence (was it during ramp-up, steady state, or spike?)
• Specific endpoint (is it one endpoint or all?)
• Infrastructure events (CPU spike, OOM, DB connection exhaustion?)

Step 4 — Saturation analysis

Graph CPU, memory, and connection pool over time. If any resource reached 80%+ of capacity, it is a candidate bottleneck — even if SLOs passed this run.

Step 5 — Compare to baseline run

Every run should be compared to the previous run. A 10% regression in p99 latency warrants investigation even if it is still within SLO.

Regression classification:

| Change | Classification | Action |

|---|---|---|

| p99 within 5% of previous run | Green — no regression | No action |

| p99 5–15% worse than previous | Yellow — watch | Investigate before next release |

| p99 >15% worse than previous | Red — regression | Block release, file ticket |

| Error rate increased vs previous | Red — regression | Block release |

| SLO threshold breached | Critical | Block release, page on-call |

---

9. CI Integration

Add load tests as a gated step in the release pipeline. Run the baseline scenario on every release candidate; run all scenarios weekly.

# Example: GitHub Actions step (adapt for your CI platform)
load-test:
  runs-on: ubuntu-latest
  needs: [deploy-staging]
  if: github.ref == 'refs/heads/main'
  steps:
    - uses: actions/checkout@v3

    - name: Install k6
      run: |
        curl -s https://dl.k6.io/key.gpg | sudo apt-key add -
        echo "deb https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
        sudo apt-get update && sudo apt-get install k6

    - name: Seed test data
      run: [seed command]

    - name: Run baseline load test
      run: |
        k6 run \
          --env BASE_URL=${{ secrets.LOAD_TEST_ENV_URL }} \
          --out json=results.json \
          scripts/load_test.js
      env:
        LOAD_TEST_ENV_URL: ${{ secrets.LOAD_TEST_ENV_URL }}

    - name: Check thresholds
      run: |
        # k6 exits with non-zero if any threshold fails — this step fails the build
        echo "k6 threshold check complete"

    - name: Upload results
      uses: actions/upload-artifact@v3
      if: always()
      with:
        name: load-test-results-${{ github.run_id }}
        path: results.json

    - name: Cleanup test data
      if: always()
      run: [cleanup command]

CI gates summary:

• Baseline scenario runs on every release to staging
• Full scenario suite (stress, spike, soak) runs weekly on a schedule
• Any threshold failure blocks promotion to production
• Results are archived for trend analysis

---

Quality Checks

• [ ] All key endpoints are covered by at least one test scenario — no production endpoint is untested
• [ ] Thresholds are derived from actual SLO targets, not guesses
• [ ] Test data seeding is scripted and reproducible — tests do not rely on pre-existing environment state
• [ ] The load generator runs on separate infrastructure from the service under test
• [ ] CI integration blocks promotion on threshold failure — not just records results
• [ ] Soak test has been run at least once to establish a memory and connection pool baseline
• [ ] Results comparison to previous run is part of the analysis — not just absolute pass/fail

Anti-Patterns

• [ ] Do not set thresholds without grounding them in actual SLO targets or production baselines — arbitrary numbers produce meaningless pass/fail results
• [ ] Do not run the load generator on the same host as the service under test — this contaminates both the test results and the service metrics
• [ ] Do not use production user data in load test seeding — all test data must be synthetic, tagged, and cleaned up after each run
• [ ] Do not skip the soak test on first deployment — only a soak test reveals slow memory leaks and connection pool exhaustion that short tests miss
• [ ] Do not treat a passing baseline test as evidence the service handles spikes — baseline, stress, spike, and soak scenarios test fundamentally different failure modes