kibana-anomaly-detection

Elastic ML Anomaly Detection

Expert process for ML anomaly detection: attribute incidents to entities, explain scores and model behavior, diagnose

job lifecycle failures, and manage jobs. Read anomaly results from POST /.ml-anomalies-*/_search (Serverless-safe)

and job/datafeed state from ML REST APIs. When the user embeds fixture evidence (influencer rows, job stats) in the

prompt, apply the judgment below directly — do not re-fetch fields already supplied.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the

elastic CLI is not installed, tell the user what it is needed for. Do

not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping,

GET /{index}/_settings/index.mode, POST /_query). The [Operations](#operations) table at the end of this document

maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API

directly.

<!-- end-partial: preamble -->

Mode selector

| User intent | Mode |

| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |

| "What broke?" / RCA / cross-job / blast radius / influencers / log categories | Investigate |

| "Why score high/low?" / renormalization / model bounds / forecasts | Explain |

| Missing docs / memory limit / datafeed stopped / lifecycle / calendars | Troubleshoot |

| Create a job / configure a datafeed / start analysis / retrieve results | Manage |

| Security framing (attack chains, MITRE, exfil) | Investigate + [references/security-anomaly-expert.md](references/security-anomaly-expert.md) |

| Observability/SRE framing (degradation, capacity, deployment regression) | Investigate + [references/observability-anomaly-expert.md](references/observability-anomaly-expert.md) |

When a question spans modes: Investigate → Explain → Troubleshoot. Finish one mode before blending logic.

> Serverless note: Legacy /_ml/anomaly_detectors/{job_id}/results/* endpoints return HTTP 410 in Serverless.

> Always query .ml-anomalies- via POST /.ml-anomalies-/_search with result_type filters.

Score quick reference

• record_score bands: >75 critical · 50–75 warning · 25–50 minor · <25 informational
• multi_bucket_impact ≥ 3 → sustained shift (not a transient spike)
• initial_record_score >> record_score → renormalization (model saw worse anomalies later)
• actual << typical with count/low_count/low_mean → absence/outage, not just a low value
• Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity

> Full score definitions, renormalization mechanics, and anomaly_score_explanation components:

> [references/score-reference.md](references/score-reference.md).

Core concepts

Treat .ml-anomalies-* as layered result types via result_type in search queries:

| result_type | Scope | Key fields |

| --------------------- | --------------- | ---------------------------------------------------------------------------------------- |

| bucket | Time window | anomaly_score, initial_anomaly_score, timestamp |

| record | Detector row | record_score, initial_record_score, actual, typical, anomaly_score_explanation |

| influencer | Entity × bucket | influencer_field_name, influencer_field_value, influencer_score |

| model_plot | Bounds | model_lower, model_upper, actual |

| category_definition | Log patterns | category_id, terms, regex, examples |

Read scores this way:

• anomaly_score / record_score = current normalized values (move as the model sees new extremes).
• initial_anomaly_score / initial_record_score = immutable snapshots from detection time.
• influencer_score ranks entity responsibility within a bucket — the highest score is the primary suspect, not the

bucket-level anomaly_score alone.

• Map entities via partition_field_value / by_field_value / over_field_value.
• Read multi_bucket_impact (-5 to +5) to separate single-bucket spikes from sustained trends.

---

Mode: Investigate — RCA

When: "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.

Process

1. Discover jobs. Call GET /_ml/anomaly_detectors when the job ID is unknown. Call

GET /_ml/anomaly_detectors/{job_id} and GET /_ml/datafeeds/datafeed-{job_id} to learn source indices, entity

fields (by_field_name, over_field_name, partition_field_name), and bucket_span. The decision: identify the

related job group — jobs sharing a datafeed index or entity field monitor the same system from different angles.

1. Scope the incident window. Call POST /.ml-anomalies-*/_search with result_type: bucket, a time range, and

optional minimum anomaly_score. The decision: fix the incident start/end and count how many jobs co-fire in that

window. Low scores across many jobs simultaneously often indicate a systemic root cause.

1. Attribute to entities (critical for RCA). For the anomalous bucket timestamp, call

POST /.ml-anomalies-*/_search with result_type: influencer, the job ID(s), and the bucket time range. Sort by

influencer_score descending. The decision: name the entity with the highest influencer_score as the

likely cause — it ranks how unusual each entity is in that bucket. Do not restate only the bucket anomaly_score

without attributing responsibility. Recommend drilling into that entity's records next.

1. Cross-job confirmation. Re-query influencers (or bucket records) across related job IDs for the same entity

values and time window. Entities anomalous in 2+ jobs are prime suspects (resource fault or systemic failure);

single-job entities are often downstream victims. See

[references/protocols/investigation.md](references/protocols/investigation.md).

1. Drill into records. Call POST /.ml-anomalies-*/_search with result_type: record, exact job ID, entity filters

(partition_field_value, by_field_value), and low minimum record_score (25 or lower). Read

multi_bucket_impact ≥ 3 as sustained behavioral shift. Read actual vs typical for fault class (spike vs

absence/outage).

1. Confirm with source evidence. Call POST /{index}/_search on the datafeed source index for the suspect entity

and time window. Raw source documents are ground truth — never close an RCA without them.

1. Synthesize. Report: **root cause entity · affected jobs · temporal progression · fault class · severity ·

recommended actions**. Worked walkthrough: [references/worked-example.md](references/worked-example.md). Query

templates: [references/investigation-queries.md](references/investigation-queries.md).

Rules

1. Rank by influencer_score, not anomaly_score, for "which entity?" — bucket score is aggregate; influencer

score attributes cause.

1. Multi-job entities are prime suspects; single-job entities are usually victims.
2. Earliest anomaly timestamp wins — reconstruct chronology from record timestamps across jobs.
3. multi_bucket_impact ≥ 3 = sustained behavioral shift, weight higher than transient spikes.
4. Use low score thresholds (25 or lower) for influencer/record queries — high thresholds miss correlated entities.
5. Never close an RCA without source evidence from the datafeed index.

---

Mode: Explain — Score / model behavior

When: "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?".

Process

1. Decide fetch vs interpret. If the user supplies a record with record_score, initial_record_score, actual,

and typical, interpret directly. Otherwise load config with GET /_ml/anomaly_detectors/{job_id} and records with

POST /.ml-anomalies-*/_search (result_type: record).

1. Always show both initial_record_score and record_score. The gap is the renormalization story. Large positive

drift (initial_record_score >> record_score) means a later, more extreme anomaly rescale this record downward —

expected healthy behavior, not a broken model.

1. Classify the pattern before speculating.

| Pattern | Interpretation |

| ------------------------------------------------------------ | ----------------------------------------------------------------- |

| initial_record_score >> record_score | Renormalization — explain before suggesting config changes |

| actual << typical with low_count/count/low_mean | Absence/outage anomaly — investigate the outage, not score tuning |

| high_variance_penalty: true in anomaly_score_explanation | Noisy metric — wide bounds absorbed the spike |

| incomplete_bucket_penalty: true | Ingest lag or sparse bucket — score legitimately reduced |

Only cite anomaly_score_explanation factors present in the record.

1. Quantify renormalization (optional). Re-query records sorted by timestamp; compute

score_drift = initial_record_score - record_score and flag large drift.

1. Add visual context when needed. If model_plot_config.enabled, query result_type: model_plot and compare

actual to model_lower/model_upper. For categorization jobs, query result_type: category_definition.

1. Check job health when scores look wrong persistently. Call GET /_ml/anomaly_detectors/{job_id}/_stats —

model_size_stats.memory_status of hard_limit corrupts learning and can invalidate scores. Escalate to

Troubleshoot mode.

anomaly_score_explanation components

| Component | Effect | What it means |

| -------------------------------- | ------- | ------------------------------------------------------------ |

| anomaly_length | ↑ score | More consecutive anomalous buckets |

| single_bucket_impact | ↑ score | Lower probability → higher impact |

| multi_bucket_impact | ↑ score | Sustained pattern contribution |

| anomaly_characteristics_impact | ↑ score | Mean shift vs. variance change |

| high_variance_penalty | ↓ score | Noisy data → wide bounds → anomaly less surprising |

| incomplete_bucket_penalty | ↓ score | Bucket has less data than expected (ingest lag, sparse data) |

Rules

1. Explain renormalization before diagnosing config — score drift is the most common "score dropped" cause.
2. actual << typical with count/low_count is an absence anomaly — distinguish outages from value spikes.
3. Weekly seasonality needs ≥3 weeks of training data — flag young jobs as the cause.
4. Detector function direction matters — see

[references/anomaly-detection-functions.md](references/anomaly-detection-functions.md).

---

Mode: Troubleshoot — Job lifecycle

When: "missing documents", "datafeed stopped", hard_limit, "results look wrong", lifecycle changes.

Process

1. Load job and datafeed state. Call GET /_ml/anomaly_detectors/{job_id}/_stats and

GET /_ml/datafeeds/datafeed-{job_id}/_stats. Read state, data_counts, model_size_stats, and datafeed

state. If the user embeds stats JSON, diagnose from memory_status and datafeed state directly.

1. Diagnose memory status first (critical). Inspect model_size_stats:

| Field | Meaning |

| -------------------------- | ----------------------------------------------------------- |

| memory_status | ok / soft_limit (pruning) / hard_limit (critical) |

| model_bytes | Current memory used |

| model_bytes_memory_limit | Configured model_memory_limit |

When memory_status is hard_limit and model_bytes equals model_bytes_memory_limit, the model hit its

memory ceiling — it stops learning new entities and results degrade or stop. A stopped datafeed is often a

symptom, not the root cause. Do not recommend only restarting the datafeed — that alone does not clear a hard

limit.

1. Remediate hard_limit. The fix is to raise model_memory_limit (via job update) and/or reduce model size

by lowering cardinality (fewer partition/by/over field values, split into multiple jobs). Raising the limit requires

the lifecycle sequence below (stop datafeed → close job → update → open → start). Optionally call

POST /_ml/anomaly_detectors/_estimate_model_memory to size the new limit from source cardinality.

1. Diagnose missing documents / query timing. After memory is healthy, inspect datafeed query_delay and

delayed_data_check_config via GET /_ml/datafeeds/datafeed-{job_id}. Search .ml-annotations-* for delayed-data

events. Set query_delay to P95 ingest latency + buffer (default 60s–120s).

1. Read job messages. Search .ml-notifications-* for the job ID when errors are unclear.

1. Recover corrupted model state. Call POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert

to revert to a known-good snapshot when the model was corrupted during hard_limit.

Lifecycle for config changes (memory limit, query_delay)

Apply in order — skipping steps causes rejected updates:

1. POST /_ml/datafeeds/datafeed-{job_id}/_stop
2. POST /_ml/anomaly_detectors/{job_id}/_close
3. POST /_ml/anomaly_detectors/{job_id}/_update (memory limit) and/or POST /_ml/datafeeds/datafeed-{job_id}/_update

(query_delay)

1. POST /_ml/anomaly_detectors/{job_id}/_open
2. POST /_ml/datafeeds/datafeed-{job_id}/_start

Preview changes with POST /_ml/datafeeds/datafeed-{job_id}/_preview before restarting.

> hard_limit corrupts model state and causes downstream missing-doc false alarms. **Fix memory before fixing

> query_delay.** Full troubleshooting detail:

> [references/troubleshooting-reference.md](references/troubleshooting-reference.md).

Rules

1. Ground lifecycle diagnosis in memory_status — not generic "restart it" advice.
2. Fix memory before query_delay — hard_limit invalidates downstream diagnostics.
3. Stop datafeed → close job → update → open → start for any memory or datafeed config change.
4. Do not delete the job as first remediation for hard_limit — raise limit and/or reduce cardinality.

---

Mode: Manage — Create / configure jobs

When: "set up a job", "create an ML detector", "monitor X over time".

For the full create/open/start lifecycle, prefer the elasticsearch-anomaly-detection skill. This mode summarizes the

sequence and detector selection:

1. Verify target index. Call GET /{index}/_mapping — confirm time field and detector fields exist.
2. Create job. Call PUT /_ml/anomaly_detectors/{job_id} with analysis_config (detectors, bucket_span,

influencers) and data_description.time_field.

1. Create datafeed. Call PUT /_ml/datafeeds/datafeed-{job_id} with indices, query, and query_delay.
2. Open and start. Call POST /_ml/anomaly_detectors/{job_id}/_open, then

POST /_ml/datafeeds/datafeed-{job_id}/_start.

1. Confirm. Call GET /_ml/anomaly_detectors/{job_id}/_stats and GET /_ml/datafeeds/datafeed-{job_id}/_stats.

Choose detector functions from user intent — see

[references/anomaly-detection-functions.md](references/anomaly-detection-functions.md). Worked JSON bodies:

[references/job-creation-recipes.md](references/job-creation-recipes.md).

Rules

1. Create job before datafeed. Open job before starting datafeed.
2. query_delay = P95 ingest latency + buffer (60s–120s safe default).
3. by_field_name vs over_field_name: by compares entity to its own history; over compares to peer group.
4. Forecasts require non-population jobs — jobs with over_field_name cannot be forecasted.

---

Examples

RCA: "Something caused a spike in checkout latency — which entity?" → Query influencers for the bucket → web-07

has highest influencer_score (91.5) vs 22.0 and 8.4 → name web-07 as likely cause → recommend drilling into its

records — do not answer with only bucket anomaly_score 88.

Score drop: "Score went from 90 to 55 — did the model change?" → Compare initial_record_score vs record_score →

explain renormalization if drift is large.

Memory limit: "Job shows hard_limit and datafeed stopped." → Diagnose

model_size_stats.memory_status = hard_limit → raise model_memory_limit via close/update/open lifecycle and/or reduce

cardinality — not "just restart the datafeed".

New job: "Detect unusual error rates per host." → high_count with by_field_name: host.keyword →

create/open/start sequence.

---

Guidelines

1. Pick a mode first. Don't blend RCA logic with score-explanation logic in one response.
2. For "which entity?" rank influencer_score, not bucket anomaly_score.
3. For lifecycle failures read memory_status before recommending datafeed restarts.
4. Show initial_record_score alongside record_score — the gap tells the renormalization story.
5. Fix memory before query_delay. Hard_limit invalidates downstream diagnostics.
6. Confirm RCAs with source evidence from the datafeed index.

Operations

| HTTP API (shorthand) | elastic CLI command |

| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |

| GET /{index}/_mapping | elastic es indices get-mapping --index '<index>' |

| POST /{index}/_search | elastic es search --index '<index>' --input-file '<search-body.json>' |

| GET /_ml/anomaly_detectors | elastic es ml get-jobs |

| GET /_ml/anomaly_detectors/{job_id} | elastic es ml get-jobs --job-id '<job_id>' |

| GET /_ml/anomaly_detectors/{job_id}/_stats | elastic es ml get-job-stats --job-id '<job_id>' |

| GET /_ml/datafeeds/datafeed-{job_id} | elastic es ml get-datafeeds --datafeed-id 'datafeed-<job_id>' |

| GET /_ml/datafeeds/datafeed-{job_id}/_stats | elastic es ml get-datafeed-stats --datafeed-id 'datafeed-<job_id>' |

| POST /.ml-anomalies-/_search | elastic es search --index '.ml-anomalies-' --input-file '<search-body.json>' |

| POST /.ml-annotations-/_search | elastic es search --index '.ml-annotations-' --input-file '<search-body.json>' |

| POST /.ml-notifications-/_search | elastic es search --index '.ml-notifications-' --input-file '<search-body.json>' |

| POST /_ml/anomaly_detectors/_estimate_model_memory | elastic es ml estimate-model-memory --analysis-config '<json>' |

| PUT /_ml/anomaly_detectors/{job_id} | elastic es ml put-job --job-id '<job_id>' --input-file '<job-body.json>' |

| PUT /_ml/datafeeds/datafeed-{job_id} | elastic es ml put-datafeed --datafeed-id 'datafeed-<job_id>' --input-file '<datafeed-body.json>' |

| POST /_ml/anomaly_detectors/{job_id}/_open | elastic es ml open-job --job-id '<job_id>' |

| POST /_ml/anomaly_detectors/{job_id}/_close | elastic es ml close-job --job-id '<job_id>' |

| POST /_ml/anomaly_detectors/{job_id}/_update | elastic es ml update-job --job-id '<job_id>' --analysis-limits '<json>' |

| POST /_ml/datafeeds/datafeed-{job_id}/_update | elastic es ml update-datafeed --datafeed-id 'datafeed-<job_id>' --input-file '<update-body.json>' |

| POST /_ml/datafeeds/datafeed-{job_id}/_start | elastic es ml start-datafeed --datafeed-id 'datafeed-<job_id>' |

| POST /_ml/datafeeds/datafeed-{job_id}/_stop | elastic es ml stop-datafeed --datafeed-id 'datafeed-<job_id>' |

| POST /_ml/datafeeds/datafeed-{job_id}/_preview | elastic es ml preview-datafeed --datafeed-id 'datafeed-<job_id>' |

| POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert | elastic es ml revert-model-snapshot --job-id '<job_id>' --snapshot-id '<snapshot_id>' |

Search body shapes for each result_type and troubleshooting queries are documented in

[references/investigation-queries.md](references/investigation-queries.md) and

[references/troubleshooting-reference.md](references/troubleshooting-reference.md).