observability-onboarding

Observability Onboarding

Instrument an application with the Elastic Distribution of OpenTelemetry (EDOT) and prove the telemetry arrived. The

instrumentation change is only half the job: an application can be configured perfectly and still emit nothing, so this

skill ends by querying Elasticsearch for the service's data rather than declaring success from a config diff.

The scope is application instrumentation with the EDOT SDKs. Deploying an EDOT Collector, collecting logs from files or

infrastructure, and onboarding data through Elastic Agent, Fleet, or integration packages are separate ingest paths that

this skill does not cover.

Once telemetry is flowing, use the observability-sre-triage skill to assess service health,

observability-k8s-investigation for Kubernetes-layer failures, observability-service-reliability to define SLOs

and alerts on the new signals, and observability-llm-obs for GenAI and agentic workloads.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the

elastic CLI is not installed, tell the user what it is needed for. Do

not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping,

GET /{index}/_settings/index.mode, POST /_query). The [Operations](#operations) table at the end of this document

maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API

directly.

<!-- end-partial: preamble -->

Analysis without cluster access

The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the

evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason

from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific

query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is

useful without a cluster; one that only asks for setup is not.

Only the verification step in this skill talks to Elasticsearch; it runs through POST /_query. Everything else edits

application code and configuration in the user's workspace.

Jobs to be done

• Decide whether a service needs fresh instrumentation or a migration off the classic Elastic APM agent
• Apply the EDOT configuration for the service's language and runtime
• Point telemetry at the correct OTLP destination and reject APM Server URLs
• Verify that traces, metrics, and logs land in Elasticsearch under the expected service.name
• Diagnose an instrumented service that is reporting nothing, or reporting from the wrong agent

Output discipline

Applies to every response produced under this skill.

• Make the edit. When the language, the current agent, and the destination are known, apply the change. Do not

narrate a plan and ask for approval on each file; ask only when a decision genuinely cannot be inferred.

• Do not dump documentation. Link the relevant page and state the two or three rules that apply to this service.

Pasting a whole setup guide into the reply is a defect.

• Verify before reporting success. "Configured correctly" is not the finding; "telemetry is arriving" is. If

verification returns no rows, say the service is not reporting and give the most likely cause.

• Do not speculate past the evidence. If the verification query is empty, that is one fact with several possible

causes — name the likeliest and how to distinguish it, rather than asserting one.

• End on the result. No trailing offers such as "want me to set up dashboards next?". Follow-up work belongs in a

recommendations list, phrased as a recommendation.

Process: route the request

1. Identify the language and runtime. Read the project's dependency manifest — pom.xml or build.gradle,

requirements.txt or pyproject.toml, *.csproj, composer.json. The decision: which language reference to open.

Also note how the process starts (a Dockerfile ENTRYPOINT, a Kubernetes pod spec, a systemd unit), because that is

where the agent attaches and where the environment variables must be set.

| Runtime | Reference | How EDOT attaches |

| ------- | -------------------------------------------- | ---------------------------------------------------------- |

| Java | [references/java.md](references/java.md) | -javaagent: flag or JAVA_TOOL_OPTIONS |

| Python | [references/python.md](references/python.md) | opentelemetry-instrument entrypoint wrapper |

| .NET | [references/dotnet.md](references/dotnet.md) | builder.AddElasticOpenTelemetry() in startup |

| PHP | [references/php.md](references/php.md) | native extension via OS package, then full process restart |

For a language not listed, follow the same shape — the required configuration and the following verification step are

language-independent — and use the upstream EDOT SDK documentation for the attach mechanism.

1. Detect a classic Elastic APM agent. The decision: instrument path or migrate path. Each language reference lists

the markers; the common ones are ELASTIC_APM_* environment variables and a language-specific Elastic APM package.

If any marker is present, take the migrate path — do not layer EDOT on top.

1. Resolve the telemetry destination. The decision: which URL goes in OTEL_EXPORTER_OTLP_ENDPOINT. Two valid

answers, and one common wrong one:

• The Elastic managed OTLP endpoint for the deployment or Serverless project. Correct default.
• An EDOT Collector the user already runs, when telemetry must be enriched, sampled, or fanned out before it

reaches Elastic.

• Never an APM Server URL. If the candidate contains apm-server, ends in port 8200, or has an

/intake/v2/events path, it is the classic ingest endpoint and EDOT cannot use it. On the migrate path this is the

single most common mistake — the old ELASTIC_APM_SERVER_URL value must not be carried over.

1. Apply the shared configuration below, then the language reference. The shared rules are the same for every

runtime; the reference covers only what is language-specific.

1. Verify. Do not report success from the configuration alone — run the verification process.

Required configuration

Identical across every EDOT SDK. Set exactly three environment variables:

| Variable | Value |

| ----------------------------- | --------------------------------------------------------------------------------------------------------------------- |

| OTEL_SERVICE_NAME | The service's name. This becomes service.name in Elasticsearch and is how every other Observability skill finds it. |

| OTEL_EXPORTER_OTLP_ENDPOINT | The managed OTLP endpoint or EDOT Collector URL resolved in step 3. |

| OTEL_EXPORTER_OTLP_HEADERS | The credential, in Authorization=ApiKey <key> or Authorization=Bearer <token> form. |

Two rules that hold for every language:

• Do not set OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, or OTEL_LOGS_EXPORTER. The defaults are already

correct. Setting them is the usual reason a service reports traces but no metrics or logs.

• Never run a classic Elastic APM agent and EDOT together on the same process. They double-instrument and produce

inconsistent traces. Removal and replacement belong in the same change.

Process: verify telemetry arrives

Run this after every instrument or migrate change, and as the entry point when a user reports that an instrumented

service is missing. Give the pipeline a minute or two after the deployment restarts before concluding anything.

1. Confirm documents are landing for the service. Query POST /_query, scoping to the service name that was set in

OTEL_SERVICE_NAME. The decision: is anything arriving at all, and across which signals.

   FROM traces-*,metrics-*,logs-* METADATA _index
   | WHERE @timestamp > NOW() - 15 minutes AND service.name == "<service>"
   | STATS docs = COUNT(*), last_seen = MAX(@timestamp) BY _index
   | SORT docs DESC
   | LIMIT 20

A healthy result has rows from a traces data stream, a metrics data stream, and a logs data stream, with last_seen

inside the last minute or two. Missing signals are diagnostic, not cosmetic: traces only, with no metrics or logs,

usually means an OTEL_*_EXPORTER variable was set.

1. If nothing came back, find out what name the service is reporting under. An empty result is ambiguous between "no

telemetry at all" and "telemetry arriving under a different name" — a typo in OTEL_SERVICE_NAME, or the SDK default

of unknown_service. Drop the service filter and look at what is actually arriving.

   FROM traces-*,metrics-*,logs-*
   | WHERE @timestamp > NOW() - 15 minutes
   | STATS docs = COUNT(*), last_seen = MAX(@timestamp) BY service.name
   | SORT docs DESC
   | LIMIT 25

The decision: if the service appears under an unexpected name, fix OTEL_SERVICE_NAME. If it does not appear at all,

the exporter is not reaching the destination — go back to step 3 of the routing process and re-check the endpoint and

credential.

1. Confirm EDOT is the reporter, not something else. This is what distinguishes a correct migration from a

half-finished one, and vanilla OpenTelemetry from EDOT.

   FROM traces-*
   | WHERE @timestamp > NOW() - 15 minutes AND service.name == "<service>"
   | STATS spans = COUNT(*) BY telemetry.sdk.language, telemetry.sdk.name, telemetry.distro.name, telemetry.distro.version
   | SORT spans DESC
   | LIMIT 10

telemetry.distro.name is elastic when an EDOT SDK is reporting. A null distro with

telemetry.sdk.name == "opentelemetry" means the application is on the upstream OpenTelemetry SDK rather than EDOT.

Two rows for one service — one with the Elastic distro and one without, or two different telemetry.sdk.language

values — means more than one agent is attached, or an old deployment is still running.

1. Diagnose from the shape of the result.

| Verification result | Most likely cause |

| ---------------------------------------------------------- | ------------------------------------------------------------------------- |

| No rows for the service, and the name is absent everywhere | Exporter never reached the destination — wrong endpoint or bad credential |

| Service appears under a different name | OTEL_SERVICE_NAME unset or misspelled; SDK fell back to its default |

| Traces only, no metrics or logs | An OTEL_*_EXPORTER variable was set and overrode the default |

| Rows present but last_seen is stale | The process exited, or the deployment was rolled back |

| Two distro values for one service | Classic agent and EDOT both attached, or an old replica still running |

If the field names above are absent from the mapping, confirm what is available with GET /<index>/_mapping before

concluding the telemetry is wrong.

Examples

"Add observability to my Spring Boot service" — read pom.xml to confirm Java, check for co.elastic.apm and

ELASTIC_APM_* markers, and finding none, take the instrument path in references/java.md. Add the -javaagent flag

to the container entrypoint, set the three environment variables against the managed OTLP endpoint, then run the

verification query and report which signals arrived.

"We're moving off the Elastic APM agent to EDOT, we're on .NET" — this is the migrate path. Remove the

Elastic.Apm.* packages, the AddAllElasticApm() call, and the ElasticApm block from appsettings.json in the same

change that adds Elastic.OpenTelemetry and builder.AddElasticOpenTelemetry(). Translate the configuration rather

than copying it — in particular do not reuse ELASTIC_APM_SERVER_URLS as the OTLP endpoint. Verify with step 3 and

confirm exactly one distro is reporting.

"I instrumented my Python app but nothing shows up in Elastic" — start at the verification process, not at the

configuration. Run step 1; if it is empty, run step 2 to see whether the service is reporting under a different name. If

the service is absent entirely, the two candidates are an unwrapped entrypoint (opentelemetry-instrument missing) and

an endpoint still pointing at APM Server. Check the entrypoint first — it is the more common of the two.

"Our service sends traces but we have no logs or metrics" — this is the OTEL_*_EXPORTER signature. Confirm it with

step 1, which will show a traces data stream and nothing else, then remove the OTEL_TRACES_EXPORTER,

OTEL_METRICS_EXPORTER, or OTEL_LOGS_EXPORTER variable from the deployment and re-verify.

Guidelines

• Read the setup or migration guide for the language before editing. The per-language references link the

authoritative page; the attach mechanism differs enough between runtimes that pattern-matching from another language

produces broken configurations.

• Removal and replacement go in one change. A window where both the classic agent and EDOT are attached produces

double-instrumented traces that are worse than either alone.

• Never carry the classic server URL forward. ELASTIC_APM_SERVER_URL points at APM Server;

OTEL_EXPORTER_OTLP_ENDPOINT must point at the managed OTLP endpoint or an EDOT Collector.

• Set the three variables and no more. Additional OTEL_* tuning is occasionally justified, but the exporter

variables specifically should be left at their defaults.

• Treat an empty verification result as unknown, not as failure of a specific component. Distinguish the causes with

step 2 before recommending a fix.

• Never place a credential in the skill's output or in a committed file. OTEL_EXPORTER_OTLP_HEADERS belongs in the

deployment's secret mechanism.

• Attribution: the per-language EDOT guidance consolidated here originates with the apm-agent-devs team.

Operations

| HTTP API (shorthand) | elastic CLI command |

| ----------------------- | ----------------------------------------------------- |

| GET / | elastic es info |

| POST /_query | elastic es esql query --format tsv --query '<esql>' |

| GET /<index>/_mapping | elastic es indices get-mapping --index '<index>' |