extracting-keywords

Extracting keywords, language, and embeddings

Use this for the enrichment surface around extraction: statistical keyword

extraction, language detection, and vector embeddings. Keywords and

language detection ride along with extraction and land on the result;

embeddings are produced by a dedicated embed command.

Keywords (YAKE / RAKE)

Keyword extraction is configured via the [keywords] config block (or

inline JSON) — there is no single --keywords CLI flag. When enabled,

extracted keywords appear on result.keywords. Two algorithms are

available:

• YAKE ("yake") — statistical, unsupervised single-document

extraction. Good general default.

• RAKE ("rake") — co-occurrence / phrase-based. Favors multi-word

key phrases.

> Feature-gated: keyword extraction requires the CLI to be built with the

> keywords-yake and/or keywords-rake Cargo features (both are in the

> default/full build). If the CLI was built without them, the [keywords]

> config block is silently ignored — result.keywords simply stays empty

> rather than erroring. The "yake" algorithm needs keywords-yake; "rake"

> needs keywords-rake.

Enable via inline JSON on the CLI:

kreuzberg extract paper.pdf --format json \
  --config-json '{"keywords":{"algorithm":"yake","max_keywords":15,"language":"en"}}' \
  | jq '.keywords'

Or in a config file:

[keywords]
algorithm = "rake"       # "yake" or "rake"
max_keywords = 10        # default 10
min_score = 0.0          # filter below this score (ranges differ per algorithm)
ngram_range = [1, 3]     # unigrams..trigrams (default)
language = "en"          # stopword language; omit to skip stopword filtering

kreuzberg extract report.pdf --config kreuzberg.toml --format json | jq '.keywords'

Field notes:

• max_keywords caps how many keywords are returned (default 10).
• min_score filters low-scoring keywords; note YAKE scores are

lower-is-better while RAKE scores are higher-is-better, so a single

threshold behaves differently per algorithm.

• ngram_range is [min, max]: [1,1] unigrams only, [1,2] adds

bigrams, [1,3] (default) adds trigrams.

• language enables stopword filtering for that language; omit it to

disable stopword filtering entirely.

Language detection

Language detection is a real CLI flag: --detect-language. Detected

languages appear on result.detected_languages:

kreuzberg extract multilingual.pdf --detect-language true --format json \
  | jq '.detected_languages'

In a config file it lives under [language_detection]:

[language_detection]
enabled = true
min_confidence = 0.8
detect_multiple = false

The CLI flag enables detection with min_confidence = 0.8 and

single-language mode; use the config block to detect multiple languages or

tune confidence.

Embeddings (embed command)

The standalone embed command produces vector embeddings for text from

--text (repeatable) or stdin. It does not run extraction — pipe

extracted content in if you want document embeddings.

# Local ONNX preset model (default provider)
kreuzberg embed --text "first passage" --text "second passage" --preset balanced

# Embed extracted document text
kreuzberg extract report.pdf | kreuzberg embed --preset quality

Presets for the local provider: fast, balanced (default), quality,

multilingual. Output defaults to JSON (--format json).

--provider selects the embedding source:

| Provider | Flag | Notes |

| -------- | ------------------------------------- | --------------------------------------------- |

| local | --preset <fast\|balanced\|quality\|multilingual> | Default. ONNX model, no API key. |

| llm | --model <id> --api-key <key> | liter-llm routing, e.g. openai/text-embedding-3-small. |

| plugin | --plugin <name> | A backend pre-registered in-process via the plugin API. |

# Provider-hosted embeddings via an LLM
kreuzberg embed --text "query text" \
  --provider llm --model openai/text-embedding-3-small --api-key "$OPENAI_API_KEY"

Local embedding presets must be downloaded first if not cached. Pre-warm

them with the cache command:

kreuzberg cache warm --embedding-model balanced   # one preset
kreuzberg cache warm --all-embeddings             # all four presets

Programmatic access

Keywords and detected languages live on the extraction result:

from kreuzberg import extract_file_sync, ExtractionConfig

result = extract_file_sync(
    "paper.pdf",
    config=ExtractionConfig(),  # configure keywords/language_detection on the config
)
print(result.keywords)             # extracted keywords (when enabled)
print(result.detected_languages)   # detected languages (when enabled)

See references/python-api.md and references/configuration.md in the

sibling kreuzberg skill for the keyword / language-detection config

classes and the embedding presets.

Common pitfalls

• No --keywords flag — keyword extraction is config-only. Use

--config-json '{"keywords":{...}}' or a [keywords] config block.

• min_score direction — lower is better for YAKE, higher is better

for RAKE; pick the threshold to match the algorithm.

• Embeddings ≠ extraction — embed only takes raw text. Pipe

kreuzberg extract output into it for document vectors.

• Cold embedding models — first local run downloads the preset; run

kreuzberg cache warm --all-embeddings to pre-populate.

See references/advanced-features.md for the embeddings pipeline and

references/cli-reference.md for the embed and cache warm flag sets.