keyword-extract gives an archival document two kinds of keyword, kept apart. Statistical keywords (KeyBERT by default, YAKE, or the legacy KER) describe the document and each of its pages. Controlled keywords assign each line a term of the AMČR keyword lists and the TEATER thesaurus, chosen by a language model, with the records behind the term. Every keyword names the method that produced it, with its score and rank. Use it on the ATRIUM document record, or on nlp-enrich's CoNLL-U and TEITOK files; the controlled kind needs a language-model service and is research output.
Releases · MIT licence · ATRIUM documentation
This repository was assembled on 1 October 2026 from the keyword extraction of
atrium-nlp-enrich (keywords.py) and of atrium-llm-enrich (the LLM
engine and the vocabulary code), after the meeting of 30 September
(#1); nlp-enrich keeps the LINDAT calls (UDPipe,
NameTag). The images of those two repositories that carried these stages stay published and get no new tags.
| Input | Output |
|---|---|
the ATRIUM document record: its lines[].text (and lemma, upos where present) |
statistical keywords of the document and of each page; the record with its keywords block |
| the same record, with a language-model service configured | per line, an AMČR/TEATER term with its ids, the Czech and English keywords of the line, entity links; the enrichment block |
a plain text (/extract_keywords_text) |
its statistical keywords |
nlp-enrich's CoNLL-U or TEITOK file, a line table (CSV), a .txt |
the same statistical keywords, from the service (file) or the command line |
| for the batch command line: a folder of any of the above | a summary CSV, one CSV per document and per page, the records (outputs) |
The record follows the hub's document contract;
lines labelled Trash, Garbage, Inverted or Empty are left out. keywords.py and the service read every input
with the same code (kw_inputs.py); what only one of them does, and why:
the command line and the service.
The service from the published image, on a sentence, with YAKE, which needs no model.
DEFAULT_KW_METHOD=yake keeps the service from downloading KeyBERT's embedding model when it starts:
docker run --rm -p 8000:8000 -e DEFAULT_KW_METHOD=yake \
ghcr.io/ufal/atrium-keyword-extract-api:<version> # the release without its v
curl --fail-with-body -sS -X POST localhost:8000/extract_keywords_text -H 'Content-Type: application/json' \
-d '{"text": "Archeologický výzkum odkryl zahloubený objekt se sídlištní keramikou.", "method": "yake", "kind": "statistical", "num_keywords": 5}'Expected result (abridged):
{"kind": "statistical", "method_used": "yake", "words": 8,
"keywords": [{"keyword": "Archeologický výzkum odkryl", "score": 1.0, "method": "yake", "rank": 1}, "…"],
"kinds": [{"kind": "statistical", "status": "ok", "detail": null}]}POST /extract_keywords takes a record (-F "document_json=@<doc>.document.json;type=application/json") and also
returns per-page keywords and the record with its blocks. KeyBERT, the default method, loads its embedding model from
the Hugging Face Hub at start-up. The controlled kind (kind=controlled or both) calls OpenRouter or a local
Ollama, set by LLM_BACKEND; without one it answers 501, or reports itself skipped under kind=both. Every
option: service/README.md.
The statistical kind runs one of three methods over the record's text, for the whole document and page by page; scores compare only within a method. The controlled kind sends each line, with a prompt that carries the whole vocabulary (about 58k tokens), to a language model whose answer is constrained to the vocabulary's 4,711 terms, and writes the chosen term with its AMČR and TEATER ids. The details, and the batch and research paths: docs/how-it-works.md.
keyword-extract comes after nlp-enrich in the scanned-document pipeline, and after digital-convert on the
born-digital route; it reads the record's lines. The legacy method (KER) counts lemmas, which nlp-enrich's CoNLL-U
and TEITOK files carry; no ATRIUM tool writes them into the record today. It writes the record's keywords and
enrichment blocks, which
nlp-enrich's /project_record can carry into the TEITOK header. The vocabulary build is also released as
atrium-vocabulary-<version>.zip, which the translator and the end-to-end tests read. The whole chain: the
hub's workflow page. Its SSH Open Marketplace
record: Keyword extraction from archival documents (statistical and vocabulary-controlled) -
ATRIUM.
- The controlled kind is not evaluated. The rubric is decided, but the annotated gold set does not exist yet, so treat its output as research output (#2).
- A constrained answer stays in the vocabulary, not on topic: a model can be forced onto an unrelated term that shares the start of the word it began (Known limits).
- The prompt is about 58k tokens. A model served with a smaller window sees only part of the vocabulary; the paradata records how many terms were cut.
- YAKE is AGPL-3.0: a run that uses it lists YAKE among its components, so the run's calculated licence summary becomes AGPL-3.0; that summary is not the licence of the keywords (licences).
- The research path (
llm_run.py) needs a GPU and local weights; the service needs neither.
| To | Read |
|---|---|
| set it up, run the service, the batch CLI, the images | docs/usage.md |
| deploy and operate the service, its options, limits, errors | service/README.md |
| see the API contract | service/openapi.json |
| understand the two kinds and the vocabulary | docs/how-it-works.md |
| read the batch CLI's outputs and scores | docs/reference/outputs.md |
| see the licences of a run | docs/reference/licences.md |
| build or review the vocabulary | data_samples/vocab/RUNBOOK.md |
| change the controlled kind's prompt | prompts/RUNBOOK.md |
| contribute, test, release | CONTRIBUTING.md |
The documentation follows the ATRIUM documentation style.
The code is MIT (LICENSE). It uses KeyBERT with sentence-transformers (MIT, Apache-2.0), YAKE (AGPL-3.0) and its own lemma-frequency method named after ÚFAL's KER (MIT), and the AMČR and TEATER vocabularies (CC0); the language model of the controlled kind is the deployment's choice, and every run records its id. The inventory is para_config.txt; what a run records is in docs/reference/licences.md. Cite with CITATION.cff.
Developed by ÚFAL, Charles University. Funded by ATRIUM (GA 101132163), with the support of the LINDAT/CLARIAH-CZ research infrastructure and the AIS CR infrastructure (LM2023031). Contact: the issue tracker.