lvnlp

Latvian NLP tools for morphosyntactic parsing.

Installation

Install from PyPI:

pip install lvnlp

Or install the latest development version from GitHub:

pip install git+https://github.com/LUMII-AILab/lvnlp.git

Morphosyntactic parsing

from lvnlp.parser import Parser
from lvnlp.parser.tokenize import tokenize_sentences
from lvnlp.parser.utils import to_conll

parser = Parser.from_pretrained()
sentences = parser.parse(tokenize_sentences('Jānis brauca uz Rīgu. Viņš atgriezās vakarā.'))
print(to_conll(sentences))

The parser can optionally use the Latvian morphological analyzer during decoding. This mainly improves XPOS tagging and lemmatization: analyzer candidates are scored with the model's tag and lemma-rule probabilities, the best candidate's lemma is adopted, and its tag replaces the model's tag when it scores within a margin of it.

Corpus tagging

lvnlp tag retags a corpus file in one streaming pass. Vertical (.vert) and jsonl files are read and written, optionally gzipped; the format follows the file name.

lvnlp tag -i corpus.vert.gz -o tagged.vert.gz                        # form, lemma, xpos
lvnlp tag -i corpus.vert -o tagged.vert --columns 6                  # + upos, head, deprel
lvnlp tag -i corpus.vert -o tagged.vert --in-place                   # keep the input columns, overwrite the tagger's
lvnlp tag -i docs.jsonl -o tagged.jsonl.gz --device cuda:0*3         # {"id", "text"} documents; three model workers
lvnlp tag -i corpus.vert -o tagged.vert --analyzer --decode-workers 2  # rescore with the morphological analyzer
lvnlp tag -i corpus.vert -o tagged.vert --device cpu*8 --cpu-threads 1

The CLI can also run without installing: uv fetches the package into a cached environment on first use.

# install uv once, then open a new shell
curl -LsSf https://astral.sh/uv/install.sh | sh

uvx lvnlp tag -i corpus.vert.gz -o tagged.vert.gz --device cuda:0
uvx lvnlp==0.2.0 tag -i corpus.vert.gz -o tagged.vert.gz --device cuda:0   # specific version
uvx --from 'git+https://github.com/LUMII-AILab/lvnlp.git@main' lvnlp tag -i corpus.vert.gz -o tagged.vert.gz   # development version

Main options: --device cuda:0, cuda:0*N (N model workers sharing one GPU), cuda:0,cuda:1 or cpu*N; --decode-workers K adds K processes per model worker that decode and rescore batches, which is what makes --analyzer fast; --batch-size (128) and --buffer-words (20000) size the forward passes; --limit N stops after N documents.

Steady-state throughput of lv_ud2.18_lv-deberta-base (bf16 on GPU, fp32 on CPU):

setup	plain	with --analyzer	memory
one A100: --device cuda:0 (+ --decode-workers 2 with the analyzer)	33k words/s	21k words/s	14 GB VRAM, 2-4 GB RAM
one A100: --device cuda:0*3 (+ --decode-workers 2 with the analyzer)	53k words/s	47k words/s	35 GB VRAM, 6-11 GB RAM
1 CPU core: --device cpu --cpu-threads 1	73 words/s	~70 words/s	2 GB RAM
8 CPU cores: --device cpu*8 --cpu-threads 1 --buffer-words 1000	~0.6k words/s	~0.6k words/s	16 GB RAM
64 CPU cores: --device cpu*64 --cpu-threads 1 --buffer-words 1000	4.7k words/s	~5k words/s	126 GB RAM

On CPU the forward pass dominates, so the analyzer costs little there; each CPU process needs about 2 GB RAM.

Citation

@inproceedings{znotins-2026-improving,
  title = {Improving Latvian Morphosyntactic Parsing with Pretrained Encoders and Analyzer-Constrained Decoding},
  author = {Znotins, Arturs},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026)},
  month = {May},
  year = {2026},
  pages = {11724--11734},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  editor = {Piperidis, Stelios and Bel, Núria and van den Heuvel, Henk and Ide, Nancy and Krek, Simon and Toral, Antonio},
  doi = {10.63317/5khpzsaiqrzw},
  url = {https://lrec.elra.info/lrec2026-main-918}
}

Acknowledgements

This work was supported by the EU Recovery and Resilience Facility project Language Technology Initiative (2.3.1.1.i.0/1/22/I/CFLA/002).