Scaling Unsupervised Word Alignment to Documents with Structural Constraints

This repository contains the experimental code for the paper Scaling Unsupervised Word Alignment to Documents via Structural Constraints. To word align your own documents, use the CTFAlign package.

Installation

Python 3.11 recommended.

pip install -r reqs.txt

A HuggingFace token is read from .keys/hf.txt, needed only for gated models when re-running the encoding step.

Reproducing content of the paper based on existing results

All the word alignment predictions from the experiments reported in the paper are included in this repository and the the tables and figures can be reproduced by following the steps listed below.

Tables

All table scripts print LaTeX to stdout.

Paper table	Command
1 — GPT-5.4-mini AER (tab:gpt-results)	python scripts/make_gpt_table.py
2 — Translation coverage ROC AUC (tab:auc-results-token)	python scripts/make_wmt_auc_table.py, or see Translation coverage
3 — SwissGov-RSD test Spearman (tab:diffalign-spearman-test)	see SwissGov-RSD
Dataset statistics (tab:alignment-data)	python scripts/dataset_stats.py
WMT statistics (tab:wmt-overview)	python scripts/wmt_stats.py --full
Pilot models (tab:pilot-results-avg)	python scripts/make_pilot_table.py
SwissGov-RSD dev Spearman (tab:diffalign-spearman-dev)	see SwissGov-RSD
k/w search (tab:k-search)	python scripts/make_k_search_table.py
Dev results (tab:alignment-results-dev)	python scripts/make_dev_table.py
Test results (tab:alignment-results)	python scripts/make_test_table.py

Figures

Paper figure	How
1 — SimAlign / MDPAlign / CTFAlign on en-ja	python scripts/figure1.py → figures/figure1/
2 — MDPAlign masks (fig:mdpalign)	figures/alignment-viewer.html, exported by hand
3 — CTFAlign schematic (fig:ctf)	figures/ctfalign-figure-editor.html, exported by hand
4 — Test AER bars (fig:test-results)	python scripts/plot_test_bars.py --metric AER --outfile results/test_bars.pdf
Layer sweep (fig:layer-results)	python scripts/plot_layer_f1.py
k/w sweep (fig:k-comb)	python scripts/plot_k_sweep_combined.py
mmBERT failure case (fig:fail)	figures/alignment-viewer.html, exported by hand

Re-running the pipeline from scratch

The embeddings and similarity matrices produced during our experiments are not included in this repository but can be reproduced by following the instructions below. Steps 1 and 2 need a GPU; everything after is CPU-only.

Downloading and Processing Data

To download the sentence-level data and rebuild the documents from the original sources:

# EN-FR, EN-RO  (Mihalcea & Pedersen 2003)
wget --no-check-certificate https://web.eecs.umich.edu/~mihalcea/wpt/data/English-French.trial.tar.gz
wget --no-check-certificate https://web.eecs.umich.edu/~mihalcea/wpt/data/English-French.test.tar.gz
wget --no-check-certificate https://web.eecs.umich.edu/~mihalcea/wpt/data/Romanian-English.trial.tar.gz
wget --no-check-certificate https://web.eecs.umich.edu/~mihalcea/wpt/data/Romanian-English.test.tar.gz
# EN-JA  (Neubig, KFTT)
wget http://www.phontron.com/kftt/download/kftt-alignments.tar.gz
# LA-GR  (Ugarit gold standards)
git clone --depth 1 https://github.com/UgaritAlignment/Alignment-Gold-Standards

mkdir -p raw_data && for f in *.tar.gz; do tar -xzf "$f" -C raw_data; done
mv Alignment-Gold-Standards raw_data
python scripts/preprocessing.py         # -> data/{sentences,documents}/

Two sources need a manual download into raw_data/:

• EN-CZ — https://lindat.mff.cuni.cz/repository/bitstreams/21d1bff9-a215-43d8-a516-5f5d39851b36/download
• EN-ZH — https://nlp.csai.tsinghua.edu.cn/~ly/systems/TsinghuaAligner/download_evalset.php

Overview of all word alignment data sources:

Lang pair	Source	License
en-ja	Neubig (2011)	CC BY-SA 3.0
la-gr	Yousef et al. (2022)	CC BY-SA 4.0
en-fr, en-ro	Mihalcea & Pedersen (2003)	Not specified by the source
en-zh	Liu and Sun (2015)	Not specified by the source
en-cz	Mareček (2011)	CC BY-SA 4.0

The reconstructed document-level word alignment datasets are made available: https://huggingface.co/datasets/ZurichNLP/document-level-word-alignment

Word Alignment

Method Names

The paper's three methods map to --mask values as follows. Directory names under preds/ and results/ use the same strings.

Paper	--mask	--k means	Value used in the paper
SimAlign (baseline)	—	—	—
MDPAlign strict	mdpalign_strict	band half-width, absolute token count	k=50 (dev), k=25 (test)
MDPAlign fuzzy	mdpalign_fuzzy	Gaussian σ, absolute token count	k=150
CTFAlign	ctfalign	buffer half-width w, in coarse blocks	w=8

Argmax vs Itermax is --method simalign-argmax / simalign-itermax (--max_count sets the iteration count; the paper uses 2).

Commands

# 1. Token embeddings -> cosine similarity matrices  [GPU]
sbatch run_sim_matrices.sh Qwen3-Embedding-4B
#      run_sim_matrices.sh <model-key> [lang-pair]; see the table in that file

# 2. Alignment predictions from the similarity matrices  [CPU]
python scripts/get_predictions.py \
  --model_name Qwen3-Embedding-4B --method simalign-argmax \
  --granularity documents --split test --mask ctfalign --k 8 --best_layer_only

# 3. Evaluate -> results/
python scripts/eval.py \
  --path_to_preds_folder "preds/simalign-argmax_mask=ctfalign_k=8.0" \
  --model_name Qwen3-Embedding-4B --granularity documents --split test

# 4. Regenerate the tables and figures as above

Translation coverage

The same four steps as above, except that step 2 produces continuous scores (1 - max similarity per word) instead of binary alignments, and step 3 evaluates them against the MQM labels:

# 0. WMT 2024 MQM data
mkdir -p wmt && cd wmt
wget https://raw.githubusercontent.com/google/wmt-mqm-human-evaluation/refs/heads/main/generalMT2024/mqm_generalMT2024_ende.tsv
wget https://raw.githubusercontent.com/google/wmt-mqm-human-evaluation/refs/heads/main/generalMT2024/mqm_generalMT2024_jazh.tsv
cd .. && python scripts/preprocess_wmt.py --full   # -> wmt/processed_full/

# 1. Token embeddings -> cosine similarity matrices  [GPU]
python scripts/get_sim_matrices_qwen4b.py \
  --model_name Qwen/Qwen3-Embedding-4B --layer 20 --granularity documents \
  --data_path wmt/processed_full \
  --sim_matrices_path wmt/similarity_matrices \
  --embeddings_path wmt/embeddings

# 2. Continuous scores instead of alignments  [CPU]
python scripts/get_continuous_scores.py \
  --model_name Qwen3-Embedding-4B --granularity documents --split test \
  --mask ctfalign --width 8 --coarse_method argmax \
  --sim_matrices_path wmt/similarity_matrices \
  --output_folder wmt/continuous_preds_full/

# 3. Evaluate -> Table 2, one row per method
#    (--granularity sentences for the segment-level block)
python scripts/eval_hallucination.py \
  --hallucination_preds_folder "wmt/continuous_preds_full/ctfalign_argmax_width=8" \
  --gold_folder wmt/processed_full/ \
  --model_name Qwen3-Embedding-4B --granularity documents --split test

# 4. Regenerate the table as above

Swap the step 2 flags and the corresponding step 3 folder for the other rows:

Table 2 row	Step 2 flags	Folder
DiffAlign (baseline)	--mask none	none
DiffAlign (MDP+Argmax)	--mask mdpalign_fuzzy --k 150	mdpalign_fuzzy_k=150.0
DiffAlign (CTF+Argmax)	--mask ctfalign --width 8 --coarse_method argmax	ctfalign_argmax_width=8
DiffAlign (CTF+Itermax)	--mask ctfalign --width 8 --coarse_method itermax	ctfalign_itermax_width=8

Each run prints both sides per language pair: source is omissions, target is additions. The reported metric is the token ROC AUC.

python scripts/make_wmt_auc_table.py runs all of the above and emits the whole table.

SwissGov-RSD

See [swissgov_rsd/README.md](swissgov_rsd/README.md) for provenance and licensing.

The same steps as above, except that there is no step 0 (the gold data is vendored in swissgov_rsd/) and steps 1 and 2 are fused: predict_encoder.py embeds and scores in a single pass. The scripts resolve the gold labels relative to their own location, so run them from swissgov_rsd/.

cd swissgov_rsd

# 1+2. Embeddings -> continuous scores, in one pass  [GPU]
PYTHONPATH=. python scripts/predict_encoder.py Qwen/Qwen3-Embedding-4B \
  --type ctfalign --layer 20 --width 8 --test_data swissgov --split test
#      -> data/evaluation/encoder_predictions/{split}/

# 3. Evaluate -> Table 3, one row per method (--split dev for the dev table)
PYTHONPATH=. python scripts/evaluate_predictions_admin.py \
  "data/evaluation/encoder_predictions/test/DiffAlignCTFAlign(model=Qwen_Qwen3-Embedding-4B, layer=20, width=8)_admin_" \
  --split test

# 4. No table script; the evaluation output is the table

Swap the step 1+2 --type and the corresponding step 3 filename prefix for the other rows:

Table 3 row	--type	Prediction file prefix
DiffAlign (baseline)	diffalign	DiffAlign(model=Qwen_Qwen3-Embedding-4B, layer=20_admin_
DiffAlign (MDP+Argmax)	mdpalign_fuzzy	DiffAlignMDPAlignFuzzy(model=Qwen_Qwen3-Embedding-4B, layer=20, k=150.0)_admin_
DiffAlign (CTF+Argmax)	ctfalign	DiffAlignCTFAlign(model=Qwen_Qwen3-Embedding-4B, layer=20, width=8)_admin_
DiffAlign (CTF+Itermax)	ctfalign_itermax	DiffAlignCTFAlignItermax(model=Qwen_Qwen3-Embedding-4B, layer=20, width=8)_admin_

The prefixes come from each recognizer's __str__, so renaming a recognizer renames its prediction files.

Citation

@misc{wastl2026scalingunsupervisedwordalignment,
      title={Scaling Unsupervised Word Alignment to Documents via Structural Constraints}, 
      author={Michelle Wastl and Jannis Vamvas and Rico Sennrich},
      year={2026},
      eprint={2608.21023},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.21023}, 
}