Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ jobs:
- "3.11"
- "3.12"
- "3.13"
- "3.14"

steps:
- name: Checkout the repository
Expand Down
62 changes: 31 additions & 31 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,40 +5,40 @@ on:
branches: [master]

permissions:
contents: write
contents: read
pages: write
id-token: write

jobs:
docs:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest

strategy:
matrix:
python-version:
- "3.13"

steps:
- uses: actions/checkout@v4

- name: Install uv and set the python version
uses: astral-sh/setup-uv@v5
with:
python-version: ${{ matrix.python-version }}

- run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV

- uses: actions/cache@v4
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
key: mkdocs-material-${{ env.cache_id }}
path: .cache
restore-keys: |
mkdocs-material-

- name: Install dependencies
run: uv sync --locked --only-group docs

- name: Run docs build
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: bash ./scripts/docs_build.sh
- name: Configure GitHub Pages
uses: actions/configure-pages@v6

- name: Checkout the repository
uses: actions/checkout@v7

- name: Install uv and set Python
uses: astral-sh/setup-uv@v5
with:
python-version: "3.14"

- name: Install documentation dependencies
run: uv sync --locked --no-default-groups --group docs

- name: Build documentation
run: bash ./scripts/docs_build.sh

- name: Upload GitHub Pages artifact
uses: actions/upload-pages-artifact@v5
with:
path: site

- name: Deploy GitHub Pages
id: deployment
uses: actions/deploy-pages@v5
2 changes: 1 addition & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v6.1.0
with:
python-version: '3.13'
python-version: '3.14'

- name: Install build tools
run: |
Expand Down
37 changes: 37 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# AGENTS.md

## Project shape

- This is a single Python package, not a monorepo. Runtime code is under `src/dataguard`; tests are under `tests`; documentation is under `docs`; documentation examples are sourced from `notebooks`.
- The distribution is named `peh-dataguard`, but the import name is `dataguard`. The public exports are `Validator`, `Filter`, and `ErrorCollector` from `src/dataguard/__init__.py`.
- Python `>=3.10` is supported; CI tests Python `3.10` through `3.14`. Ruff targets Python 3.10, so do not use syntax or APIs newer than the minimum unless the project metadata is changed.
- The main flow is configuration mapping -> recursive Pydantic expression/schema models -> Polars expression compilation -> Pandera validation for `Validator` or direct Polars filtering for `Filter`. The relevant wiring is in `config/config_reader.py`, `core/check/`, `core/filter/`, and `core/models/`.
- `CaseCheckExpression` is recursive: `conjunction` and `disjunction` require at least two expressions, while `condition` requires exactly two. Built-in check names are mapped to Polars methods; custom callable checks are supported for validation, not filtering.
- `Validator` normally collects Polars/Pandera/configuration errors in the shared `ErrorCollector`; pass `collect_exceptions=False` when an exception should propagate. `Filter.apply` preserves `DataFrame` versus `LazyFrame` input type.

## Toolchain and commands

- Use `uv` and commit `uv.lock` changes generated by `uv lock`; do not edit the lockfile by hand. The direct dependency bounds intentionally block next-minor releases while allowing patch updates.
- Reproduce the CI environment with `uv sync --locked --all-extras --dev`. The docs environment is separate: `uv sync --locked --no-default-groups --group docs`.
- Run the full test and coverage suite with `make test` (it runs `uv run pytest -s -vv --cov=src/` and then `coverage html`). It modifies the tracked `.coverage` file; restore it before committing unless the coverage data is an intentional change.
- Run a focused test with `make test-set SET='tests/tests_filter/test_filter.py::test_filter_apply_dataframe'`, or use `uv run pytest` with the desired node id. Tests are local/unit-style and use Hypothesis; no external service is configured.
- Run lint with `make lint`. It runs Ruff checks twice; it does not check formatting. For a non-mutating formatting check use `uv run ruff format --check .`. `make format` runs Ruff autofixes and formatting, so it changes files.
- `bash scripts/pre_commit.sh` is the repository wrapper for `format -> lint -> test`; it is mutating because it runs the formatter first. There is no configured mypy/pyright type-check step.
- Build the sdist and wheel with `uv build`. Keep `dist/`, `site/`, `htmlcov/`, caches, and virtual environments out of commits.

## State and testing gotchas

- `ErrorCollector` is cached/shared; clear it between tests that inspect collected reports. The validator test fixture demonstrates this cleanup.
- `parse_checks()` destructively pops `name`, `error_level`, and `error_msg` from each check mapping. Do not reuse an input mapping after parsing if its original contents matter.
- `expression_mapper` is global and `map_command()` mutates parsed expression objects. Filter tests reset this mapping; preserve that isolation when adding tests or changing command registration.
- `CaseCheckExpression` and `SimpleCheckExpression` are the backend-neutral expression layer. Keep built-in lowering shared between validation and filtering rather than duplicating command logic.

## Documentation and release operations

- Zensical, not the `mkdocs` CLI, builds the site from the MkDocs-compatible `mkdocs.yml`. Use `make serve` for a local preview and `bash scripts/docs_build.sh` for the strict clean build; the script uses `uv run --no-default-groups --group docs` and builds `site/`.
- The Markdown guides embed code with `--8<--` snippets from `notebooks/`. Edit the notebook source and rebuild rather than copying code into the generated guide. `docs/changelog.md` similarly includes the root `CHANGELOG.md`.
- API documentation is generated from source docstrings through `mkdocstrings` (`paths: [../src]`), so update the relevant source docstrings when changing public APIs.
- The `DOCS` workflow runs only for pushes to `master` and deploys the built `site/` directory through GitHub Pages Actions. GitHub Pages must be configured to use GitHub Actions as its source.
- The publish workflow builds on branch pushes, publishes to TestPyPI on tag pushes, and publishes to PyPI when a GitHub release is published. A normal branch push alone does not publish the package.
- For a release, update `project.version` in `pyproject.toml`, regenerate `uv.lock`, run `uv build`, and tag the matching commit; do not create a release with mismatched project and lock metadata.
- `presentation.md` is a standalone 16:9 Marp deck; no Makefile or CI target renders it, so validate it separately if it is changed.
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Changelog

All notable changes to DataGuard are documented in this file. The format is
based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.6.1] - 2026-09-25

This maintenance release updates packaging, supported Python versions,
dependencies, CI, and documentation without changing the public runtime API.

### Added

- Added end-user installation instructions for `uv` and `pip` to the
documentation and README.
- Added this changelog and exposed it in the documentation navigation.

### Changed

- Bumped the package version to `0.6.1`.
- Added Python 3.14 to the CI, documentation, and publishing workflows and
updated the contributor setup.
- Migrated the documentation from MkDocs Material to Zensical, including the
GitHub Pages deployment workflow.
- Declared direct Polars and Pydantic runtime dependencies and added
next-minor upper bounds for direct runtime, development, build, and
documentation dependencies.
- Regenerated `uv.lock` with the validated dependency graph.
- Updated the README with installation, development, documentation, and
release-history links.

[Unreleased]: https://github.com/eu-parc/dataguard/compare/v0.6.1...HEAD
[0.6.1]: https://github.com/eu-parc/dataguard/compare/v0.6.0...v0.6.1
91 changes: 66 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
# DataGuard

<p align="center">
<a href="https://github.com/eu-parc/dataguard/actions?query=workflow%3ACI" targe>
<a href="https://github.com/eu-parc/dataguard/actions?query=workflow%3ACI" target="_blank" rel="noopener">
<img src="https://github.com/eu-parc/dataguard/actions/workflows/ci.yml/badge.svg" alt="CI">
</a>
<a href="https://github.com/eu-parc/dataguard/actions?query=workflow%3ADOCS" targe>
<img src="https://github.com/eu-parc/dataguard/actions/workflows/docs.yml/badge.svg" alt="CI">
<a href="https://github.com/eu-parc/dataguard/actions?query=workflow%3ADOCS" target="_blank" rel="noopener">
<img src="https://github.com/eu-parc/dataguard/actions/workflows/docs.yml/badge.svg" alt="Documentation build">
</a>
<p/>
</p>

Source: https://github.com/eu-parc/dataguard
Documentation: https://eu-parc.github.io/dataguard/
- Source: [GitHub repository](https://github.com/eu-parc/dataguard)
- Documentation: [DataGuard documentation](https://eu-parc.github.io/dataguard/)
- Changelog: [CHANGELOG.md](https://github.com/eu-parc/dataguard/blob/master/CHANGELOG.md)

DataGuard supports Python 3.10 and later.

## Features

Expand All @@ -24,29 +27,67 @@ Documentation: https://eu-parc.github.io/dataguard/
- **Detailed error reporting**: Collect and format validation errors with custom messages and severity levels
- **Polars support**: Built on Polars for efficient data processing at scale

## Installation
To install the library for development.
## Installation

The package is published on PyPI as `peh-dataguard` and imported as
`dataguard`.

### Install with uv

For a `uv`-managed project, add DataGuard as a dependency:

```bash
uv add peh-dataguard
```

To install it into the current environment without changing a project file:

```bash
uv pip install peh-dataguard
```

### Install with pip

Create and activate a virtual environment, then install the package:

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install peh-dataguard
```

On Windows PowerShell, activate the environment with
`.venv\Scripts\Activate.ps1` instead of `source`.

Verify that the public API can be imported:

```bash
python -c "from dataguard import Filter, Validator; print('DataGuard is ready')"
```

For a `uv`-managed project, use `uv run python ...` instead of `python ...`.

### Synchronize the project dependencies

```bash
$ uv sync
```
## Development

#### Tests
For a source checkout, install the development dependencies with Python 3.14:

```bash
$ make test
```
```bash
git clone https://github.com/eu-parc/dataguard.git
cd dataguard
uv sync --python 3.14
```

#### Format
Run the project checks with:

```bash
$ make format
```
```bash
make test
make lint
make format
```

#### lint
To preview the Zensical documentation locally:

```bash
$ make lint
```
```bash
make serve
```
3 changes: 3 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Changelog

--8<-- "CHANGELOG.md:2"
45 changes: 43 additions & 2 deletions docs/getting_started.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,47 @@
# Getting Started

## Dataguard workflow
## Installation

The PyPI distribution is named `peh-dataguard`, while the Python import
name is `dataguard`.

### Install with `uv`

For a `uv`-managed project, add DataGuard as a dependency:

```bash
uv add peh-dataguard
```

To install it into the current environment without changing a project file:

```bash
uv pip install peh-dataguard
```

### Install with `pip`

Create and activate a virtual environment, then install the package:

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install peh-dataguard
```

On Windows PowerShell, activate the environment with
`.venv\Scripts\Activate.ps1` instead of `source`.

### Verify the installation

```bash
python -c "from dataguard import Filter, Validator; print('DataGuard is ready')"
```

For a `uv`-managed project, use `uv run python ...` instead of `python ...`.

## DataGuard workflow

```py title="getting_started.py" linenums="1" hl_lines="3-9 13 15"
--8<-- "notebooks/getting_started.py:1:15"
Expand All @@ -15,7 +56,7 @@ While this example uses empty lists and an empty DataFrame for simplicity, it il

## Validating real-world constraints

Consider an `age` column configuration that demonstrates Dataguard's data quality enforcement capabilities:
Consider an `age` column configuration that demonstrates DataGuard's data quality enforcement capabilities:

- **Type Safety**: Enforcing `integer` data type prevents string or float contamination
- **Null Prevention**: `nullable: False` ensures no missing age values slip through
Expand Down
6 changes: 4 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Welcome to Dataguard
# Welcome to DataGuard

Dataguard is a Python package for performing data validation and filtering on dataframe objects.
DataGuard is a Python package for performing data validation and filtering on dataframe objects.

It provides a flexible and extensible framework for defining validation rules and filter expressions, ensuring that your data meets quality and consistency requirements.

Expand All @@ -10,3 +10,5 @@ For filtering data, please see [How to filter](how-to-filter.md)

For API documentation, please see the [API Reference](api.md).

For release history, see the [Changelog](changelog.md).

4 changes: 4 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,11 @@ site_name: DataGuard
repo_url: https://github.com/eu-parc/dataguard
repo_name: eu-parc/dataguard
edit_uri: edit/master/docs/
# Zensical intentionally reads this MkDocs-compatible configuration.
# The classic variant preserves the existing Material-style presentation.
theme:
name: material
variant: classic

plugins:
- search
Expand Down Expand Up @@ -39,6 +42,7 @@ markdown_extensions:
nav:
- Home: index.md
- Getting Started: getting_started.md
- Changelog: changelog.md
- User Guide:
- How to create checks: how-to-create_checks.md
- How to filter: how-to-filter.md
Expand Down
Loading
Loading