Skip to content

docs: align Wrangle README with Chances product format #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mikkokotila merged 1 commit into from
Oct 2, 2026
Merged

docs: align Wrangle README with Chances product format #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
307 changes: 194 additions & 113 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,49 +1,83 @@
# Wrangle

[![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/autonomio/wrangle/badge)](https://scorecard.dev/viewer/?uri=github.com/autonomio/wrangle)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/15135/badge)](https://www.bestpractices.dev/en/projects/15135)

Turn research files into a checked analysis table. Keep the preparation protocol
in a readable YAML file, reuse it for the next batch, and see what changed.
Wrangle serves researchers directly and agents working on their behalf.

## Install

From this checkout, using Python 3.10 or later:
<div align="center">
<br />
<a href="https://github.com/autonomio"><img src="https://avatars.githubusercontent.com/u/28189776?v=4" alt="Autonomio" width="150" height="150" /></a>
<br />
</div>
<br />
<div align="center"><b>Wrangle turns research files into checked analysis tables and reusable preparation evidence.</b></div>

<div align="center">
<a href="#wrangle">Wrangle</a> •
<a href="#what-wrangle-is-not">Scope</a> •
<a href="#capabilities">Capabilities</a> •
<a href="#first-successful-preparation">First Preparation</a> •
<a href="#choose-the-next-task">Learn More</a>
</div>
<br />
<div align="center">
<a href="https://www.bestpractices.dev/projects/15135"><img src="https://www.bestpractices.dev/projects/15135/badge" alt="OpenSSF Best Practices" /></a>
<a href="https://scorecard.dev/viewer/?uri=github.com/autonomio/wrangle"><img src="https://api.scorecard.dev/projects/github.com/autonomio/wrangle/badge" alt="OpenSSF Scorecard" /></a>
<a href="https://pypi.org/project/wrangle/"><img src="https://img.shields.io/pypi/v/wrangle?label=pypi" alt="PyPI version" /></a>
<a href="wrangle/docs/getting_started.md"><img src="https://img.shields.io/badge/docs-manual-blue" alt="Wrangle documentation" /></a>
<a href="https://github.com/autonomio/wrangle/actions/workflows/ci-push.yml"><img src="https://github.com/autonomio/wrangle/actions/workflows/ci-push.yml/badge.svg?branch=master&amp;event=push" alt="Master tests and builds" /></a>
</div>

<hr />

<a id="wrangle"></a>

# Wrangle — Scientific data preparation

*Research files prepared through a readable YAML protocol, with checked tables and execution receipts.*

Wrangle brings inspection, cleaning, matching, reshaping, and validation into
one preparation protocol. Declare the research decisions, prepare the table,
and keep the evidence needed to review the result and reuse the protocol.
Researchers, notebooks, and agents use the same native Polars engine.

## What Wrangle Is Not

Wrangle owns data preparation, protocol checks, and retained evidence.
Researchers own observation identity, units, missingness, exclusions, metadata
matches, and statistical choices. Wrangle does not infer those decisions or
certify their scientific suitability. It does not build or fit models.
All transformations and data checks compile to Polars expressions and LazyFrame
plans; Python handles orchestration, files, and receipts.

## Capabilities

| Research task | Supported capability |
| --- | --- |
| Understand incoming files | Inspect columns, missingness, example rows, summaries, and schema changes |
| Prepare instrument measurements | Preserve identifiers; explicitly parse numbers, dates, missing codes, and text |
| Match samples and metadata | Joins with enforced cardinality, ordering, and unmatched-observation policies |
| Combine batches or reshape observations | Concatenation, pivoting, unpivoting, nested fields, and declared output identity |
| Apply a research protocol | Category mappings, derived fields, unit conversions, and recorded exclusions |
| Prepare longitudinal or grouped data | Group summaries, temporal matching, lag, and rolling calculations |
| Reuse reference-batch decisions | Frozen imputation and standardization parameters; declared sampling and partitions |
| Check and retain an analysis table | Output contracts, readable reports, reusable YAML, and hash receipts |

Use the [recipe manual](wrangle/docs/recipes.md#select-the-operation) to choose
a workflow and the [operation catalog](wrangle/docs/operations.json) for exact
arguments, preconditions, and stable failures.

## First successful preparation

This README describes the **1.0 API in this repository**. Historical 0.7 releases
on PyPI use a different interface. Start from the current source checkout:

```sh
git clone https://github.com/autonomio/wrangle.git
cd wrangle
python -m pip install .
```

Wrangle uses Polars for data preparation and a strict YAML parser for recipe files.
Excel support is an optional extra. Wrangle does not build or fit models.
Use Python 3.10 or later. Installation supplies Polars and the strict YAML parser.
For Excel, install `python -m pip install '.[excel]'` from the checkout and select
a worksheet explicitly. Choose new directories for examples and outputs;
existing destinations are rejected.

## Start with your own files

Run this in a terminal:

```sh
wrangle start measurements.csv --metadata metadata.csv --output my-protocol
```

The guide asks what a row represents, which fields identify it, how measurement
numbers and units should be represented, and what missing values, exclusions and
metadata matches mean. Enter column numbers or names; press Enter if you are not
sure. Wrangle records your answers and creates a commented `recipe.yaml`.

Unanswered decisions block preparation. A complete draft is ready to run checks;
preparation validates the data. The guide prints the exact preparation
command; run it and review the resulting report. The same recipe can then prepare
the next batch. See [Getting started](wrangle/docs/getting_started.md).

For Excel, install `wrangle[excel]` and supply `--sheet "Sheet name"`; the metadata
file uses `--metadata-sheet "Sheet name"`. Worksheet selection is explicit.
Agents use `wrangle start ... --json` to receive the same questions without prompts
and provide researcher-approved decisions through `--answers answers.yaml`.

## Try a complete example

Run these commands in a terminal:
Run this complete example in a terminal:

```sh
wrangle example my-study
Expand All @@ -53,25 +87,22 @@ wrangle prepare recipe.yaml --source measurements=samples.csv --source metadata=
wrangle inspect prepared
```

The example joins sample metadata, applies a declared instrument quality rule,
and converts milligrams to grams. It starts with three samples and produces:
Expected result: three input samples become two accepted samples, with metadata
attached and mass converted from milligrams to grams:

| sample_id | mass_g | qc | group |
| --- | ---: | --- | --- |
| 001 | 1.0 | pass | control |
| 003 | 3.0 | pass | treated |

Sample `002` is excluded because its instrument quality result is `fail`.
The terminal explains the changes. Open `prepared/report.txt` for the saved
summary, including checks and any unresolved declarations.

The example's `recipe.yaml` includes comments explaining each decision.
Use [Getting started](wrangle/docs/getting_started.md) to adapt it to your files;
its quality rule, units and limits are examples, not recommendations for your study.
Sample `002` is excluded by the example's declared instrument quality rule.
Open `prepared/report.txt` to review the checks and exclusion. The commented
`recipe.yaml` explains the supplied decisions; its quality rule, units, and limits
belong to this example and must be adapted to your study.

## Use the same recipe in Python
### Use the same recipe in Python

From the example directory:
From the `my-study` directory:

```python
import wrangle as wr
Expand All @@ -87,82 +118,132 @@ print(result.summary())
result.data # A Polars table with the two accepted samples.
```

For a recipe using only one input file, `wr.prepare("samples.csv", "recipe.yaml")`
is sufficient; that recipe need not declare an `input` source name.
Programmatic Python callers may also supply a mapping instead of a YAML file.
`inspect` observes without changing the source. CSV fields start as text,
preserving identifiers such as `001`; the recipe explicitly converts measurements.
`prepare` returns checked `data` and a `receipt`. A one-source recipe can use
`wr.prepare("samples.csv", "recipe.yaml")`; Python callers may also supply a mapping.

## Your files and decisions
### Start with your own files

`inspect` reports columns, missing values and example rows without changing the
source. CSV fields start as text, preserving sample identifiers such as `001`.
Your recipe explicitly converts measurement fields to numbers.

`prepare` reads local CSV, TSV, Parquet, Arrow IPC or NDJSON files, native Polars
tables, an explicitly selected Excel sheet, or a verified prepared directory.
Multiple sources are named as in the example. Author recipe files as `.yaml` or
`.yml`; JSON recipe files are not an input format.
```sh
wrangle start measurements.csv --metadata metadata.csv --output my-protocol
```

You decide what identifies an observation, what missing values mean, which
observations to exclude, how metadata must match, and which units or statistical
methods apply. Wrangle checks those decisions and stops with an explanation when
the data contradict them. A successful check establishes your declared protocol;
it does not establish the scientific correctness of a chosen rule.
Omit `--metadata metadata.csv` for one file. The guide asks what a row represents,
which fields identify it, how numbers and units should be represented, and what
missing values, exclusions, and metadata matches mean. Enter column numbers or
names; press Enter if you are unsure. Your answers become a commented YAML protocol.

## Keep and reuse the result
Unanswered decisions block preparation. A complete draft is ready to run checks;
preparation validates the data. Run the command printed by the guide and review
the resulting report. Reuse the same recipe for the next batch.
For Excel, supply `--sheet "Sheet name"` and, when needed,
`--metadata-sheet "Sheet name"`. See [Getting started](wrangle/docs/getting_started.md).

Passing `output="new-directory"` saves `data.parquet`, `recipe.yaml`, `report.txt`
and `receipt.json` together. Existing directories are never overwritten. The YAML
is the reusable protocol; the report is for reading; the JSON receipt records
inputs, output, checks, exclusions, parameters and hashes for verification.
Sources remain unchanged. Use `result.write("new-directory")` to save an unsaved
Python result, or use a verified preparation directory as a later source.
## Retain the evidence

For large local files, add `--execution disk --output new-directory` to the CLI,
or use `wr.prepare(sources, "recipe.yaml", execution="disk", output="new-directory")`.
Complete inputs and intermediates stay on disk; the result's `data` is a Polars
LazyFrame. Global joins, sorting and grouped calculations can still need substantial
RAM. `inspect` remains eager. See [disk execution](wrangle/docs/recipes.md#disk-execution).
| File | Retained evidence |
| --- | --- |
| `data.parquet` | Checked analysis table |
| `recipe.yaml` | Resolved, reusable preparation protocol |
| `report.txt` | Readable summary of checks, changes, exclusions, and unresolved declarations |
| `receipt.json` | Input and output hashes, protocol identity, checks, exclusions, and fitted parameters |

## Researchers and agents share one engine
Python's `output="new-directory"` and the CLI's `--output new-directory` save
these files together. Use `result.write("new-directory")` to save an unsaved
Python result. Sources remain unchanged; existing outputs are never overwritten.
A saved preparation directory can be reused as a source after evidence verification.

The CLI prints readable results and recovery guidance by default. Add `--json`
to `start`, `inspect`, `catalog`, `prepare` or `example` for structured output and errors.
The same protocol serves shell scripts and agents:

```sh
wrangle inspect samples.csv --json
wrangle catalog join --json
wrangle prepare recipe.yaml --source measurements=samples.csv --source metadata=metadata.csv --json
```

Recipes, checks and receipts are identical across Python, human CLI output and
agent CLI output. Internal receipts and the generated operation catalog use JSON;
recipe authoring uses YAML only. Comments and formatting do not change recipe identity.

Agents start at [AGENTS.md](AGENTS.md), also shipped inside the installed package.
The [recipe manual](wrangle/docs/recipes.md) gives exact grammar, boundaries and
recovery; the [operation catalog](wrangle/docs/operations.json) is generated from
the engine. The [human workflow](wrangle/docs/human_workflow.py) verifies the
complete example through both presentations. Dedicated agent tools are unnecessary.
Read [migration](wrangle/docs/migration.md) when upgrading from older Wrangle.

## Development and security

[OpenSSF Best Practices progress](https://www.bestpractices.dev/en/projects/15135)
records the published Silver answers, measured checks and owner attestations.

[Contributing](CONTRIBUTING.md) covers installation, required checks and review;
[governance](GOVERNANCE.md), [conduct](CODE_OF_CONDUCT.md), the [roadmap](ROADMAP.md)
and [release notes](CHANGELOG.md) explain how the project is maintained.
Report vulnerabilities through [the security policy](SECURITY.md).

The [architecture](wrangle/docs/architecture.md) and
[security boundaries](wrangle/docs/security.md) describe the actual guarantees.
[Release verification](wrangle/docs/security/releases.md) checks artifact identity;
[OpenSSF evidence](wrangle/docs/security/openssf-evidence.yaml) distinguishes
published controls from work still awaiting verification.

Verified examples and the agent manual ship with the package. All transformations
and data checks use native Polars expressions and plans. Python handles files,
orchestration and receipts. Preparation never uploads data or accesses the network.
The CLI is readable by default; `--json` supplies structured results and errors.
Agents use `wrangle start ... --json` for the same questions without prompts and
`--answers answers.yaml` for researcher-approved decisions. Preparation uses
identical checks and evidence in Python and both CLI presentations.
Recipe files are YAML only (`.yaml` or `.yml`); internal receipts and catalogs use
JSON. Comments and formatting do not change protocol identity.

## Risk Boundary

Passing checks establishes conformity to your declared protocol. It does not
establish the scientific correctness of a chosen rule. Hash receipts check
consistency with retained evidence; they do not establish authorship or constitute
an independent audit. Unresolved guided decisions stop execution with
`UNRESOLVED_PROTOCOL`; answer them rather than removing the blockers.

Preparation accepts local CSV, TSV, Parquet, Arrow IPC, NDJSON, explicitly selected
Excel sheets, native Polars tables, and verified preparation bundles. It never
uploads data or accesses the network. Recipes cannot execute arbitrary callbacks.
Read the [security boundaries](wrangle/docs/security.md) and
[recipe contracts and recovery](wrangle/docs/recipes.md).

For large local files, use `--execution disk --output new-directory`, or
`wr.prepare(sources, "recipe.yaml", execution="disk", output="new-directory")`.
Complete input snapshots, intermediates, and evidence stay on disk; `result.data`
is a Polars LazyFrame. Global joins, sorting, and grouped calculations can still
need substantial RAM. Inspection and guided start remain eager.
See [disk execution](wrangle/docs/recipes.md#disk-execution).

## Choose the next task

| Job | Start here |
| --- | --- |
| Prepare your own files with guided decisions | [Getting started](wrangle/docs/getting_started.md) |
| Try the complete human and agent example | [Verified first study](wrangle/docs/human_workflow.py) |
| Keep and reuse a research batch | [Research batch](wrangle/docs/research_batch.py) |
| Join, reshape, or summarize observations | [Preparation workflows](wrangle/docs/preparation_workflows.py) |
| Reuse fitted reference parameters | [Frozen parameters](wrangle/docs/frozen_parameters.py) |
| Prepare large files on disk | [Disk workflow](wrangle/docs/disk_preparation.py) |
| Select an operation and its preconditions | [Operation catalog](wrangle/docs/operations.json) |
| Integrate Python or an agent | [Agent entrypoint](AGENTS.md) and [recipe manual](wrangle/docs/recipes.md) |
| Run commands and recover from errors | [Command line](wrangle/docs/recipes.md#inspection-and-cli) |
| Replace a historical method | [Migration](wrangle/docs/migration.md) |

Installed agents start at `Path(wrangle.__file__).parent / "AGENTS.md"`;
the installed manual and generated operation contracts live beside it under `docs/`.
The catalog is generated from the engine, and shipped examples are executed in
release checks. Stable `WrangleError` codes and details guide recovery without
substituting a scientific method. Dedicated Wrangle agent tools are unnecessary.

<a id="contribute-support-and-cite"></a>

## Contributing

Start with [CONTRIBUTING.md](CONTRIBUTING.md) for setup, checks, and review.
Propose work through [Wrangle issues](https://github.com/autonomio/wrangle/issues).
[Governance](GOVERNANCE.md), [conduct](CODE_OF_CONDUCT.md), the [roadmap](ROADMAP.md),
and [release notes](CHANGELOG.md) describe project maintenance;
[architecture](wrangle/docs/architecture.md) describes the engine.

## Support

Use [Wrangle issues](https://github.com/autonomio/wrangle/issues) for bug reports,
feature requests, and usage questions. Include a minimal YAML protocol, Wrangle
and Python versions, operating system, and the error code and details.
Use a small shareable fixture when a report needs source data.

## Vulnerabilities

Report suspected vulnerabilities privately through
[GitHub Security Advisories](https://github.com/autonomio/wrangle/security/advisories/new).
Use [SECURITY.md](SECURITY.md) for supported versions and response policy, and
[release verification](wrangle/docs/security/releases.md) to check artifact identity.
[OpenSSF evidence](wrangle/docs/security/openssf-evidence.yaml) records the published
Silver answers, distinguishing measured checks from owner attestations.

## Citations

Identify Wrangle, its version, and the retained preparation protocol and receipt
when citing a research workflow. Link the corresponding
[release](https://github.com/autonomio/wrangle/releases) to identify the software used.
No DOI is supplied.

## License

[MIT License](LICENSE).
5 changes: 4 additions & 1 deletion scripts/build_catalog.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,10 @@
content = json.dumps(document, indent=2, ensure_ascii=False, sort_keys=True) + "\n"
readme_path = ROOT / "wrangle" / "docs" / "README.md"
readme = (ROOT / "README.md").read_text(encoding="utf-8")
readme = readme.replace("](AGENTS.md)", "](../AGENTS.md)").replace("](wrangle/docs/", "](").replace("[MIT License](LICENSE).", "MIT License; see the distribution's license metadata.")
readme = (readme.replace("](AGENTS.md)", "](../AGENTS.md)")
.replace("](wrangle/docs/", "](")
.replace('href="wrangle/docs/', 'href="')
.replace("[MIT License](LICENSE).", "MIT License; see the distribution's license metadata."))
readme += "\n## Agent navigation\n\nThis installed manual is relative to the package root. In the operation catalog,\n`returns` identifies the output kind and `recipe.eligibility` distinguishes table\nsteps from direct calls. Read each operation's `validation` before executing it.\nAn aggregate changes the individual-record key: declare step.key for the checked\noutput observation unit.\n"
POLICIES = ('CONTRIBUTING.md', 'SECURITY.md', 'GOVERNANCE.md', 'CODE_OF_CONDUCT.md', 'ROADMAP.md', 'CHANGELOG.md')
for name in POLICIES:
Expand Down
Loading
Loading