From a44056b140bd5af58864d2a2929951d7ed66ffda Mon Sep 17 00:00:00 2001 From: Mikko Kotila Date: Fri, 2 Oct 2026 08:31:16 +0300 Subject: [PATCH] docs: align Wrangle README with Chances product format --- README.md | 307 +++++++++++++++++++++++++-------------- scripts/build_catalog.py | 5 +- wrangle/docs/README.md | 307 +++++++++++++++++++++++++-------------- 3 files changed, 392 insertions(+), 227 deletions(-) diff --git a/README.md b/README.md index 8f4e6dc..fa43aa5 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,83 @@ -# Wrangle - -[![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/autonomio/wrangle/badge)](https://scorecard.dev/viewer/?uri=github.com/autonomio/wrangle) -[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/15135/badge)](https://www.bestpractices.dev/en/projects/15135) - -Turn research files into a checked analysis table. Keep the preparation protocol -in a readable YAML file, reuse it for the next batch, and see what changed. -Wrangle serves researchers directly and agents working on their behalf. - -## Install - -From this checkout, using Python 3.10 or later: +
+
+ Autonomio +
+
+
+
Wrangle turns research files into checked analysis tables and reusable preparation evidence.
+ +
+ Wrangle • + Scope • + Capabilities • + First Preparation • + Learn More +
+
+
+ OpenSSF Best Practices + OpenSSF Scorecard + PyPI version + Wrangle documentation + Master tests and builds +
+ +
+ + + +# Wrangle — Scientific data preparation + +*Research files prepared through a readable YAML protocol, with checked tables and execution receipts.* + +Wrangle brings inspection, cleaning, matching, reshaping, and validation into +one preparation protocol. Declare the research decisions, prepare the table, +and keep the evidence needed to review the result and reuse the protocol. +Researchers, notebooks, and agents use the same native Polars engine. + +## What Wrangle Is Not + +Wrangle owns data preparation, protocol checks, and retained evidence. +Researchers own observation identity, units, missingness, exclusions, metadata +matches, and statistical choices. Wrangle does not infer those decisions or +certify their scientific suitability. It does not build or fit models. +All transformations and data checks compile to Polars expressions and LazyFrame +plans; Python handles orchestration, files, and receipts. + +## Capabilities + +| Research task | Supported capability | +| --- | --- | +| Understand incoming files | Inspect columns, missingness, example rows, summaries, and schema changes | +| Prepare instrument measurements | Preserve identifiers; explicitly parse numbers, dates, missing codes, and text | +| Match samples and metadata | Joins with enforced cardinality, ordering, and unmatched-observation policies | +| Combine batches or reshape observations | Concatenation, pivoting, unpivoting, nested fields, and declared output identity | +| Apply a research protocol | Category mappings, derived fields, unit conversions, and recorded exclusions | +| Prepare longitudinal or grouped data | Group summaries, temporal matching, lag, and rolling calculations | +| Reuse reference-batch decisions | Frozen imputation and standardization parameters; declared sampling and partitions | +| Check and retain an analysis table | Output contracts, readable reports, reusable YAML, and hash receipts | + +Use the [recipe manual](wrangle/docs/recipes.md#select-the-operation) to choose +a workflow and the [operation catalog](wrangle/docs/operations.json) for exact +arguments, preconditions, and stable failures. + +## First successful preparation + +This README describes the **1.0 API in this repository**. Historical 0.7 releases +on PyPI use a different interface. Start from the current source checkout: ```sh +git clone https://github.com/autonomio/wrangle.git +cd wrangle python -m pip install . ``` -Wrangle uses Polars for data preparation and a strict YAML parser for recipe files. -Excel support is an optional extra. Wrangle does not build or fit models. +Use Python 3.10 or later. Installation supplies Polars and the strict YAML parser. +For Excel, install `python -m pip install '.[excel]'` from the checkout and select +a worksheet explicitly. Choose new directories for examples and outputs; +existing destinations are rejected. -## Start with your own files - -Run this in a terminal: - -```sh -wrangle start measurements.csv --metadata metadata.csv --output my-protocol -``` - -The guide asks what a row represents, which fields identify it, how measurement -numbers and units should be represented, and what missing values, exclusions and -metadata matches mean. Enter column numbers or names; press Enter if you are not -sure. Wrangle records your answers and creates a commented `recipe.yaml`. - -Unanswered decisions block preparation. A complete draft is ready to run checks; -preparation validates the data. The guide prints the exact preparation -command; run it and review the resulting report. The same recipe can then prepare -the next batch. See [Getting started](wrangle/docs/getting_started.md). - -For Excel, install `wrangle[excel]` and supply `--sheet "Sheet name"`; the metadata -file uses `--metadata-sheet "Sheet name"`. Worksheet selection is explicit. -Agents use `wrangle start ... --json` to receive the same questions without prompts -and provide researcher-approved decisions through `--answers answers.yaml`. - -## Try a complete example - -Run these commands in a terminal: +Run this complete example in a terminal: ```sh wrangle example my-study @@ -53,25 +87,22 @@ wrangle prepare recipe.yaml --source measurements=samples.csv --source metadata= wrangle inspect prepared ``` -The example joins sample metadata, applies a declared instrument quality rule, -and converts milligrams to grams. It starts with three samples and produces: +Expected result: three input samples become two accepted samples, with metadata +attached and mass converted from milligrams to grams: | sample_id | mass_g | qc | group | | --- | ---: | --- | --- | | 001 | 1.0 | pass | control | | 003 | 3.0 | pass | treated | -Sample `002` is excluded because its instrument quality result is `fail`. -The terminal explains the changes. Open `prepared/report.txt` for the saved -summary, including checks and any unresolved declarations. - -The example's `recipe.yaml` includes comments explaining each decision. -Use [Getting started](wrangle/docs/getting_started.md) to adapt it to your files; -its quality rule, units and limits are examples, not recommendations for your study. +Sample `002` is excluded by the example's declared instrument quality rule. +Open `prepared/report.txt` to review the checks and exclusion. The commented +`recipe.yaml` explains the supplied decisions; its quality rule, units, and limits +belong to this example and must be adapted to your study. -## Use the same recipe in Python +### Use the same recipe in Python -From the example directory: +From the `my-study` directory: ```python import wrangle as wr @@ -87,46 +118,43 @@ print(result.summary()) result.data # A Polars table with the two accepted samples. ``` -For a recipe using only one input file, `wr.prepare("samples.csv", "recipe.yaml")` -is sufficient; that recipe need not declare an `input` source name. -Programmatic Python callers may also supply a mapping instead of a YAML file. +`inspect` observes without changing the source. CSV fields start as text, +preserving identifiers such as `001`; the recipe explicitly converts measurements. +`prepare` returns checked `data` and a `receipt`. A one-source recipe can use +`wr.prepare("samples.csv", "recipe.yaml")`; Python callers may also supply a mapping. -## Your files and decisions +### Start with your own files -`inspect` reports columns, missing values and example rows without changing the -source. CSV fields start as text, preserving sample identifiers such as `001`. -Your recipe explicitly converts measurement fields to numbers. - -`prepare` reads local CSV, TSV, Parquet, Arrow IPC or NDJSON files, native Polars -tables, an explicitly selected Excel sheet, or a verified prepared directory. -Multiple sources are named as in the example. Author recipe files as `.yaml` or -`.yml`; JSON recipe files are not an input format. +```sh +wrangle start measurements.csv --metadata metadata.csv --output my-protocol +``` -You decide what identifies an observation, what missing values mean, which -observations to exclude, how metadata must match, and which units or statistical -methods apply. Wrangle checks those decisions and stops with an explanation when -the data contradict them. A successful check establishes your declared protocol; -it does not establish the scientific correctness of a chosen rule. +Omit `--metadata metadata.csv` for one file. The guide asks what a row represents, +which fields identify it, how numbers and units should be represented, and what +missing values, exclusions, and metadata matches mean. Enter column numbers or +names; press Enter if you are unsure. Your answers become a commented YAML protocol. -## Keep and reuse the result +Unanswered decisions block preparation. A complete draft is ready to run checks; +preparation validates the data. Run the command printed by the guide and review +the resulting report. Reuse the same recipe for the next batch. +For Excel, supply `--sheet "Sheet name"` and, when needed, +`--metadata-sheet "Sheet name"`. See [Getting started](wrangle/docs/getting_started.md). -Passing `output="new-directory"` saves `data.parquet`, `recipe.yaml`, `report.txt` -and `receipt.json` together. Existing directories are never overwritten. The YAML -is the reusable protocol; the report is for reading; the JSON receipt records -inputs, output, checks, exclusions, parameters and hashes for verification. -Sources remain unchanged. Use `result.write("new-directory")` to save an unsaved -Python result, or use a verified preparation directory as a later source. +## Retain the evidence -For large local files, add `--execution disk --output new-directory` to the CLI, -or use `wr.prepare(sources, "recipe.yaml", execution="disk", output="new-directory")`. -Complete inputs and intermediates stay on disk; the result's `data` is a Polars -LazyFrame. Global joins, sorting and grouped calculations can still need substantial -RAM. `inspect` remains eager. See [disk execution](wrangle/docs/recipes.md#disk-execution). +| File | Retained evidence | +| --- | --- | +| `data.parquet` | Checked analysis table | +| `recipe.yaml` | Resolved, reusable preparation protocol | +| `report.txt` | Readable summary of checks, changes, exclusions, and unresolved declarations | +| `receipt.json` | Input and output hashes, protocol identity, checks, exclusions, and fitted parameters | -## Researchers and agents share one engine +Python's `output="new-directory"` and the CLI's `--output new-directory` save +these files together. Use `result.write("new-directory")` to save an unsaved +Python result. Sources remain unchanged; existing outputs are never overwritten. +A saved preparation directory can be reused as a source after evidence verification. -The CLI prints readable results and recovery guidance by default. Add `--json` -to `start`, `inspect`, `catalog`, `prepare` or `example` for structured output and errors. +The same protocol serves shell scripts and agents: ```sh wrangle inspect samples.csv --json @@ -134,35 +162,88 @@ wrangle catalog join --json wrangle prepare recipe.yaml --source measurements=samples.csv --source metadata=metadata.csv --json ``` -Recipes, checks and receipts are identical across Python, human CLI output and -agent CLI output. Internal receipts and the generated operation catalog use JSON; -recipe authoring uses YAML only. Comments and formatting do not change recipe identity. - -Agents start at [AGENTS.md](AGENTS.md), also shipped inside the installed package. -The [recipe manual](wrangle/docs/recipes.md) gives exact grammar, boundaries and -recovery; the [operation catalog](wrangle/docs/operations.json) is generated from -the engine. The [human workflow](wrangle/docs/human_workflow.py) verifies the -complete example through both presentations. Dedicated agent tools are unnecessary. -Read [migration](wrangle/docs/migration.md) when upgrading from older Wrangle. - -## Development and security - -[OpenSSF Best Practices progress](https://www.bestpractices.dev/en/projects/15135) -records the published Silver answers, measured checks and owner attestations. - -[Contributing](CONTRIBUTING.md) covers installation, required checks and review; -[governance](GOVERNANCE.md), [conduct](CODE_OF_CONDUCT.md), the [roadmap](ROADMAP.md) -and [release notes](CHANGELOG.md) explain how the project is maintained. -Report vulnerabilities through [the security policy](SECURITY.md). - -The [architecture](wrangle/docs/architecture.md) and -[security boundaries](wrangle/docs/security.md) describe the actual guarantees. -[Release verification](wrangle/docs/security/releases.md) checks artifact identity; -[OpenSSF evidence](wrangle/docs/security/openssf-evidence.yaml) distinguishes -published controls from work still awaiting verification. - -Verified examples and the agent manual ship with the package. All transformations -and data checks use native Polars expressions and plans. Python handles files, -orchestration and receipts. Preparation never uploads data or accesses the network. +The CLI is readable by default; `--json` supplies structured results and errors. +Agents use `wrangle start ... --json` for the same questions without prompts and +`--answers answers.yaml` for researcher-approved decisions. Preparation uses +identical checks and evidence in Python and both CLI presentations. +Recipe files are YAML only (`.yaml` or `.yml`); internal receipts and catalogs use +JSON. Comments and formatting do not change protocol identity. + +## Risk Boundary + +Passing checks establishes conformity to your declared protocol. It does not +establish the scientific correctness of a chosen rule. Hash receipts check +consistency with retained evidence; they do not establish authorship or constitute +an independent audit. Unresolved guided decisions stop execution with +`UNRESOLVED_PROTOCOL`; answer them rather than removing the blockers. + +Preparation accepts local CSV, TSV, Parquet, Arrow IPC, NDJSON, explicitly selected +Excel sheets, native Polars tables, and verified preparation bundles. It never +uploads data or accesses the network. Recipes cannot execute arbitrary callbacks. +Read the [security boundaries](wrangle/docs/security.md) and +[recipe contracts and recovery](wrangle/docs/recipes.md). + +For large local files, use `--execution disk --output new-directory`, or +`wr.prepare(sources, "recipe.yaml", execution="disk", output="new-directory")`. +Complete input snapshots, intermediates, and evidence stay on disk; `result.data` +is a Polars LazyFrame. Global joins, sorting, and grouped calculations can still +need substantial RAM. Inspection and guided start remain eager. +See [disk execution](wrangle/docs/recipes.md#disk-execution). + +## Choose the next task + +| Job | Start here | +| --- | --- | +| Prepare your own files with guided decisions | [Getting started](wrangle/docs/getting_started.md) | +| Try the complete human and agent example | [Verified first study](wrangle/docs/human_workflow.py) | +| Keep and reuse a research batch | [Research batch](wrangle/docs/research_batch.py) | +| Join, reshape, or summarize observations | [Preparation workflows](wrangle/docs/preparation_workflows.py) | +| Reuse fitted reference parameters | [Frozen parameters](wrangle/docs/frozen_parameters.py) | +| Prepare large files on disk | [Disk workflow](wrangle/docs/disk_preparation.py) | +| Select an operation and its preconditions | [Operation catalog](wrangle/docs/operations.json) | +| Integrate Python or an agent | [Agent entrypoint](AGENTS.md) and [recipe manual](wrangle/docs/recipes.md) | +| Run commands and recover from errors | [Command line](wrangle/docs/recipes.md#inspection-and-cli) | +| Replace a historical method | [Migration](wrangle/docs/migration.md) | + +Installed agents start at `Path(wrangle.__file__).parent / "AGENTS.md"`; +the installed manual and generated operation contracts live beside it under `docs/`. +The catalog is generated from the engine, and shipped examples are executed in +release checks. Stable `WrangleError` codes and details guide recovery without +substituting a scientific method. Dedicated Wrangle agent tools are unnecessary. + + + +## Contributing + +Start with [CONTRIBUTING.md](CONTRIBUTING.md) for setup, checks, and review. +Propose work through [Wrangle issues](https://github.com/autonomio/wrangle/issues). +[Governance](GOVERNANCE.md), [conduct](CODE_OF_CONDUCT.md), the [roadmap](ROADMAP.md), +and [release notes](CHANGELOG.md) describe project maintenance; +[architecture](wrangle/docs/architecture.md) describes the engine. + +## Support + +Use [Wrangle issues](https://github.com/autonomio/wrangle/issues) for bug reports, +feature requests, and usage questions. Include a minimal YAML protocol, Wrangle +and Python versions, operating system, and the error code and details. +Use a small shareable fixture when a report needs source data. + +## Vulnerabilities + +Report suspected vulnerabilities privately through +[GitHub Security Advisories](https://github.com/autonomio/wrangle/security/advisories/new). +Use [SECURITY.md](SECURITY.md) for supported versions and response policy, and +[release verification](wrangle/docs/security/releases.md) to check artifact identity. +[OpenSSF evidence](wrangle/docs/security/openssf-evidence.yaml) records the published +Silver answers, distinguishing measured checks from owner attestations. + +## Citations + +Identify Wrangle, its version, and the retained preparation protocol and receipt +when citing a research workflow. Link the corresponding +[release](https://github.com/autonomio/wrangle/releases) to identify the software used. +No DOI is supplied. + +## License [MIT License](LICENSE). diff --git a/scripts/build_catalog.py b/scripts/build_catalog.py index 95bc3c9..e2a6098 100644 --- a/scripts/build_catalog.py +++ b/scripts/build_catalog.py @@ -16,7 +16,10 @@ content = json.dumps(document, indent=2, ensure_ascii=False, sort_keys=True) + "\n" readme_path = ROOT / "wrangle" / "docs" / "README.md" readme = (ROOT / "README.md").read_text(encoding="utf-8") -readme = readme.replace("](AGENTS.md)", "](../AGENTS.md)").replace("](wrangle/docs/", "](").replace("[MIT License](LICENSE).", "MIT License; see the distribution's license metadata.") +readme = (readme.replace("](AGENTS.md)", "](../AGENTS.md)") + .replace("](wrangle/docs/", "](") + .replace('href="wrangle/docs/', 'href="') + .replace("[MIT License](LICENSE).", "MIT License; see the distribution's license metadata.")) readme += "\n## Agent navigation\n\nThis installed manual is relative to the package root. In the operation catalog,\n`returns` identifies the output kind and `recipe.eligibility` distinguishes table\nsteps from direct calls. Read each operation's `validation` before executing it.\nAn aggregate changes the individual-record key: declare step.key for the checked\noutput observation unit.\n" POLICIES = ('CONTRIBUTING.md', 'SECURITY.md', 'GOVERNANCE.md', 'CODE_OF_CONDUCT.md', 'ROADMAP.md', 'CHANGELOG.md') for name in POLICIES: diff --git a/wrangle/docs/README.md b/wrangle/docs/README.md index ff7502a..d239337 100644 --- a/wrangle/docs/README.md +++ b/wrangle/docs/README.md @@ -1,49 +1,83 @@ -# Wrangle - -[![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/autonomio/wrangle/badge)](https://scorecard.dev/viewer/?uri=github.com/autonomio/wrangle) -[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/15135/badge)](https://www.bestpractices.dev/en/projects/15135) - -Turn research files into a checked analysis table. Keep the preparation protocol -in a readable YAML file, reuse it for the next batch, and see what changed. -Wrangle serves researchers directly and agents working on their behalf. - -## Install - -From this checkout, using Python 3.10 or later: +
+
+ Autonomio +
+
+
+
Wrangle turns research files into checked analysis tables and reusable preparation evidence.
+ +
+ Wrangle • + Scope • + Capabilities • + First Preparation • + Learn More +
+
+
+ OpenSSF Best Practices + OpenSSF Scorecard + PyPI version + Wrangle documentation + Master tests and builds +
+ +
+ + + +# Wrangle — Scientific data preparation + +*Research files prepared through a readable YAML protocol, with checked tables and execution receipts.* + +Wrangle brings inspection, cleaning, matching, reshaping, and validation into +one preparation protocol. Declare the research decisions, prepare the table, +and keep the evidence needed to review the result and reuse the protocol. +Researchers, notebooks, and agents use the same native Polars engine. + +## What Wrangle Is Not + +Wrangle owns data preparation, protocol checks, and retained evidence. +Researchers own observation identity, units, missingness, exclusions, metadata +matches, and statistical choices. Wrangle does not infer those decisions or +certify their scientific suitability. It does not build or fit models. +All transformations and data checks compile to Polars expressions and LazyFrame +plans; Python handles orchestration, files, and receipts. + +## Capabilities + +| Research task | Supported capability | +| --- | --- | +| Understand incoming files | Inspect columns, missingness, example rows, summaries, and schema changes | +| Prepare instrument measurements | Preserve identifiers; explicitly parse numbers, dates, missing codes, and text | +| Match samples and metadata | Joins with enforced cardinality, ordering, and unmatched-observation policies | +| Combine batches or reshape observations | Concatenation, pivoting, unpivoting, nested fields, and declared output identity | +| Apply a research protocol | Category mappings, derived fields, unit conversions, and recorded exclusions | +| Prepare longitudinal or grouped data | Group summaries, temporal matching, lag, and rolling calculations | +| Reuse reference-batch decisions | Frozen imputation and standardization parameters; declared sampling and partitions | +| Check and retain an analysis table | Output contracts, readable reports, reusable YAML, and hash receipts | + +Use the [recipe manual](recipes.md#select-the-operation) to choose +a workflow and the [operation catalog](operations.json) for exact +arguments, preconditions, and stable failures. + +## First successful preparation + +This README describes the **1.0 API in this repository**. Historical 0.7 releases +on PyPI use a different interface. Start from the current source checkout: ```sh +git clone https://github.com/autonomio/wrangle.git +cd wrangle python -m pip install . ``` -Wrangle uses Polars for data preparation and a strict YAML parser for recipe files. -Excel support is an optional extra. Wrangle does not build or fit models. +Use Python 3.10 or later. Installation supplies Polars and the strict YAML parser. +For Excel, install `python -m pip install '.[excel]'` from the checkout and select +a worksheet explicitly. Choose new directories for examples and outputs; +existing destinations are rejected. -## Start with your own files - -Run this in a terminal: - -```sh -wrangle start measurements.csv --metadata metadata.csv --output my-protocol -``` - -The guide asks what a row represents, which fields identify it, how measurement -numbers and units should be represented, and what missing values, exclusions and -metadata matches mean. Enter column numbers or names; press Enter if you are not -sure. Wrangle records your answers and creates a commented `recipe.yaml`. - -Unanswered decisions block preparation. A complete draft is ready to run checks; -preparation validates the data. The guide prints the exact preparation -command; run it and review the resulting report. The same recipe can then prepare -the next batch. See [Getting started](getting_started.md). - -For Excel, install `wrangle[excel]` and supply `--sheet "Sheet name"`; the metadata -file uses `--metadata-sheet "Sheet name"`. Worksheet selection is explicit. -Agents use `wrangle start ... --json` to receive the same questions without prompts -and provide researcher-approved decisions through `--answers answers.yaml`. - -## Try a complete example - -Run these commands in a terminal: +Run this complete example in a terminal: ```sh wrangle example my-study @@ -53,25 +87,22 @@ wrangle prepare recipe.yaml --source measurements=samples.csv --source metadata= wrangle inspect prepared ``` -The example joins sample metadata, applies a declared instrument quality rule, -and converts milligrams to grams. It starts with three samples and produces: +Expected result: three input samples become two accepted samples, with metadata +attached and mass converted from milligrams to grams: | sample_id | mass_g | qc | group | | --- | ---: | --- | --- | | 001 | 1.0 | pass | control | | 003 | 3.0 | pass | treated | -Sample `002` is excluded because its instrument quality result is `fail`. -The terminal explains the changes. Open `prepared/report.txt` for the saved -summary, including checks and any unresolved declarations. - -The example's `recipe.yaml` includes comments explaining each decision. -Use [Getting started](getting_started.md) to adapt it to your files; -its quality rule, units and limits are examples, not recommendations for your study. +Sample `002` is excluded by the example's declared instrument quality rule. +Open `prepared/report.txt` to review the checks and exclusion. The commented +`recipe.yaml` explains the supplied decisions; its quality rule, units, and limits +belong to this example and must be adapted to your study. -## Use the same recipe in Python +### Use the same recipe in Python -From the example directory: +From the `my-study` directory: ```python import wrangle as wr @@ -87,46 +118,43 @@ print(result.summary()) result.data # A Polars table with the two accepted samples. ``` -For a recipe using only one input file, `wr.prepare("samples.csv", "recipe.yaml")` -is sufficient; that recipe need not declare an `input` source name. -Programmatic Python callers may also supply a mapping instead of a YAML file. +`inspect` observes without changing the source. CSV fields start as text, +preserving identifiers such as `001`; the recipe explicitly converts measurements. +`prepare` returns checked `data` and a `receipt`. A one-source recipe can use +`wr.prepare("samples.csv", "recipe.yaml")`; Python callers may also supply a mapping. -## Your files and decisions +### Start with your own files -`inspect` reports columns, missing values and example rows without changing the -source. CSV fields start as text, preserving sample identifiers such as `001`. -Your recipe explicitly converts measurement fields to numbers. - -`prepare` reads local CSV, TSV, Parquet, Arrow IPC or NDJSON files, native Polars -tables, an explicitly selected Excel sheet, or a verified prepared directory. -Multiple sources are named as in the example. Author recipe files as `.yaml` or -`.yml`; JSON recipe files are not an input format. +```sh +wrangle start measurements.csv --metadata metadata.csv --output my-protocol +``` -You decide what identifies an observation, what missing values mean, which -observations to exclude, how metadata must match, and which units or statistical -methods apply. Wrangle checks those decisions and stops with an explanation when -the data contradict them. A successful check establishes your declared protocol; -it does not establish the scientific correctness of a chosen rule. +Omit `--metadata metadata.csv` for one file. The guide asks what a row represents, +which fields identify it, how numbers and units should be represented, and what +missing values, exclusions, and metadata matches mean. Enter column numbers or +names; press Enter if you are unsure. Your answers become a commented YAML protocol. -## Keep and reuse the result +Unanswered decisions block preparation. A complete draft is ready to run checks; +preparation validates the data. Run the command printed by the guide and review +the resulting report. Reuse the same recipe for the next batch. +For Excel, supply `--sheet "Sheet name"` and, when needed, +`--metadata-sheet "Sheet name"`. See [Getting started](getting_started.md). -Passing `output="new-directory"` saves `data.parquet`, `recipe.yaml`, `report.txt` -and `receipt.json` together. Existing directories are never overwritten. The YAML -is the reusable protocol; the report is for reading; the JSON receipt records -inputs, output, checks, exclusions, parameters and hashes for verification. -Sources remain unchanged. Use `result.write("new-directory")` to save an unsaved -Python result, or use a verified preparation directory as a later source. +## Retain the evidence -For large local files, add `--execution disk --output new-directory` to the CLI, -or use `wr.prepare(sources, "recipe.yaml", execution="disk", output="new-directory")`. -Complete inputs and intermediates stay on disk; the result's `data` is a Polars -LazyFrame. Global joins, sorting and grouped calculations can still need substantial -RAM. `inspect` remains eager. See [disk execution](recipes.md#disk-execution). +| File | Retained evidence | +| --- | --- | +| `data.parquet` | Checked analysis table | +| `recipe.yaml` | Resolved, reusable preparation protocol | +| `report.txt` | Readable summary of checks, changes, exclusions, and unresolved declarations | +| `receipt.json` | Input and output hashes, protocol identity, checks, exclusions, and fitted parameters | -## Researchers and agents share one engine +Python's `output="new-directory"` and the CLI's `--output new-directory` save +these files together. Use `result.write("new-directory")` to save an unsaved +Python result. Sources remain unchanged; existing outputs are never overwritten. +A saved preparation directory can be reused as a source after evidence verification. -The CLI prints readable results and recovery guidance by default. Add `--json` -to `start`, `inspect`, `catalog`, `prepare` or `example` for structured output and errors. +The same protocol serves shell scripts and agents: ```sh wrangle inspect samples.csv --json @@ -134,36 +162,89 @@ wrangle catalog join --json wrangle prepare recipe.yaml --source measurements=samples.csv --source metadata=metadata.csv --json ``` -Recipes, checks and receipts are identical across Python, human CLI output and -agent CLI output. Internal receipts and the generated operation catalog use JSON; -recipe authoring uses YAML only. Comments and formatting do not change recipe identity. - -Agents start at [AGENTS.md](../AGENTS.md), also shipped inside the installed package. -The [recipe manual](recipes.md) gives exact grammar, boundaries and -recovery; the [operation catalog](operations.json) is generated from -the engine. The [human workflow](human_workflow.py) verifies the -complete example through both presentations. Dedicated agent tools are unnecessary. -Read [migration](migration.md) when upgrading from older Wrangle. - -## Development and security - -[OpenSSF Best Practices progress](https://www.bestpractices.dev/en/projects/15135) -records the published Silver answers, measured checks and owner attestations. - -[Contributing](project/CONTRIBUTING.md) covers installation, required checks and review; -[governance](project/GOVERNANCE.md), [conduct](project/CODE_OF_CONDUCT.md), the [roadmap](project/ROADMAP.md) -and [release notes](project/CHANGELOG.md) explain how the project is maintained. -Report vulnerabilities through [the security policy](project/SECURITY.md). - -The [architecture](architecture.md) and -[security boundaries](security.md) describe the actual guarantees. -[Release verification](security/releases.md) checks artifact identity; -[OpenSSF evidence](security/openssf-evidence.yaml) distinguishes -published controls from work still awaiting verification. - -Verified examples and the agent manual ship with the package. All transformations -and data checks use native Polars expressions and plans. Python handles files, -orchestration and receipts. Preparation never uploads data or accesses the network. +The CLI is readable by default; `--json` supplies structured results and errors. +Agents use `wrangle start ... --json` for the same questions without prompts and +`--answers answers.yaml` for researcher-approved decisions. Preparation uses +identical checks and evidence in Python and both CLI presentations. +Recipe files are YAML only (`.yaml` or `.yml`); internal receipts and catalogs use +JSON. Comments and formatting do not change protocol identity. + +## Risk Boundary + +Passing checks establishes conformity to your declared protocol. It does not +establish the scientific correctness of a chosen rule. Hash receipts check +consistency with retained evidence; they do not establish authorship or constitute +an independent audit. Unresolved guided decisions stop execution with +`UNRESOLVED_PROTOCOL`; answer them rather than removing the blockers. + +Preparation accepts local CSV, TSV, Parquet, Arrow IPC, NDJSON, explicitly selected +Excel sheets, native Polars tables, and verified preparation bundles. It never +uploads data or accesses the network. Recipes cannot execute arbitrary callbacks. +Read the [security boundaries](security.md) and +[recipe contracts and recovery](recipes.md). + +For large local files, use `--execution disk --output new-directory`, or +`wr.prepare(sources, "recipe.yaml", execution="disk", output="new-directory")`. +Complete input snapshots, intermediates, and evidence stay on disk; `result.data` +is a Polars LazyFrame. Global joins, sorting, and grouped calculations can still +need substantial RAM. Inspection and guided start remain eager. +See [disk execution](recipes.md#disk-execution). + +## Choose the next task + +| Job | Start here | +| --- | --- | +| Prepare your own files with guided decisions | [Getting started](getting_started.md) | +| Try the complete human and agent example | [Verified first study](human_workflow.py) | +| Keep and reuse a research batch | [Research batch](research_batch.py) | +| Join, reshape, or summarize observations | [Preparation workflows](preparation_workflows.py) | +| Reuse fitted reference parameters | [Frozen parameters](frozen_parameters.py) | +| Prepare large files on disk | [Disk workflow](disk_preparation.py) | +| Select an operation and its preconditions | [Operation catalog](operations.json) | +| Integrate Python or an agent | [Agent entrypoint](../AGENTS.md) and [recipe manual](recipes.md) | +| Run commands and recover from errors | [Command line](recipes.md#inspection-and-cli) | +| Replace a historical method | [Migration](migration.md) | + +Installed agents start at `Path(wrangle.__file__).parent / "AGENTS.md"`; +the installed manual and generated operation contracts live beside it under `docs/`. +The catalog is generated from the engine, and shipped examples are executed in +release checks. Stable `WrangleError` codes and details guide recovery without +substituting a scientific method. Dedicated Wrangle agent tools are unnecessary. + + + +## Contributing + +Start with [CONTRIBUTING.md](project/CONTRIBUTING.md) for setup, checks, and review. +Propose work through [Wrangle issues](https://github.com/autonomio/wrangle/issues). +[Governance](project/GOVERNANCE.md), [conduct](project/CODE_OF_CONDUCT.md), the [roadmap](project/ROADMAP.md), +and [release notes](project/CHANGELOG.md) describe project maintenance; +[architecture](architecture.md) describes the engine. + +## Support + +Use [Wrangle issues](https://github.com/autonomio/wrangle/issues) for bug reports, +feature requests, and usage questions. Include a minimal YAML protocol, Wrangle +and Python versions, operating system, and the error code and details. +Use a small shareable fixture when a report needs source data. + +## Vulnerabilities + +Report suspected vulnerabilities privately through +[GitHub Security Advisories](https://github.com/autonomio/wrangle/security/advisories/new). +Use [SECURITY.md](project/SECURITY.md) for supported versions and response policy, and +[release verification](security/releases.md) to check artifact identity. +[OpenSSF evidence](security/openssf-evidence.yaml) records the published +Silver answers, distinguishing measured checks from owner attestations. + +## Citations + +Identify Wrangle, its version, and the retained preparation protocol and receipt +when citing a research workflow. Link the corresponding +[release](https://github.com/autonomio/wrangle/releases) to identify the software used. +No DOI is supplied. + +## License MIT License; see the distribution's license metadata.