diff --git a/.claude/commands/convert-definition-lists.md b/.claude/commands/convert-definition-lists.md new file mode 100644 index 0000000000..6262366254 --- /dev/null +++ b/.claude/commands/convert-definition-lists.md @@ -0,0 +1,275 @@ +--- +description: Convert Markdown definition lists to heading format for any documentation page +--- + +# Convert Definition Lists to Headings + +Convert Markdown definition list syntax to heading format across all documentation pages including CLI options, environment variables, configuration settings, and API parameters. + +## Task + +Search for and convert all definition list entries from the MyST definition list syntax: +```markdown +`term` +: Description text here. +``` + +To heading format: +```markdown +##### `term` + +Description text here. +``` + +This applies to various documentation contexts: +- CLI options and flags +- Environment variables +- Configuration settings +- API/function parameters +- Type definitions +- Any other documented items using definition lists + +## Conversion Rules + +### Top-Level Definition Lists (h5 headings) + +Use h5 headings (`#####`) for top-level items that are primary documentation entries. + +1. **Simple definitions:** + ```markdown + `TERM_NAME` + : Description text here. + ``` + Converts to: + ```markdown + ##### `TERM_NAME` + + Description text here. + ``` + +2. **With default values or metadata:** + ```markdown + `NXF_OPTS` (`-Xms512m`) + : JVM options for Nextflow execution. + ``` + Converts to: + ```markdown + ##### `NXF_OPTS` (`-Xms512m`) + + JVM options for Nextflow execution. + ``` + +3. **Multi-line descriptions:** + ```markdown + `NXF_WORK` + : Directory where working files are stored. + + Default: `./work` + ``` + Converts to: + ```markdown + ##### `NXF_WORK` + + Directory where working files are stored. + + Default: `./work` + ``` + +### Examples by Documentation Type + +**Environment Variables:** +```markdown +`NXF_HOME` +: Nextflow home directory +``` +→ `##### `NXF_HOME`` + +**CLI Options:** +```markdown +`-with-docker` +: Enable Docker container execution +``` +→ `##### `-with-docker`` + +**Configuration Settings:** +```markdown +`process.executor` +: Defines the executor to use +``` +→ `##### `process.executor`` + +## Embedded Definition Lists (h6 headings) + +Embedded definition lists are indented and represent nested or sub-items within a section. These should be converted to h6 headings to preserve hierarchy. + +### Pattern + +```markdown + `nestedItem` + : Description text here (default: *value*) +``` + +### Conversion Format + +**Primary conversion to h6 heading (default):** +```markdown +###### `nestedItem` + +Description text here (default: *value*) +``` + +### Examples by Context + +**Function Parameters:** +```markdown + `maxDepth: int` + : Maximum number of directory levels to visit (default: *no limit*) +``` +→ +```markdown +###### `maxDepth: int` + +Maximum number of directory levels to visit (default: *no limit*) +``` + +**Configuration Sub-options:** +```markdown + `retry.maxAttempts` + : Maximum number of retry attempts +``` +→ +```markdown +###### `retry.maxAttempts` + +Maximum number of retry attempts +``` + +**Type Properties:** +```markdown + `name: String` + : The name of the resource +``` +→ +```markdown +###### `name: String` + +The name of the resource +``` + +### Alternative Formats + +For very compact lists or where h6 headings are too prominent: + +**Option A: Bold inline format:** +```markdown +- **`item`** - Description text here +``` + +**Option B: Simple list:** +```markdown +- `item`: Description text here +``` + +**When to use alternatives:** Only when the documentation context specifically requires a more compact format, or when there are many simple items in quick succession. + +## Steps + +1. **Search for definition lists** using Grep: + - Top-level: lines with backtick-wrapped terms followed by `: ` on next line (no leading whitespace) + - Embedded: indented lines (with leading spaces) with backtick-wrapped terms followed by `: ` + +2. **Determine heading level** based on indentation and context: + - **No indentation** → Top-level item → h5 heading (`#####`) + - **Indented (2-4 spaces)** → Nested item → h6 heading (`######`) + +3. **For each match:** + - Read surrounding context to capture full definition including multi-line descriptions + - Note any special formatting within descriptions (code blocks, lists, emphasis) + +4. **Convert using Edit tool:** + - **Top-level**: Replace with `##### ` + term, remove `: ` prefix, add blank line + - **Embedded**: Replace with `###### ` + term, remove indentation and `: ` prefix, add blank line + - Preserve all formatting within description text + +5. **Handle edge cases:** + - Items with no description (rare but possible) + - Nested content like code blocks, lists, or tables within descriptions + - Multiple consecutive definition list items + - Mixed indentation levels + - Special characters in terms (backticks, colons, etc.) + +## Pattern Matching + +### Top-Level Definition Lists (h5) + +**Pattern:** +``` +`TERM` +: Description +``` + +**Characteristics:** +- Line starts with backtick (no leading whitespace) +- Term can be: environment variable, CLI option, config setting, type name, etc. +- Next line starts with `: ` (colon + space) +- Description may span multiple lines + +**Common forms:** +- `` `ENV_VAR` `` - Environment variables +- `` `-option` `` - CLI flags +- `` `config.setting` `` - Configuration options +- `` `TypeName` `` - Type definitions +- `` `functionName()` `` - Function/method names + +### Embedded Definition Lists (h6) + +**Pattern:** +``` + `nestedTerm` + : Description +``` + +**Characteristics:** +- Line starts with 2-4 spaces of indentation +- Term wrapped in backticks +- Next line indented with `: ` prefix +- Represents nested or sub-items + +**Common forms:** +- `` `paramName: type` `` - Function parameters +- `` `property` `` - Object properties +- `` `subOption` `` - Nested configuration options +- `` `field` `` - Data structure fields + +## Important Notes + +### Formatting Preservation +- Preserve all formatting within descriptions: code blocks, lists, emphasis, links, etc. +- Keep default values in parentheses or emphasis exactly as-is +- Maintain relative indentation within multi-line descriptions + +### Heading Levels +- **Top-level (no indent)** → h5 (`#####`) - one blank line between heading and description +- **Embedded (indented)** → h6 (`######`) - remove indentation, add one blank line between heading and description + +### Common Uses Across Documentation +- **Environment variables** (env-vars pages) +- **CLI options and flags** (CLI reference pages) +- **Configuration settings** (config reference pages) +- **Function/method parameters** (API documentation) +- **Type properties** (type reference pages) +- **Feature flags** (feature documentation) + +### Best Practices +- Work systematically top-to-bottom to avoid missing entries +- Read surrounding context to determine correct heading level +- Verify document structure and hierarchy after conversion +- Test rendering in documentation preview if available +- Use `replace_all: false` in Edit tool to handle each occurrence individually when patterns repeat + +### Context Awareness +When uncertain about conversion format, consider: +- Where does this item appear in the document hierarchy? +- Is it a primary entry or a nested sub-item? +- What's the surrounding content structure? +- Would h6 preserve or break the logical flow? diff --git a/.claude/commands/convert-myst-admonitions.md b/.claude/commands/convert-myst-admonitions.md new file mode 100644 index 0000000000..4f8f2dffca --- /dev/null +++ b/.claude/commands/convert-myst-admonitions.md @@ -0,0 +1,58 @@ +--- +description: Convert MyST-style version admonitions to JSX tag format +--- + +# Convert MyST Admonitions to JSX Tags + +Convert all version admonitions in MDX files from the old MyST colon-fence format to the new JSX tag-based format. + +## Task + +Search for and convert all instances of: +- `:::{versionadded} X.Y.Z` → `` +- `:::{deprecated} X.Y.Z` → `` +- `:::{versionchanged} X.Y.Z` → `` + +## Conversion Rules + +1. **Simple admonitions (empty body):** + ```markdown + :::{versionadded} X.Y.Z + ::: + ``` + Converts to: + ```markdown + + ``` + +2. **Admonitions with description (multi-line body):** + ```markdown + :::{deprecated} X.Y.Z + Some description text here. + ::: + ``` + Converts to: + ```markdown + + Some description text here. + + ``` + +## Steps + +1. Use Grep to find all occurrences of `:::{versionadded}`, `:::{deprecated}`, and `:::{versionchanged}` patterns +2. Read the context around each match to understand the full admonition structure +3. Convert each admonition using Edit tool with appropriate old_string/new_string +4. For duplicate patterns that appear multiple times, include enough surrounding context to make each match unique +5. Verify the file has the correct imports at the top: + ```javascript + ``` +6. After all conversions, verify no old-format admonitions remain using Grep + +## Important Notes + +- Preserve all surrounding formatting, spacing, and content exactly as-is +- Handle each occurrence individually with sufficient context for unique matching +- Use self-closing tags (`/>`) for empty admonitions +- Use container tags with content for admonitions with descriptions +- Work systematically from top to bottom of the file to avoid confusion with line number shifts \ No newline at end of file diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index a933fc7d01..1eb020932e 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -9,14 +9,19 @@ jobs: docs-build: name: Build runs-on: ubuntu-latest + defaults: + run: + working-directory: docs steps: - uses: actions/checkout@v4 - - uses: actions/setup-python@v4 + - uses: actions/setup-node@v4 with: - python-version: '3.9' + node-version: '20' + cache: 'npm' + cache-dependency-path: docs/package-lock.json - - name: Test docs build - run: | - cd docs/ - pip install -r requirements.txt - make clean html + - name: Install dependencies + run: npm ci + + - name: Build docs + run: npm run build diff --git a/.gitignore b/.gitignore index e9619ea503..8c8e7b8ab9 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,7 @@ .cache/* .vagrant/* .cache/* +.netlify/ .nextflow* .node-nextflow* .devcontainer @@ -45,4 +46,5 @@ plugins-prod /test-module /results /x/* -mise.toml \ No newline at end of file +mise.toml +docs/.netlify/ diff --git a/docs/.dockerignore b/docs/.dockerignore deleted file mode 100644 index 8896a7a43b..0000000000 --- a/docs/.dockerignore +++ /dev/null @@ -1,2 +0,0 @@ -** -!requirements.txt \ No newline at end of file diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 0000000000..48fcd6ff6b --- /dev/null +++ b/docs/.gitignore @@ -0,0 +1,25 @@ +# Dependencies +/node_modules + +# Production +/build + +# Generated files +.docusaurus +.cache-loader +/nextflow_stable +/versioned_docs +/versioned_sidebars +/versions.json + +# Misc +.DS_Store +.env +.env.local +.env.development.local +.env.test.local +.env.production.local + +npm-debug.log* +yarn-debug.log* +yarn-error.log* diff --git a/docs/Dockerfile b/docs/Dockerfile deleted file mode 100644 index a961249faf..0000000000 --- a/docs/Dockerfile +++ /dev/null @@ -1,14 +0,0 @@ -FROM mambaorg/micromamba:1.5.8 - -RUN micromamba install --yes --name base --channel conda-forge \ - make=4.3 \ - python=3.7 \ - conda-forge:git=2.45.0 \ - && \ - micromamba clean --all --yes - -COPY requirements.txt . - -RUN eval "$(micromamba shell hook --shell=bash)" && \ - micromamba activate && \ - pip install -r requirements.txt diff --git a/docs/Makefile b/docs/Makefile deleted file mode 100644 index 8817d40c01..0000000000 --- a/docs/Makefile +++ /dev/null @@ -1,153 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -PAPER = -BUILDDIR = _build - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . -# the i18n builder cannot share the environment and doctrees with the others -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext - -help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " texinfo to make Texinfo files" - @echo " info to make Texinfo files and run them through makeinfo" - @echo " gettext to make PO message catalogs" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - -clean: - -rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Blow.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Blow.qhc" - -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/Blow" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Blow" - @echo "# devhelp" - -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -texinfo: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo - @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." - @echo "Run \`make' in that directory to run these through makeinfo" \ - "(use \`make info' here to do that automatically)." - -info: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo "Running Texinfo files through makeinfo..." - make -C $(BUILDDIR)/texinfo info - @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." - -gettext: - $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale - @echo - @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." diff --git a/docs/README.md b/docs/README.md index f0b98c510d..7b789cd9e2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,47 +1,239 @@ -# Nextflow Documentation +# Nextflow documentation -Nextflow documentation is written using [Sphinx](http://www.sphinx-doc.org/), [MyST](https://myst-parser.readthedocs.io/en/latest/) which is an extended version of Markdown for Sphinx, and the [Read The Docs theme for Sphinx](https://github.com/readthedocs/sphinx_rtd_theme). +This directory contains the official Nextflow documentation, built with [Docusaurus 3.9.2](https://docusaurus.io/) using the Seqera preset theme. +## Technology stack -## Dependencies +- **Docusaurus 3.9.2**: Modern static site generator +- **MDX**: Markdown with JSX/React component support +- **Seqera Preset**: Custom Docusaurus theme and configuration +- **Node.js 20+**: Required runtime environment +- **Tailwind CSS**: Utility-first CSS framework +- **DaisyUI**: Component library -The most convenient approach is to create a Conda environment with Python 3.7 (other versions may work but haven't been tested). - -The build dependencies can be installed with `pip`: +## Quick start ```bash cd docs -pip install -r requirements.txt +npm install +npm start ``` -Alternatively, you can use the Dockerfile to build the docs in a container (see below). +This starts a local development server at `http://localhost:3000` with hot reload. Most changes are reflected live without restarting the server. +## Available commands -## Contributing +| Command | Description | +|---------|-------------| +| `npm start` | Start local development server with hot reload | +| `npm run build` | Build production static site to `build/` directory | +| `npm run serve` | Serve the built site locally for testing | +| `npm run clear` | Clear Docusaurus cache (use if build issues occur) | +| `npm run swizzle` | Eject Docusaurus components for customization | + +## Directory structure + +``` +docs/ +├── docs/ # Documentation content (.mdx files) +│ ├── _static/ # Static assets for documentation pages +│ ├── developer/ # Developer/contributor documentation +│ ├── guides/ # How-to guides +│ ├── migrations/ # Version migration guides +│ ├── plugins/ # Plugin documentation +│ ├── reference/ # Language and runtime reference +│ ├── snippets/ # Code snippet examples +│ └── tutorials/ # Step-by-step tutorials +├── src/ # Custom React components and styling +│ ├── components/ # Custom React components +│ └── custom.css # Custom CSS styles +├── static/ # Static assets served at root +├── build/ # Generated static site (git-ignored) +├── docusaurus.config.js # Main Docusaurus configuration +├── sidebars.js # Sidebar navigation structure +└── package.json # Node.js dependencies + +../netlify.toml # Netlify deployment configuration (at repo root) +``` -To edit and contribute to the documentation, you only need a text editor to change the appropriate `.md` files in this directory. +## Writing documentation -Once you have made your changes, run the following command to build the HTML files: +### File format -```bash -make clean html +- Use `.mdx` extension for all documentation files +- MDX supports standard Markdown plus JSX/React components +- Files are automatically processed and converted to HTML pages + +### Navigation + +- Add new pages to `sidebars.js` to appear in navigation +- Use relative paths for internal links: `[link text](./other-page.mdx)` +- Organize pages into categories using the sidebar structure + +### Admonitions + +Docusaurus provides built-in admonitions for callouts: + +```markdown +:::note +This is a note +::: + +:::tip +This is a helpful tip +::: + +:::warning +This is a warning +::: + +:::danger +This is a danger warning +::: ``` -Alternatively, you can use the Dockerfile to build the docs in a container: +### Version tags -```bash -docker build -t nextflow/sphinx:5.3.0 . -docker run -v $(pwd):/tmp nextflow/sphinx:5.3.0 -- make html +Use custom version tags for tracking feature changes: + +```jsx + + + +``` + +### Code blocks + +Code blocks support syntax highlighting: + +````markdown +```groovy +workflow { + println "Hello, Nextflow!" +} ``` +```` + +Supported languages include: `groovy`, `bash`, `python`, `javascript`, `java`, `yaml`, `json`, and many more. + +### Images and assets + +- Place images in `docs/_static/` or `static/` +- Reference from docs using relative paths: + - From `docs/_static/`: `![alt text](./_static/image.png)` + - From `static/`: `![alt text](/image.png)` (served at root) + +## Versioning + +This section explains how versioning is implemented. + +### Files + +- **`versions.json`**: Contains an array of version strings (e.g., `["24.10", "24.04"]`). +- **`latest_version.js`**: Exports the latest version from `versions.json` (first element in the array). +- **`docusaurus.config.js`**: Configuration file that includes versioning settings. + +### Configuration + +The versioning configuration in `docusaurus.config.js` includes: + +- `includeCurrentVersion`: Set to `true` by default. Can be disabled with `INCLUDE_NEXT=false` environment variable for PR previews. +- `lastVersion`: Points to the latest version from `latest_version.js`, or `undefined` if no versions exist. -Then start up a local http server and open `localhost:8080` in your browser to verify the changes: +### Creating a new version + +To create a new version of the documentation, use the Docusaurus versioning command: ```bash -python -m http.server 8080 --directory _build/html/ +npm run docusaurus docs:version +``` + +This will: + +1. Create a copy of the current docs in `versioned_docs/version-/` +2. Create a copy of the current sidebar in `versioned_sidebars/version--sidebars.json` +3. Add the version to `versions.json` + +### Directory structure + +When versions are created, the structure will be: + +``` +docs/ # Current/next version (latest documentation) +versioned_docs/ + version-24.10/ # October 2024 stable release docs + version-24.04/ # April 2024 stable release docs +versioned_sidebars/ + version-24.10-sidebars.json # Sidebar for 24.10 + version-24.04-sidebars.json # Sidebar for 24.04 +versions.json # ["24.10", "24.04"] +latest_version.js # Exports "24.10" ``` +### Controlling version switcher visibility + +The version switcher will only appear when you have 2+ versions. This includes both: +- Versioned snapshots in `versions.json` (e.g., "24.10", "24.04") +- The "next" version (current docs folder) if `includeCurrentVersion: true` + +#### Custom theme components + +Two theme components are swizzled (customized) from the Seqera theme: + +1. **`src/theme/DocSidebar/Desktop/index.tsx`** - Parent component that controls when to show the version switcher + - Modified to check for `/nextflow` paths + - Renders the VersionSwitcher component when on Nextflow docs pages + +2. **`src/theme/DocSidebar/Desktop/Content/VersionSwitcher/index.tsx`** - The version switcher component itself + - Uses `"default"` plugin ID instead of `"platform-enterprise"` + - Checks for `/nextflow` paths + - Uses Docusaurus hooks: + - `useVersions("default")` - Gets all available versions + - `useDocsVersion()` - Gets the current version + - `useDocsPreferredVersion("default")` - Manages user's preferred version selection + +## Contributing + +We welcome documentation contributions! Please: + +1. **Fork and create a branch** for your changes +2. **Test locally** using `npm run build` +3. **Follow existing patterns** for consistency +4. **Update sidebars.js** if adding new pages +5. **Check spelling and grammar** +6. **Submit a pull request** with a clear description + +See the main [CONTRIBUTING.md](../CONTRIBUTING.md) for general contribution guidelines. + +## Troubleshooting + +### Build fails + +1. Clear the cache: `npm run clear` +2. Reinstall dependencies: `rm -rf node_modules && npm install` +3. Check for syntax errors in MDX files +4. Verify all internal links are correct + +### Hot reload not working + +1. Restart the development server +2. Clear browser cache +3. Check for JavaScript errors in browser console + +### Missing styling + +1. Ensure custom CSS imports in `docusaurus.config.js` +2. Check Tailwind configuration +3. Clear Docusaurus cache + +### Node version issues + +Ensure you're using Node.js 20 or higher: + +```bash +node --version +``` ## License -Nextflow documentation is distributed under -[Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license](https://creativecommons.org/licenses/by-sa/4.0/). +Nextflow documentation is distributed under the [Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/) license. \ No newline at end of file diff --git a/docs/_static/.DS_Store b/docs/_static/.DS_Store deleted file mode 100644 index 3dba88a671..0000000000 Binary files a/docs/_static/.DS_Store and /dev/null differ diff --git a/docs/conf.py b/docs/conf.py deleted file mode 100644 index 9420b4d7ed..0000000000 --- a/docs/conf.py +++ /dev/null @@ -1,424 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Nextflow documentation build configuration file, created by -# sphinx-quickstart on Sat May 5 16:57:28 2012. -# -# This file is execfile()d with the current directory set to its containing dir. -# -# Note that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import sys, os -import sphinx_rtd_theme -import datetime -import subprocess -import shlex - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -#sys.path.insert(0, os.path.abspath('.')) - -# -- General configuration ----------------------------------------------------- - -# If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be extensions -# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = [ - 'sphinx.ext.mathjax', - 'sphinxcontrib.mermaid', - 'sphinxext.rediraffe', - 'sphinx_rtd_theme', - 'myst_parser', - 'sphinx_tabs.tabs', - 'sphinx_copybutton' -] - -myst_enable_extensions = ['colon_fence', 'deflist', 'dollarmath'] - -myst_heading_anchors = 3 - -rediraffe_redirects = { - 'getstarted.md': 'install.md', - 'basic.md': 'overview.md', - 'tracing.md': 'reports.md', - 'mail.md': 'notifications.md', - 'operator.md': 'reference/operator.md', - 'dsl1.md': 'migrations/dsl1.md', - 'updating-syntax.md': 'strict-syntax.md', - 'updating-spot-retries.md': 'guides/updating-spot-retries.md', - 'metrics.md': 'tutorials/metrics.md', - 'data-lineage.md' : 'tutorials/data-lineage.md', - 'workflow-outputs.md': 'tutorials/workflow-outputs.md', - 'flux.md': 'tutorials/flux.md', - 'developer/plugins.md': 'plugins/developing-plugins.md', - 'plugins.md': 'plugins/plugins.md', - 'channel.md': 'workflow.md', - 'module.md': 'modules/modules.md' -} - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix of source filenames. -source_suffix = '.md' - -# The encoding of source files. -#source_encoding = 'utf-8-sig' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'Nextflow' -copyright = str(datetime.date.today().year) + u', Seqera Labs, S.L' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. - -# AUTO-GENERATED BY GIT -# Check if we are checked out at a specific version or not -process = subprocess.Popen(shlex.split("git tag --points-at HEAD"), stdout=subprocess.PIPE) - -# The full version, including alpha/beta/rc tags. -release = process.communicate()[0].decode("utf-8").strip() - -# The short X.Y version. -version = release.replace("-edge", "") - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -html_title = f"Nextflow {release} documentation" - -# Get the current sha if not checked out at a specific version -if len(release) == 0: - process = subprocess.Popen(shlex.split("git rev-parse --short HEAD"), stdout=subprocess.PIPE) - current_sha = process.communicate()[0].decode("utf-8").strip() - release = f"dev ({current_sha})" - html_title = "Nextflow documentation" - - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -#language = None - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -exclude_patterns = ['_build', '**README.md'] - -# The reST default role (used for this markup: `text`) to use for all documents. -#default_role = None - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'default' - -# A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] - -# Add copyable selector -copybutton_selector = ".copyable pre" - -# -- Options for HTML output --------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. - -html_theme = "sphinx_rtd_theme" - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -html_theme_options = { - 'logo_only': True, - 'display_version': False -} - -html_context = { - "display_github": True, - "github_user": "nextflow-io", - "github_repo": "nextflow", - "github_version": "master", - "conf_py_path": "/docs/" -} - -# Nextflow theme -html_css_files = ['theme.css'] - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -html_logo = '_static/nextflow-logo.png' - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -html_favicon = '_static/favicon.ico' - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -html_use_smartypants = False - -# Custom sidebar templates, maps document names to template names. -html_sidebars = { - '**': ['globaltoc.html'] -} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -html_show_sourcelink = False - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Output file base name for HTML help builder. -htmlhelp_basename = 'Nextflowdoc' - - -# -- Options for LaTeX output -------------------------------------------------- - -latex_elements = { -# The paper size ('letterpaper' or 'a4paper'). -#'papersize': 'letterpaper', - -# The font size ('10pt', '11pt' or '12pt'). -#'pointsize': '10pt', - -# Additional stuff for the LaTeX preamble. -#'preamble': '', -} - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, author, documentclass [howto/manual]). -latex_documents = [ - ('index', 'Nextflow.tex', u'Nextflow Documentation', - u'Paolo Di Tommaso', 'manual'), -] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -latex_logo = '_static/nextflow-logo.png' - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# If true, show page references after internal links. -#latex_show_pagerefs = False - -# If true, show URL addresses after external links. -#latex_show_urls = False - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_domain_indices = True - - -# -- Options for manual page output -------------------------------------------- - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -man_pages = [ - ('index', 'nextflow', u'Nextflow Documentation', - [u'Paolo Di Tommaso'], 1) -] - -# If true, show URL addresses after external links. -#man_show_urls = False - - -# -- Options for Texinfo output ------------------------------------------------ - -# Grouping the document tree into Texinfo files. List of tuples -# (source start file, target name, title, author, -# dir menu entry, description, category) -texinfo_documents = [ - ('index', 'Nextflow', u'Nextflow Documentation', - u'Paolo Di Tommaso', 'Nextflow', 'One line description of project.', - 'Miscellaneous'), -] - -# Documents to append as an appendix to all manuals. -#texinfo_appendices = [] - -# If false, no module index is generated. -#texinfo_domain_indices = True - -# How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' - - -# -- Options for Epub output --------------------------------------------------- - -# Bibliographic Dublin Core info. -epub_title = u'Nextflow' -epub_author = u'Paolo Di Tommaso' -epub_publisher = u'Paolo Di Tommaso' -epub_copyright = u'Copyright 2013-2024, Seqera Labs' - -# The language of the text. It defaults to the language option -# or en if the language is not set. -#epub_language = '' - -# The scheme of the identifier. Typical schemes are ISBN or URL. -#epub_scheme = '' - -# The unique identifier of the text. This can be a ISBN number -# or the project homepage. -#epub_identifier = '' - -# A unique identification for the text. -#epub_uid = '' - -# A tuple containing the cover image and cover page html template filenames. -#epub_cover = () - -# HTML files that should be inserted before the pages created by sphinx. -# The format is a list of tuples containing the path and title. -#epub_pre_files = [] - -# HTML files that should be inserted after the pages created by sphinx. -# The format is a list of tuples containing the path and title. -#epub_post_files = [] - -# A list of files that should not be packed into the epub file. -#epub_exclude_files = [] - -# The depth of the table of contents in toc.ncx. -#epub_tocdepth = 3 - -# Allow duplicate toc entries. -#epub_tocdup = True - -# see also: https://github.com/pygments/pygments/blob/master/pygments/lexers/jvm.py -from pygments.lexer import RegexLexer -from sphinx.highlighting import lexers - -class NextflowLexer(RegexLexer): - """ - For Nextflow source code. - """ - - import re - from pygments.lexer import bygroups, using, this, default - from pygments.token import Comment, Operator, Keyword, Name, String, Number, Whitespace - from pygments.util import shebang_matches - - name = 'Nextflow' - url = 'https://nextflow.io/' - aliases = ['nextflow', 'nf'] - filenames = ['*.nf'] - mimetypes = ['text/x-nextflow'] - # version_added = '1.5' - - flags = re.MULTILINE | re.DOTALL - - tokens = { - 'root': [ - # Nextflow allows a file to start with a shebang - (r'#!(.*?)$', Comment.Preproc, 'base'), - default('base'), - ], - 'base': [ - (r'[^\S\n]+', Whitespace), - (r'(//.*?)(\n)', bygroups(Comment.Single, Whitespace)), - (r'/\*.*?\*/', Comment.Multiline), - # keywords: go before method names to avoid lexing "throw new XYZ" - # as a method signature - (r'(assert|catch|else|if|instanceof|new|return|throw|try|in|as)\b', Keyword), - (r'(channel|log)', Name.Namespace), - # method names - (r'^(\s*(?:[a-zA-Z_][\w.\[\]]*\s+)+?)' # return arguments - r'(' - r'[a-zA-Z_]\w*' # method name - r'|"(?:\\\\|\\[^\\]|[^"\\])*"' # or double-quoted method name - r"|'(?:\\\\|\\[^\\]|[^'\\])*'" # or single-quoted method name - r')' - r'(\s*)(\()', # signature start - bygroups(using(this), Name.Function, Whitespace, Operator)), - (r'@[a-zA-Z_][\w.]*', Name.Decorator), - (r'(def|enum|include|from|output|params|process|workflow)\b', Keyword.Declaration), - (r'(boolean|byte|char|double|float|int|long|short|void)\b', Keyword.Type), - (r'(true|false|null)\b', Keyword.Constant), - (r'""".*?"""', String.Double), - (r"'''.*?'''", String.Single), - (r'"(\\\\|\\[^\\]|[^"\\])*"', String.Double), - (r"'(\\\\|\\[^\\]|[^'\\])*'", String.Single), - (r'/(\\\\|\\[^\\]|[^/\\])*/', String), - (r"'\\.'|'[^\\]'|'\\u[0-9a-fA-F]{4}'", String.Char), - (r'(\.)([a-zA-Z_]\w*)', bygroups(Operator, Name.Attribute)), - (r'[a-zA-Z_]\w*:', Name.Label), - (r'[a-zA-Z_$]\w*', Name), - (r'[~^*!%&\[\](){}<>|+=:;,./?-]', Operator), - (r'[0-9][0-9]*\.[0-9]+([eE][0-9]+)?[fd]?', Number.Float), - (r'0x[0-9a-fA-F]+', Number.Hex), - (r'[0-9]+L?', Number.Integer), - (r'\n', Whitespace) - ], - } - - def analyse_text(text): - return shebang_matches(text, r'nextflow') - -lexers['nextflow'] = NextflowLexer() diff --git a/docs/developer/diagram.md b/docs/developer/diagram.md deleted file mode 100644 index 90300113c2..0000000000 --- a/docs/developer/diagram.md +++ /dev/null @@ -1,10 +0,0 @@ -(diagram-page)= - -# Workflow Diagram - -The following diagram is a high-level overview of the Nextflow source code in a similar style as the {ref}`workflow diagram ` visualization for Nextflow pipelines. Each node and subgraph is a class. Arrows depict the flow of data and/or communication between classes. - -In general, nodes with sharp corners are "record" classes that simply hold information, while nodes with rounded edges are "function" classes that transform some input into an output. Subgraphs are either long-running classes, i.e. "places where things happen", or one of the other two types for which it was useful to expand and show internal details. - -```{mermaid} diagrams/overview.mmd -``` diff --git a/docs/developer/nextflow.cli.md b/docs/developer/nextflow.cli.md deleted file mode 100644 index 8a93b5c76c..0000000000 --- a/docs/developer/nextflow.cli.md +++ /dev/null @@ -1,17 +0,0 @@ - -# `nextflow.cli` - -The `nextflow.cli` package implements the command line interface. - -## Class Diagram - -```{mermaid} diagrams/nextflow.cli.mmd -``` - -```{note} -Some classes may be excluded from the above diagram for brevity. -``` - -## Notes - -The `Launcher` class is the entrypoint for Nextflow. It uses [JCommander](https://jcommander.org/) to parse the command-line arguments. Additionally, there is a class for each subcommand which implements the application logic of that command. By far the most complex command is `CmdRun`. diff --git a/docs/developer/nextflow.cloud.aws.md b/docs/developer/nextflow.cloud.aws.md deleted file mode 100644 index 30195218b2..0000000000 --- a/docs/developer/nextflow.cloud.aws.md +++ /dev/null @@ -1,13 +0,0 @@ - -# `nextflow.cloud.aws` - -The `nextflow.cloud.aws` package implements the AWS Batch executor. - -## Class Diagram - -```{mermaid} diagrams/nextflow.cloud.aws.mmd -``` - -```{note} -Some classes may be excluded from the above diagrams for brevity. -``` diff --git a/docs/developer/nextflow.cloud.aws.nio.md b/docs/developer/nextflow.cloud.aws.nio.md deleted file mode 100644 index 78bf60525f..0000000000 --- a/docs/developer/nextflow.cloud.aws.nio.md +++ /dev/null @@ -1,17 +0,0 @@ - -# `nextflow.cloud.aws.nio` - -The `nextflow.cloud.aws.nio` package implements the S3 filesystem. - -## Class Diagram - -```{mermaid} diagrams/nextflow.cloud.aws.nio.mmd -``` - -```{note} -Some classes may be excluded from the above diagrams for brevity. -``` - -## Notes - -The S3 filesystem translates Java Path API calls into S3 API calls, which allows Nextflow to interact with S3 objects through the same interface for local files. diff --git a/docs/developer/nextflow.cloud.azure.md b/docs/developer/nextflow.cloud.azure.md deleted file mode 100644 index 649e243cc0..0000000000 --- a/docs/developer/nextflow.cloud.azure.md +++ /dev/null @@ -1,13 +0,0 @@ - -# `nextflow.cloud.azure` - -The `nextflow.cloud.azure` package implements the Azure Batch executor. - -## Class Diagram - -```{mermaid} diagrams/nextflow.cloud.azure.mmd -``` - -```{note} -Some classes may be excluded from the above diagrams for brevity. -``` diff --git a/docs/developer/nextflow.cloud.google.md b/docs/developer/nextflow.cloud.google.md deleted file mode 100644 index bf793fb6ce..0000000000 --- a/docs/developer/nextflow.cloud.google.md +++ /dev/null @@ -1,13 +0,0 @@ - -# `nextflow.cloud.google` - -The `nextflow.cloud.google` package implements the Google Batch executor. - -## Class Diagram - -```{mermaid} diagrams/nextflow.cloud.google.mmd -``` - -```{note} -Some classes may be excluded from the above diagrams for brevity. -``` diff --git a/docs/developer/nextflow.dag.md b/docs/developer/nextflow.dag.md deleted file mode 100644 index b08328c32a..0000000000 --- a/docs/developer/nextflow.dag.md +++ /dev/null @@ -1,19 +0,0 @@ - -# `nextflow.dag` - -The `nextflow.dag` package implements the workflow DAG and renderers for several diagram formats. - -## Class Diagram - -```{mermaid} diagrams/nextflow.dag.mmd -``` - -```{note} -Some classes may be excluded from the above diagram for brevity. -``` - -## Notes - -The workflow DAG defines the network of processes, channels, and operators that comprise a workflow. It is produced by the execution of the Nextflow script. See [nextflow.script](nextflow.script.md) for more details. - -Implementations of the `DagRenderer` interface define how to render the workflow DAG to a particular diagram format. See {ref}`workflow-diagram` for more details. diff --git a/docs/developer/nextflow.executor.md b/docs/developer/nextflow.executor.md deleted file mode 100644 index 8be432393e..0000000000 --- a/docs/developer/nextflow.executor.md +++ /dev/null @@ -1,21 +0,0 @@ - -# `nextflow.executor` - -The `nextflow.executor` package defines the executor interface and implements several built-in executors. - -## Class Diagram - -```{mermaid} diagrams/nextflow.executor.mmd -``` - -```{note} -Some classes may be excluded from the above diagram for brevity. -``` - -## Notes - -The `Executor` class is the base class for all Nextflow executors. The main purpose of an `Executor` is to submit tasks to an underlying compute environment, such as an HPC scheduler or cloud batch executor. It uses a `TaskMonitor` to manage the lifecycle of all tasks and a `TaskHandler` to manage each individual task. Most executors use the same polling monitor, but each executor implements its own task handler to customize it for a particular compute environment. See [nextflow.processor](nextflow.processor.md) for more details about these classes. - -The built-in executors include the local executor (`LocalExecutor`) and the various grid executors (SLURM, PBS, LSF, etc), all of which extend `AbstractGridExecutor`. The `LocalExecutor` implements both "script" tasks (processes with a `script` or `shell` block) and "native" tasks (processes with an `exec` block). - -The `BashWrapperBuilder` is used by executors to generate the wrapper script (`.command.run`) for a task, from a template script called `command-run.txt`, as well as the task configuration and the execution environment. diff --git a/docs/developer/nextflow.extension.md b/docs/developer/nextflow.extension.md deleted file mode 100644 index 7bafc14a81..0000000000 --- a/docs/developer/nextflow.extension.md +++ /dev/null @@ -1,19 +0,0 @@ - -# `nextflow.extension` - -The `nextflow.extension` package implements the channel operators and other extension methods. - -## Class Diagram - -```{mermaid} diagrams/nextflow.extension.mmd -``` - -```{note} -Some classes may be excluded from the above diagram for brevity. -``` - -## Notes - -Operators are implemented using the [GPars](http://gpars.org/1.2.1/guide/guide/dataflow.html) dataflow library. In general, an operator consumes one or more `DataflowReadChannel`s and produces one or more `DataflowWriteChannel`s. See {ref}`operator-page` for details about each operator. - -Other notable classes include `Bolts` and `FilesEx`, which implement various extension methods used throughout the Nextflow codebase. If you see a method that doesn't appear to be implemented by the calling object, it may be implemented in one of these extension classes. diff --git a/docs/developer/nextflow.k8s.md b/docs/developer/nextflow.k8s.md deleted file mode 100644 index 1f51e0dac5..0000000000 --- a/docs/developer/nextflow.k8s.md +++ /dev/null @@ -1,17 +0,0 @@ - -# `nextflow.k8s` - -The `nextflow.k8s` package implements the Kubernetes executor and the `kuberun` command. - -## Class Diagram - -```{mermaid} diagrams/nextflow.k8s.mmd -``` - -```{note} -Some classes may be excluded from the above diagram for brevity. -``` - -## Notes - -The Kubernetes integration uses the K8s HTTP API to interact with K8s clusters, and relies on the `kubectl` command and `~/.kube/config` file for authentication. diff --git a/docs/developer/nextflow.md b/docs/developer/nextflow.md deleted file mode 100644 index d6f07fe880..0000000000 --- a/docs/developer/nextflow.md +++ /dev/null @@ -1,21 +0,0 @@ - -# `nextflow` - -The `nextflow` package contains various top-level classes. - -## Class Diagram - -```{mermaid} diagrams/nextflow.mmd -``` - -```{note} -Some classes may be excluded from the above diagram for brevity. -``` - -## Notes - -The `Nextflow` class implements several methods that are exposed to Nextflow scripts. See {ref}`stdlib-namespaces-global` for details. - -The `Channel` class implements the channel factory methods, and it is exposed directly to Nextflow scripts. See {ref}`channel-factory` for details. - -The `Session` class is the top-level representation of a Nextflow run, or "session". See [nextflow.script](nextflow.script.md) for more details about how a `Session` is created. diff --git a/docs/developer/nextflow.script.md b/docs/developer/nextflow.script.md deleted file mode 100644 index 7c76f1c2bd..0000000000 --- a/docs/developer/nextflow.script.md +++ /dev/null @@ -1,21 +0,0 @@ - -# `nextflow.script` - -The `nextflow.script` package implements the parsing and execution of Nextflow scripts. - -## Class Diagram - -```{mermaid} diagrams/nextflow.script.mmd -``` - -```{note} -Some classes may be excluded from the above diagram for brevity. -``` - -## Notes - -The execution of a Nextflow pipeline occurs in two phases. In the first phase, Nextflow parses and runs the script (using the language extensions in [nextflow.ast](nextflow.ast.md) and [nextflow.extension](nextflow.extension.md)), which produces the workflow DAG. In the second phase, Nextflow executes the workflow. - -```{note} -In DSL1, there was no separation between workflow construction and execution -- dataflow operators were executed as soon as they were constructed. DSL2 introduced lazy execution in order to separate process definition from execution, and thereby facilitate subworkflows and modules. -``` diff --git a/docs/developer/nextflow.secret.md b/docs/developer/nextflow.secret.md deleted file mode 100644 index db7b84f1a8..0000000000 --- a/docs/developer/nextflow.secret.md +++ /dev/null @@ -1,17 +0,0 @@ - -# `nextflow.secret` - -The `nextflow.secret` package defines the secrets provider interface and implements the built-in local secrets store. - -## Class Diagram - -```{mermaid} diagrams/nextflow.secret.mmd -``` - -```{note} -Some classes may be excluded from the above diagram for brevity. -``` - -## Notes - -The default secrets provider simply stores key-value pairs in a local JSON file. diff --git a/docs/developer/packages.md b/docs/developer/packages.md deleted file mode 100644 index 364d61960b..0000000000 --- a/docs/developer/packages.md +++ /dev/null @@ -1,12 +0,0 @@ -(packages-page)= - -# Packages - -The following subpages correspond to packages in the Nextflow source code: - -```{toctree} -:glob: -:maxdepth: 1 - -nextflow* -``` diff --git a/docs/docs/.netlify/netlify.toml b/docs/docs/.netlify/netlify.toml new file mode 100644 index 0000000000..9e7af7d4d0 --- /dev/null +++ b/docs/docs/.netlify/netlify.toml @@ -0,0 +1,37 @@ +headersOrigin = "config" +plugins = [] +redirects = [] + +[functions] + +[functions."*"] + +[build] +publish = "/Users/chris.hakkaart/workspace/architecture/nextflow/docs/build" +publishOrigin = "config" +commandOrigin = "config" +base = "/Users/chris.hakkaart/workspace/architecture/nextflow/docs" +command = "npm run build" +ignore = "git diff --quiet $CACHED_COMMIT_REF $COMMIT_REF ." + +[build.environment] +NODE_VERSION = "22" +NODE_OPTIONS = "--max-old-space-size=2048" + +[build.processing] + +[build.processing.css] + +[build.processing.html] + +[build.processing.images] + +[build.processing.js] + +[build.services] + +[[headers]] +for = "/*" + +[headers.values] +X-Robots-Tag = "noindex" \ No newline at end of file diff --git a/docs/LICENCE.txt b/docs/docs/LICENCE.txt similarity index 100% rename from docs/LICENCE.txt rename to docs/docs/LICENCE.txt diff --git a/docs/_static/dag.mmd b/docs/docs/_static/dag.mmd similarity index 100% rename from docs/_static/dag.mmd rename to docs/docs/_static/dag.mmd diff --git a/docs/_static/degular/Degular-Bold.woff b/docs/docs/_static/degular/Degular-Bold.woff similarity index 100% rename from docs/_static/degular/Degular-Bold.woff rename to docs/docs/_static/degular/Degular-Bold.woff diff --git a/docs/_static/degular/Degular-Bold.woff2 b/docs/docs/_static/degular/Degular-Bold.woff2 similarity index 100% rename from docs/_static/degular/Degular-Bold.woff2 rename to docs/docs/_static/degular/Degular-Bold.woff2 diff --git a/docs/_static/degular/Degular-BoldItalic.woff b/docs/docs/_static/degular/Degular-BoldItalic.woff similarity index 100% rename from docs/_static/degular/Degular-BoldItalic.woff rename to docs/docs/_static/degular/Degular-BoldItalic.woff diff --git a/docs/_static/degular/Degular-BoldItalic.woff2 b/docs/docs/_static/degular/Degular-BoldItalic.woff2 similarity index 100% rename from docs/_static/degular/Degular-BoldItalic.woff2 rename to docs/docs/_static/degular/Degular-BoldItalic.woff2 diff --git a/docs/_static/degular/Degular-Italic.woff b/docs/docs/_static/degular/Degular-Italic.woff similarity index 100% rename from docs/_static/degular/Degular-Italic.woff rename to docs/docs/_static/degular/Degular-Italic.woff diff --git a/docs/_static/degular/Degular-Italic.woff2 b/docs/docs/_static/degular/Degular-Italic.woff2 similarity index 100% rename from docs/_static/degular/Degular-Italic.woff2 rename to docs/docs/_static/degular/Degular-Italic.woff2 diff --git a/docs/_static/degular/Degular-Regular.woff b/docs/docs/_static/degular/Degular-Regular.woff similarity index 100% rename from docs/_static/degular/Degular-Regular.woff rename to docs/docs/_static/degular/Degular-Regular.woff diff --git a/docs/_static/degular/Degular-Regular.woff2 b/docs/docs/_static/degular/Degular-Regular.woff2 similarity index 100% rename from docs/_static/degular/Degular-Regular.woff2 rename to docs/docs/_static/degular/Degular-Regular.woff2 diff --git a/docs/_static/favicon.ico b/docs/docs/_static/favicon.ico similarity index 100% rename from docs/_static/favicon.ico rename to docs/docs/_static/favicon.ico diff --git a/docs/_static/nextflow-k8s-min.png b/docs/docs/_static/nextflow-k8s-min.png similarity index 100% rename from docs/_static/nextflow-k8s-min.png rename to docs/docs/_static/nextflow-k8s-min.png diff --git a/docs/_static/nextflow-logo-bg-dark.png b/docs/docs/_static/nextflow-logo-bg-dark.png similarity index 100% rename from docs/_static/nextflow-logo-bg-dark.png rename to docs/docs/_static/nextflow-logo-bg-dark.png diff --git a/docs/_static/nextflow-logo-bg-light.png b/docs/docs/_static/nextflow-logo-bg-light.png similarity index 100% rename from docs/_static/nextflow-logo-bg-light.png rename to docs/docs/_static/nextflow-logo-bg-light.png diff --git a/docs/_static/nextflow-logo.png b/docs/docs/_static/nextflow-logo.png similarity index 100% rename from docs/_static/nextflow-logo.png rename to docs/docs/_static/nextflow-logo.png diff --git a/docs/_static/report-resource-cpu-noheader.png b/docs/docs/_static/report-resource-cpu-noheader.png similarity index 100% rename from docs/_static/report-resource-cpu-noheader.png rename to docs/docs/_static/report-resource-cpu-noheader.png diff --git a/docs/_static/report-resource-cpu.png b/docs/docs/_static/report-resource-cpu.png similarity index 100% rename from docs/_static/report-resource-cpu.png rename to docs/docs/_static/report-resource-cpu.png diff --git a/docs/_static/report-resource-io-read.png b/docs/docs/_static/report-resource-io-read.png similarity index 100% rename from docs/_static/report-resource-io-read.png rename to docs/docs/_static/report-resource-io-read.png diff --git a/docs/_static/report-resource-io-write.png b/docs/docs/_static/report-resource-io-write.png similarity index 100% rename from docs/_static/report-resource-io-write.png rename to docs/docs/_static/report-resource-io-write.png diff --git a/docs/_static/report-resource-job-duration.png b/docs/docs/_static/report-resource-job-duration.png similarity index 100% rename from docs/_static/report-resource-job-duration.png rename to docs/docs/_static/report-resource-job-duration.png diff --git a/docs/_static/report-resource-memory-pctram.png b/docs/docs/_static/report-resource-memory-pctram.png similarity index 100% rename from docs/_static/report-resource-memory-pctram.png rename to docs/docs/_static/report-resource-memory-pctram.png diff --git a/docs/_static/report-resource-memory-ram.png b/docs/docs/_static/report-resource-memory-ram.png similarity index 100% rename from docs/_static/report-resource-memory-ram.png rename to docs/docs/_static/report-resource-memory-ram.png diff --git a/docs/_static/report-resource-memory-vmem.png b/docs/docs/_static/report-resource-memory-vmem.png similarity index 100% rename from docs/_static/report-resource-memory-vmem.png rename to docs/docs/_static/report-resource-memory-vmem.png diff --git a/docs/_static/report-resources-min.png b/docs/docs/_static/report-resources-min.png similarity index 100% rename from docs/_static/report-resources-min.png rename to docs/docs/_static/report-resources-min.png diff --git a/docs/_static/report-summary-min.png b/docs/docs/_static/report-summary-min.png similarity index 100% rename from docs/_static/report-summary-min.png rename to docs/docs/_static/report-summary-min.png diff --git a/docs/_static/report-tasks-min.png b/docs/docs/_static/report-tasks-min.png similarity index 100% rename from docs/_static/report-tasks-min.png rename to docs/docs/_static/report-tasks-min.png diff --git a/docs/docs/_static/seqera-logo-dark.svg b/docs/docs/_static/seqera-logo-dark.svg new file mode 100644 index 0000000000..4f0ed3546c --- /dev/null +++ b/docs/docs/_static/seqera-logo-dark.svg @@ -0,0 +1,16 @@ + + + + + + + + + + + + + + + + diff --git a/docs/docs/_static/seqera-logo-light.svg b/docs/docs/_static/seqera-logo-light.svg new file mode 100644 index 0000000000..4982acc896 --- /dev/null +++ b/docs/docs/_static/seqera-logo-light.svg @@ -0,0 +1,16 @@ + + + + + + + + + + + + + + + + diff --git a/docs/_static/seqera-logo.svg b/docs/docs/_static/seqera-logo.svg similarity index 100% rename from docs/_static/seqera-logo.svg rename to docs/docs/_static/seqera-logo.svg diff --git a/docs/_static/timeline-min.png b/docs/docs/_static/timeline-min.png similarity index 100% rename from docs/_static/timeline-min.png rename to docs/docs/_static/timeline-min.png diff --git a/docs/_static/workflow-notification-min.png b/docs/docs/_static/workflow-notification-min.png similarity index 100% rename from docs/_static/workflow-notification-min.png rename to docs/docs/_static/workflow-notification-min.png diff --git a/docs/_templates/layout.html b/docs/docs/_templates/layout.html similarity index 100% rename from docs/_templates/layout.html rename to docs/docs/_templates/layout.html diff --git a/docs/amazons3.md b/docs/docs/amazons3.mdx similarity index 83% rename from docs/amazons3.md rename to docs/docs/amazons3.mdx index a551a1043a..2c1a04c18f 100644 --- a/docs/amazons3.md +++ b/docs/docs/amazons3.mdx @@ -1,4 +1,7 @@ -(amazons3-page)= +--- +title: Amazon S3 +description: Use Amazon S3 buckets as a file system in Nextflow pipelines. +--- # Amazon S3 @@ -20,7 +23,7 @@ The usual file operations can be applied to a path handle with the above notatio println file('s3://my-bucket/data/sequences.fa').text ``` -See {ref}`working-with-files` and the {ref}`stdlib-types-path` reference to learn more about available file operations. +See [Working with files][working-with-files] and the [Path][stdlib-types-path] reference to learn more about available file operations. ## Security credentials @@ -35,9 +38,9 @@ The AWS access and secret keys can be specified by using the `aws` section in th ```groovy aws { - accessKey = '' - secretKey = '' - region = '' + accessKey = '' + secretKey = '' + region = '' } ``` @@ -85,10 +88,10 @@ in your Nextflow configuration as shown below: ```groovy aws { - accessKey = '' - secretKey = '' + accessKey = '' + secretKey = '' client { - endpoint = '' + endpoint = '' s3PathStyleAccess = true } } @@ -96,4 +99,9 @@ aws { ## Advanced configuration -Read {ref}`AWS configuration` section to learn more about advanced S3 client configuration options. +See [AWS configuration][config-aws] for more information about advanced S3 client configuration options. + + +[config-aws]: ./reference/config#aws +[stdlib-types-path]: ./reference/stdlib-types#path +[working-with-files]: ./working-with-files \ No newline at end of file diff --git a/docs/aws.md b/docs/docs/aws.mdx similarity index 86% rename from docs/aws.md rename to docs/docs/aws.mdx index 8118c3131a..88cab1c7f5 100644 --- a/docs/aws.md +++ b/docs/docs/aws.mdx @@ -1,8 +1,11 @@ -(aws-page)= +--- +title: Amazon Web Services +description: Configure and run Nextflow pipelines on AWS. +--- # Amazon Web Services -:::{tip} +:::tip This page describes how to manually set up and use Nextflow with AWS Cloud. You may be interested in using [Batch Forge](https://docs.seqera.io/platform/latest/compute-envs/aws-batch) in [Seqera Platform](https://seqera.io/platform/), which automatically creates the required AWS infrastructure for you with minimal intervention. @@ -14,7 +17,7 @@ Nextflow uses the [AWS security credentials](https://docs.aws.amazon.com/general The AWS credentials are selected from the following sources, in order of descending priority: -1. Nextflow configuration file - `aws.accessKey` and `aws.secretKey`. See {ref}`AWS configuration` for more details. +1. Nextflow configuration file - `aws.accessKey` and `aws.secretKey`. See [aws][config-aws] for more details. 2. A custom profile in `$HOME/.aws/credentials` and/or `$HOME/.aws/config`. The profile can be supplied from the `aws.profile` config option, or the `AWS_PROFILE` or `AWS_DEFAULT_PROFILE` environmental variables. @@ -24,8 +27,8 @@ The AWS credentials are selected from the following sources, in order of descend 5. Single Sign-On (SSO) credentials. See the [AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html) for more details. - :::{versionadded} 23.07.0-edge - ::: + + 6. EC2 instance profile credentials. See the [AWS documentation](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) and [this blog post](https://aws.amazon.com/blogs/security/granting-permission-to-launch-ec2-instances-with-iam-roles-passrole-permission/) for more details. @@ -88,7 +91,7 @@ Minimal permissions policies to be attached to the AWS account used by Nextflow Alternatively, you can use AWS provided `AmazonEC2ContainerRegistryReadOnly` managed policy. -:::{note} +:::note If you are running Fargate or Fargate Spot, you may need the following policies in addition to the listed above: ```json "ec2:DescribeSubnets" @@ -112,13 +115,13 @@ Depending on the pipeline configuration, the above actions can be done all in a ```json { "Version": "2012-10-17", - "Id": "", + "Id": "", "Statement": [ { - "Sid": "", + "Sid": "", "Effect": "Allow", "Principal": { - "AWS": "" + "AWS": "" }, "Action": [ "s3:GetObject", @@ -127,7 +130,7 @@ Depending on the pipeline configuration, the above actions can be done all in a "s3:PutObjectTagging", "s3:AbortMultipartUpload" ], - "Resource": "arn:aws:s3:::/*" + "Resource": "arn:aws:s3:::/*" }, { "Sid": "AllowSSLRequestsOnly", @@ -135,8 +138,8 @@ Depending on the pipeline configuration, the above actions can be done all in a "Principal": "*", "Action": "s3:*", "Resource": [ - "arn:aws:s3:::", - "arn:aws:s3:::/*" + "arn:aws:s3:::", + "arn:aws:s3:::/*" ], "Condition": { "Bool": { @@ -150,8 +153,6 @@ Depending on the pipeline configuration, the above actions can be done all in a See the [bucket policy documentation](https://docs.aws.amazon.com/config/latest/developerguide/s3-bucket-policy.html) for additional details. -(aws-batch)= - ## AWS Batch [AWS Batch](https://aws.amazon.com/batch/) is a managed computing service that enables the execution of containerized workloads on AWS cloud infrastructure. It dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized compute resources) based on the volume and specific resource requirements of the submitted jobs. @@ -159,19 +160,17 @@ See the [bucket policy documentation](https://docs.aws.amazon.com/config/latest/ Nextflow natively supports AWS Batch, allowing you to run pipeline tasks as Batch jobs in the cloud. -See {ref}`AWS Batch executor ` to learn more about the `awsbatch` executor. - -(aws-batch-cli)= +See [AWS Batch][awsbatch-executor] to learn more about the `awsbatch` executor. ### AWS CLI -:::{tip} -The recommended approach for running Nextflow pipelines with AWS Batch is to use {ref}`wave-page` and {ref}`fusion-page` for S3 access, rather than relying on the AWS CLI. This approach removes the requirement to install and configure the AWS CLI in containers or custom AMIs. +:::tip +The recommended approach for running Nextflow pipelines with AWS Batch is to use [Wave][wave-page] and [Fusion][fusion-page] for S3 access, rather than relying on the AWS CLI. This approach removes the requirement to install and configure the AWS CLI in containers or custom AMIs. ::: Nextflow uses the [AWS command line tool](https://aws.amazon.com/cli/) (`aws`) to stage input files and output files between S3 and the task containers. -The `aws` command can be made available by either (1) installing it in the container image(s) or (2) installing it in a {ref}`custom AMI ` to be used instead of the default AMI when configuring AWS Batch. +The `aws` command can be made available by either (1) installing it in the container image(s) or (2) installing it in a [custom AMI][aws-custom-ami]` to be used instead of the default AMI when configuring AWS Batch. ### Get started @@ -186,7 +185,7 @@ The `aws` command can be made available by either (1) installing it in the conta 3. In the AWS Console, create an S3 bucket for the work directory (see below). You can also create separate buckets for input data and results, as needed. -4. Make sure that every process in your pipeline specifies a Docker container with the {ref}`process-container` directive. +4. Make sure that every process in your pipeline specifies a Docker container with the [container][process-container] directive. 5. Make sure that all of your container images are published in a Docker registry that can be reached by AWS Batch, such as [Docker Hub](https://hub.docker.com/), [Quay](https://quay.io/), or [Elastic Container Registry](https://aws.amazon.com/ecr/). @@ -194,9 +193,9 @@ The `aws` command can be made available by either (1) installing it in the conta To configure your pipeline for AWS Batch: -1. Specify the AWS Batch {ref}`executor ` -2. Specify the AWS Batch queue with the {ref}`process-queue` directive -3. Specify any Batch job container options with the {ref}`process-containerOptions` directive. +1. Specify the AWS Batch [executor][awsbatch-executor] +2. Specify the AWS Batch queue with the [queue][process-queue] directive +3. Specify any Batch job container options with the [containerOptions][process-containerOptions] directive. An example `nextflow.config` file is shown below: @@ -217,16 +216,13 @@ aws { } ``` -:::{tip} -Each process can be configured with its own queue by using the {ref}`process-queue` directive in the process definition or via {ref}`config-process-selectors` in your Nextflow configuration. +:::tip +Each process can be configured with its own queue by using the [queue][process-queue] directive in the process definition or via [Process selectors][config-process-selectors] in your Nextflow configuration. ::: ## Container Options -:::{versionadded} 21.12.1-edge -::: - -The {ref}`process-containerOptions` directive can be used to control the properties of the container execution associated with each Batch job. +The [containerOptions][process-containerOptions] directive can be used to control the properties of the container execution associated with each Batch job. The following container options are currently supported: @@ -267,8 +263,6 @@ containerOptions '--ulimit nofile=1280:2560 --ulimit nproc=16:32 --privileged' Check the [AWS documentation](https://docs.aws.amazon.com/batch/latest/APIReference/API_ContainerProperties.html) for further details. -(aws-custom-ami)= - ## Custom AMI There are several reasons why you might need to create your own [AMI (Amazon Machine Image)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) to use in your Compute Environments: @@ -283,7 +277,7 @@ There are several reasons why you might need to create your own [AMI (Amazon Mac From the EC2 Dashboard, select **Launch Instance**, then select **Browse more AMIs**. In the new page, select **AWS Marketplace AMIs**, and then search for `Amazon ECS-Optimized Amazon Linux 2023 (AL2023) x86_64 AMI`. Select the AMI and continue as usual to configure and launch the instance. -:::{note} +:::note The selected instance has a root volume of 30GB. Make sure to increase its size or add a second EBS volume with enough storage for real genomic workloads. ::: @@ -293,17 +287,16 @@ Finally, select **Create Image** from the EC2 Dashboard to create a new AMI from The new AMI ID needs to be specified when creating the Batch Compute Environment. -:::{warning} +:::warning Any additional software must be installed on the EC2 instance *before* creating the AMI. ::: -(id2)= ### AWS CLI installation -:::{tip} +:::tip The need for the AWS CLI is considered a legacy requirement for the deployment of Nextflow pipelines with AWS Batch. -Instead, consider using {ref}`wave-page` and {ref}`fusion-page` to facilitate access to S3 without using the AWS CLI. +Instead, consider using [Wave][wave-page] and [Fusion][fusion-page] to facilitate access to S3 without using the AWS CLI. ::: The [AWS CLI](https://aws.amazon.com/cli) should be installed in your custom AMI using a self-contained package manager such as [Conda](https://conda.io). That way, you can control which version of Python is used by the AWS CLI (which is written in Python). @@ -328,7 +321,7 @@ $ ./miniconda/bin/aws --version aws-cli/1.29.20 Python/3.11.4 Linux/4.14.318-241.531.amzn2.x86_64 botocore/1.31.20 ``` -:::{note} +:::note The `aws` tool will be placed in a directory named `bin` in the main installation folder. Modifying this directory structure after the tool is installed will cause it to not work properly. ::: @@ -340,11 +333,7 @@ aws.batch.cliPath = '/home/ec2-user/miniconda/bin/aws' Replace the path above with the one matching the location where the `aws` tool is installed in your AMI. -:::{versionchanged} 19.07.0 -The `executor.awscli` config option was replaced by `aws.batch.cliPath`. -::: - -:::{warning} +:::warning The grandparent directory of the `aws` tool will be mounted into the container at the same path as the host, e.g. `/home/ec2-user/miniconda`, which will shadow existing files in the container. Make sure you use a path that is not already present in the container. ::: @@ -389,11 +378,11 @@ To test the installation: curl -s http://localhost:51678/v1/metadata | python -mjson.tool ``` -:::{note} +:::note The `AmazonEC2ContainerServiceforEC2Role` policy must be attached to the instance role in order to be able to connect the EC2 instance created by the Compute Environment to the ECS container. ::: -:::{note} +:::note The `AmazonEC2ContainerRegistryReadOnly` policy should be attached to the instance role in order to get read-only access to Amazon EC2 Container Registry repositories. ::: @@ -405,7 +394,7 @@ Nextflow automatically creates the Batch [Job definitions](http://docs.aws.amazo However, sometimes you may still need to specify a custom **Job Definition** to fine tune the configuration of a specific job, for example to define custom mount paths. -To do that, first create a **Job Definition** in the AWS Console (or by other means). Note the name of the Job definition you created. You can then associate a process execution with this Job definition by using the {ref}`process-container` directive and specifying, in place of the container image name, the Job definition name prefixed by `job-definition://`, as shown below: +To do that, first create a **Job Definition** in the AWS Console (or by other means). Note the name of the Job definition you created. You can then associate a process execution with this Job definition by using the [container][process-container] directive and specifying, in place of the container image name, the Job definition name prefixed by `job-definition://`, as shown below: ```groovy process.container = 'job-definition://your-job-definition-name' @@ -415,7 +404,7 @@ process.container = 'job-definition://your-job-definition-name' Nextflow allows the use of multiple executors in the same workflow application. This feature enables the deployment of hybrid workloads in which some jobs are executed in the local computer or local computing cluster and some jobs are offloaded to AWS Batch. -To enable this feature, use one or more {ref}`config-process-selectors` in your Nextflow configuration to apply the AWS Batch configuration to the subset of processes that you want to offload. For example: +To enable this feature, use one or more [Process selectors][config-process-selectors] in your Nextflow configuration to apply the AWS Batch configuration to the subset of processes that you want to offload. For example: ```groovy process { @@ -431,27 +420,24 @@ aws { } ``` -With the above configuration, processes with the `bigTask` {ref}`process-label` will run on AWS Batch, while the remaining processes will run in the local computer. +With the above configuration, processes with the `bigTask` [label][process-label] will run on AWS Batch, while the remaining processes will run in the local computer. Then launch the pipeline with the -bucket-dir option to specify an AWS S3 path for the jobs computed with AWS Batch and, optionally, the -work-dir to specify the local storage for the jobs computed locally: ```bash -nextflow run