Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/content/docs/components/ip-addresses/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ to successfully connect to your system. This applies to all components including

**Important:** These IP addresses can change in the future! For your convenience, you can programmatically
fetch and process the [list of existing IP addresses in JSON format](/components/ip-addresses/kbc-public-ip.json).
Below are listed the available [Keboola Stack endpoints](https://developers.keboola.com/overview/api/#regions-and-endpoints).
Below are listed the available [Keboola Stack endpoints](https://developers.keboola.com/overview/api/#stacks-and-endpoints).

For ease of identification, our outbound IP addresses on AWS stacks (except for legacy services) now have reverse DNS records.
Each IP address has a unique name like `outbound-if-issue-contact-support-at-keboola-com.keboola.com`, embedding a reference for a support email.
Expand Down
12 changes: 6 additions & 6 deletions src/content/docs/extend/common-interface/config-file/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,9 @@ The `image_parameters` contents are configured in the [component settings](https
text fields: **Image Parameters** and **Stack Parameters**.

Both JSONs are merged into the `image_parameters` of the configuration file. The *Stack Parameters*
provide different values for different [Keboola Stacks](/overview/api/#regions-and-endpoints). Values in
provide different values for different [Keboola Stacks](https://developers.keboola.com/overview/api/#stacks-and-endpoints). Values in
*Stack Parameters* are merged with those in *Image Parameters* with *Stack Parameters* having a higher priority.
*Stack Parameters* are indexed with [Storage URL](/overview/api/#regions-and-endpoints) or the given region.
*Stack Parameters* are indexed with [Storage URL](https://developers.keboola.com/overview/api/#stacks-and-endpoints) or the given region.

Given the following *Image Parameters*:

Expand Down Expand Up @@ -114,7 +114,7 @@ As with configurations, the encrypted values must be prefixed with the hash sign
you **have to encrypt values manually via the API** -- they will not be encrypted automatically when you store *Stack Parameters*!
When using the [encryption API](https://api.keboola.com/?service=encryption#post-/encrypt), provide only the `componentId`
parameter (using `projectId` or `configId` will make the cipher unusable).
Also take care to use the correct [API URL](/overview/api/#regions-and-endpoints) to obtain
Also take care to use the correct [API URL](https://developers.keboola.com/overview/api/#stacks-and-endpoints) to obtain
ciphers for each region you need.

## State File
Expand Down Expand Up @@ -247,15 +247,15 @@ A sample configuration file might look like this:
"source": "destination.csv",
"destination": "out.c-main.test",
"incremental": false,
"colummns": [],
"columns": [],
"primary_key": [],
"delete_where": [],
"delimiter": ",",
"enclosure": "\""
},
{
"source": "write-alwayss.csv",
"destination": "out.c-main.output-even-on-error"
"source": "write-always.csv",
"destination": "out.c-main.output-even-on-error",
"write_always": true
}
],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ components that modify external resources (e.g., database writers) and those tha

### Is a Component Executed in a Branch Context?

When the [Job Queue](/extend/job-queue/) executes a job in a branch, it sets the [`KBC_BRANCHID` environment variable](/extend/common-interface/environment/#environment-variables) to the current branch ID, which is unique accross the stack.
When the [Job Queue](/extend/job-queue/) executes a job in a branch, it sets the [`KBC_BRANCHID` environment variable](/extend/common-interface/environment/#environment-variables) to the current branch ID, which is unique across the stack.

Typically, the fact that a component is executed in a branch is not very important to the component itself. It behaves the same way, and the Keboola job runner does all the heavy lifting.

Expand Down
6 changes: 3 additions & 3 deletions src/content/docs/extend/common-interface/environment/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,14 @@ The following environment variables are injected into the container:

- `KBC_DATADIR`: Always `/data/` in Keboola; use this variable during component development to create development and testing environments.
- `KBC_RUNID`: The RunId from Storage; links all events within an API call (useful for logging).
- `KBC_PROJECTID`: The ID of the project in Keboola within a [Keboola stack](/overview/api/#regions-and-endpoints).
- `KBC_STACKID`: The ID of the [Keboola stack](/overview/api/#regions-and-endpoints).
- `KBC_PROJECTID`: The ID of the project in Keboola within a [Keboola stack](https://developers.keboola.com/overview/api/#stacks-and-endpoints).
- `KBC_STACKID`: The ID of the [Keboola stack](https://developers.keboola.com/overview/api/#stacks-and-endpoints).
- `KBC_CONFIGID`: The ID of the configuration, or a hash of configuration data if the configuration is not named (e.g., when `configData` is used in an [API call](https://api.keboola.com/?service=job-queue#post-/jobs)).
- `KBC_CONFIGVERSION`: The version of the configuration, or empty if unnamed (when `configData` is used in the [API call](https://api.keboola.com/?service=job-queue#post-/jobs)).
- `KBC_COMPONENTID`: The ID of the component.
- `KBC_CONFIGROWID`: The ID of the configuration row, if available.
- `KBC_BRANCHID`: The ID of the [development branch](https://api.keboola.com/?service=storage#get-/v2/storage/dev-branches/-id-).
- `KBC_STAGING_FILE_PROVIDER`: Either `aws` or `azure`, depending on the type of [stack](/overview/api/#regions-and-endpoints) the container is running. This value refers to the file storage used during [file import/export operations](/storage/api/import-export/).
- `KBC_STAGING_FILE_PROVIDER`: Either `aws` or `azure`, depending on the type of [stack](https://developers.keboola.com/overview/api/#stacks-and-endpoints) the container is running. This value refers to the file storage used during [file import/export operations](/storage/api/import-export/).
- `KBC_PROJECT_FEATURE_GATES`: A comma-separated list of feature gates activated for the current project. Feature gates are considered internal and may change or disappear without notice. We recommend checking with our support team before relying on any feature gates.
- `KBC_COMPONENT_RUN_MODE`: Either `run` or `debug`. The value `debug` is used when the job is run in debug mode ([learn more](/extend/component/running/#debugging)). This variable can be helpful, for example, to enable more verbose logging.
- `KBC_DATA_TYPE_SUPPORT`: Either `authoritative`, `hints`, or `none`:
Expand Down
16 changes: 8 additions & 8 deletions src/content/docs/extend/common-interface/folders/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ The component can run arbitrary queries against the database. The database crede
"host": "database.example.com",
"warehouse": "test",
"database": "my-db",
"schema": "my-schema"
"schema": "my-schema",
"user": "john-doe",
"password": "secret"
}
Expand Down Expand Up @@ -214,7 +214,7 @@ and unloaded from when the job finishes (when staging storage output is set).
The workspace is created just before the job starts and is deleted when the job is terminated.

If this option is enabled, the data and the manifests will be loaded to the azure storage blob container under the
data folder similarly to how it does when using the default [local filesystem](extend/common-interface/folders/#root-folder-data).
data folder similarly to how it does when using the default [local filesystem](/extend/common-interface/folders/#root-folder-data).

### Files
Files are loaded into the workspace as `[file name]/[file ID]`. For example, if a file 'test.txt' with ID '12345' is in
Expand All @@ -226,17 +226,17 @@ the input mapping then the file will appear in the storage blob container with U
Synapse only exports tables as sliced files.
So for example, if you set as table input mapping the table `in.c-main.my-input` as source and `my-input.csv` as
destination then in the ABS workspace you will find it with the following structure:
- [containerName]/data/in/tables/my-inpupt.csv/[random identifier1].txt
- [containerName]/data/in/tables/my-inpupt.csv/[random identifier2].txt
- [containerName]/data/in/tables/my-inpupt.csv/[random identifier3].txt
- [containerName]/data/in/tables/my-input.csv/[random identifier1].txt
- [containerName]/data/in/tables/my-input.csv/[random identifier2].txt
- [containerName]/data/in/tables/my-input.csv/[random identifier3].txt

### Mappings

To sum up, below is a sample storage configuration and where the files are written from and to:

| Direction | Source | Destination |
| --- | --- | --- |
| input | in.c-main.my-table-from-abs-workspace | Many slices like `[abs-workspace-root]/data/in/tables/my-inpupt-table.csv/[random identifier].txt` |
| input | in.c-main.my-table-from-abs-workspace | Many slices like `[abs-workspace-root]/data/in/tables/my-input-table.csv/[random identifier].txt` |
| input | file with tag `my-input-files` named `input-file.txt` | `[abs-workspace-root]/data/in/files/test.txt/12345` |
| output | `[abs-workspace-root]/data/out/tables/my-output-table.csv` | out.c-main.my-table-from-abs-workspace |
| output | `[abs-workspace-root]/data/out/files/my-file.txt` | file `my-file.txt` with tag `uploaded-from-abs-workspace` |
Expand All @@ -257,7 +257,7 @@ To sum up, below is a sample storage configuration and where the files are writt
"tags": ["my-input-files"]
}
]
}
},
"output": {
"tables": [
{
Expand Down Expand Up @@ -295,7 +295,7 @@ shown below.
"authorization": {
"workspace": {
"container": "azure-storage-blob-container",
"connectionString": "azure-storage-blob-SAS-connection-string",
"connectionString": "azure-storage-blob-SAS-connection-string"
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/extend/common-interface/logging/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ slug: 'extend/common-interface/logging'
---


There are two main, mutually exclusive, ways in which your component can display events Keboola end-users:
There are two main, mutually exclusive, ways in which your component can display events to Keboola end-users:

1. Using [standard output and standard error](https://en.wikipedia.org/wiki/Standard_streams)
2. Using [Graylog GELF](http://docs.graylog.org/en/3.1/pages/gelf.html) compatible logger
Expand All @@ -28,7 +28,7 @@ of the component and forwards the STDOUT content live to [Storage API Events](ht
(log level `info`). The content of STDERR is collected and added (if not empty) as the last event of the job with level `error`.
The events are displayed in a [Job detail](/management/jobs/).

The entire output from a component is filter for sensitive values. The [Job Queue](/extend/job-queue/)
The entire output from a component is filtered for sensitive values. The [Job Queue](/extend/job-queue/)
keeps track of all encrypted values and if it encounters them in the component output, it replaces
them by `[hidden]` placeholder. This prevents accidental leaking of sensitive information for
example in exception traces.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ contents:
}
```

The `name` node refers to the name of the component configuration.
The `name` node refers to the name of the table.
The `metadata` and `column_metadata` fields contain
Metadata for the table and its columns.
The `metadata` field corresponds to the [Table Metadata API call](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/tables/-id-/metadata).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ You have created an empty transformation.
/admin/projects/{PROJECT_ID}/transformations-v2/{COMPONENT_ID}/{CONFIGURATION_ID}
```

**Set the code pattern to the transformation via [Storage API](/overview/api/).**
**Set the code pattern to the transformation via [Storage API](https://developers.keboola.com/overview/api/).**

Load the configuration in the JSON format via the [Configuration Detail](https://api.keboola.com/?service=storage#get-/v2/storage/branch/-branchId-/components/-componentId-/configs/-configurationId-) API call.

Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/extend/component/deployment/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ component repository and tag were automatically set:

The component is now runnable in Keboola. You can view all settings in our
[example repository](https://github.com/keboola/ex-docs-tutorial). You can also
review [Travis Configuration](https://travis-ci.org/keboola/ex-docs-tutorial/).
review [Travis Configuration](https://app.travis-ci.com/keboola/ex-docs-tutorial).

*Note that it takes up to **5 minutes** before the changes in the Developer Portal propagate to all Keboola instances in all regions.*

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ locally with `pip3 install keboola.component`.

A generated [API documentation](https://htmlpreview.github.io/?https://raw.githubusercontent.com/keboola/python-component/main/docs/api-html/component/interface.html)
is available for the package, and an actual working example can be found in our
[Python template](https://bitbucket.org/kds_consulting_team/kbc-python-template/src/master/src/component.py).
[Python template](https://github.com/keboola/cookiecutter-python-component/blob/main/%7B%7Bcookiecutter.repository_folder_name%7D%7D/src/component.py).

### Initialization

Expand Down Expand Up @@ -112,7 +112,7 @@ ci = CommonInterface()

# A ValueError error is raised if the config.json file does not exists in the data dir.
# Checks for required parameters and throws ValueError if any is missing.
ci.validate_configuration(REQUIRED_PARAMETERS)
ci.validate_configuration_parameters(REQUIRED_PARAMETERS)

# print Keboola Project ID from the environment variable if present:
logging.info(ci.environment_variables.project_id)
Expand All @@ -121,7 +121,7 @@ logging.info(ci.environment_variables.project_id)
logging.info(ci.configuration.parameters[SOME_PARAMETER])
```

The above would read the `somemyParameter_user_parameter` parameter from the user-supplied configuration:
The above would read the `myParameter` parameter from the user-supplied configuration:

```json
{
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/extend/component/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ As a developer, you implement the application logic in a language of your choice
git repository. The component must adhere to our [common interface](/extend/common-interface/).
To start quickly, use our component generators for [Python](https://github.com/keboola/cookiecutter-python-component) or [PHP, CI pipelines](https://github.com/keboola/component-generator) that can generate a skeleton of the component for you. We also provide libraries to help you with implementation in
[Python](https://github.com/keboola/python-component) and
[PHP](https://github.com/keboola/php-docker-application).
[PHP](https://github.com/keboola/php-component).
Check our example component in [Python](https://github.com/keboola/cookiecutter-python-component/tree/main/%7B%7Bcookiecutter.repository_folder_name%7D%7D).

The main part of the [common interface](/extend/common-interface/) is the specification how
Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/extend/component/running/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ a [data folder and configuration file](/extend/common-interface/). The above men
[Run Job API call](https://api.keboola.com/?service=job-queue#post-/jobs)
is designed to do that.

We recommend that you use the [API console or Postman](/overview/api/) to call the API.
We recommend that you use the [API console or Postman](https://developers.keboola.com/overview/api/) to call the API.
A [collection of examples](https://documenter.getpostman.com/view/3086797/kbc-samples/77h845D?version=latest#9b9f3e7b-de3b-4c90-bad6-a8760e3852eb) of the
Run Job API call in debug mode is available in Postman Docs.

Expand Down Expand Up @@ -141,7 +141,7 @@ destination to which the table will be downloaded for the component; it will the

The entire `configData.storage` node is generated by the UI. The node `parameters` contains arbitrary
parameters which are passed to the component. The URL of the request
is `https://queue.keboola.com/jobs` (in the [US Region](/overview/api/#regions-and-endpoints)).
is `https://queue.keboola.com/jobs` (in the [US Region](https://developers.keboola.com/overview/api/#stacks-and-endpoints)).

The request is same to the one used when running a job.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ initialized by reading the `data` directory (`CommonInterface()`). Its property
configuration parameters as a dictionary. Apart from that the [Python library](https://github.com/keboola/python-component)
provides methods to handle input / output files and many more (see [here](/extend/component/implementation/python/) for more information).

Similar library is currently available also for the [R language](https://github.com/keboola/r-docker-application).
A similar library is available for the [R language](https://github.com/keboola/r-docker-application).
It does no magic or rocket science, so you can read the [config file](/extend/common-interface/config-file/) directly if you wish.

Commit and push the code in your repository and tag it with a [normal version tag](https://semver.org/#spec-item-2).
Expand Down Expand Up @@ -133,7 +133,7 @@ configuration, the following schema can be created:

In the schema the two properties `sound` and `repeat` are declared along with the specification
of their form input fields.
You can test the above schema [online](http://jeremydorn.com/json-editor/) ([alternative](https://mozilla-services.github.io/react-jsonschema-form/)) and verify that the
You can test the above schema [online](https://www.jeremydorn.com/json-editor/) ([alternative](https://mozilla-services.github.io/react-jsonschema-form/)) and verify that the
form generated from it produces the desired JSON structure. Once satisfied with the result,
simply paste the schema into the **Configuration schema** in your component properties in the
[Developer Portal](https://components.keboola.com/).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ to launch [Bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) instead. The [
ensure that the container runs in an interactive mode.
You can then inspect the container contents: 'ls /data/'.

### Step 4 -- Modify
### Step 5 -- Modify
Chances are that you want to modify the component code often. If you modify the component code, you have to rebuild the
docker image. To avoid the slow and tedious work, run the image with the following command:

Expand Down
6 changes: 3 additions & 3 deletions src/content/docs/extend/component/tutorial/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Before you continue with this tutorial, make sure you
- can log in to one of the Keboola [stacks](/overview/#stacks)
- have a [Github](https://github.com/) account.

*Note: Even though the tutorial assumes using [GitHub](https://github.com/) + [Travis](https://travis-ci.org/) services, they are not required for extending Keboola.
*Note: Even though the tutorial assumes using [GitHub](https://github.com/) + [Travis](https://www.travis-ci.com/) services, they are not required for extending Keboola.
We use them because we like them the most. The [deployment documentation](/extend/component/deployment/) shows how to configure,
for example, [Bitbucket](/extend/component/deployment/#bitbucket-integration) and [GitLab](/extend/component/deployment/#gitlab-integration)
integrations.*
Expand Down Expand Up @@ -118,7 +118,7 @@ cookiecutter gh:keboola/cookiecutter-python-component```
- You can execute the component via normal local environment without docker installed.
- Set any additional dependencies for your project in `requirements.txt`

More information on the template [here](https://bitbucket.org/kds_consulting_team/kbc-python-template/src/master/README.md)
More information on the template [here](https://github.com/keboola/cookiecutter-python-component)

## CI Setup
- Bitbucket: Enable [pipelines](https://confluence.atlassian.com/bitbucket/get-started-with-bitbucket-pipelines-792298921.html) in the repository.
Expand All @@ -133,7 +133,7 @@ cookiecutter gh:keboola/cookiecutter-python-component```
- `KBC_DEVELOPERPORTAL_VENDOR` - dev portal vendor
- `KBC_STORAGE_TOKEN` - (optional) in case you wish to run KBC automated tests

[https://github.com/keboola/cookiecutter-python-component]()
[https://github.com/keboola/cookiecutter-python-component](https://github.com/keboola/cookiecutter-python-component)

### PHP, R or just CI pipeline for GitHub or Bitbucket

Expand Down
Loading
Loading