feat(kfp-api): add s3-credentials relation to relate with s3-integrator by mvlassis · Pull Request #956 · canonical/kfp-operators

mvlassis · 2026-06-18T12:46:03Z

Summary

Add s3-credentials relation using the s3 interface.
Implemented the RelationCountGate component manually
The s3 endpoint is parsed into the host, port and TLS flag kfp expects (OBJECTSTORECONFIG_*)
Update the S3 service to include the tls-ca-chain, similar to the structure for the S3 manager in object-storage-integrators
Add integration tests, split into 2 distinct files:
- test_charm_s3.py, which tests the relation with s3-integrator:s3-credentials
- test_charm_object_storage.py, which tests the relation with minio:object-storage
Add test for migrating from minio-operator to s3-integrator in integration-object-storage

Removal

This PR removes the minio-service service created by kfp-api. The context is:

This PR was initially created to fix [feature] remove hard-coded configs in KFP V2 launcher kubeflow/pipelines#9689, which has been fixed
The fix is the introduction of a kfp-launcher configMap that can specify the endpoint (among others) instead of using the default hardcoded value
As of the merge of feat: Update manifests to match upstream kfp 2.16.0 release #870, we now always set the endpoint in kfp-launcher to override the default value, which has been http://seaweedfs.kubeflow:8333 as of Kubeflow pipelines 2.15.0

Notes

Split out of feat: support s3-integrator as an object storage backend #955. Companion PRs add the same support to kfp-ui and kfp-profile-controller.
The root tox.ini change is identical across the three split PRs (conflict-free); the ci.yaml matrix enables integration-s3 only for kfp-api here, so merging the companion PRs will require a small exclude-list merge.
I removed all references of mariadb and replaced them with mysql-k8s :)

Closes #964

Add the `s3` interface (via an `s3-credentials` relation) so kfp-api can use an s3-integrator as an alternative to the in-cluster MinIO (`object-storage`) backend. Exactly one of the two relations may be related at a time; relating both blocks the charm. The s3 endpoint is parsed into the host, port and TLS flag kfp expects, and bucket creation is skipped for s3 (the integrator manages its own buckets). Integration tests are split into object-storage and s3 suites, wired up through a new `integration-s3` tox env and CI matrix entry.

ckfbot · 2026-06-18T12:46:40Z

🤖 Backport labels populated

Labels to this pull request were added automatically by the populate-labels.yaml action.

When the PR is merged, backport PRs according to the labels will be automatically created. To skip the backport creation, remove any unneeded labels before merging the PR.

…t-storage env

Copilot

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

NohaIhab

thanks @mvlassis, left some comments

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

NohaIhab

thanks @mvlassis

dariofaccin

Left some suggestions and questions.
In the integration tests we are deploying grafana-agent, checking alert rules (test_alert_rules), metrics (test_metrics_endpoint), logging (test_logging) and container security context (test_container_security_context) in both object-storage and s3 scenarios. Does it provide more coverage or is it CI time wasted?

Co-authored-by: Dario Faccin <35623244+dariofaccin@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

mvlassis · 2026-06-25T11:37:45Z

We discussed this question by @dariofaccin:

In the integration tests we are deploying grafana-agent, checking alert rules (test_alert_rules), metrics (test_metrics_endpoint), logging (test_logging) and container security context (test_container_security_context) in both object-storage and s3 scenarios. Does it provide more coverage or is it CI time wasted?

And the decision was to keep these tests, as it is the same pattern we use for the test_charm_ambient.py tests. The future plan is to deprecate the object-storage integration in the future, and thus remove the test_charm_object_storage.py file completely.

Copilot

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

dariofaccin

LGTM. Thanks.

ckfbot added backport track/2.16 Backport to track/2.16 backport track/2.15 Backport to track/2.15 backport track/2.5 Backport to track/2.5 labels Jun 18, 2026

mvlassis mentioned this pull request Jun 18, 2026

feat: support s3-integrator as an object storage backend #955

Closed

mvlassis temporarily deployed to edge-pr June 18, 2026 12:56 — with GitHub Actions Inactive

ci(kfp-api): run s3 in default integration env, add integration-objec…

514a98a

…t-storage env

mvlassis temporarily deployed to edge-pr June 18, 2026 13:29 — with GitHub Actions Inactive

Update tox docs

3cf4b5c

mvlassis temporarily deployed to edge-pr June 18, 2026 14:12 — with GitHub Actions Inactive

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread tox.ini Outdated

Comment thread charms/kfp-api/src/charm.py Outdated

Comment thread charms/kfp-api/src/charm.py Outdated

Comment thread charms/kfp-api/src/services/s3.py

mvlassis and others added 3 commits June 24, 2026 11:54

Fix comment

79d3841

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

Error handling for host

e419e06

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

Fix linting

2b3c801

NohaIhab reviewed Jun 24, 2026

View reviewed changes

Comment thread charms/kfp-api/src/charm.py Outdated

Comment thread charms/kfp-api/src/charm.py Outdated

Comment thread charms/kfp-api/src/charm.py Outdated

Comment thread charms/kfp-api/tests/integration/test_charm_object_storage.py

mvlassis added 4 commits June 24, 2026 14:34

Also show missing fields in s3-credentials relation

7f33cbe

Better variable name

11b7726

Emit BlockedStatus on a TLS error

684dcd0

Use mysql instead of mariadb

5698acb

Copilot AI reviewed Jun 24, 2026

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread charms/kfp-api/src/services/s3.py

Comment thread charms/kfp-api/src/services/s3.py Outdated

Comment thread charms/kfp-api/src/charm.py

Comment thread charms/kfp-api/tox.ini

mvlassis and others added 2 commits June 24, 2026 14:52

Return None tls-ca-chain for minio

9e1c851

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

fix: add connect/read timeouts to cached S3 client property

f97978a

NohaIhab approved these changes Jun 24, 2026

View reviewed changes

dariofaccin requested changes Jun 25, 2026

View reviewed changes

mvlassis and others added 4 commits June 25, 2026 14:27

Address comments

0058b73

Update charms/kfp-api/src/services/s3.py

0782d0f

Co-authored-by: Dario Faccin <35623244+dariofaccin@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

Address comments

088a9c8

Address comments

714f587

mvlassis added 2 commits June 25, 2026 14:38

Address comments

1102e34

Address comments

d164ce5

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread charms/kfp-api/src/charm.py

Comment thread charms/kfp-api/src/charm.py Outdated

Comment thread bundle.yaml Outdated

mvlassis and others added 4 commits June 25, 2026 14:30

fix: Address comments

a222d7a

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

Use profile: testing for mysql-k8s

e858c78

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Manos Vlassis <57320708+mvlassis@users.noreply.github.com>

Remove uneccessary minio-service

26d21b3

Mocked resource handlere

f44ddfd

dariofaccin approved these changes Jun 25, 2026

View reviewed changes

NohaIhab approved these changes Jun 25, 2026

View reviewed changes

mvlassis mentioned this pull request Jun 26, 2026

Add s3 interface to kfp-api to relate with s3-integrator #964

Closed

Uh oh!

Conversation

mvlassis commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Removal

Notes

Uh oh!

ckfbot commented Jun 18, 2026

🤖 Backport labels populated

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NohaIhab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NohaIhab left a comment

Choose a reason for hiding this comment

Uh oh!

dariofaccin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mvlassis commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dariofaccin left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mvlassis commented Jun 18, 2026 •

edited

Loading