diff --git a/daprdocs/content/en/concepts/dapr-services/placement.md b/daprdocs/content/en/concepts/dapr-services/placement.md index e285e7216eb..e0061ec04b8 100644 --- a/daprdocs/content/en/concepts/dapr-services/placement.md +++ b/daprdocs/content/en/concepts/dapr-services/placement.md @@ -5,18 +5,46 @@ linkTitle: "Placement" description: "Overview of the Dapr Placement service" --- +{{% alert title="Deprecation notice" color="warning" %}} +Starting with Dapr v1.19, actor placement can be served by the [Scheduler service]({{% ref scheduler %}}), enabled by the opt-in flag `global.scheduler.placement.enabled=true` (`false` by default). The standalone Placement service is planned for deprecation in Dapr v1.21, when the Scheduler serves actor placement by default. +{{% /alert %}} + The Dapr Placement service is used to calculate and distribute distributed hash tables for the location of [Dapr actors]({{% ref actors %}}) running in [self-hosted mode]({{% ref self-hosted %}}) or on [Kubernetes]({{% ref kubernetes %}}). Grouped by namespace, the hash tables map actor types to pods or processes so a Dapr application can communicate with the actor. Anytime a Dapr application activates a Dapr actor, the Placement service updates the hash tables with the latest actor location. ## Self-hosted mode -The Placement service Docker container is started automatically as part of [`dapr init`]({{% ref self-hosted-with-docker %}}). It can also be run manually as a process if you are running in [slim-init mode]({{% ref self-hosted-no-docker %}}). +The Placement service Docker container is started automatically as part of [`dapr init`]({{% ref self-hosted-with-docker %}}), unless you initialize with `dapr init --scheduler-placement`, which has the [Scheduler service serve actor placement](#serving-placement-from-the-scheduler-service) instead. It can also be run manually as a process if you are running in [slim-init mode]({{% ref self-hosted-no-docker %}}). ## Kubernetes mode The Placement service is deployed as part of `dapr init -k`, or via the Dapr Helm charts. You can run Placement in high availability (HA) mode. [Learn more about setting HA mode in your Kubernetes service.]({{% ref "kubernetes-production#individual-service-ha-helm-configuration" %}}) +Alternatively, actor placement can be served by the [Scheduler service]({{% ref scheduler %}}) instead of the standalone Placement service. See [Serving placement from the Scheduler service](#serving-placement-from-the-scheduler-service). + For more information on running Dapr on Kubernetes, visit the [Kubernetes hosting page]({{% ref kubernetes %}}). +## Serving placement from the Scheduler service + +The Scheduler service can serve actor placement itself, so the standalone Placement service does not run. + +In Kubernetes mode, set the Helm value: + +``` +global.scheduler.placement.enabled=true +``` + +With this setting, the Placement StatefulSet is not deployed and the Scheduler runs with `--placement-enabled=true`. + +In self-hosted mode, initialize Dapr with `dapr init --scheduler-placement` (requires Dapr 1.19 or later). The Placement container is not started, and the CLI starts the Scheduler container with `--placement-enabled=true` for you. No further configuration is needed. Only in [slim-init mode]({{% ref self-hosted-no-docker %}}), where you start the binaries yourself, the flag skips installing the Placement binary and you pass `--placement-enabled=true` when running the Scheduler binary. Applications and sidecars need no configuration of their own: each sidecar takes actor placement from whichever service the control plane advertises, so the cluster always has exactly one placement authority. Toggling the setting in either direction requires no sidecar restarts. + +The two services place actors with different algorithms: the Placement service uses a consistent hash ring, while the Scheduler uses rendezvous hashing. Both give the same guarantees (every sidecar deterministically agrees on a single host for each actor ID, and ownership spreads across hosts), but the resulting actor-to-host assignments differ. Switching the placement authority in either direction therefore reassigns actors once: affected actors are deactivated and reactivate on their new hosts with their state intact, as in any actor rebalance. + +{{% alert title="Important" color="warning" %}} +Complete your Dapr version rollout before enabling this setting. Sidecars running an older Dapr version can only use the Placement service: with it undeployed, their Actor and Workflow APIs stall until the pod is upgraded to a version that supports scheduler placement. No actor state is lost. While the Placement service is still running, the Scheduler waits for every older sidecar to disconnect before signalling it to stand down, so the cluster keeps a single placement authority throughout the rollout. Once no Placement service remains, a connected older sidecar is logged as a warning instead, since nothing is left to serve it. +{{% /alert %}} + +The [`/placement/state` API](#placement-tables) is only available from the standalone Placement service and is not served by the Scheduler. + ## Placement tables There is an [HTTP API `/placement/state` for Placement service]({{% ref placement_api %}}) that exposes placement table information. The API is exposed on the sidecar on the same port as the healthz. This is an unauthenticated endpoint, and is disabled by default. You need to set `DAPR_PLACEMENT_METADATA_ENABLED` environment or `metadata-enabled` command line args to true to enable it. If you are using helm you just need to set `dapr_placement.metadataEnabled` to true. @@ -105,6 +133,8 @@ global.actors.enabled=false The Placement service is not deployed with this setting in Kubernetes mode. This not only disables actor deployment, but also disables workflows, given that workflows use actors. This setting only applies in Kubernetes mode, however initializing Dapr with `--slim` excludes the Placement service from being deployed in self-hosted mode. +The Placement service is also not deployed when actor placement is served by the Scheduler service (`global.scheduler.placement.enabled=true`). Actors and workflows keep working in that case. See [Serving placement from the Scheduler service](#serving-placement-from-the-scheduler-service). + For more information on running Dapr on Kubernetes, visit the [Kubernetes hosting page](https://docs.dapr.io/operations/hosting/kubernetes/). diff --git a/daprdocs/content/en/concepts/dapr-services/scheduler.md b/daprdocs/content/en/concepts/dapr-services/scheduler.md index 879600c151f..9f604355f03 100644 --- a/daprdocs/content/en/concepts/dapr-services/scheduler.md +++ b/daprdocs/content/en/concepts/dapr-services/scheduler.md @@ -10,6 +10,8 @@ The Dapr Scheduler service is used to schedule different types of jobs, running - Actor reminder jobs (used by the actor reminders) - Actor reminder jobs created by the Workflow API (which uses actor reminders) +The Scheduler service can also serve actor placement in place of the standalone [Placement service]({{% ref placement %}}). See [Serving actor placement](#serving-actor-placement). + There is no concept of a leader Scheduler instance. All Scheduler service replicas are considered peers. All receive jobs to be scheduled for execution and the jobs are allocated between the available Scheduler service replicas for load balancing of the trigger events. @@ -199,9 +201,36 @@ curl -s http://localhost:2379/metrics Fine tune the embedded etcd to your needs by [reviewing and configuring the Scheduler's etcd flags as needed](https://github.com/dapr/dapr/blob/master/charts/dapr/README#dapr-scheduler-options). +## Serving actor placement + +The Scheduler can serve actor placement itself, so the standalone Placement service does not run. + +In Kubernetes mode, set the Helm value: + +``` +global.scheduler.placement.enabled=true +``` + +In self-hosted mode, initialize Dapr with the `--scheduler-placement` flag, which skips the Placement container and starts the Scheduler with `--placement-enabled=true`: + +```bash +dapr init --scheduler-placement +``` + +Sidecars take actor placement from whichever service the control plane advertises, so applications need no configuration of their own and the cluster always has exactly one placement authority. Toggling the setting in either direction requires no sidecar restarts. [Learn more about serving placement from the Scheduler service.]({{% ref "placement#serving-placement-from-the-scheduler-service" %}}) + +The dissemination behavior of scheduler-served placement can be tuned with the following Scheduler flags: + +``` +--placement-enabled bool When enabled, this scheduler serves actor placement to daprd sidecars, replacing the standalone placement service. All scheduler replicas in the cluster must set the same value. (default false) +--placement-disseminate-timeout duration The timeout for a placement dissemination round to daprd sidecars. Sidecars which fail to acknowledge within the timeout have their placement stream closed. (default 8s) +--placement-disseminate-coalesce-window duration The window in which placement membership changes are coalesced into a single dissemination round. 0 disseminates immediately. (default 0) +``` + ## Disabling the Scheduler service If you are not using any features that require the Scheduler service (Jobs API, Actor Reminders, or Workflows), you can disable it by setting `global.scheduler.enabled=false`. +The Scheduler service cannot be disabled while it serves actor placement (`global.scheduler.placement.enabled=true`). For more information on running Dapr on Kubernetes, visit the [Kubernetes hosting page]({{% ref kubernetes %}}). ## Flag tuning @@ -286,4 +315,4 @@ dapr_scheduler.etcdMaxTxnOps=10000 ## Related links - [Learn more about the Jobs API.]({{% ref jobs_api %}}) -- [Learn more about Actor Reminders.]{{% ref "actors-features-concepts#reminders" %}}) +- [Learn more about Actor Reminders.]({{% ref "actors-features-concepts#reminders" %}}) diff --git a/daprdocs/content/en/developing-applications/building-blocks/actors/actors-features-concepts.md b/daprdocs/content/en/developing-applications/building-blocks/actors/actors-features-concepts.md index 2ae32429af6..e3ff16ff986 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/actors/actors-features-concepts.md +++ b/daprdocs/content/en/developing-applications/building-blocks/actors/actors-features-concepts.md @@ -52,7 +52,7 @@ This simplifies some choices, but also carries some consideration: - Because actors are randomly placed, it should be expected that actor operations always require network communication, including serialization and deserialization of method call data, incurring latency and overhead. {{% alert title="Note" color="primary" %}} - Note: The Dapr actor Placement service is only used for actor placement and therefore is not needed if your services are not using Dapr actors. The Placement service can run in all [hosting environments]({{% ref hosting %}}), including self-hosted and Kubernetes. + Note: The Dapr actor Placement service is only used for actor placement and therefore is not needed if your services are not using Dapr actors. The Placement service can run in all [hosting environments]({{% ref hosting %}}), including self-hosted and Kubernetes. In both self-hosted and Kubernetes modes, actor placement can also be served by the [Scheduler service]({{% ref "placement#serving-placement-from-the-scheduler-service" %}}) instead of a standalone Placement service. This is transparent to applications and sidecars. {{% /alert %}} ## Actor communication diff --git a/daprdocs/content/en/developing-applications/building-blocks/actors/actors-runtime-config.md b/daprdocs/content/en/developing-applications/building-blocks/actors/actors-runtime-config.md index 8df504dd7e2..13b94b10e08 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/actors/actors-runtime-config.md +++ b/daprdocs/content/en/developing-applications/building-blocks/actors/actors-runtime-config.md @@ -29,6 +29,8 @@ Starting in Dapr v1.18, daprd clamps the effective drain timeout: The clamp is applied at both registration sites: the global `drainOngoingCallTimeout` and any per-actor-type `drainOngoingCallTimeout` set under `entitiesConfig`. The configuration values your app reports to daprd via the actor config endpoint are unchanged; only the effective in-process value used during drain is clamped. If you see the warning in daprd logs, lower the configured value so that it is comfortably below the placement dissemination timeout, or raise the placement dissemination timeout on the control plane. +> **Note:** The daprd-side placement dissemination timeout discussed here (default 30 seconds) is distinct from the control plane's dissemination round timeouts: `dapr_placement.disseminateTimeout` on the Placement service, and `--placement-disseminate-timeout` on the Scheduler when it [serves placement]({{% ref "placement#serving-placement-from-the-scheduler-service" %}}). Both default to `8s`. + ## Examples {{< tabpane text=true >}} diff --git a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-deploy.md b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-deploy.md index bf344aa11fd..688d999d4a4 100644 --- a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-deploy.md +++ b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-deploy.md @@ -248,6 +248,8 @@ dapr-sidecar-injector-8555576b6f-29cqm 1/1 Running 0 40s dapr-sentry-9435776c7f-8f7yd 1/1 Running 0 40s ``` +> **Note:** The `dapr-placement` pod is not present when actor placement is served by the Scheduler service (`global.scheduler.placement.enabled=true`). + ### Uninstall Dapr on Kubernetes ```bash diff --git a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-overview.md b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-overview.md index 629866e4f1b..e72cfabaa3c 100644 --- a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-overview.md +++ b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-overview.md @@ -12,9 +12,9 @@ Dapr can be configured to run on any supported versions of Kubernetes. To achiev |-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `dapr-operator` | Manages [component]({{% ref components %}}) updates and Kubernetes services endpoints for Dapr (state stores, pub/subs, etc.) | | `dapr-sidecar-injector` | Injects Dapr into [annotated](#adding-dapr-to-a-kubernetes-deployment) deployment pods and adds the environment variables `DAPR_HTTP_PORT` and `DAPR_GRPC_PORT` to enable user-defined applications to easily communicate with Dapr without hard-coding Dapr port values. | -| `dapr-placement` | Used for [actors]({{% ref actors %}}) only. Creates mapping tables that map actor instances to pods | +| `dapr-placement` | Used for [actors]({{% ref actors %}}) only. Creates mapping tables that map actor instances to pods. Not deployed when actor placement is served by the Scheduler service (`global.scheduler.placement.enabled=true`) | | `dapr-sentry` | Manages mTLS between services and acts as a certificate authority. For more information read the [security overview]({{% ref "security-concept.md" %}}) | -| `dapr-scheduler` | Provides distributed job scheduling capabilities used by the Jobs API, Workflow API, and Actor Reminders | +| `dapr-scheduler` | Provides distributed job scheduling capabilities used by the Jobs API, Workflow API, and Actor Reminders. Can also serve [actor placement]({{% ref "placement#serving-placement-from-the-scheduler-service" %}}) in place of `dapr-placement` | diff --git a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md index b692490a684..9db0442a72e 100644 --- a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md +++ b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md @@ -122,6 +122,8 @@ To scale scheduler and placement to three instances independently of the `global --wait ``` +> **Note:** When actor placement is served by the Scheduler service (`global.scheduler.placement.enabled=true`), the Placement service is not deployed and `dapr_placement.ha` has no effect. The Scheduler replica count then determines placement availability. + ### Spreading Placement and Scheduler replicas With HA enabled, the default Helm values add pod anti-affinity so Scheduler and Placement service replicas are attempted to be scheduled across different failure domains if cluster resourcing allows. This is configured using the following Helm chart [options](https://github.com/dapr/dapr/blob/master/charts/dapr/README.md) @@ -276,7 +278,7 @@ Update pods that are running Dapr to pick up the new version of the Dapr runtime Enabling HA mode for an existing Dapr deployment requires two steps: -1. Delete the existing placement stateful set. +1. Delete the existing placement stateful set. Skip this step if the placement StatefulSet is not deployed, for example when actor placement is served by the Scheduler service (`global.scheduler.placement.enabled=true`). ```bash kubectl delete statefulset.apps/dapr-placement-server -n dapr-system @@ -324,6 +326,8 @@ When running in production, it's recommended to configure the Placement service 2. `dapr_placement.keepAliveTimeout` sets the timeout period for Dapr sidecars to respond to the Placement service's [keep alive](https://grpc.io/docs/guides/keepalive/) pings before the Placement service closes the connection. Lower values will lead to shorter actor rebalancing time in case of pod loss/restart, but higher network traffic during normal operation. Accepts values between `1s` and `10s`. Default is `3s`. 3. `dapr_placement.disseminateTimeout` sets the timeout period for dissemination to be delayed after actor membership change (usually related to pod restarts) to avoid excessive dissemination during multiple pod restarts. Higher values will reduce the frequency of dissemination, but delay the table dissemination. Accepts values between `1s` and `3s`. Default is `2s`. +When actor placement is served by the Scheduler service (`global.scheduler.placement.enabled=true`), the `dapr_placement.*` values above do not apply because the Placement service is not deployed. Scheduler-served placement is tuned with the Scheduler flags `--placement-disseminate-timeout` (the time a dissemination round waits for every sidecar to acknowledge before evicting the slow ones, default `8s`, a different knob than `dapr_placement.disseminateTimeout`) and `--placement-disseminate-coalesce-window` (the window in which membership changes are folded into a single dissemination round, default `0` for immediate dissemination). [Learn more about serving placement from the Scheduler service.]({{% ref "placement#serving-placement-from-the-scheduler-service" %}}) + ## Service account tokens diff --git a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-upgrade.md b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-upgrade.md index 963cdb5bdac..419c38e7338 100644 --- a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-upgrade.md +++ b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-upgrade.md @@ -121,6 +121,18 @@ As of version 1.0.0 onwards, existing certificate values will automatically be r [Enable high availability mode in an existing Dapr deployment with a few additional steps.]({{% ref "kubernetes-production.md#enabling-high-availability-in-an-existing-dapr-deployment" %}}) +## Enable scheduler placement in an existing Dapr deployment + +Actor placement can be served by the Scheduler service instead of the standalone Placement service by upgrading with `global.scheduler.placement.enabled=true`. The placement StatefulSet is removed by the same upgrade, and running sidecars adopt scheduler placement on their own: no sidecar restarts are needed, in either direction. [Learn more about serving placement from the Scheduler service.]({{% ref "placement#serving-placement-from-the-scheduler-service" %}}) + +{{% alert title="Important" color="warning" %}} +Complete your Dapr version rollout before enabling this setting. Sidecars running an older Dapr version can only use the Placement service: with it undeployed, their Actor and Workflow APIs stall until the pod is upgraded to a version that supports scheduler placement. No actor state is lost. While the Placement service is still running, the Scheduler waits for every older sidecar to disconnect before signalling it to stand down, so the cluster keeps a single placement authority throughout the rollout. Once no Placement service remains, a connected older sidecar is logged as a warning instead, since nothing is left to serve it. +{{% /alert %}} + +To roll back, upgrade with the setting `false`: the placement StatefulSet is redeployed and the schedulers hand placement back to it, again without sidecar restarts. In both directions, the change [reassigns actors once]({{% ref "placement#serving-placement-from-the-scheduler-service" %}}) because the two services place actors with different algorithms. Actor state is unaffected. + +Because `helm upgrade` resets values not passed on the command line, pass the setting on every subsequent upgrade (or keep it in your values file), otherwise the placement StatefulSet is redeployed and placement hands back to it. + ## Related links - [Dapr on Kubernetes]({{% ref kubernetes-overview.md %}}) diff --git a/daprdocs/content/en/operations/security/mtls.md b/daprdocs/content/en/operations/security/mtls.md index 19655a5e181..c1403add6de 100644 --- a/daprdocs/content/en/operations/security/mtls.md +++ b/daprdocs/content/en/operations/security/mtls.md @@ -69,6 +69,8 @@ kubectl rollout restart deploy/dapr-operator -n kubectl rollout restart statefulsets/dapr-placement-server -n ``` +> **Note:** When actor placement is served by the Scheduler service (`global.scheduler.placement.enabled=true`), the `dapr-placement-server` StatefulSet is not deployed: skip its restart command above. + *Note: the control plane Sidecar Injector service does not need to be redeployed* ### Disabling mTLS with Helm @@ -213,6 +215,8 @@ kubectl rollout restart deploy/myapp If the Root or Issuer certs are about to expire, you can update them and restart the required system services. +> **Note:** When actor placement is served by the Scheduler service (`global.scheduler.placement.enabled=true`), the `dapr-placement-server` StatefulSet is not deployed: skip its restart commands below. + {{% alert title="Avoiding downtime when rotating certificates" color="warning" %}} To avoid downtime when rotating expiring certificates your new certificates must be signed with the same private root key as the previous certificates. This is not currently possible using self-signed certificates generated by Dapr. {{% /alert %}} @@ -309,6 +313,7 @@ Once Sentry has been completely restarted run: ```bash kubectl rollout restart deploy/dapr-operator -n kubectl rollout restart statefulsets/dapr-placement-server -n +kubectl rollout restart statefulsets/dapr-scheduler-server -n ``` Next, you must restart all Dapr-enabled pods. diff --git a/daprdocs/content/en/reference/api/placement_api.md b/daprdocs/content/en/reference/api/placement_api.md index 6b02dd2cbfb..dbbe0b5ec85 100644 --- a/daprdocs/content/en/reference/api/placement_api.md +++ b/daprdocs/content/en/reference/api/placement_api.md @@ -6,6 +6,10 @@ description: "Detailed documentation on the Placement API" weight: 1100 --- +{{% alert title="Deprecation notice" color="warning" %}} +The standalone Placement service is planned for deprecation in Dapr v1.21, when the [Scheduler service]({{% ref scheduler %}}) serves actor placement by default. Until then, scheduler placement is enabled by the opt-in flag `global.scheduler.placement.enabled=true` (`false` by default). This API is not served by the Scheduler. +{{% /alert %}} + Dapr has an HTTP API `/placement/state` for Placement service that exposes placement table information. The API is exposed on the sidecar on the same port as the healthz. This is an unauthenticated endpoint, and is disabled by default. To enable the placement metadata in self-hosted mode you can either set`DAPR_PLACEMENT_METADATA_ENABLED` environment variable or `metadata-enabled` command line args on the Placement service to `true` to. See [how to run the Placement service in self-hosted mode]({{% ref "self-hosted-no-docker.md#enable-actors" %}}). @@ -16,6 +20,8 @@ When running placement in [multi-tenant mode]({{% ref namespaced-actors.md %}}), If you are using Helm for deployment of the Placement service on Kubernetes then to enable the placement metadata, set `dapr_placement.metadataEnabled` to `true`. +> **Note:** This API is only available from the standalone Placement service. When actor placement is [served by the Scheduler service]({{% ref "placement#serving-placement-from-the-scheduler-service" %}}), the `/placement/state` endpoint is not available. + ## Usecase The placement table API can be used for retrieving the current placement table, which contains all the actors registered. This can be helpful for debugging and allows tools to extract and present information about actors.