The code in this repository implements a Temporal CustomDataStore using Yugabyte YCQL.
Temporal users running self-hosted instances can now leverage the power of Yugabyte YCQL as their primary data store. This integration offers a leap in performance, reliability, and ease of operation, surpassing alternatives such as Apache Cassandra. For Yugabyte users, integrating Temporal's durable execution model into their stack is now a seamless process, maximizing their existing investment and expertise.
You may start a local instance of temporal-yugabyte using docker compose:
docker compose -f docker/docker-compose/quick-start.yml upThis command will deploy a Temporal instance with Yugabyte as the primary datastore, and Elasticsearch as the visibility store. You may find the Temporal UI at http://localhost:8080 and the Temporal API at localhost:7233.
All features are code-complete and stable. This project is used in production. All code changes are verified through an extensive unit and integration test suite via continuous integration (CI) to ensure the quality of each change. See the Testing Guide to run that suite locally.
The Manetu Platform relies heavily on various third-party frameworks and services, including Yugabyte and Temporal. The throughput, latency, scalability, and reliability of these components have a direct effect on the overall performance of the Manetu product. Therefore, we make every effort to tailor their use and configuration to achieve optimal results.
Temporal supports using either PostgreSQL or Cassandra as its primary data store. Since Yugabyte supports both PostgreSQL and Cassandra CQL APIs, it may appear straightforward to use Yugabyte as a replacement for the native versions.
However, there were a few issues:
-
While Temporal worked out of the box with Yugabyte's PostgreSQL API, the Temporal CQL driver offers substantial performance benefits over the SQL driver.
-
There are subtle differences in Yugabyte's YCQL APIs, which makes it incompatible with the current CQL driver in Temporal (See Issue #3511 for details)
Our situation presented a choice: run two databases - Yugabyte as our platform's primary datastore and Cassandra as the Temporal datastore - or sponsor an effort to add a YCQL driver to Temporal. As you've read this far, our decision should be clear. This code is a YCQL driver for Temporal, enabling us (and you!) to run a converged stack with a single database layer provided by Yugabyte without sacrificing performance or compatibility.
Our initial approach was to try a minor inline modification to the existing CQL code in the upstream Temporal project similar to conditional changes made to support ScyllaDB. However, our analysis concluded that the required changes to the DDL/DML to adapt and optimize for YCQL were too extensive, making a conditionalized approach impractical. We also determined that the Temporal core supports a first-class ability to register CustomDataStore types, making it conducive to working with an arbitrary persistence layer without requiring any changes to the core code.
Therefore, this project started with a fork of the Temporal Cassandra driver and heavily modified the code to create a YCQL-optimized variant. This project incorporates the upstream Temporal core as an unmodified dependency and registers a new CustomDataStore type, preserving all other upstream features intact.
The result is a drop-in replacement for the Temporal server binary/container, offering all the features of the upstream Temporal service, plus the addition of a new storage option for Yugabyte YCQL, configured as a custom datastore option.
A few of the key differences between the original upstream CQL code and the new Yugabyte driver are as follows:
- Use of the Yugabyte-optimized gocql client.
- Replacement of most of the logged batches with YCQL transactions.
- Splitting the executions table into multiple domain-specific subtables since Yugabyte supports inter-table transactions.
- Optimizing the cluster columns by collapsing most of them into composite partition keys to eliminate extra b-tree lookups that are no longer required to deal with Cassandra's limited transaction model.
- Removal of non-conforming code such as TTLs on secondary indices.
The result is a Temporal server that supports connecting to a Yugabyte YCQL database as its primary data store via a driver registered as a CustomDataStore type. We have found that the resulting service is faster, more reliable, and easier to manage than the Apache Cassandra cluster that it replaces. We are sharing the code with the community in the hope that it will be helpful to others in both the Temporal and Yugabyte communities.
Manetu publishes the compiled artifacts as Docker containers for convenient deployment in containerized environments such as Kubernetes. Generally speaking, the deployment of Yugabyte YCQL enabled Temporal is similar to existing self-hosted Temporal deployment methods, with the following adjustments:
- Substitute the Temporal server instances with a binary/container from this project
- Add the configuration for Yugabyte YCQL as a CustomDataStore
- Apply the Yugabyte-specific schema
The following guide is geared towards users deploying to Kubernetes using the Temporal Helm chart, but the steps outlined should be easily adaptable to other deployment flows.
Substitute the container image used for the various Temporal services (Frontend, History, Matching, etc) with the temporal-yugabyte image from our published releases:
server:
image:
repository: ghcr.io/manetu/temporal-yugabyte:latestThe following assumes that you are using PR #658
server:
config:
persistence:
defaultStore: yugabyte
default:
driver: custom
additionalStores:
yugabyte:
customDatastore:
name: "yugabyte"
options:
hosts: "yugabyte-yb-tservers"
keyspace: "mykeyspace"
replicationFactor: 3
user: cassandra
tls:
enabled: true
enableHostVerification: false
caFile: /etc/yugabyte/ca.crtYou will need to deploy the DDL update to Yugabyte manually. You may leverage the admin-tools image supplied by this project for convenience.
apiVersion: batch/v1
kind: Job
metadata:
name: temporal-schema-init
spec:
template:
metadata:
name: temporal-schema-init
spec:
restartPolicy: OnFailure
securityContext: {}
serviceAccountName: default
containers:
- name: temporal-defaultstore-init
image: ghcr.io/manetu/temporal-yugabyte:latest-admin-tools
imagePullPolicy: IfNotPresent
command:
- /bin/bash
- -c
- |
set -eux
temporal-cassandra-tool create -k $CASSANDRA_KEYSPACE --replication-factor "3"
temporal-cassandra-tool setup-schema -v "0.0"
temporal-cassandra-tool update-schema --schema-dir $TEMPORAL_SCHEMA_PATH/yugabyte/temporal/versioned
env:
- name: CASSANDRA_HOST
value: "yugabyte-yb-tservers"
- name: CASSANDRA_PORT
value: "9042"
- name: CASSANDRA_KEYSPACE
value: "mykeyspace"
- name: CASSANDRA_USER
value: cassandra
- name: CASSANDRA_PASSWORD
valueFrom:
secretKeyRef:
name: temporal-yugabyte-secret
key: password
- name: CASSANDRA_ENABLE_TLS
value: "true"
- name: CASSANDRA_TLS_CA
value: /etc/yugabyte/ca.crt
- name: CASSANDRA_TLS_DISABLE_HOST_VERIFICATION
value: "true"
volumeMounts:
- mountPath: /etc/yugabyte
name: yugabyte-secrets
readOnly: true
resources: {}
See Benchmarking
This repository is self-contained and can be used as is for development, though the process is still slightly cumbersome. Improvements welcome.
- Development Guide - local setup, running the server in a debugger, and profiling
- Testing Guide - running the unit, core persistence, and workflow integration suites, and reproducing CI locally