Skip to content

fixed test flakiness in replication tests #2943

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zachmu wants to merge 13 commits into
base: main
Choose a base branch
from
Open

fixed test flakiness in replication tests #2943

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a04a905
fixed test flakiness caused by eventual consistency of asynchronously…
zachmu Jul 16, 2026
0ba9028
more generous timeouts for replication test
zachmu Jul 16, 2026
324daf8
new dolt
zachmu Jul 16, 2026
40d51b1
merge main
zachmu Jul 17, 2026
6601ef6
new dolt]
zachmu Jul 17, 2026
36724f3
more retries
zachmu Jul 17, 2026
4e364ef
new dolt
zachmu Jul 17, 2026
5569846
new dolt
zachmu Jul 17, 2026
5d864de
new dolt
zachmu Jul 17, 2026
3e09f9e
merge main
zachmu Jul 17, 2026
e77b70a
turn macos back on
zachmu Jul 17, 2026
97f2bc1
new dolt
zachmu Jul 17, 2026
dac6ead
new dolt
zachmu Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci-go-sql-server-driver.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ ubuntu-22.04 ]
# os: [ ubuntu-22.04 ]
# TODO: failing on MacOS due to races in syncig the default postgres db with no writes
# os: [ ubuntu-22.04, macos-latest ]
os: [ ubuntu-22.04, macos-latest ]
steps:
- uses: actions/checkout@v6
- name: Set up Go
Expand Down
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ require (
github.com/PuerkitoBio/goquery v1.8.1
github.com/cockroachdb/apd/v3 v3.2.3
github.com/cockroachdb/errors v1.7.5
github.com/dolthub/dolt/go v0.40.5-0.20260717180637-9606cfb035fc
github.com/dolthub/dolt/go v0.40.5-0.20260717212323-c764f22aebc6
github.com/dolthub/eventsapi_schema v0.0.0-20260715220557-d9b4a1c6b4d4
github.com/dolthub/flatbuffers/v23 v23.3.3-dh.2
github.com/dolthub/go-mysql-server v0.20.1-0.20260717163909-4add05e419ac
github.com/dolthub/go-mysql-server v0.20.1-0.20260717195005-e77c2d65f1a5
github.com/dolthub/pg_query_go/v6 v6.0.0-20251215122834-fb20be4254d1
github.com/dolthub/sqllogictest/go v0.0.0-20260624223518-788480b24166
github.com/dolthub/vitess v0.0.0-20260624214226-81d034e0fde8
Expand Down
8 changes: 4 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,8 +246,8 @@ github.com/dolthub/aws-sdk-go-ini-parser v0.0.0-20250305001723-2821c37f6c12 h1:I
github.com/dolthub/aws-sdk-go-ini-parser v0.0.0-20250305001723-2821c37f6c12/go.mod h1:rN7X8BHwkjPcfMQQ2QTAq/xM3leUSGLfb+1Js7Y6TVo=
github.com/dolthub/dolt-mcp v0.3.4 h1:AyG5cw+fNWXDHXujtQnqUPZrpWtPg6FN6yYtjv1pP44=
github.com/dolthub/dolt-mcp v0.3.4/go.mod h1:bCZ7KHvDYs+M0e+ySgmGiNvLhcwsN7bbf5YCyillLrk=
github.com/dolthub/dolt/go v0.40.5-0.20260717180637-9606cfb035fc h1:2dZ6Xc86zOFf6ubkGDwCHsWfPWao7yxWAXfFdkeA8OM=
github.com/dolthub/dolt/go v0.40.5-0.20260717180637-9606cfb035fc/go.mod h1:CT3SR83WMpcBoEWYc7RSpupK2EiPu1aI7JfeLKPRsf4=
github.com/dolthub/dolt/go v0.40.5-0.20260717212323-c764f22aebc6 h1:SRrDAFGTAed8ZLNQp9KrQSVQZe86SsTA7h0INd0a+G8=
github.com/dolthub/dolt/go v0.40.5-0.20260717212323-c764f22aebc6/go.mod h1:sYSKrZZTCRlhbTXRRpeb7U9cDo2+9MDnd/lvmRYp+zg=
github.com/dolthub/eventsapi_schema v0.0.0-20260715220557-d9b4a1c6b4d4 h1:0mg9QEFdkkBwJMxvz1tCjHYmfG2iIC6aShj1InDq9/M=
github.com/dolthub/eventsapi_schema v0.0.0-20260715220557-d9b4a1c6b4d4/go.mod h1:SSLraQS/jGLYFgff3vuZ+JbVUct6vyEeMzjLBqWqoyM=
github.com/dolthub/flatbuffers/v23 v23.3.3-dh.2 h1:u3PMzfF8RkKd3lB9pZ2bfn0qEG+1Gms9599cr0REMww=
Expand All @@ -256,8 +256,8 @@ github.com/dolthub/fslock v0.0.5 h1:QoXhBhgY1oumHE26qyE7tgmXUT8qjJwxsIzo54O/B/k=
github.com/dolthub/fslock v0.0.5/go.mod h1:sdofYYqE0D79zNZyB4/kmlnsQOVap1C2yByjGKSirEM=
github.com/dolthub/go-icu-regex v0.0.0-20260610153742-72563bc7ca83 h1:FEMjCGEroDnY/BXyAffVZxUpXhP2GpoUJyyq5KaLn8c=
github.com/dolthub/go-icu-regex v0.0.0-20260610153742-72563bc7ca83/go.mod h1:F3cnm+vMRK1HaU6+rNqQrOCyR03HHhR1GWG2gnPOqaE=
github.com/dolthub/go-mysql-server v0.20.1-0.20260717163909-4add05e419ac h1:Bird54IhXCOIg+OlBnaJ/v7zrdAvHiTuXXhMhp+GSxM=
github.com/dolthub/go-mysql-server v0.20.1-0.20260717163909-4add05e419ac/go.mod h1:AnD2jKQZCf09sM2JhV/cTEtsoEhPNg6pVWjRCBox4Zw=
github.com/dolthub/go-mysql-server v0.20.1-0.20260717195005-e77c2d65f1a5 h1:4kF8QNxodC25HuY+XpcMyv1aQ9bngSlQxCO7SPK9oo4=
github.com/dolthub/go-mysql-server v0.20.1-0.20260717195005-e77c2d65f1a5/go.mod h1:AnD2jKQZCf09sM2JhV/cTEtsoEhPNg6pVWjRCBox4Zw=
github.com/dolthub/gozstd v0.0.0-20240423170813-23a2903bca63 h1:OAsXLAPL4du6tfbBgK0xXHZkOlos63RdKYS3Sgw/dfI=
github.com/dolthub/gozstd v0.0.0-20240423170813-23a2903bca63/go.mod h1:lV7lUeuDhH5thVGDCKXbatwKy2KW80L4rMT46n+Y2/Q=
github.com/dolthub/ishell v0.0.0-20260414231531-5f031e3e9037 h1:oIW9HwuWrhxv+4HZxA+QQSKHLqWFyXZ2FmNjUYwkdiM=
Expand Down
7 changes: 6 additions & 1 deletion integration-tests/go-sql-server-driver/driver/cmd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,12 @@ var DelvePath string
const TestUserName = "Bats Tests"
const TestEmailAddress = "bats@email.fake"

const ConnectAttempts = 50
// ConnectAttempts and RetrySleepDuration are used by the test driver to retry connections to a doltgres server
// that is starting up. Besides server startup, the retries must also absorb cluster replication being
// eventually consistent: connections default to the "postgres" database, and on a freshly-booted standby
// that database only exists once its first push from the primary lands (which on a slow CI host can take
// many seconds of first-contact setup). ~30 seconds, matching statusEventualConsistencyAttempts.
const ConnectAttempts = 600
const RetrySleepDuration = 50 * time.Millisecond

// EnvDoltgresBinPath is the environment variable used to locate the doltgres
Expand Down
30 changes: 28 additions & 2 deletions integration-tests/go-sql-server-driver/testdef.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
"io"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"text/template"
Expand Down Expand Up @@ -59,7 +60,11 @@ type Test struct {
// Set this environment variable to effectively disable timeouts for debugging.
const debugEnvKey = "DOLTGRES_SQL_SERVER_TEST_DEBUG"

var timeout = 20 * time.Second
// timeout bounds each individual query in a test. It must comfortably exceed the
// dolt_cluster_ack_writes_timeout_secs values used in the cluster replication tests (60s): a write on the
// primary legitimately blocks for up to that long waiting for standbys to acknowledge replication, e.g.
// when a standby is slow to complete first contact (jwks setup plus create-on-push) on a loaded CI host.
var timeout = 90 * time.Second

func init() {
_, ok := os.LookupEnv(debugEnvKey)
Expand Down Expand Up @@ -629,12 +634,33 @@ func RetryTestRun(t *testing.T, attempts int, test func(TestingT)) {
rtt.try(attempts, test)
}

// statusEventualConsistencyAttempts bounds how long RunQuery waits for a cluster
// replication status query to converge to its expected result.
// Retries are quite forgiving in these tests because the default `postgres` database in most of these tests is never
// written to, which means it get replicated async by a background thread, rather than by a user write.
// At driver.RetrySleepDuration (50ms) per attempt this is roughly 30 seconds, which also makes status
// queries usable as explicit quiesce barriers before graceful role transitions (which have a fixed
// 10-second internal budget for standbys to catch up).
const statusEventualConsistencyAttempts = 600

func RunQuery(t *testing.T, conn *sql.Conn, q driver.Query, ports *DynamicResources) {
RetryTestRun(t, q.RetryAttempts, func(t TestingT) {
attempts := q.RetryAttempts
// Replication status is eventually consistent, so status queries always get at
// least the generous eventual-consistency budget, even if the test set a smaller
// (or no) retry_attempts.
if observesReplicationStatus(q) && attempts < statusEventualConsistencyAttempts {
attempts = statusEventualConsistencyAttempts
}
RetryTestRun(t, attempts, func(t TestingT) {
RunQueryAttempt(t, conn, q, ports)
})
}

// observesReplicationStatus reports whether the query reads the dolt_cluster_status system table
func observesReplicationStatus(q driver.Query) bool {
return q.Query != "" && strings.Contains(strings.ToLower(q.Query), "dolt_cluster_status")
}

func RunQueryAttempt(t TestingT, conn *sql.Conn, q driver.Query, ports *DynamicResources) {
args := make([]any, len(q.Args))
for i := range q.Args {
Expand Down
Loading
Loading