Flaky CI: intermittent tests-without-auth failures (boost timeDecay ~1-in-6, plus two confirmed serialization bugs) #470

Summary

tests-without-auth fails intermittently across all Weaviate versions and both Node versions, on unrelated PRs. Re-running the identical commit turns it green, so the code under test is not the cause.

Most recent evidence: run 33513846023 failed, then passed on all 8 jobs after a re-run with no code change.

At least three distinct flakes are involved. One is now understood in detail; the other two are not.

1. query > boost > 'timeDecay' — understood, ~1-in-6 failure rate

test/collections/query/integration.test.ts

AssertionError: expected [ …(3) ] to not deeply equal [ …(3) ]

The assertion is:

const baseline = ids(await collection.query.nearVector([1, 2, 3]));
const boosted  = ids(await collection.query.nearVector([1, 2, 3], { boost }));
expect(boosted).not.toEqual(baseline);

Measured behaviour

Reproduced locally against semitechnologies/weaviate:1.38.2, rebuilding the collection each trial:

• 5/30 trials (16.7%) failed on unmodified main.
• Across rebuilds the boosted ordering spans all six permutations of the three objects, near-uniformly. 1/6 = 16.7%, matching the observed rate.
• Within a single collection, both baseline and boosted are perfectly stable (0/30 unstable). This is why re-running the same query never reproduces it — only rebuilding the collection does.

Ruled out

Assessment

The test's premise is the problem: it assumes a boost must reorder three near-tied results, which the server does not guarantee. Retuning the input values will not hold — 467e5a7 already attempted exactly that in July, and this is the same test failing again.

Suggested direction: assert something deterministic (an expected ordering over well-separated data, or scores/membership) rather than "the order changed". The other three boost cases in the same it.each (filter, numericDecay, numericProperty) have never flaked.

2. Two confirmed bugs found while investigating

These are independent of the flake and are separately actionable.

2a. decay is silently dropped from Boost.timeDecay / Boost.numericDecay

The public type uses decay; the generated proto field is decayValue:

• src/collections/types/query.ts:211,223 — decay?: number
• src/proto/v1/search_get.ts:351-367 — decayValue?: number
• src/collections/serialize/index.ts:1645,1659 — fromPartial({ ...timeDecay, curve })

fromPartial discards the unknown decay key, so the value never reaches the wire. Observed payload:

{"timeDecay":{"property":"createdAt","origin":"2024-01-01T00:00:00Z","scale":"35d","curve":3}}

With a two-line mapping (decayValue: decay) it becomes:

{"timeDecay":{"property":"createdAt","origin":"...","scale":"35d","curve":3,"decayValue":0.2}}

Any user passing decay today has it ignored, with no error. Note: applying this fix did not make the ordering deterministic, so it is not the cause of the flake above.

2b. Debug logging shipped in the client

src/collections/serialize/index.ts:1679

console.error(JSON.stringify(out));

Inside Serialize.boostGRPC — writes to stderr on every boosted query.

3. Not yet understood

3a. data > dryRun deleteMany — TypeError: Stringified UUID is invalid

❯ stringify node_modules/uuid/dist-node/stringify.js:31:15
❯ src/collections/deserialize/index.ts:536:19   id: stringify(obj.uuid),

The server returns a BatchDeleteObject with no uuid; the proto defaults it to new Uint8Array(0) (src/proto/v1/batch_delete.ts:303,332), unsafeStringify then indexes byteToHex[undefined] and produces an invalid string. Not reproduced locally; why the server omits the field is unresolved. Regardless, the call site is unguarded and turns this into an opaque TypeError.

3b. collections/journey.test.ts — fetch failed in beforeAll

WeaviateStartUpError: Weaviate failed to startup with message: fetch failed   (src/connection/helpers.ts:123)

This suite connects to a fixed external WCD cluster rather than the CI container, so all seven matrix jobs hit the same shared cluster concurrently and a transient network failure kills the suite (observed with zero failed assertions: 1204 passed | 83 skipped).

Separately, afterAll at test/collections/journey.test.ts:16 is unguarded:

afterAll(() => client.collections.delete(collectionName));

When beforeAll fails, client is undefined and this throws a second, misleading TypeError: Cannot read properties of undefined (reading 'collections'), obscuring the real cause. A client?. guard would fix that much.

Worth considering whether a shared, externally-mutable cluster belongs in the blocking CI path at all — an upgrade to it recently reddened every open PR simultaneously (fixed in #469).