feat(codecs, databricks_zerobus sink): Unity Catalog VARIANT support

[yorickvanzweeden]

Summary

Hi Vector team,

Databricks released another fix for their Zerobus Ingestion SDK. This PR:

• Adds support for Unity Catalog VARIANT columns in the databricks_zerobus sink.
• The implementation converts JSON values into Databricks’ Parquet VARIANT binary representation and recursively supports VARIANT fields nested in structs, lists, fixed-size lists, large lists, and maps.
• This also upgrades the Zerobus SDK to 2.4.0, which includes the Arrow schema support required for VARIANT columns.

Please let me know if you require any changes, or have any questions.

Vector configuration

The sink can be configured as follows:

 [sinks.databricks]
 type = "databricks_zerobus"
 inputs = ["events"]
 ingestion_endpoint = "https://<workspace-id>.zerobus.<region>.cloud.databricks.com"
 unity_catalog_endpoint = "https://<workspace>.cloud.databricks.com"
 table_name = "<catalog>.<schema>.<table>"
 
 [sinks.databricks.auth]
 strategy = "oauth"
 client_id = "${DATABRICKS_CLIENT_ID}"
 client_secret = "${DATABRICKS_CLIENT_SECRET}"

When using ${...} environment variables, Vector environment interpolation must be explicitly enabled or replaced with an appropriate secret-injection mechanism.

How did you test this PR?

• Ran cargo fmt --all -- --check.
• Ran focused VARIANT codec tests: 3 passed.
• Ran the full Arrow codec test suite: 16 passed.
• Ran Databricks Zerobus sink tests: 39 passed.
• Ran Clippy with -D warnings for the codec and Zerobus sink.
• Built Vector with databricks-zerobus-ingest-sdk 2.4.0.
• Tested top-level VARIANT object encoding.
• Tested nested VARIANT values in structs, lists, and maps.
• Tested invalid VARIANT JSON handling.
• Tested preservation of VARIANT child nullability.
• Regenerated third-party license metadata after the dependency updates.

Additionally, I ran an internal test against our Databricks instance.

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Related: [Rust] Fix variant Arrow schema databricks/zerobus-sdk#525

[thomasqueirozb]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 938623cd33

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Preserve string scalars when encoding VARIANT values

When a VARIANT column receives a normal Vector string value, this branch treats that string as already-serialized JSON instead of as the JSON string scalar it is. In practice inputs such as "hello" are rejected as invalid JSON, while strings like "42" or "true" are silently stored as numeric/boolean VARIANTs rather than strings; this corrupts or drops rows for VARIANT columns containing scalar string data. Since non-string serde_json::Values are serialized below, string values need a path that preserves them as strings unless the user has explicitly provided pre-serialized JSON.

Useful? React with 👍 / 👎.

[yorickvanzweeden]

Declining this one. Strings are intentionally treated as serialized JSON so values such as {"name":"Alice"} become VARIANT objects rather than quoted strings. Native objects, arrays, numbers, and booleans are serialized directly. A VARIANT string scalar can still be provided as valid JSON string text, for example "\"hello\"". Supporting both interpretations would require an explicit input mode; guessing would make one of them silently incorrect.

[chatgpt-codex-connector]

Avoid treating ordinary binary structs as VARIANT

Because this generic Arrow/Parquet encoder now infers VARIANT solely from the child field shape, any user schema with a normal Struct<metadata: LargeBinary, value: LargeBinary> is rewritten as a Parquet VARIANT instead of preserving the two binary fields. This affects non-Databricks users of the same serializer, such as user-provided Arrow schemas for ClickHouse or Parquet output, where an event object containing metadata and value would be serialized as a new VARIANT payload and the original binary fields are lost. Please gate this conversion on an explicit VARIANT marker/configuration rather than the plain struct shape.

Useful? React with 👍 / 👎.

[yorickvanzweeden]

Fixed. The generic Arrow encoder no longer identifies VARIANT from the metadata/value struct shape. VARIANT fields must now be passed explicitly by logical field path. With no paths configured—the default for Parquet, ClickHouse, and other users—the struct is encoded normally. I also added regression coverage for an ordinary struct with the same physical shape.

[flaviofcruz]

Duck typing here might be a bit risky in case a user defines a struct type in a table with exactly the same fields. It probably would make sense to attach an annotation that the Field is variant before checking it but I think it would require changes in the zerobus server and SDK.

[yorickvanzweeden]

Fixed without changing the server or SDK. We retain the logical VARIANT paths from the Unity Catalog schema and pass them separately to Vector’s encoder. The Arrow wire schema remains unchanged. The encoder validates each marked path against the expected physical representation, but never infers VARIANT from that representation alone.

Forgot to push, will do in about 12 hours. Let me know if this solution fits the bill