feat: BigQuery Data Warehouse Connector — Phase 1 (Schema, Types, Validation)#391
Open
Shrotriya-lalit wants to merge 5 commits into
Open
feat: BigQuery Data Warehouse Connector — Phase 1 (Schema, Types, Validation)#391Shrotriya-lalit wants to merge 5 commits into
Shrotriya-lalit wants to merge 5 commits into
Conversation
Phase 1 of the BigQuery warehouse connector. Adds Prisma models for BigQueryConnection, BigQuerySync and BigQuerySyncRun with four supporting enums, two Prisma migrations, typed JSON column via PrismaJson namespace, Zod schemas (zBigQuerySyncConfig + column mapping variants) in @openpanel/validation, and the @google-cloud/bigquery dependency.
- Add zBqColRef validator for column references (supports dot-notation for STRUCT nested fields like user.profile.email) - Add mappingType discriminator to both mapping schemas so the union is discriminated and TypeScript can narrow the type cleanly - Add superRefine cross-validation: append mode events syncs must declare an insertTime column (the TIMESTAMP cursor) - Update plan with verified BigQuery Node.js client type mappings: INT64 needs wrapIntegers:true, TIMESTAMP/.value not .toISOString(), DATETIME has no timezone, BYTES→Buffer, Big for NUMERIC/BIGNUMERIC
Real-world orgs connect multiple data sources to one project (e.g. jm-ebg and jm-ebg-cdp on the same ROAS project). The original @unique on projectId wrongly enforced a single connection per project. - Remove @unique from BigQueryConnection.projectId - Add name String field (user label, e.g. "CDP Source") - Replace with @@unique([projectId, name]) — names unique within project - Change Project.bigQueryConnection? → bigQueryConnections[]
Schema additions: - BigQueryConnection: gcpRegion (GDPR region compliance), lastTestedAt/lastTestStatus (connection health) - BigQuerySync: lastSyncStatus typed as enum (was String?), errorRetryCount+isErrorPaused circuit breaker, partitionFilter for cost-safe full-refresh on partitioned tables - BigQuerySyncRun: rowCount BigInt (INT max ~2.1B insufficient), bytesProcessed for cost tracking Zod additions: - zGcpProjectId: GCP project ID format regex (rejects project numbers and display names) - zBqIdentifier: dataset/table name validator (no hyphens, per BQ naming rules) - zServiceAccountJson: SA JSON structure check (rejects authorized_user creds before encryption) - zBigQueryConnectionCreate: connection creation schema with name/region/SA JSON - zBigQuerySyncConfig: dataset/tableName now use zBqIdentifier, partitionFilter field added
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds BigQuery connector support: new Prisma models and enums, multiple SQL migrations (foundation, multi-connection, hardening, referential integrity), Zod validation schemas/types for BigQuery configs, and dependency/type wiring including the BigQuery SDK. ChangesBigQuery Connector Infrastructure
sequenceDiagram
participant Client
participant API
participant Database
participant BigQuery
Client->>API: create/trigger BigQuery sync (projectId, connection)
API->>Database: read BigQueryConnection / BigQuerySync
API->>BigQuery: authenticate with serviceAccountJson and run query
BigQuery->>API: return rows / bytesProcessed
API->>Database: insert BigQuerySyncRun (status, rowCount, bytesProcessed)
🎯 4 (Complex) | ⏱️ ~45 minutes
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/db/package.json`:
- Line 17: Update the `@google-cloud/bigquery` dependency in
packages/db/package.json from ^7.9.1 to ^8.3.1 (look for the dependency key
"`@google-cloud/bigquery`") and run your test suite and any integration checks
that exercise BigQuery calls to catch breaking API changes; after updating,
re-run npm audit / GitHub advisory checks and address any newly surfaced
advisories or required code adjustments in functions that call BigQuery client
methods (e.g., places creating BigQuery clients or invoking methods like
dataset/table/query) to align with the v8 API.
In
`@packages/db/prisma/migrations/20260608090217_add_bigquery_connector/migration.sql`:
- Around line 50-53: The migration creates bigquery_sync_runs with a projectId
column but no FK, allowing runs to be assigned to the wrong project; add
referential integrity by adding a foreign key on bigquery_sync_runs.projectId
referencing the canonical projects table (projects.id) and also ensure
run-to-sync consistency by adding a composite foreign key (syncId, projectId)
referencing bigquery_syncs(id, projectId) (first add a unique constraint on
bigquery_syncs(id, projectId) if needed); update the CREATE TABLE for
bigquery_sync_runs to include these constraints (names like
fk_bigquery_sync_runs_project and fk_bigquery_sync_runs_sync_project) so every
run is tied to an existing project and the syncId matches that project.
- Around line 69-73: The two separate FKs allow mismatched project vs connection
tenants on bigquery_syncs; drop the existing bigquery_syncs_connectionId_fkey
and replace it with a composite foreign key (e.g.
bigquery_syncs_projectId_connectionId_fkey) on (projectId, connectionId) that
REFERENCES "public"."bigquery_connections"(projectId, id) WITH ON DELETE CASCADE
ON UPDATE CASCADE so the referenced connection must belong to the same project;
keep or leave the existing projectId -> projects FK
(bigquery_syncs_projectId_fkey) as-is.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 90092f45-30c5-4c2c-aaa2-84bd90a711d7
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (8)
packages/db/package.jsonpackages/db/prisma/migrations/20260607120000_add_event_meta_description/migration.sqlpackages/db/prisma/migrations/20260608090217_add_bigquery_connector/migration.sqlpackages/db/prisma/migrations/20260608091000_bigquery_multi_connection/migration.sqlpackages/db/prisma/migrations/20260608120000_bigquery_harden/migration.sqlpackages/db/prisma/schema.prismapackages/db/src/types.tspackages/validation/src/index.ts
Comment on lines
+50
to
+53
| CREATE TABLE "public"."bigquery_sync_runs" ( | ||
| "id" UUID NOT NULL DEFAULT gen_random_uuid(), | ||
| "syncId" UUID NOT NULL, | ||
| "projectId" TEXT NOT NULL, |
Contributor
There was a problem hiding this comment.
Constrain bigquery_sync_runs.projectId to prevent mismatched run attribution.
Line 53 stores projectId, but there is no FK for it, and Line 76 only validates syncId. A run row can carry a wrong project id, breaking run-history correctness and tenant-scoped queries.
Suggested migration direction
+-- Keep sync/project coupling enforceable
+ALTER TABLE "public"."bigquery_syncs"
+ ADD CONSTRAINT "bigquery_syncs_id_projectId_key" UNIQUE ("id", "projectId");
+
+-- Enforce run project consistency with parent sync
+ALTER TABLE "public"."bigquery_sync_runs"
+ ADD CONSTRAINT "bigquery_sync_runs_syncId_projectId_fkey"
+ FOREIGN KEY ("syncId", "projectId")
+ REFERENCES "public"."bigquery_syncs"("id", "projectId")
+ ON DELETE CASCADE ON UPDATE CASCADE;
+
+-- Optional explicit FK for direct project integrity
+ALTER TABLE "public"."bigquery_sync_runs"
+ ADD CONSTRAINT "bigquery_sync_runs_projectId_fkey"
+ FOREIGN KEY ("projectId")
+ REFERENCES "public"."projects"("id")
+ ON DELETE CASCADE ON UPDATE CASCADE;Also applies to: 75-76
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@packages/db/prisma/migrations/20260608090217_add_bigquery_connector/migration.sql`
around lines 50 - 53, The migration creates bigquery_sync_runs with a projectId
column but no FK, allowing runs to be assigned to the wrong project; add
referential integrity by adding a foreign key on bigquery_sync_runs.projectId
referencing the canonical projects table (projects.id) and also ensure
run-to-sync consistency by adding a composite foreign key (syncId, projectId)
referencing bigquery_syncs(id, projectId) (first add a unique constraint on
bigquery_syncs(id, projectId) if needed); update the CREATE TABLE for
bigquery_sync_runs to include these constraints (names like
fk_bigquery_sync_runs_project and fk_bigquery_sync_runs_sync_project) so every
run is tied to an existing project and the syncId matches that project.
Comment on lines
+69
to
+73
| -- AddForeignKey | ||
| ALTER TABLE "public"."bigquery_syncs" ADD CONSTRAINT "bigquery_syncs_connectionId_fkey" FOREIGN KEY ("connectionId") REFERENCES "public"."bigquery_connections"("id") ON DELETE CASCADE ON UPDATE CASCADE; | ||
|
|
||
| -- AddForeignKey | ||
| ALTER TABLE "public"."bigquery_syncs" ADD CONSTRAINT "bigquery_syncs_projectId_fkey" FOREIGN KEY ("projectId") REFERENCES "public"."projects"("id") ON DELETE CASCADE ON UPDATE CASCADE; |
Contributor
There was a problem hiding this comment.
Enforce project/connection tenant consistency on bigquery_syncs.
Line 70 and Line 73 create independent foreign keys, so a sync can reference a connectionId from project B while storing projectId from project A. That breaks tenant isolation and can misroute data.
Suggested migration direction
+-- Ensure referenced columns are jointly unique
+ALTER TABLE "public"."bigquery_connections"
+ ADD CONSTRAINT "bigquery_connections_id_projectId_key" UNIQUE ("id", "projectId");
+
+-- Enforce sync belongs to the same project as its connection
+ALTER TABLE "public"."bigquery_syncs"
+ ADD CONSTRAINT "bigquery_syncs_connectionId_projectId_fkey"
+ FOREIGN KEY ("connectionId", "projectId")
+ REFERENCES "public"."bigquery_connections"("id", "projectId")
+ ON DELETE CASCADE ON UPDATE CASCADE;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@packages/db/prisma/migrations/20260608090217_add_bigquery_connector/migration.sql`
around lines 69 - 73, The two separate FKs allow mismatched project vs
connection tenants on bigquery_syncs; drop the existing
bigquery_syncs_connectionId_fkey and replace it with a composite foreign key
(e.g. bigquery_syncs_projectId_connectionId_fkey) on (projectId, connectionId)
that REFERENCES "public"."bigquery_connections"(projectId, id) WITH ON DELETE
CASCADE ON UPDATE CASCADE so the referenced connection must belong to the same
project; keep or leave the existing projectId -> projects FK
(bigquery_syncs_projectId_fkey) as-is.
Shrotriya-lalit
force-pushed
the
feat/bigquery-connector-phase1
branch
from
154e5e2 to
7920219
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/db/prisma/migrations/20260608091000_bigquery_multi_connection/migration.sql`:
- Around line 6-9: The migration creates bigquery_connections.name with an
empty-string backfill which leaves existing rows with '' and still allows future
empty-string inserts; update the migration to backfill a meaningful non-empty
value for existing rows (e.g., update "bigquery_connections" set "name" =
concat('connection_', id) or another deterministic non-empty token for rows
where name = '') and then add a DB-level constraint to prevent empty names
(e.g., ALTER TABLE "bigquery_connections" ADD CONSTRAINT ... CHECK
(char_length(name) > 0) or name <> '') so future inserts cannot use ''. Ensure
the ALTER that drops the default comes after the backfill and the CHECK
constraint is added as part of the migration so both existing and new rows are
validated.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ce3b42d9-fd05-4483-8001-e2ff1812b45c
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (8)
packages/db/package.jsonpackages/db/prisma/migrations/20260607120000_add_event_meta_description/migration.sqlpackages/db/prisma/migrations/20260608090217_add_bigquery_connector/migration.sqlpackages/db/prisma/migrations/20260608091000_bigquery_multi_connection/migration.sqlpackages/db/prisma/migrations/20260608120000_bigquery_harden/migration.sqlpackages/db/prisma/schema.prismapackages/db/src/types.tspackages/validation/src/index.ts
✅ Files skipped from review due to trivial changes (1)
- packages/db/src/types.ts
🚧 Files skipped from review as they are similar to previous changes (6)
- packages/db/package.json
- packages/db/prisma/migrations/20260607120000_add_event_meta_description/migration.sql
- packages/db/prisma/schema.prisma
- packages/validation/src/index.ts
- packages/db/prisma/migrations/20260608090217_add_bigquery_connector/migration.sql
- packages/db/prisma/migrations/20260608120000_bigquery_harden/migration.sql
Comment on lines
+6
to
+9
| ALTER TABLE "public"."bigquery_connections" ADD COLUMN "name" TEXT NOT NULL DEFAULT ''; | ||
|
|
||
| -- Remove the default now that column exists | ||
| ALTER TABLE "public"."bigquery_connections" ALTER COLUMN "name" DROP DEFAULT; |
Contributor
There was a problem hiding this comment.
Backfill leaves invalid empty names and the DB still permits future empty-string inserts.
Line 6 seeds existing rows with '', and after Line 9 those rows remain empty. That conflicts with the non-empty connection-name contract in validation and can leak invalid records into future flows. Please backfill to a non-empty value and enforce non-empty at the database layer.
Suggested migration adjustment
-ALTER TABLE "public"."bigquery_connections" ADD COLUMN "name" TEXT NOT NULL DEFAULT '';
-
--- Remove the default now that column exists
-ALTER TABLE "public"."bigquery_connections" ALTER COLUMN "name" DROP DEFAULT;
+ALTER TABLE "public"."bigquery_connections" ADD COLUMN "name" TEXT;
+UPDATE "public"."bigquery_connections"
+SET "name" = 'default'
+WHERE "name" IS NULL OR btrim("name") = '';
+ALTER TABLE "public"."bigquery_connections" ALTER COLUMN "name" SET NOT NULL;
+ALTER TABLE "public"."bigquery_connections"
+ADD CONSTRAINT "bigquery_connections_name_nonempty_chk"
+CHECK (btrim("name") <> '');📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ALTER TABLE "public"."bigquery_connections" ADD COLUMN "name" TEXT NOT NULL DEFAULT ''; | |
| -- Remove the default now that column exists | |
| ALTER TABLE "public"."bigquery_connections" ALTER COLUMN "name" DROP DEFAULT; | |
| ALTER TABLE "public"."bigquery_connections" ADD COLUMN "name" TEXT; | |
| UPDATE "public"."bigquery_connections" | |
| SET "name" = 'default' | |
| WHERE "name" IS NULL OR btrim("name") = ''; | |
| ALTER TABLE "public"."bigquery_connections" ALTER COLUMN "name" SET NOT NULL; | |
| ALTER TABLE "public"."bigquery_connections" | |
| ADD CONSTRAINT "bigquery_connections_name_nonempty_chk" | |
| CHECK (btrim("name") <> ''); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@packages/db/prisma/migrations/20260608091000_bigquery_multi_connection/migration.sql`
around lines 6 - 9, The migration creates bigquery_connections.name with an
empty-string backfill which leaves existing rows with '' and still allows future
empty-string inserts; update the migration to backfill a meaningful non-empty
value for existing rows (e.g., update "bigquery_connections" set "name" =
concat('connection_', id) or another deterministic non-empty token for rows
where name = '') and then add a DB-level constraint to prevent empty names
(e.g., ALTER TABLE "bigquery_connections" ADD CONSTRAINT ... CHECK
(char_length(name) > 0) or name <> '') so future inserts cannot use ''. Ensure
the ALTER that drops the default comes after the backfill and the CHECK
constraint is added as part of the migration so both existing and new rows are
validated.
…d dependency upgrade
- Upgrade @google-cloud/bigquery from ^7.9.1 to ^8.3.1 (latest stable)
- Add FK bigquery_sync_runs.projectId -> projects(id) (was missing, allowing orphan runs)
- Add composite FK bigquery_syncs(projectId, connectionId) -> bigquery_connections(projectId, id)
to prevent cross-tenant data: a sync can no longer reference a connection from a different project
- Add UNIQUE INDEX bigquery_connections(projectId, id) to back the composite FK
- Add CHECK(char_length(name) > 0) on bigquery_connections to enforce non-empty names at DB level
- Backfill any dev rows with empty name using concat('connection_', id)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces the foundational schema, migrations, and Zod validators for a BigQuery Data Warehouse Connector — a Mixpanel Warehouse Connector equivalent for OpenPanel. No UI or sync behavior yet; this PR establishes the type-safe, production-grade foundation that Phases 2–6 will build on.
What's included
Prisma schema
BigQueryConnection— stores GCP service account credentials (encrypted) per project. Supports multiple named connections per project (@@unique([projectId, name])), matching how Mixpanel allows e.g.jm-ebgandjm-ebg-cdpunder the same projectBigQuerySync— defines a recurring sync job (source dataset/table, column mapping, schedule, mode)BigQuerySyncRun— audit log for every execution (status, row count, bytes processed, error)BigQuerySyncMappingType,BigQuerySyncMode,BigQuerySyncSchedule,BigQuerySyncRunStatusMigrations
20260608090217_add_bigquery_connector— creates all tables, enums, indexes, FK cascades20260608091000_bigquery_multi_connection— multi-connection support (@@unique([projectId, name]))20260608120000_bigquery_harden— production-grade hardening fields (see below)Zod validators (
packages/validation)zBigQueryConnectionCreate— connection creation with name, SA JSON structure check, GCP regionzServiceAccountJson— validates SA key structure before encryption (rejectsauthorized_usercreds)zGcpProjectId— GCP project ID format regex (rejects project numbers / display names)zBqIdentifier— dataset/table name validator (BigQuery only allows[a-zA-Z0-9_])zBigQuerySyncConfig— sync job definition withdiscriminatedUnioncolumn mappingzBigQueryColumnMappingEvents/zBigQueryColumnMappingProfilesDependency
@google-cloud/bigquery@^7.9.1added topackages/dbProduction-grade hardening applied
gcpRegionon connectionslastTestedAt+lastTestStatuserrorRetryCount+isErrorPausedpartitionFilteron syncsbytesProcessedon runsrowCount BigInt(wasInt)lastSyncStatustyped as enumString?— arbitrary values could be storedVerification
DB tables confirmed via
psql:bigquery_connectionswith@@unique([projectId, name])bigquery_syncswithlastSyncStatusasBigQuerySyncRunStatusenumbigquery_sync_runswithrowCount BIGINTWhat's next
listConnections,connect,testConnection,disconnect), settings UI tab🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes