chore: add Redshift support by hanslemm · Pull Request #453 · brooklyn-data/dbt_artifacts

hanslemm · 2024-10-21T11:30:23Z

Overview

This is an update of @brendan-cook-87 PR for Redshift support.

Just changed the data type to SUPER for ARRAY and JSON.

Also merged the latest updates of the package.

I've tested it using Redshift and works fine.

Anyone wanting to test this can just add the package to your dbt project while this is not merged via:

# packages.yml
packages:
  - git: https://github.com/hanslemm/dbt_artifacts.git
    revision: REDSHIFT_SUPPORT

Update type - breaking / non-breaking

What does this solve?

Outstanding questions

What databases have you tested with?

jomach · 2025-10-29T07:09:10Z

I'm still getting errors:

07:01:49  Failure in model model_executions (models/sources/model_executions.sql)
07:01:49    Database Error in model model_executions (models/sources/model_executions.sql)
  column "adapter_response" is of type character varying but expression is of type super
  compiled code at target/run/dbt_artifacts/models/sources/model_executions.sql


      insert into "personal_jorge_machado_extern"."public"."tests" ("run_started_at", "command_invocation_id", "node_id", "name", "depends_on_nodes", "package_name", "test_path", "tags", "all_results")
    (
        select "run_started_at", "command_invocation_id", "node_id", "name", "depends_on_nodes", "package_name", "test_path", "tags", "all_results"
        from "tests__dbt_tmp065924265201"
    )

hanslemm · 2025-10-29T08:18:31Z

It seems the error is related to a test run. Over the last year using my package I realized that at least one test out of the standard dbt column tests doesn't work with it (I don't remember now which one, I think it is related with column name test).
Can you check which test is the culprit via isolating each one and running the model? I can try to implement a fix afterwards.

jomach · 2025-10-31T07:31:22Z

So I deleted all tables and runned again. Now it does not fail but the tables are empty :D
I also tried another run with dbt build. Also empty tables..

 dbt run --select dbt_artifacts --no-version-check
07:29:27  Running with dbt=1.10.10
07:29:28  Registered adapter: redshift=1.9.5
07:29:28  Found 110 models, 1 snapshot, 6 seeds, 396 data tests, 14 sources, 1043 macros, 19 unit tests
07:29:28  
07:29:28  Concurrency: 6 threads (target='dev')
07:29:28  
07:29:30  1 of 34 START sql incremental model public.exposures ........................... [RUN]
07:29:30  2 of 34 START sql incremental model public.invocations ......................... [RUN]
07:29:30  3 of 34 START sql incremental model public.model_executions .................... [RUN]
07:29:30  4 of 34 START sql incremental model public.models .............................. [RUN]
07:29:30  5 of 34 START sql incremental model public.seed_executions ..................... [RUN]
07:29:30  6 of 34 START sql incremental model public.seeds ............................... [RUN]
07:29:37  6 of 34 OK created sql incremental model public.seeds .......................... [SUCCESS in 7.22s]
07:29:37  7 of 34 START sql incremental model public.snapshot_executions ................. [RUN]
07:29:37  1 of 34 OK created sql incremental model public.exposures ...................... [SUCCESS in 7.47s]
07:29:37  8 of 34 START sql incremental model public.snapshots ........................... [RUN]
07:29:38  4 of 34 OK created sql incremental model public.models ......................... [SUCCESS in 7.67s]
07:29:38  9 of 34 START sql incremental model public.sources ............................. [RUN]
07:29:38  5 of 34 OK created sql incremental model public.seed_executions ................ [SUCCESS in 7.92s]
07:29:38  3 of 34 OK created sql incremental model public.model_executions ............... [SUCCESS in 7.93s]
07:29:38  2 of 34 OK created sql incremental model public.invocations .................... [SUCCESS in 7.93s]
07:29:38  10 of 34 START sql incremental model public.test_executions .................... [RUN]
07:29:38  11 of 34 START sql incremental model public.tests .............................. [RUN]
07:29:38  12 of 34 START sql view model public.stg_dbt__seeds ............................ [RUN]
07:29:39  12 of 34 OK created sql view model public.stg_dbt__seeds ....................... [SUCCESS in 1.64s]
07:29:39  13 of 34 START sql view model public.stg_dbt__exposures ........................ [RUN]
07:29:41  13 of 34 OK created sql view model public.stg_dbt__exposures ................... [SUCCESS in 1.45s]
07:29:41  14 of 34 START sql view model public.stg_dbt__models ........................... [RUN]
07:29:41  7 of 34 OK created sql incremental model public.snapshot_executions ............ [SUCCESS in 3.99s]
07:29:41  15 of 34 START sql view model public.stg_dbt__seed_executions .................. [RUN]
07:29:43  14 of 34 OK created sql view model public.stg_dbt__models ...................... [SUCCESS in 2.00s]
07:29:43  15 of 34 OK created sql view model public.stg_dbt__seed_executions ............. [SUCCESS in 1.80s]
07:29:43  16 of 34 START sql view model public.stg_dbt__model_executions ................. [RUN]
07:29:43  17 of 34 START sql view model public.stg_dbt__invocations ...................... [RUN]
07:29:45  17 of 34 OK created sql view model public.stg_dbt__invocations ................. [SUCCESS in 2.46s]
07:29:45  18 of 34 START sql view model public.dim_dbt__seeds ............................ [RUN]
07:29:45  16 of 34 OK created sql view model public.stg_dbt__model_executions ............ [SUCCESS in 2.47s]
07:29:45  19 of 34 START sql view model public.dim_dbt__exposures ........................ [RUN]
07:29:46  9 of 34 OK created sql incremental model public.sources ........................ [SUCCESS in 7.95s]
07:29:46  20 of 34 START sql view model public.stg_dbt__snapshot_executions .............. [RUN]
07:29:46  8 of 34 OK created sql incremental model public.snapshots ...................... [SUCCESS in 8.44s]
07:29:46  21 of 34 START sql view model public.dim_dbt__models ........................... [RUN]
07:29:46  11 of 34 OK created sql incremental model public.tests ......................... [SUCCESS in 8.14s]
07:29:46  22 of 34 START sql view model public.fct_dbt__seed_executions .................. [RUN]
07:29:46  10 of 34 OK created sql incremental model public.test_executions ............... [SUCCESS in 8.54s]
07:29:46  23 of 34 START sql view model public.fct_dbt__invocations ...................... [RUN]
07:29:48  19 of 34 OK created sql view model public.dim_dbt__exposures ................... [SUCCESS in 2.35s]
07:29:48  24 of 34 START sql view model public.dim_dbt__current_models ................... [RUN]
07:29:48  21 of 34 OK created sql view model public.dim_dbt__models ...................... [SUCCESS in 2.05s]
07:29:48  18 of 34 OK created sql view model public.dim_dbt__seeds ....................... [SUCCESS in 2.45s]
07:29:48  20 of 34 OK created sql view model public.stg_dbt__snapshot_executions ......... [SUCCESS in 2.33s]
07:29:48  27 of 34 START sql view model public.stg_dbt__snapshots ........................ [RUN]
07:29:48  26 of 34 START sql view model public.stg_dbt__sources .......................... [RUN]
07:29:48  25 of 34 START sql view model public.fct_dbt__model_executions ................. [RUN]
07:29:49  22 of 34 OK created sql view model public.fct_dbt__seed_executions ............. [SUCCESS in 2.67s]
07:29:49  23 of 34 OK created sql view model public.fct_dbt__invocations ................. [SUCCESS in 2.27s]
07:29:49  28 of 34 START sql view model public.stg_dbt__tests ............................ [RUN]
07:29:49  29 of 34 START sql view model public.stg_dbt__test_executions .................. [RUN]
07:29:51  25 of 34 OK created sql view model public.fct_dbt__model_executions ............ [SUCCESS in 2.90s]
07:29:51  27 of 34 OK created sql view model public.stg_dbt__snapshots ................... [SUCCESS in 2.90s]
07:29:51  26 of 34 OK created sql view model public.stg_dbt__sources ..................... [SUCCESS in 2.90s]
07:29:51  30 of 34 START sql view model public.fct_dbt__snapshot_executions .............. [RUN]
07:29:51  24 of 34 OK created sql view model public.dim_dbt__current_models .............. [SUCCESS in 3.00s]
07:29:51  31 of 34 START sql view model public.dim_dbt__snapshots ........................ [RUN]
07:29:51  32 of 34 START sql view model public.dim_dbt__sources .......................... [RUN]
07:29:52  29 of 34 OK created sql view model public.stg_dbt__test_executions ............. [SUCCESS in 3.51s]
07:29:52  33 of 34 START sql view model public.fct_dbt__test_executions .................. [RUN]
07:29:52  28 of 34 OK created sql view model public.stg_dbt__tests ....................... [SUCCESS in 3.51s]
07:29:52  34 of 34 START sql view model public.dim_dbt__tests ............................ [RUN]
07:29:53  31 of 34 OK created sql view model public.dim_dbt__snapshots ................... [SUCCESS in 2.37s]
07:29:53  32 of 34 OK created sql view model public.dim_dbt__sources ..................... [SUCCESS in 2.37s]
07:29:53  30 of 34 OK created sql view model public.fct_dbt__snapshot_executions ......... [SUCCESS in 2.37s]
07:29:54  33 of 34 OK created sql view model public.fct_dbt__test_executions ............. [SUCCESS in 1.80s]
07:29:54  34 of 34 OK created sql view model public.dim_dbt__tests ....................... [SUCCESS in 1.80s]
07:29:54  
07:29:54  Finished running 11 incremental models, 23 view models in 0 hours 0 minutes and 25.99 seconds (25.99s).
07:29:55  
07:29:55  Completed successfully
07:29:55  
07:29:55  Done. PASS=34 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=34

jomach · 2025-10-31T09:05:42Z

ok, After adding the upload hook everything works fine.

jomach · 2025-11-06T09:47:47Z

Is it planned to get this merged ? We wanted to use this in production...

hanslemm · 2025-11-06T16:21:46Z

Over a year and none of the main branch maintainers left a comment @jomach, so I don't think we will have it in the official package sadly.

jomach · 2025-11-06T16:24:12Z

Can we ping someone ? The builds are failing

hanslemm · 2025-11-06T16:31:53Z

maybe @jared-rimmer can take a look?

brendan-cook-87 · 2025-11-06T20:53:47Z

Over a year and none of the main branch maintainers left a comment @jomach, so I don't think we will have it in the official package sadly.

Yeah. They've never been responsive at all lol. I gave up with the same issues of completely unrelated builds failing...

jomach · 2025-11-06T21:20:23Z

I'm switching to elementary

Update type_helpers.sql

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

dbt-core's MissingArgumentsPropertyInGenericTestDeprecation flags the old top-level generic-test-argument style; dbt >=1.12 with require_generic_test_arguments_property: true parses only the nested `arguments:` form. 11 sites across 10 bundled fct_dbt__* models used the old style (dbt_artifacts.is_between's min_value/max_value, and one accepted_values' values). Ran dbt Labs' dbt-autofix (`deprecations --path .`) to nest them under `arguments:`; kept only the models/*.yml changes (pure re-indentation plus the nesting) and reverted the tool's unrelated dbt_project.yml edit (a `False`->`false` casing tweak and a speculative `flags:` block), since this repo's own dbt_project.yml behavior flags are a separate decision for a different change, not implied by fixing these 11 deprecation sites. Verified via `uvx --from 'dbt-core>=1.12,<1.13' --with 'dbt-postgres>=1.11,<1.12' dbt parse --target postgres --project-dir integration_test_project --profiles-dir integration_test_project --no-partial-parse --show-all-deprecations`: zero MissingArgumentsPropertyInGenericTestDeprecation remain (the one remaining ProjectFlagsMovedDeprecation is pre-existing, unrelated, and lives in integration_test_project/profiles.yml, not touched here). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CNVbr2GyVityjqt3LPAnU

One row per refable node describing the relation the most recent successful execution wrote, across models, seeds and snapshots. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CNVbr2GyVityjqt3LPAnU

Prints dim_dbt__current_relations as a versioned JSON contract, documented in the README. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CNVbr2GyVityjqt3LPAnU

resource_types is validated against the known set and target_name is escaped as a SQL string literal before either reaches the query text. database/schema reach the query indirectly via api.Relation.create(), whose quoting does not escape an embedded quote character either, so they get the same treatment: validated against a safe identifier pattern before being used to build the relation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CNVbr2GyVityjqt3LPAnU

Quote-doubling isn't a safe escape for target_name on every adapter this package supports: Snowflake, BigQuery, Spark and Databricks treat backslash as a string-literal escape character, so a value ending in an odd run of backslashes could desynchronise a doubled quote. Stop interpolating target_name into SQL entirely and filter matching rows in Jinja instead. Widen the database/schema identifier pattern to allow the characters real warehouse identifiers use (including the hyphens common in BigQuery project IDs) while still excluding quotes, semicolons, whitespace and backslashes. Treat an empty resource_types list as "export nothing" rather than building an invalid empty IN (). Make the top docstring precise about when the macro raises. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CNVbr2GyVityjqt3LPAnU

…er nulls last Ranking success rows by node_id alone collapsed several targets writing to the same artifacts tables down to a single winner, so a node whose latest success ran on a different target than the one export_state's target_name argument requests was dropped instead of falling back to its own latest success on that target. Join stg_dbt__invocations in before ranking and partition by (node_id, target_name), making that the view's grain: one row per node per target. Add current_relation_id, a surrogate key of that grain (following the lineage_edge_id precedent) so it stays testable without a dbt_utils dependency, and update the schema tests and the raw-SQL uniqueness test to the new grain. Also order nulls last in the same ranking: NULLS FIRST is Postgres/Redshift's default for DESC, so a success with a null completion time would otherwise outrank a real one. The case-expression form is portable to every adapter this package supports, including SQL Server, which has no NULLS LAST clause. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CNVbr2GyVityjqt3LPAnU

…mestamps dim_dbt__current_relations is now grained on (node_id, target_name). export_state's row loop filtered by target_name after the view had already collapsed to one row per node_id, so a node whose winning row belonged to a different target than requested was silently dropped rather than falling back to its own latest success on the requested target. With the view now returning up to one row per node per target, the loop only needs to pick the right row: a straight match when target_name is given (now correct, since every matching row is genuinely that node's row for that target), and the most recently completed success across targets when it isn't, preserving the single-target-project default of one row per node. Also fix last_success_at: `| string` produced Postgres's space-separated, offset-less format, contradicting the ISO 8601 promise in the README and the consuming repo's spec. A value with no tzinfo is UTC wall-clock time (every adapter this package writes to records query_completed_at from dbt's own UTC run-results timing), so attach UTC explicitly before calling isoformat(), which gives the `T` separator and offset uniformly across adapters. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CNVbr2GyVityjqt3LPAnU

add redshift support

92a6876

hanslemm had a problem deploying to Approve Integration Tests October 21, 2024 11:30 — with GitHub Actions Failure

Merge branch 'main' into REDSHIFT_SUPPORT

8ba69db

hanslemm had a problem deploying to Approve Integration Tests October 21, 2024 11:33 — with GitHub Actions Failure

chore(profiles): clean up

6ad19c4

hanslemm had a problem deploying to Approve Integration Tests October 21, 2024 11:35 — with GitHub Actions Failure

chore(type_helpers): redshift SUPER type for JSON and ARRAY

2be379f

hanslemm had a problem deploying to Approve Integration Tests October 21, 2024 11:38 — with GitHub Actions Failure

hanslemm changed the title ~~add redshift support~~ chore: add Redshift support Oct 21, 2024

hanslemm marked this pull request as ready for review October 21, 2024 12:13

hanslemm mentioned this pull request Oct 21, 2024

Support Redshift #60

Open

Merge branch 'brooklyn-data:main' into REDSHIFT_SUPPORT

dfe7572

hanslemm had a problem deploying to Approve Integration Tests December 19, 2024 17:19 — with GitHub Actions Failure

Update dbt_project.yml

72f5e4a

hanslemm had a problem deploying to Approve Integration Tests December 19, 2024 17:21 — with GitHub Actions Failure

Merge branch 'main' into REDSHIFT_SUPPORT

42b11cd

hanslemm had a problem deploying to Approve Integration Tests June 12, 2025 13:52 — with GitHub Actions Failure

hanslemm had a problem deploying to Approve Integration Tests September 23, 2025 08:12 — with GitHub Actions Failure

Merge branch 'main' into REDSHIFT_SUPPORT

637586e

hanslemm and others added 10 commits August 12, 2026 08:52

Merge branch 'REDSHIFT_SUPPORT' into REDSHIFT_SUPPORT

c58f50d

Merge pull request #1 from shiv-io/REDSHIFT_SUPPORT

2021662

Update type_helpers.sql

chore: merge upstream 2.11.0 into REDSHIFT_SUPPORT

91ad526

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat: export_state run-operation for deferral state

54bb9cb

Prints dim_dbt__current_relations as a versioned JSON contract, documented in the README. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CNVbr2GyVityjqt3LPAnU

Navigation Menu

Uh oh!

chore: add Redshift support - #453

chore: add Redshift support#453
hanslemm wants to merge 23 commits into
brooklyn-data:mainfrom
hanslemm:REDSHIFT_SUPPORT

hanslemm commented Oct 21, 2024 •

edited

Loading

Uh oh!

jomach commented Oct 29, 2025 •

edited

Loading

Uh oh!

hanslemm commented Oct 29, 2025 •

edited

Loading

Uh oh!

jomach commented Oct 31, 2025 •

edited

Loading

Uh oh!

jomach commented Oct 31, 2025

Uh oh!

jomach commented Nov 6, 2025

Uh oh!

hanslemm commented Nov 6, 2025

Uh oh!

jomach commented Nov 6, 2025

Uh oh!

hanslemm commented Nov 6, 2025

Uh oh!

brendan-cook-87 commented Nov 6, 2025

Uh oh!

jomach commented Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

hanslemm commented Oct 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Update type - breaking / non-breaking

What does this solve?

Outstanding questions

What databases have you tested with?

Uh oh!

jomach commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hanslemm commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jomach commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jomach commented Oct 31, 2025

Uh oh!

jomach commented Nov 6, 2025

Uh oh!

hanslemm commented Nov 6, 2025

Uh oh!

jomach commented Nov 6, 2025

Uh oh!

hanslemm commented Nov 6, 2025

Uh oh!

brendan-cook-87 commented Nov 6, 2025

Uh oh!

jomach commented Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hanslemm commented Oct 21, 2024 •

edited

Loading

jomach commented Oct 29, 2025 •

edited

Loading

hanslemm commented Oct 29, 2025 •

edited

Loading

jomach commented Oct 31, 2025 •

edited

Loading