Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/invalid-input-error-cause.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@slonik/pg-driver": minor
---

fix: `InvalidInputError` raised for SQLSTATE 22P02 (invalid text representation) includes the original error as `cause`, like the other errors that the driver maps.
5 changes: 5 additions & 0 deletions .changeset/one-first-data-integrity-error.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"slonik": minor
---

fix: `oneFirst` throws `DataIntegrityError` (instead of `UnexpectedStateError`) when the query returns multiple columns, like `maybeOneFirst`, `anyFirst` and `manyFirst`.
5 changes: 5 additions & 0 deletions .changeset/remove-query-method-values-parameter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"slonik": minor
---

remove the `values` parameter from the types of the query methods (e.g. `pool.query(query, values)`). The parameter was accepted, but ignored; values are bound using the `sql` tag.
5 changes: 5 additions & 0 deletions .changeset/reuse-connection-after-result-errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"slonik": minor
---

fix: a query that fails with `NotFoundError`, `DataIntegrityError` or `SchemaValidationError` (e.g. `pool.one()` that finds no rows) no longer destroys its connection; the connection is returned to the pool. This applies to queries and transactions on the pool. Connections used by `pool.connect()` are still destroyed when the routine fails.
6 changes: 6 additions & 0 deletions .changeset/stream-errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
"@slonik/pg-driver": minor
"slonik": minor
---

fix: `stream()` rejects with the error of a `transformRow` or `transformRowAsync` interceptor (e.g. a `SchemaValidationError` of a result parser). Previously, the error was an unhandled rejection, `stream()` never settled, and the connection was never released. `stream()` also waits for the promise returned by the stream handler (e.g. an `async` function that uses `for await`). When the consumer of a stream fails, the driver closes the cursor, so that the connection can be used again (e.g. to roll back the transaction).
5 changes: 5 additions & 0 deletions .changeset/transaction-options.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"slonik": minor
---

add the `retryLimit` option to `transaction()`, e.g. `pool.transaction(handler, { retryLimit: 3 })`. Passing the retry limit as the second argument (`pool.transaction(handler, 3)`) is deprecated. Also fixes `pool.transaction()` dropping the retry limit when a `beforePoolConnection` interceptor returns another pool: the options are now passed to that pool.
94 changes: 57 additions & 37 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -389,9 +389,7 @@ To begin with, Slonik does not allow running plain-text queries.
connection.query("SELECT 1");
```

The above invocation would produce an error:

> TypeError: Query must be constructed using `sql` tagged template literal.
Query methods only accept queries constructed using the `sql` tagged template literal, so TypeScript rejects the above invocation.

This means that the only way to run a query is by constructing it using [`sql` tagged template literal](https://github.com/gajus/slonik#value-placeholders-tagged-template-literals), e.g.

Expand Down Expand Up @@ -465,7 +463,8 @@ Supported parameters:
| `sslmode` | [`sslmode`](https://www.postgresql.org/docs/current/libpq-ssl.html#LIBPQ-SSL-PROTECTION) (supported values: `disable`, `no-verify`, `require`) | `disable` |
| `sslrootcert` | The location of the [root certificate](<(https://www.postgresql.org/docs/current/libpq-ssl.html#LIBQ-SSL-CERTIFICATES)>) file. | |

Note that unless listed above, other [libpq parameters](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS) are not supported.
> [!NOTE]
> Unless listed above, other [libpq parameters](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS) are not supported.

Examples of valid DSNs:

Expand Down Expand Up @@ -521,7 +520,8 @@ const pool = await createPool("postgres://", {
});
```

> **Note:** If you are new to Slonik, then you should read [Integrating Slonik with Express.js](https://dev.to/gajus/integrating-slonik-with-expressjs-33kn).
> [!NOTE]
> If you are new to Slonik, then you should read [Integrating Slonik with Express.js](https://dev.to/gajus/integrating-slonik-with-expressjs-33kn).

Instance of Slonik connection pool can be then used to create a new connection, e.g.

Expand Down Expand Up @@ -568,7 +568,8 @@ const main = async () => {
main();
```

Note: `pool.end()` does not terminate active connections/ transactions.
> [!NOTE]
> `pool.end()` does not terminate active connections/ transactions.

### Describing the current state of the connection pool

Expand Down Expand Up @@ -635,7 +636,8 @@ const main = async () => {
main();
```

Note: `pool.end()` does not terminate active connections/ transactions.
> [!NOTE]
> `pool.end()` does not terminate active connections/ transactions.

### API

Expand Down Expand Up @@ -808,7 +810,7 @@ Read: [Protecting against unsafe connection handling](#protecting-against-unsafe

The `DatabasePool` extends `DatabasePoolEventEmitter` and exposes the following events:

- `error`: `(error: SlonikError) => void` – emitted for all errors that happen within the pool.
- `error`: `(error: SlonikError) => void` – emitted when a query (including `stream`) fails with a `SlonikError`. Database errors that Slonik does not map to its own [error classes](#error-handling) are not emitted.

```ts
import { createPool, type DatabasePoolEventEmitter } from "slonik";
Expand Down Expand Up @@ -842,7 +844,8 @@ The primary difference between Slonik and `pg-promise`:
- Slonik does not allow to execute raw text queries. Slonik queries can only be constructed using [`sql` tagged template literals](#tagged-template-literals). This design [protects against unsafe value interpolation](#protecting-against-unsafe-value-interpolation).
- Slonik implements [interceptor API](#interceptors) (middleware). Middlewares allow to modify connection handling, override queries and modify the query results. Example Slonik interceptors include [field name transformation](https://github.com/gajus/slonik-interceptor-field-name-transformation), [query normalization](https://github.com/gajus/slonik-interceptor-query-normalisation) and [query benchmarking](https://github.com/gajus/slonik-interceptor-query-benchmarking).

Note: Author of `pg-promise` has [objected to the above claims](https://github.com/gajus/slonik/issues/122). I have removed a difference that was clearly wrong. I maintain that the above two differences remain valid differences: even though `pg-promise` might have substitute functionality for variable interpolation and interceptors, it implements them in a way that does not provide the same benefits that Slonik provides, namely: guaranteed security and support for extending library functionality using multiple plugins.
> [!NOTE]
> Author of `pg-promise` has [objected to the above claims](https://github.com/gajus/slonik/issues/122). I have removed a difference that was clearly wrong. I maintain that the above two differences remain valid differences: even though `pg-promise` might have substitute functionality for variable interpolation and interceptors, it implements them in a way that does not provide the same benefits that Slonik provides, namely: guaranteed security and support for extending library functionality using multiple plugins.

Other differences are primarily in how the equivalent features are implemented, e.g.

Expand Down Expand Up @@ -885,7 +888,8 @@ Example:
}
```

Note: Unlike [`pg-types`](https://github.com/brianc/node-pg-types) that uses OIDs to identify types, Slonik identifies types using their names.
> [!NOTE]
> Unlike [`pg-types`](https://github.com/brianc/node-pg-types) that uses OIDs to identify types, Slonik identifies types using their names.

Use this query to find type names:

Expand Down Expand Up @@ -1069,18 +1073,25 @@ pool.connect();

Executed after query has been executed and before rows were transformed using `transformRow`.

Note: When query is executed using `stream`, then `afterQuery` is called with empty result set.
> [!NOTE]
> Not executed for queries executed using `stream`.

#### <code>beforeQueryExecution</code>

This function can optionally return a direct result of the query which will cause the actual query never to be executed.

> [!NOTE]
> Not executed for queries executed using `stream`.

#### <code>beforeQueryResult</code>

Executed just before the result is returned to the client.

Use this method to capture the result that will be returned to the client.

> [!NOTE]
> Not executed for queries executed using `stream`.

#### <code>beforeTransformQuery</code>

Executed before `transformQuery`. Use this interceptor to capture the original query (e.g. for logging purposes).
Expand Down Expand Up @@ -1131,8 +1142,9 @@ Transforms row.

Use `transformRow` to modify the query result.

Note: `transformRow` is synchronous. Use [`transformRowAsync`](#transformrowasync) if the
transformation is asynchronous.
> [!NOTE]
> `transformRow` is synchronous. Use [`transformRowAsync`](#transformrowasync) if the
> transformation is asynchronous.

#### <code>transformRowAsync</code>

Expand Down Expand Up @@ -1187,13 +1199,12 @@ On the other hand, `beforePoolConnection` makes it easy to route based on conven

The first option is self-explanatory to implement, but this recipe demonstrates my convention for using `beforePoolConnection` to route queries.

Note: How you determine which queries are safe to route to a read-only instance is outside of scope for this documentation.

Note: `beforePoolConnection` only works for connections initiated by a query, i.e. `pool#query` and not `pool#connect()`.

Note: `pool#transaction` triggers `beforePoolConnection` but has no `query`.

Note: This particular implementation does not handle [`SELECT INTO`](https://www.postgresql.org/docs/current/sql-selectinto.html).
> [!NOTE]
>
> - How you determine which queries are safe to route to a read-only instance is outside of scope for this documentation.
> - `beforePoolConnection` only works for connections initiated by a query, i.e. `pool#query` and not `pool#connect()`.
> - `pool#transaction` triggers `beforePoolConnection` but has no `query`.
> - This particular implementation does not handle [`SELECT INTO`](https://www.postgresql.org/docs/current/sql-selectinto.html).

```ts
const readOnlyPool = await createPool("postgres://read-only");
Expand Down Expand Up @@ -1408,7 +1419,8 @@ Conversely you can allow unknown keys to be passed through by using [`.passthrou
z.object({ foo: z.string() }).passthrough();
```

_Note_: Using `.passthrough()` is not recommended as it reduces type safety and may lead to unexpected behaviour.
> [!NOTE]
> Using `.passthrough()` is not recommended as it reduces type safety and may lead to unexpected behaviour.

### Handling schema validation errors

Expand Down Expand Up @@ -1573,25 +1585,18 @@ query with 'baz' value binding.

### Manually constructing the query

Manually constructing queries is not allowed.

There is an internal mechanism that checks to see if query was created using `sql` tagged template literal, i.e.
Query methods only accept queries constructed using the `sql` tagged template literal, so TypeScript rejects a manually constructed query, e.g.

```ts
const query = {
sql: "SELECT 1 AS id FROM foo WHERE bar = $1",
type: "SQL",
values: ["baz"],
};

connection.query(query);
```

Will result in an error:

> Query must be constructed using `sql` tagged template literal.

This is a security measure designed to prevent unsafe query execution.
The `sql` tag functions must be called as tagged template literals. Calling one as a regular function, e.g. `sql.unsafe("SELECT 1")`, throws `InvalidInputError` ("Function must be called as a template literal."). This prevents passing a plain string (e.g. user input) as the query text.

Furthermore, a query object constructed using `sql` tagged template literal is [frozen](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/freeze) to prevent further manipulation.

Expand Down Expand Up @@ -2648,8 +2653,7 @@ const row = await connection.one(sql.typeAlias("foo")`SELECT foo`);
// row.foo is the result of the `foo` column value of the first row.
```

> Note:
>
> [!NOTE]
> I've been asked "What makes this different from [knex.js](http://knexjs.org/) `knex('foo').limit(1)`?".
> `knex('foo').limit(1)` simply generates "SELECT \* FROM foo LIMIT 1" query.
> `knex` is a query builder; it does not assert the value of the result.
Expand Down Expand Up @@ -2723,8 +2727,7 @@ const memberCounts = await connection.record(sql.type(

Returns an empty object if query returns no rows.

> Note:
>
> [!NOTE]
> `record` is not just a convenience method – it asserts the integrity of the result.
> Building the record manually (e.g. using `Object.fromEntries`) silently drops rows
> that share the same key, e.g. as a result of an incorrect `GROUP BY` clause.
Expand Down Expand Up @@ -2776,6 +2779,8 @@ await connection.stream(sql.typeAlias("foo")`SELECT foo`, async (stream) => {
});
```

If the function passed to `stream` returns a promise (e.g. it is an `async` function), `stream` waits for the promise. If the stream fails (e.g. because an interceptor fails to transform a row), `stream` rejects with the error.

### <code>transaction</code>

`transaction` method is used wrap execution of queries in `START TRANSACTION` and `COMMIT` or `ROLLBACK`. `COMMIT` is called if the transaction handler returns a promise that resolves; `ROLLBACK` is called otherwise.
Expand Down Expand Up @@ -2853,7 +2858,7 @@ await connection.transaction(async (t1) => {
await t1.transaction(async (t2) => {
await t2.query(sql.typeAlias("void")`INSERT INTO qux (quux) VALUES ('corge')`);

await t1.transaction(async (t3) => {
await t2.transaction(async (t3) => {
await t3.query(sql.typeAlias("void")`INSERT INTO uier (grault) VALUES ('garply')`);

return Promise.reject(new Error("foo"));
Expand Down Expand Up @@ -2882,7 +2887,21 @@ Transactions that are failing with [Transaction Rollback](https://www.postgresql

A failing transaction will be rolled back and the callback function passed to the transaction method call will be executed again. Nested transactions are also retried until the retry limit is reached. If the nested transaction keeps failing with a [Transaction Rollback](https://www.postgresql.org/docs/current/errcodes-appendix.html) error, then the parent transaction will be retried until the retry limit is reached.

How many times a transaction is retried is controlled using `transactionRetryLimit` configuration (default: 5) and the `transactionRetryLimit` parameter of the `transaction` method (default: undefined). If a `transactionRetryLimit` is given to the method call then it is used otherwise the `transactionRetryLimit` configuration is used.
How many times a transaction is retried is controlled using `transactionRetryLimit` configuration (default: 5). Use the `retryLimit` option of the `transaction` method to override it for a transaction, e.g.

```ts
await connection.transaction(
async (transactionConnection) => {
await transactionConnection.query(sql.unsafe`INSERT INTO foo (bar) VALUES ('baz')`);
},
{
retryLimit: 3,
},
);
```

> [!NOTE]
> Passing the retry limit as the second argument (`connection.transaction(handler, 3)`) is deprecated.

#### Transaction events

Expand Down Expand Up @@ -3249,7 +3268,8 @@ Query-level logging can be added using [`slonik-interceptor-query-logging`](http

### Capture stack trace

Note: Requires [`slonik-interceptor-query-logging`](https://github.com/gajus/slonik-interceptor-query-logging).
> [!NOTE]
> Requires [`slonik-interceptor-query-logging`](https://github.com/gajus/slonik-interceptor-query-logging).

Enabling `captureStackTrace` configuration will create a stack trace before invoking the query and include the stack trace in the logs, e.g.

Expand Down
11 changes: 10 additions & 1 deletion packages/pg-driver/src/factories/createPgDriverFactory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ const wrapError = (error: Error, query: null | Query) => {
}

if (error.code === "22P02") {
return new InvalidInputError(error.message);
return new InvalidInputError(error.message, { cause: error });
}

if (error.code === "25P03") {
Expand Down Expand Up @@ -447,6 +447,15 @@ export const createPgDriverFactory = (): DriverFactory => {
});

stream.pipe(transform);

// `pipe` does not destroy the query stream when `transform` is destroyed (e.g. because
// the consumer failed to process a row). That would leave the cursor open and block
// every subsequent query of the connection; destroying the query stream closes it.
transform.once("close", () => {
if (!stream.destroyed) {
stream.destroy();
}
});
};

const typeParsers = streamOptions?.typeParsers;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { createPoolWithSpy } from "../../../helpers.test/createPoolWithSpy.js";
import { createTestRunner } from "../../../helpers.test/createTestRunner.js";
import type { DatabasePool } from "../../../types.js";
import { createPgDriverFactory } from "@slonik/pg-driver";
import { createSqlTag } from "@slonik/sql-tag";
import * as sinon from "sinon";
Expand Down Expand Up @@ -29,6 +30,35 @@ test("`beforePoolConnection` is called before `connect`", async (t) => {
t.true(beforePoolConnection.calledBefore(spy.acquire));
});

test("`beforePoolConnection` that returns a pool runs the transaction using that pool, with the same options", async (t) => {
const transaction = sinon.stub().resolves("foo");

const { pool } = await createPoolWithSpy(t.context.dsn, {
driverFactory,
interceptors: [
{
beforePoolConnection: () => {
return {
transaction,
} as unknown as DatabasePool;
},
name: "foo",
},
],
});

const handler = async () => {
return "bar";
};

t.is(await pool.transaction(handler, { retryLimit: 3 }), "foo");
t.deepEqual(transaction.firstCall.args, [handler, { retryLimit: 3 }]);

// Deprecated: the retry limit as the second argument.
t.is(await pool.transaction(handler, 2), "foo");
t.deepEqual(transaction.secondCall.args, [handler, 2]);
});

test("pool connection lifecycle reuses the same context object", async (t) => {
const connectionContexts = new WeakMap<object, string>();
const beforePoolConnection = sinon.stub().callsFake((connectionContext) => {
Expand Down
7 changes: 4 additions & 3 deletions packages/slonik/src/binders/bindPool.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { transaction } from "../connectionMethods/transaction.js";
import { createConnection } from "../factories/createConnection.js";
import type { ConnectionPool } from "../factories/createConnectionPool.js";
import { getTransactionRetryLimit } from "../routines/getTransactionRetryLimit.js";
import type {
ClientConfiguration,
DatabasePool,
Expand Down Expand Up @@ -216,7 +217,7 @@ export const bindPool = (
streamQuery,
);
},
transaction: (transactionHandler, transactionRetryLimit) => {
transaction: (transactionHandler, options) => {
return createConnection(
parentLog,
pool,
Expand All @@ -228,11 +229,11 @@ export const bindPool = (
connection,
clientConfiguration,
transactionHandler,
transactionRetryLimit,
getTransactionRetryLimit(options),
);
},
(newPool) => {
return newPool.transaction(transactionHandler);
return newPool.transaction(transactionHandler, options);
},
);
},
Expand Down
Loading
Loading