You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/configuration/pgdog.toml/rewrite.md
+37-10Lines changed: 37 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,29 +17,56 @@ split_inserts = "error"
17
17
18
18
| Field | Description | Default |
19
19
| --- | --- | --- |
20
-
|`enabled`| Master toggle; when `false`, PgDog parses but never applies rewrite plans. |`false`|
21
-
|`shard_key`|Behaviour when an `UPDATE` changes a sharding key.<br>`error` rejects the statement.<br>`rewrite` migrates the row between shards.<br>`ignore` forwards it unchanged. |`"error"`|
22
-
|`split_inserts`|Behaviour when a sharded table receives a multi-row `INSERT`.<br>`error` rejects the statement.<br>`rewrite` fans the rows out to their shards.<br>`ignore` forwards it unchanged. |`"error"`|
20
+
|`enabled`| Master toggle: when `false`, PgDog parses but never applies rewrite plans. |`false`|
21
+
|`shard_key`|Behavior when an `UPDATE` changes a sharding key: `error` rejects the statement,<br>`rewrite` migrates the row between shards,<br>`ignore` forwards it unchanged. |`"error"`|
22
+
|`split_inserts`|Behavior when a sharded table receives a multi-row `INSERT`: `error` rejects the statement, `rewrite` fans the rows out to their shards, `ignore` forwards it unchanged. |`"error"`|
23
23
24
24
!!! note "Two-phase commit"
25
-
PgDog recommends enabling [`general.two_phase_commit`](general.md#two_phase_commit) when either policy is set to `rewrite`. Without it, rewrites are committed shard-by-shard and can leave partial changes if a shard fails.
25
+
PgDog recommends enabling [two-phase commit](../../features/sharding/2pc.md) when either policy is set to `rewrite`. Without it, rewrites are committed shard-by-shard and can leave partial changes if a shard fails.
26
26
27
27
## Runtime overrides
28
28
29
-
The admin database exposes these toggles via `SET`:
29
+
The admin database exposes these toggles via the `SET` command:
30
30
31
31
```postgresql
32
32
SET rewrite_enabled TO true; -- mirrors [rewrite].enabled
33
33
SET rewrite_shard_key_updates TO rewrite; -- error | rewrite | ignore
34
34
SET rewrite_split_inserts TO rewrite; -- error | rewrite | ignore
35
35
```
36
36
37
-
Switches apply to subsequent sessions once the cluster reloads configuration. Session-level overrides allow canary testing before persisting them in `pgdog.toml`.
37
+
The setting changes are applied immediately. These overrides allow canary testing before persisting them in `pgdog.toml`.
38
38
39
39
## Limitations
40
40
41
-
* Shard-key rewrites require the `WHERE` clause to resolve to a single row; otherwise PgDog rolls back and raises `rewrite.shard_key="rewrite" is not yet supported ...`.
42
-
* Split INSERT rewrites must run outside explicit transactions so PgDog can orchestrate per-shard `BEGIN`/`COMMIT` cycles. Inside a transaction PgDog returns `25001` and leaves the client transaction intact.
43
-
* Both features fall back to `error` semantics while `rewrite.enabled = false` or when PgDog cannot determine a target shard.
41
+
### Sharding key updates
44
42
45
-
See [feature docs](../../features/sharding/sharding-functions.md#rewrite-behaviour) for walkthroughs of these flows.
43
+
Sharding key rewrites in an `UPDATE` clause have to resolve to a single row. If the sharding key isn't unique or the `WHERE` clause has an incorrect `OR` condition, for example, PgDog will rollback the transaction and raise an error.
44
+
45
+
For example:
46
+
47
+
```postgresql
48
+
UPDATE users SET id = 5 WHERE admin = true;
49
+
```
50
+
51
+
On a single-shard deployment, this would raise a unique index violation error. On a cross-shard deployment, the PgDog rewrite engine will block cross-shard updates that could potentially affect multiple rows.
52
+
53
+
### Multi-tuple inserts
54
+
55
+
`INSERT` statements with multiple tuples have to be executed outside of an explicit transaction. PgDog needs to start a cross-shard transaction to safely commit the rows to multiple shards, and an existing transaction will interfere with its internal state.
56
+
57
+
For example:
58
+
59
+
```postgresql
60
+
BEGIN;
61
+
INSERT INTO users VALUES ($1, $2), ($3, $4);
62
+
```
63
+
64
+
This scenario will raise an error (code `25001`).
65
+
66
+
### Default behavior
67
+
68
+
Both split inserts and sharding key updates fallback to raising an error if `enabled` is set to `false`.
Copy file name to clipboardExpand all lines: docs/features/sharding/sharding-functions.md
+60-9Lines changed: 60 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,11 +3,11 @@ icon: material/function
3
3
---
4
4
# Sharding functions
5
5
6
-
The sharding function inside PgDog transforms column values in SQL queries to specific shard numbers. They are in turn used for routing queries to one or more databases in the [configuration](../../configuration/pgdog.toml/databases.md).
6
+
The sharding function inside PgDog transforms column values in SQL queries to specific shard numbers, which are in turn used for routing queries to one or more databases in the [configuration](../../configuration/pgdog.toml/databases.md).
7
7
8
8
## How it works
9
9
10
-
PgDog sharding function is based on PostgreSQL declarative partitions. This choice is intentional: it allows data to be sharded both inside PgDog and inside PostgreSQL, with the use of the same partition functions.
10
+
The PgDog sharding function is based on PostgreSQL declarative partitions. This choice is intentional: it allows data to be sharded both inside PgDog and inside PostgreSQL, with the use of the same partition functions.
11
11
12
12
PgDog supports all three PostgreSQL partition functions and uses them for sharding data between nodes:
13
13
@@ -81,14 +81,14 @@ values = [1, 2, 3]
81
81
shard = 0
82
82
```
83
83
84
-
This example will route all queries with `user_id`equals to one, two or three to shard zero. Unlike [hash](#hash) sharding, a value <-> shard mapping is required for _all_ values of the sharding key. If a value is used that doesn't have a mapping, the query will be sent to [all shards](cross-shard.md).
84
+
This example will route all queries with `user_id`equal to one, two, or three to shard zero. Unlike [hash](#hash) sharding, a value <-> shard mapping is required for _all_ values of the sharding key. If a value is used that doesn't have a mapping, the query will be sent to [all shards](cross-shard.md).
85
85
86
86
!!! note "Required configuration"
87
87
The `[[sharded_tables]]` configuration entry is still required for list and range sharding. It specifies the data type of the column, which tells PgDog how to parse its value at runtime.
88
88
89
89
## Range
90
90
91
-
Sharding by range function is similar to [list](#list) sharding function, except instead of specifying the values explicitly, you can specify a bounding range. All values which are included in the range will be sent to the specified shard, for example:
91
+
Sharding by range is similar to [list](#list) sharding, except instead of specifying the values explicitly, you can specify a bounding range. All values that are included in the range will be sent to the specified shard, for example:
92
92
93
93
```toml
94
94
[[sharded_mappings]]
@@ -180,14 +180,65 @@ shard = 0
180
180
181
181
This will send all queries that don't specify a schema or use a schema without a mapping to shard zero.
182
182
183
-
## Rewrite behaviour
183
+
## Rewrite behavior
184
184
185
-
PgDog can transparently move writes between shards when [`rewrite`](../../configuration/pgdog.toml/rewrite.md) is enabled.
185
+
PgDog can transparently move writes between shards when the [`rewrite`](../../configuration/pgdog.toml/rewrite.md) feature is enabled.
186
186
187
-
***Shard-key updates** (`rewrite.shard_key = "rewrite"`) delete the matching row from its current shard and re-insert it on the shard implied by the new key. Exactly one row must match the `WHERE` clause; PgDog aborts rewrites that affect multiple rows or unresolved shards.
188
-
***Split INSERTs** (`rewrite.split_inserts = "rewrite"`) decompose multi-row `INSERT` statements so each shard receives only the rows it owns. PgDog opens per-shard transactions and can escalate to two-phase commit when configured to preserve atomicity across shards.
187
+
### Sharding key updates
189
188
190
-
Both features require `rewrite.enabled = true`, operate only on sharded tables, and fall back to returning errors when PgDog cannot determine a safe rewrite plan. Running them alongside [`general.two_phase_commit`](../../configuration/pgdog.toml/general.md#two_phase_commit) is recommended to guarantee atomic outcomes.
189
+
Sharding key updates handle the situation when a query is changing the value of the sharding key, which could require the row to be moved to a different shard.
190
+
191
+
For example:
192
+
193
+
```postgresql
194
+
UPDATE users SET id = $2 WHERE id = $1;
195
+
```
196
+
197
+
If configured, PgDog can rewrite this query into three statements, executed inside a cross-shard transaction:
198
+
199
+
```postgresql
200
+
SELECT * FROM users WHERE id = $1; /* query 1 */
201
+
DELETE FROM users WHERE id = $1; /* query 2 */
202
+
INSERT INTO users VALUES ($1, $2); /* row fetched in query #1 */
203
+
```
204
+
205
+
#### Limitations
206
+
207
+
The row returned by _query 1_, constructed from the `WHERE` clause of the original `UPDATE` statement, has to match exactly one row. If that's not the case, the operation will be aborted and PgDog will raise an error.
208
+
209
+
### Multi-tuple inserts
210
+
211
+
Multi-tuple `INSERT` statements may write rows that belong on separate shards. To handle this situation, if configured, PgDog can rewrite the statement into multiple, single-tuple statements, and send them to their respective shards in a cross-shard transaction.
212
+
213
+
For example:
214
+
215
+
```postgresql
216
+
INSERT INTO users (id, email) VALUES ($1, $2), ($3, $4);
217
+
```
218
+
219
+
This statement will be rewritten into the following two queries:
220
+
221
+
```postgresql
222
+
INSERT INTO users (id, email) VALUES ($1, $2);
223
+
INSERT INTO users (id, email) VALUES ($3, $4);
224
+
```
225
+
226
+
#### Limitations
227
+
228
+
Since PgDog starts a cross-shard transaction to make this operation atomic, the original `INSERT` statement must not be sent inside an explicit transaction by the client. If that's the case, PgDog will abort the operation and return an error.
229
+
230
+
### Configuration
231
+
232
+
Both features require the `enabled` flag to be set to `true`, for example:
233
+
234
+
```toml
235
+
[rewrite]
236
+
enabled = true
237
+
split_inserts = "rewrite"
238
+
shard_key = "rewrite"
239
+
```
240
+
241
+
If a safe rewrite plan cannot be determined, PgDog will abort the transaction and return an error. To guarantee cross-shard atomicity of the operation, consider enabling [two-phase commit](2pc).
Copy file name to clipboardExpand all lines: docs/features/transaction-mode.md
+27-1Lines changed: 27 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -57,10 +57,36 @@ This is performed efficiently, and server parameters are updated only if they di
57
57
58
58
1. The database has a primary and replica(s)
59
59
2. The database has more than one shard
60
-
3. [`prepared_statements`](../configuration/pgdog.toml/general.md#prepared_statements) setting is set to `"full"`
60
+
3. [`prepared_statements`](../configuration/pgdog.toml/general.md#prepared_statements) is set to `"full"`
61
+
4. [`query_parser_enabled`](../configuration/pgdog.toml/general.md#query_parser_enabled) is set to `true`
61
62
62
63
This is to avoid unnecessary overhead of using `pg_query` (however small), when we don't absolutely have to.
63
64
65
+
## Advisory locks
66
+
67
+
Advisory locks are an implementation of distributed locking in PostgreSQL. They are set on the server connection and released when the client removes the lock or disconnects.
68
+
69
+
For example:
70
+
71
+
```postgresql
72
+
SELECT pg_advisory_lock(1234);
73
+
```
74
+
75
+
In transaction mode, server connections are re-used between clients, so additional care needs to be taken to keep the server connection tied to the client that created the lock.
76
+
77
+
PgDog is able to detect advisory lock usage and will pin the server connection to the client connection until one of the following conditions is met:
78
+
79
+
1. The client releases the lock with `pg_advisory_unlock`
80
+
2. The client disconnects
81
+
82
+
!!! note "Performance"
83
+
If multiple clients use advisory locks and don't release them quickly, the effectiveness of transaction pooling will be reduced because server connections will not be effectively re-used between client transactions.
84
+
85
+
### Limitations
86
+
87
+
PgDog doesn't keep track of multiple advisory locks inside client connections. If a client acquires two different locks, for example, and only releases one, the server connection will still be returned back to the pool with the acquired lock.
88
+
89
+
64
90
### Connection parameters
65
91
66
92
Most Postgres connection drivers support passing parameters in the connection URL. Using the special `options` setting, each parameter is set using the `-c` flag, for example:
0 commit comments