diff --git a/learn/multi_search/implement_sharding.mdx b/learn/multi_search/implement_sharding.mdx index 84532d863..b707e9a15 100644 --- a/learn/multi_search/implement_sharding.mdx +++ b/learn/multi_search/implement_sharding.mdx @@ -11,6 +11,9 @@ Sharding is the process of splitting an index containing many documents into mul This guide walks you through activating the `/network` route, configuring the network object, and performing remote federated searches. + +Sharding is an Enterprise Edition feature. You are free to use it for evaluation purposes. Please [reach out to us](mailto:sales@meilisearch.com) before using it in production. + ## Configuring multiple instances @@ -19,7 +22,7 @@ To minimize issues and limit unexpected behavior, instance, network, and index c ## Prerequisites -- Multiple Meilisearch projects (instances) running Meilisearch >=v1.13 +- Multiple Meilisearch projects (instances) running Meilisearch >=v1.19 ## Activate the `/network` endpoint @@ -48,6 +51,7 @@ Next, you must configure the network object. It consists of the following fields - `remotes`: defines a list with the required information to access each remote instance - `self`: specifies which of the configured `remotes` corresponds to the current instance +- `sharding`: whether to use sharding. ### Setting up the list of remotes @@ -93,32 +97,54 @@ curl \ Meilisearch processes searches on the remote that corresponds to `self` locally instead of making a remote request. +### Enabling sharding + +Finally enable the automatic sharding of documents by Meilisearch on all instances: + +```sh +curl \ + -X PATCH 'MEILISEARCH_URL/network' \ + -H 'Content-Type: application/json' \ + --data-binary '{ + "sharding": true + }' +``` + ### Adding or removing an instance Changing the topology of the network involves moving some documents from an instance to another, depending on your hashing scheme. As Meilisearch does not provide atomicity across multiple instances, you will need to either: -1. accept search downtime while migrating documents -2. accept some documents will not appear in search results during the migration +1. accept search downtime while migrating documents +2. accept some documents will not appear in search results during the migration 3. accept some duplicate documents may appear in search results during the migration #### Reducing downtime If your disk space allows, you can reduce the downtime by applying the following algorithm: -1. Create a new temporary index in each remote instance -2. Compute the new instance for each document -3. Send the documents to the temporary index of their new instance -4. Once Meilisearch has copied all documents to their instance of destination, swap the new index with the previously used index -5. Delete the temporary index after the swap -6. Update network configuration and search queries across all instances +1. Create a new temporary index in each remote instance +2. Compute the new instance for each document +3. Send the documents to the temporary index of their new instance +4. Once Meilisearch has copied all documents to their instance of destination, swap the new index with the previously used index +5. Delete the temporary index after the swap +6. Update network configuration and search queries across all instances + +## Create indexes + +Create the same empty indexes with the same settings on all instances. +Keeping the settings and indexes in sync is important to avoid errors and unexpected behavior, though not strictly required. + +## Add documents + +Pick a single instance to send all your documents to. Documents will be replicated to the other instances. -## Create indexes and add documents +Each instance will index the documents they are responsible for and ignore the others. -Create the same empty indexes with the same settings on all instances. Keeping the settings and indexes in sync is important to avoid errors and unexpected behavior, though not strictly required. +You *may* send send the same document to multiple instances, the task will be replicated to all instances, and only the instance responsible for the document will index it. -Distribute your documents across all instances. Do not send the same document to multiple instances as this may lead to duplicate search results. Similarly, you should ensure all future versions of a document are sent to the same instance. Meilisearch recommends you hash their primary key using [rendezvous hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing). +Similarly, you may send any future versions of any document to the instance you picked, and only the correct instance will process that document. ### Updating index settings diff --git a/reference/api/network.mdx b/reference/api/network.mdx index cacac632b..d28f8ce27 100644 --- a/reference/api/network.mdx +++ b/reference/api/network.mdx @@ -40,14 +40,17 @@ Do not enable the `network` feature if you rely on the value of attributes not p ```json { "self": "ms-00", + "sharding": false, "remotes": { "ms-00": { "url": "http://ms-1235.example.meilisearch.io", - "searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas" + "searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas", + "writeApiKey": "O2OaIHgwGuHNx9duH6kSe1YJ55Bh0dXvLhbr8FQVvr3vRVViBO" }, "ms-01": { "url": "http://ms-4242.example.meilisearch.io", - "searchApiKey": "hrVu-OMcjPGElK7692K7bwriBoGyHXTMvB5NmZkMKqQ" + "searchApiKey": "hrVu-OMcjPGElK7692K7bwriBoGyHXTMvB5NmZkMKqQ", + "writeApiKey": "bd1ldDoFlfyeoFDe8f3GVNiE8AHX86chmFuzOW7nWYUbPa7ww3" } } } @@ -59,6 +62,12 @@ Do not enable the `network` feature if you rely on the value of attributes not p **Default value**: `null`
**Description**: A string indicating the name of the current instance +### `sharding` + +**Type**: Boolean
+**Default value**: `false`
+**Description**: A boolean indicating whether sharding should be enabled on the network + ### `remotes` **Type**: Object
@@ -70,7 +79,8 @@ Do not enable the `network` feature if you rely on the value of attributes not p ```json "ms-00": { "url": "http://ms-1235.example.meilisearch.io", - "searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas" + "searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas", + "writeApiKey": "O2OaIHgwGuHNx9duH6kSe1YJ55Bh0dXvLhbr8FQVvr3vRVViBO" } ``` @@ -86,6 +96,12 @@ Do not enable the `network` feature if you rely on the value of attributes not p **Default value**: `null`
**Description**: An API key with search permissions +##### `writeApiKey` + +**Type**: String
+**Default value**: `null`
+**Description**: An API key with `documents.*` permissions + ## Get the network object @@ -101,14 +117,17 @@ Returns the current value of the instance's network object. ```json { "self": "ms-00", + "sharding": false, "remotes": { "ms-00": { "url": "http://ms-1235.example.meilisearch.io", - "searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas" + "searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas", + "writeApiKey": "O2OaIHgwGuHNx9duH6kSe1YJ55Bh0dXvLhbr8FQVvr3vRVViBO" }, "ms-01": { "url": "http://ms-4242.example.meilisearch.io", - "searchApiKey": "hrVu-OMcjPGElK7692K7bwriBoGyHXTMvB5NmZkMKqQ" + "searchApiKey": "hrVu-OMcjPGElK7692K7bwriBoGyHXTMvB5NmZkMKqQ", + "writeApiKey": "bd1ldDoFlfyeoFDe8f3GVNiE8AHX86chmFuzOW7nWYUbPa7ww3" } } } @@ -122,13 +141,14 @@ Update the `self` and `remotes` fields of the network object. Updates to the network object are **partial**. Only provide the fields you intend to update. Fields not present in the payload will remain unchanged. -To reset `self` and `remotes` to their original value, set them to `null`. To remove a single `remote` from your network, set the value of its name to `null`. +To reset `self`, `sharding` and `remotes` to their original value, set them to `null`. To remove a single `remote` from your network, set the value of its name to `null`. ### Body | Name | Type | Default value | Description | | :-------------------------------- | :----- | :------------ | :---------------------------------- | | **[`self`](#self)** | String | `null` | The name of the current instance | +| **[`sharding`](#sharding)** | Boolean| `false` | Whether sharding should be enabled on the network | | **[`remotes`](#remotes)** | String | `null` | A list of remote objects describing accessible Meilisearch instances | ### Example @@ -140,14 +160,17 @@ To reset `self` and `remotes` to their original value, set them to `null`. To re ```json { "self": "ms-00", + "sharding": true, "remotes": { "ms-00": { "url": "http://INSTANCE_URL", - "searchApiKey": "INSTANCE_API_KEY" + "searchApiKey": "INSTANCE_API_KEY", + "writeApiKey": "INSTANCE_WRITE_API_KEY" }, "ms-01": { "url": "http://ANOTHER_INSTANCE_URL", - "searchApiKey": "ANOTHER_INSTANCE_API_KEY" + "searchApiKey": "ANOTHER_INSTANCE_API_KEY", + "writeApiKey": "ANOTHER_INSTANCE_WRITE_API_KEY" } } } diff --git a/reference/api/tasks.mdx b/reference/api/tasks.mdx index 35cc86e90..3411be5e2 100644 --- a/reference/api/tasks.mdx +++ b/reference/api/tasks.mdx @@ -187,6 +187,49 @@ The `details` object is set to `null` for `snapshotCreation` tasks. | **`type`** | The [error type](/reference/errors/overview#errors) | | **`link`** | A link to the relevant section of the documentation | +### `network` + + + +**Type**: Object
+**Description**: If the task was replicated from another remote or to other remotes, `network` will contain information about the remote task uids corresponding to this task. Otherwise, missing in task object. + +`network` either has a single key that is either `origin` or `remotes`. + +| Name | Description | +| :-------------| :----------------------------------------------------------------------| +| **`origin`** | The task and remote from which this task was replicated **from** | +| **`remotes`** | This task was replicated **to** these tasks and remotes | + +`origin` is itself an object with keys: + +| Name | Description | +| :--------------- | :--------------------------------------------| +| **`remoteName`** | The name of the [remote](/reference/api/network) | +| **`taskUid`** | The uid of the task of origin | + +`remotes` is itself an object whose keys are the [remotes](/reference/api/network) and values an object with a single key that is either `task_uid` or `error`: + +| Name | Description | +| :------------- | :---------------------------------------------------------------------------------------------| +| **`task_uid`** | The uid of the replicated task | +| **`error`** | A human-readable error message indicating why the task could not be replicated to that remote | + + +This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it: + +```sh +curl \ + -X PATCH 'MEILISEARCH_URL/experimental-features/' \ + -H 'Content-Type: application/json' \ + --data-binary '{ + "network": true + }' +``` + + + + ### `duration` **Type**: String
diff --git a/snippets/samples/code_samples_network_get.mdx b/snippets/samples/code_samples_network_get.mdx new file mode 100644 index 000000000..5dcfd2a61 --- /dev/null +++ b/snippets/samples/code_samples_network_get.mdx @@ -0,0 +1,6 @@ + + +```go Go +client.GetNetwork(); +``` + \ No newline at end of file diff --git a/snippets/samples/code_samples_network_patch_1.mdx b/snippets/samples/code_samples_network_patch_1.mdx new file mode 100644 index 000000000..912f5609b --- /dev/null +++ b/snippets/samples/code_samples_network_patch_1.mdx @@ -0,0 +1,14 @@ + + +```go Go +client.UpdateNetwork(&meilisearch.Network{ + Self: meilisearch.String("TEST"), + Remotes: meilisearch.NewOpt(map[string]meilisearch.Opt[meilisearch.Remote]{ + "ms-00": meilisearch.NewOpt(meilisearch.Remote{ + URL: meilisearch.String("https://example.com"), + SearchAPIKey: meilisearch.String("TEST"), + }, + }, +}); +``` + \ No newline at end of file diff --git a/snippets/samples/code_samples_network_patch_2.mdx b/snippets/samples/code_samples_network_patch_2.mdx new file mode 100644 index 000000000..33ffeb2a8 --- /dev/null +++ b/snippets/samples/code_samples_network_patch_2.mdx @@ -0,0 +1,9 @@ + + +```go Go +client.UpdateNetwork(&meilisearch.Network{ + Self: meilisearch.String("TEST"), + Remotes: meilisearch.Null[map[string]meilisearch.Opt[meilisearch.Remote]](), +}); +``` + \ No newline at end of file