Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 38 additions & 12 deletions learn/multi_search/implement_sharding.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ Sharding is the process of splitting an index containing many documents into mul

This guide walks you through activating the `/network` route, configuring the network object, and performing remote federated searches.

<NoticeTag label="Enterprise Edition" />
Sharding is an Enterprise Edition feature. You are free to use it for evaluation purposes. Please [reach out to us](mailto:sales@meilisearch.com) before using it in production.

<Tip>
## Configuring multiple instances

Expand All @@ -19,7 +22,7 @@ To minimize issues and limit unexpected behavior, instance, network, and index c

## Prerequisites

- Multiple Meilisearch projects (instances) running Meilisearch >=v1.13
- Multiple Meilisearch projects (instances) running Meilisearch >=v1.19

## Activate the `/network` endpoint

Expand Down Expand Up @@ -48,6 +51,7 @@ Next, you must configure the network object. It consists of the following fields

- `remotes`: defines a list with the required information to access each remote instance
- `self`: specifies which of the configured `remotes` corresponds to the current instance
- `sharding`: whether to use sharding.

### Setting up the list of remotes

Expand Down Expand Up @@ -93,32 +97,54 @@ curl \

Meilisearch processes searches on the remote that corresponds to `self` locally instead of making a remote request.

### Enabling sharding

Finally enable the automatic sharding of documents by Meilisearch on all instances:

```sh
curl \
-X PATCH 'MEILISEARCH_URL/network' \
-H 'Content-Type: application/json' \
--data-binary '{
"sharding": true
}'
```

### Adding or removing an instance

Changing the topology of the network involves moving some documents from an instance to another, depending on your hashing scheme.

As Meilisearch does not provide atomicity across multiple instances, you will need to either:

1. accept search downtime while migrating documents
2. accept some documents will not appear in search results during the migration
1. accept search downtime while migrating documents
2. accept some documents will not appear in search results during the migration
3. accept some duplicate documents may appear in search results during the migration

#### Reducing downtime

If your disk space allows, you can reduce the downtime by applying the following algorithm:

1. Create a new temporary index in each remote instance
2. Compute the new instance for each document
3. Send the documents to the temporary index of their new instance
4. Once Meilisearch has copied all documents to their instance of destination, swap the new index with the previously used index
5. Delete the temporary index after the swap
6. Update network configuration and search queries across all instances
1. Create a new temporary index in each remote instance
2. Compute the new instance for each document
3. Send the documents to the temporary index of their new instance
4. Once Meilisearch has copied all documents to their instance of destination, swap the new index with the previously used index
5. Delete the temporary index after the swap
6. Update network configuration and search queries across all instances

## Create indexes

Create the same empty indexes with the same settings on all instances.
Keeping the settings and indexes in sync is important to avoid errors and unexpected behavior, though not strictly required.

## Add documents

Pick a single instance to send all your documents to. Documents will be replicated to the other instances.

## Create indexes and add documents
Each instance will index the documents they are responsible for and ignore the others.

Create the same empty indexes with the same settings on all instances. Keeping the settings and indexes in sync is important to avoid errors and unexpected behavior, though not strictly required.
You *may* send send the same document to multiple instances, the task will be replicated to all instances, and only the instance responsible for the document will index it.

Distribute your documents across all instances. Do not send the same document to multiple instances as this may lead to duplicate search results. Similarly, you should ensure all future versions of a document are sent to the same instance. Meilisearch recommends you hash their primary key using [rendezvous hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing).
Similarly, you may send any future versions of any document to the instance you picked, and only the correct instance will process that document.

### Updating index settings

Expand Down
39 changes: 31 additions & 8 deletions reference/api/network.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,17 @@ Do not enable the `network` feature if you rely on the value of attributes not p
```json
{
"self": "ms-00",
"sharding": false,
"remotes": {
"ms-00": {
"url": "http://ms-1235.example.meilisearch.io",
"searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas"
"searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas",
"writeApiKey": "O2OaIHgwGuHNx9duH6kSe1YJ55Bh0dXvLhbr8FQVvr3vRVViBO"
},
"ms-01": {
"url": "http://ms-4242.example.meilisearch.io",
"searchApiKey": "hrVu-OMcjPGElK7692K7bwriBoGyHXTMvB5NmZkMKqQ"
"searchApiKey": "hrVu-OMcjPGElK7692K7bwriBoGyHXTMvB5NmZkMKqQ",
"writeApiKey": "bd1ldDoFlfyeoFDe8f3GVNiE8AHX86chmFuzOW7nWYUbPa7ww3"
}
}
}
Expand All @@ -59,6 +62,12 @@ Do not enable the `network` feature if you rely on the value of attributes not p
**Default value**: `null`<br />
**Description**: A string indicating the name of the current instance

### `sharding`

**Type**: Boolean<br />
**Default value**: `false`<br />
**Description**: A boolean indicating whether sharding should be enabled on the network

### `remotes`

**Type**: Object<br />
Expand All @@ -70,7 +79,8 @@ Do not enable the `network` feature if you rely on the value of attributes not p
```json
"ms-00": {
"url": "http://ms-1235.example.meilisearch.io",
"searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas"
"searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas",
"writeApiKey": "O2OaIHgwGuHNx9duH6kSe1YJ55Bh0dXvLhbr8FQVvr3vRVViBO"
}
```

Expand All @@ -86,6 +96,12 @@ Do not enable the `network` feature if you rely on the value of attributes not p
**Default value**: `null`<br />
**Description**: An API key with search permissions

##### `writeApiKey`

**Type**: String<br />
**Default value**: `null`<br />
**Description**: An API key with `documents.*` permissions

## Get the network object

<RouteHighlighter method="GET" path="/network"/>
Expand All @@ -101,14 +117,17 @@ Returns the current value of the instance's network object.
```json
{
"self": "ms-00",
"sharding": false,
"remotes": {
"ms-00": {
"url": "http://ms-1235.example.meilisearch.io",
"searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas"
"searchApiKey": "Ecd1SDDi4pqdJD6qYLxD3y7VZAEb4d9j6LJgt4d6xas",
"writeApiKey": "O2OaIHgwGuHNx9duH6kSe1YJ55Bh0dXvLhbr8FQVvr3vRVViBO"
},
"ms-01": {
"url": "http://ms-4242.example.meilisearch.io",
"searchApiKey": "hrVu-OMcjPGElK7692K7bwriBoGyHXTMvB5NmZkMKqQ"
"searchApiKey": "hrVu-OMcjPGElK7692K7bwriBoGyHXTMvB5NmZkMKqQ",
"writeApiKey": "bd1ldDoFlfyeoFDe8f3GVNiE8AHX86chmFuzOW7nWYUbPa7ww3"
}
}
}
Expand All @@ -122,13 +141,14 @@ Update the `self` and `remotes` fields of the network object.

Updates to the network object are **partial**. Only provide the fields you intend to update. Fields not present in the payload will remain unchanged.

To reset `self` and `remotes` to their original value, set them to `null`. To remove a single `remote` from your network, set the value of its name to `null`.
To reset `self`, `sharding` and `remotes` to their original value, set them to `null`. To remove a single `remote` from your network, set the value of its name to `null`.

### Body

| Name | Type | Default value | Description |
| :-------------------------------- | :----- | :------------ | :---------------------------------- |
| **[`self`](#self)** | String | `null` | The name of the current instance |
| **[`sharding`](#sharding)** | Boolean| `false` | Whether sharding should be enabled on the network |
| **[`remotes`](#remotes)** | String | `null` | A list of remote objects describing accessible Meilisearch instances |

### Example
Expand All @@ -140,14 +160,17 @@ To reset `self` and `remotes` to their original value, set them to `null`. To re
```json
{
"self": "ms-00",
"sharding": true,
"remotes": {
"ms-00": {
"url": "http://INSTANCE_URL",
"searchApiKey": "INSTANCE_API_KEY"
"searchApiKey": "INSTANCE_API_KEY",
"writeApiKey": "INSTANCE_WRITE_API_KEY"
},
"ms-01": {
"url": "http://ANOTHER_INSTANCE_URL",
"searchApiKey": "ANOTHER_INSTANCE_API_KEY"
"searchApiKey": "ANOTHER_INSTANCE_API_KEY",
"writeApiKey": "ANOTHER_INSTANCE_WRITE_API_KEY"
}
}
}
Expand Down
43 changes: 43 additions & 0 deletions reference/api/tasks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,49 @@ The `details` object is set to `null` for `snapshotCreation` tasks.
| **`type`** | The [error type](/reference/errors/overview#errors) |
| **`link`** | A link to the relevant section of the documentation |

### `network`

<NoticeTag type="experimental" label="experimental" />

**Type**: Object<br />
**Description**: If the task was replicated from another remote or to other remotes, `network` will contain information about the remote task uids corresponding to this task. Otherwise, missing in task object.

`network` either has a single key that is either `origin` or `remotes`.

| Name | Description |
| :-------------| :----------------------------------------------------------------------|
| **`origin`** | The task and remote from which this task was replicated **from** |
| **`remotes`** | This task was replicated **to** these tasks and remotes |

`origin` is itself an object with keys:

| Name | Description |
| :--------------- | :--------------------------------------------|
| **`remoteName`** | The name of the [remote](/reference/api/network) |
| **`taskUid`** | The uid of the task of origin |

`remotes` is itself an object whose keys are the [remotes](/reference/api/network) and values an object with a single key that is either `task_uid` or `error`:

| Name | Description |
| :------------- | :---------------------------------------------------------------------------------------------|
| **`task_uid`** | The uid of the replicated task |
| **`error`** | A human-readable error message indicating why the task could not be replicated to that remote |

<Note>
This is an experimental feature. Use the Meilisearch Cloud UI or the experimental features endpoint to activate it:

```sh
curl \
-X PATCH 'MEILISEARCH_URL/experimental-features/' \
-H 'Content-Type: application/json' \
--data-binary '{
"network": true
}'
```
</Note>



### `duration`

**Type**: String<br />
Expand Down
6 changes: 6 additions & 0 deletions snippets/samples/code_samples_network_get.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
<CodeGroup>

```go Go
client.GetNetwork();
```
</CodeGroup>
14 changes: 14 additions & 0 deletions snippets/samples/code_samples_network_patch_1.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
<CodeGroup>

```go Go
client.UpdateNetwork(&meilisearch.Network{
Self: meilisearch.String("TEST"),
Remotes: meilisearch.NewOpt(map[string]meilisearch.Opt[meilisearch.Remote]{
"ms-00": meilisearch.NewOpt(meilisearch.Remote{
URL: meilisearch.String("https://example.com"),
SearchAPIKey: meilisearch.String("TEST"),
},
},
});
```
</CodeGroup>
9 changes: 9 additions & 0 deletions snippets/samples/code_samples_network_patch_2.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<CodeGroup>

```go Go
client.UpdateNetwork(&meilisearch.Network{
Self: meilisearch.String("TEST"),
Remotes: meilisearch.Null[map[string]meilisearch.Opt[meilisearch.Remote]](),
});
```
</CodeGroup>