Skip to content

doc: add local vs distributed storage explanation #1210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
roosterfish merged 1 commit into from
Feb 3, 2026
+79 −0
Merged

doc: add local vs distributed storage explanation #1210

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions doc/.custom_wordlist.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ MicroOVN
mTLS
multicast
NAT
NBD
Netplan
NIC
NICs
Expand All @@ -59,6 +60,7 @@ PNG
pre
preseed
QSFP
RBD
Replicable
replicable
rollout
Expand All @@ -76,6 +78,7 @@ TLS
UI's
uplink
upstream's
uptime
virtualized
VLAN
VM
Expand Down
7 changes: 7 additions & 0 deletions doc/explanation/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
---
myst:
html_meta:
description: Deep-dive explanations that introduce you to concepts in MicroCloud, such as its networking approach and local versus distributed storage.
---

(explanation)=
# Explanation

Expand All @@ -8,6 +14,7 @@ The explanatory guides in this section introduce you to the concepts used in Mic

/explanation/microcloud
/explanation/networking
/explanation/storage
Initialization process </explanation/initialization>
Security </explanation/security>
```
69 changes: 69 additions & 0 deletions doc/explanation/storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
---
myst:
html_meta:
description: Explanation about local versus distributed storage in MicroCloud and the benefits and drawbacks of each, such as speed and data redundancy.
---

(exp-storage)=
# Local versus distributed storage

This page explains the benefits and drawbacks of local versus distributed storage in MicroCloud and helps you understand when you should use each.

In MicroCloud, you can set up both types of storage, or only one, depending on your needs. For production environments, we recommend that you use both. For detailed hardware requirements for both local and distributed storage, see {ref}`reference-requirements-storage`.

(exp-storage-setup)=
## Setup

Local storage is simpler: each cluster member has its own local disk, and workloads on that member read/write directly to it. It does not provide data redundancy.

Distributed storage (using {ref}`Ceph RBD <lxd:storage-ceph>`) replicates data across multiple disks. Workloads access the shared storage over a Ceph network, set up by MicroCloud using MicroCeph. Due to this more complex setup and replication, distributed storage uses more CPU, RAM, and network bandwidth.

(exp-storage-access)=
## Access

With local storage, a workload on one cluster member cannot access data on other members unless it's copied over, shared over the network, or the workload is migrated over.

With distributed storage, workloads can access shared data cluster-wide. Access occurs over the Ceph network; RBD volumes are mapped only on the member where the workload runs.

(exp-storage-speed)=
## Speed

Local storage is often faster because each read/write operation stays on the local disk.

Distributed storage can be slower due to network latency and data replication overhead, though the speed difference can be small depending on hardware and network configuration. This overhead is most noticeable for write-heavy or latency-sensitive workloads.

(exp-storage-migration)=
## Instance migration

Instances can be {ref}`migrated between cluster members <lxd:howto-instances-migrate>`. For instances using local storage, data must also be transferred over the network along with those instances. MicroCloud (through LXD) creates a temporary snapshot file, then sets up an NBD (Network Block Device) listener to transfer disk data over the network.

Data stored in a distributed setup is accessible to all cluster members and does not need to be transferred during instance migration. This means that instances using distributed storage can be migrated more quickly.

(exp-storage-ha-redundancy)=
## High availability and data redundancy

With local storage, you risk a higher potential of data loss and workload disruption if a local disk or its host machine fails. Even if you set up a backup and recovery procedure, you'll still lose any data since the last backup, it will take longer to restore data from backups than with distributed storage.

Distributed storage avoids a single point of failure. By default, MicroCloud replicates data three times across three disks. This protects against data loss and reduces recovery time; if one disk fails, data is still available from another disk. Since the data remains available, workloads can restart on other members without needing to restore from backups. For more information about HA in MicroCloud, see {ref}`exp-microcloud-ha`.

(exp-storage-summary)=
## Recommendations

For production deployments where you need both performance and resilience, use both local and distributed storage. Choose which to use depending on the workload type:

- Use local storage for temporary workloads where you need fast disk access, and data loss is less of a concern.
- Use distributed storage when you have long-running workloads with uptime requirements that need data redundancy.

## Related topics

How-to guides

- {ref}`howto-ceph-networking`

Explanation

- {ref}`exp-microcloud`

Requirements

- {ref}`reference-requirements-storage`