diff --git a/pages/deployment/benchmarking-memgraph.mdx b/pages/deployment/benchmarking-memgraph.mdx index fbee1cb77..015b7f809 100644 --- a/pages/deployment/benchmarking-memgraph.mdx +++ b/pages/deployment/benchmarking-memgraph.mdx @@ -62,7 +62,7 @@ While `mgbench` gives you full control over benchmarking and produces raw result analyzing performance metrics at scale can be time-consuming. That’s why we created [**Benchgraph**](https://github.com/memgraph/benchgraph)—a companion visualization tool for `mgbench`. -![](/pages/memgraph-in-production/benchmarking-memgraph/benchgraph-snippet.png) +![](/pages/deployment/benchmarking-memgraph/benchgraph-snippet.png) Benchgraph is not required to use `mgbench`, but it offers a **convenient way to visualize and explore** benchmark results across a variety of run conditions: diff --git a/pages/deployment/workloads.mdx b/pages/deployment/workloads.mdx index 83202ddf6..0625c4d8d 100644 --- a/pages/deployment/workloads.mdx +++ b/pages/deployment/workloads.mdx @@ -8,10 +8,13 @@ import {CommunityLinks} from '/components/social-card/CommunityLinks' # Workloads -When deploying Memgraph in production, it is essential to consider a set of prerequisites to ensure optimal **performance**, **scalability** -and **resilience**. That includes decisions about hardware configurations and integration strategies. +When deploying Memgraph in production, it is essential to consider a set of +prerequisites to ensure optimal **performance**, **scalability** and +**resilience**. That includes decisions about hardware configurations and +integration strategies. -This section provides guides that are the starting point to production-readiness with Memgraph. +This section provides guides that are the starting point to production-readiness +with Memgraph. Before you dive into specific setups, it's important to think about: - **Hardware requirements** and **instance sizing** @@ -20,18 +23,23 @@ Before you dive into specific setups, it's important to think about: - **Data import** best practices - Connecting to **external sources** -These factors ensure Memgraph performs effectively in your environment, no matter the use case. +These factors ensure Memgraph performs effectively in your environment, no +matter the use case. ## How to use these guides -**Start with the [Deployment best practices](/deployment/best-practices)**. That page covers the **best practices** that apply to most -production deployments, regardless of your workload type and are **agnostic to specific use cases**. -Each separate guide in the Workloads section focuses on a particular type of workload or deployment scenario. -At the beginning of each guide, you’ll learn when that **use case** is a good fit for your needs and specific tailored **recommendations**. +**Start with the [Deployment best practices](/deployment/best-practices)**. That +page covers the **best practices** that apply to most production deployments, +regardless of your workload type and are **agnostic to specific use cases**. +Each separate guide in the Workloads section focuses on a particular type of +workload or deployment scenario. At the beginning of each guide, you’ll learn +when that **use case** is a good fit for your needs and specific tailored +**recommendations**. -Recommendations in those guides **override** anything written in the -general suggestions when there's a conflict, so always defer to the targeted guide when applicable. +Recommendations in those guides **override** anything written in the general +suggestions when there's a conflict, so always defer to the targeted guide when +applicable. ## Available guides @@ -42,20 +50,26 @@ Here are the currently available guides to help you deploy Memgraph effectively: Learn how to utilize memgraph across your cyber security network. ### [Memgraph in high-throughput workloads](/deployment/workloads/memgraph-in-high-throughput-workloads) -Scale your write throughput while keeping up with fast-changing, high-velocity graph data. +Scale your write throughput while keeping up with fast-changing, high-velocity +graph data. ### [Memgraph in GraphRAG use cases](/deployment/workloads/memgraph-in-graphrag) -Learn how to optimize Memgraph for Retrieval-Augmented Generation (RAG) systems using graph data. +Learn how to optimize Memgraph for Retrieval-Augmented Generation (RAG) systems +using graph data. + +### [Memgraph in mission critical workloads](/deployment/workloads/memgraph-in-mission-critical-workloads) +Learn how to configure Memgraph for high availability, strong consistency, and +fault tolerance in mission-critical production environments. ## 🚧 Guides in construction - Memgraph in transactional workloads - Memgraph in analytical workloads -- Memgraph in mission critical workloads - Memgraph in supply chain use cases - Memgraph in fraud detection use cases -If you'd like to help us **prioritize** this content, feel free to reach out on [Discord](https://discord.gg/memgraph)! +If you'd like to help us **prioritize** this content, feel free to reach out on +[Discord](https://discord.gg/memgraph)! Your feedback helps us build what matters most. diff --git a/pages/deployment/workloads/_meta.ts b/pages/deployment/workloads/_meta.ts index d091f6364..0c82c3992 100644 --- a/pages/deployment/workloads/_meta.ts +++ b/pages/deployment/workloads/_meta.ts @@ -2,4 +2,5 @@ export default { "memgraph-in-cybersecurity": "Memgraph in cybersecurity", "memgraph-in-graphrag": "Memgraph in GraphRAG use cases", "memgraph-in-high-throughput-workloads": "Memgraph in high-throughput workloads", + "memgraph-in-mission-critical-workloads": "Memgraph in mission critical workloads", } diff --git a/pages/deployment/workloads/memgraph-in-cybersecurity.mdx b/pages/deployment/workloads/memgraph-in-cybersecurity.mdx index d95328aba..78f4195d9 100644 --- a/pages/deployment/workloads/memgraph-in-cybersecurity.mdx +++ b/pages/deployment/workloads/memgraph-in-cybersecurity.mdx @@ -50,7 +50,7 @@ Here's why Memgraph is a great fit for cybersecurity use cases: While many graph databases **max out around 1,000 events per second**, Memgraph can handle **up to 50x more** (see image below), making it ideal for **high-velocity security event processing**. - ![](/pages/memgraph-in-production/benchmarking-memgraph/realistic-workload.png) + ![](/pages/deployment/benchmarking-memgraph/realistic-workload.png) - **Non-blocking reads and writes with MVCC**: Built on multi-version concurrency control (MVCC), Memgraph ensures that **security event ingestion doesn't block threat analysis** and **analysis doesn't block ingestion**, diff --git a/pages/deployment/workloads/memgraph-in-graphrag.mdx b/pages/deployment/workloads/memgraph-in-graphrag.mdx index 086ec6fe6..b09a74c05 100644 --- a/pages/deployment/workloads/memgraph-in-graphrag.mdx +++ b/pages/deployment/workloads/memgraph-in-graphrag.mdx @@ -43,7 +43,7 @@ Here's what makes Memgraph a perfect fit for GraphRAG: ## What is covered? The suggestions for GraphRAG use cases **complement** several key sections in the -[general suggestions guide](/memgraph-in-production/general-suggestions). These sections offer important context and +[general suggestions guide](/deployment/best-practices). These sections offer important context and additional best practices tailored for performance, stability, and scalability in GraphRAG systems: - [Hardware sizing](#hardware-sizing) diff --git a/pages/deployment/workloads/memgraph-in-high-throughput-workloads.mdx b/pages/deployment/workloads/memgraph-in-high-throughput-workloads.mdx index 50a941b65..251da0167 100644 --- a/pages/deployment/workloads/memgraph-in-high-throughput-workloads.mdx +++ b/pages/deployment/workloads/memgraph-in-high-throughput-workloads.mdx @@ -45,7 +45,7 @@ Here's why Memgraph is a great fit for high-throughput use cases: While many graph databases **max out around 1,000 writes per second**, Memgraph can handle **up to 50x more** (see image below), making it ideal for **high-velocity, write-intensive workloads**. - ![](/pages/memgraph-in-production/benchmarking-memgraph/realistic-workload.png) + ![](/pages/deployment/benchmarking-memgraph/realistic-workload.png) - **Non-blocking reads and writes with MVCC**: Built on multi-version concurrency control (MVCC), Memgraph ensures that **writes don’t block reads** and **reads don’t block writes**, allowing each to scale independently. @@ -71,7 +71,7 @@ Here's why Memgraph is a great fit for high-throughput use cases: ## What is covered? The suggestions for high-throughput workloads **complement** several key sections in the -[general suggestions guide](/memgraph-in-production/general-suggestions). These sections offer important context and +[best practices guide](/deployment/best-practices). These sections offer important context and additional best practices tailored for performance, stability, and scalability in high-throughput systems: - [Choosing the right Memgraph flag set](#choosing-the-right-memgraph-flag-set)
diff --git a/pages/deployment/workloads/memgraph-in-mission-critical-workloads.mdx b/pages/deployment/workloads/memgraph-in-mission-critical-workloads.mdx new file mode 100644 index 000000000..3e5815564 --- /dev/null +++ b/pages/deployment/workloads/memgraph-in-mission-critical-workloads.mdx @@ -0,0 +1,224 @@ +--- +title: Memgraph in mission-critical workloads +description: Suggestions on how to bring your Memgraph to production in mission-critical and high-availability workloads. +--- + +import { Callout } from 'nextra/components' +import { CommunityLinks } from '/components/social-card/CommunityLinks' + +# Memgraph in mission-critical workloads + + +Before diving into this guide, we recommend starting with the [Best +practices](/deployment/best-practices) + +page. It provides **foundational, use-case-agnostic advice** for deploying +Memgraph in production. + +This guide builds on that foundation, offering **additional recommendations +tailored to critical and high-availability workloads**. In cases where guidance +overlaps, consider the information here as **complementary or overriding**, +depending on the unique needs of your use case. + + +## Is this guide for you? + +This guide is for you if you're building **mission-critical systems** where +uptime, data consistency, and fault tolerance are essential. You’ll benefit +from this content if: + +- You require **high availability** and **automatic failover** for your + application. +- You need **strong consistency guarantees** even under heavy loads. +- You must **recover gracefully** from unexpected failures without data loss. +- You need to **support multi-tenant environments** securely across multiple + projects or customers. + + +If this matches your needs, this guide will help you configure and operate +Memgraph to meet the demands of **always-on production environments**. + +## Why choose Memgraph for mission-critical use cases? + +When stability, consistency, and resilience matter most, Memgraph is built to +deliver. Here's why Memgraph is a great fit for mission-critical workloads: + +- **In-memory storage engine with persistence** Memgraph keeps the working set + fully in memory for fast access, while ensuring + [durability](/fundamentals/data-durability) through **periodic snapshots** and + **write-ahead logging (WALs)** in transactional mode. + + Keeping the data in memory ensures lightning speed in times when you expect + everything to function seamlessly and without issues during peak times of your + critical service. + +- **High availability with automatic failover** Memgraph supports full [high + availability clustering](/clustering/high-availability), allowing you to + deploy **multiple instances** with automatic **leader election** and + **failover** when needed. + + Deploying Memgraph with high availability will ensure Memgraph is up and + running at all times, without compromising uptime of your services. + +- **Multi-version concurrency control (MVCC)** Built on **MVCC**, Memgraph + allows **non-blocking reads and writes**, ensuring that your system remains + **responsive** even under concurrent access. Writes are not blocking reads, + and vice versa. + +- **Snapshot isolation by default** Memgraph uses **snapshot isolation** instead + of **read-committed** isolation, preventing dirty reads and guaranteeing a + **consistent view** of the graph at all times. + +- **Replication for read scaling and redundancy** Memgraph supports + **asynchronous replication**, enabling you to **scale read workloads** + independently while ensuring **failover readiness**. For a more consistent + view of the data, it also supports **synchronous replication** which + prioritizes consistency over scalability. + +- **Fine-grained access control and security** Secure your system with + [**role-based access + control**](/database-management/authentication-and-authorization/role-based-access-control) + and [**label-based access + control**](/database-management/authentication-and-authorization/role-based-access-control#label-based-access-control) + to ensure only the right users see and manipulate data. + +## What is covered? + +The suggestions for mission-critical workloads **complement** several key +sections in the [general suggestions guide](/deployment/best-practices), with +additional best practices to ensure uptime and data protection: + +- [Choosing the right Memgraph flag set](#choose-the-right-memgraph-flag-set) + Memgraph offers flags to enhance recovery, snapshot management, and failover + capabilities. + +- [Choosing the right Memgraph storage + mode](#choose-the-right-memgraph-storage-mode) + Guidance on selecting the **safest** and **most consistent** storage + configurations. + +- [Enterprise features you might + require](#enterprise-features-you-might-require) + Overview of **replication**, **multi-tenancy**, and **automatic failover** + tools that are critical in production. + +- [Backup and recovery mechanisms](#backup-and-recovery-mechanisms) + Best practices to protect your data through snapshots, WALs, and external + backup strategies. + +- [Queries that best suit your workload](#queries-that-best-suit-your-workload) + Designing queries that maintain consistent, safe, and predictable behavior in + high-availability systems. + +## Choose the right Memgraph flag set + + +For mission-critical setups, you should configure Memgraph to optimize for **durability, fast recovery**, and **stability**. Some important flags include: + +- `--storage-snapshot-interval-sec=x` + Set how often snapshots are created. In mission-critical systems, you may want + **frequent snapshots** to minimize recovery time. + +- `--storage-wal-enabled=true` + Ensure **WALs (write-ahead logs)** are enabled to protect all transactions + between snapshots. + +- `--storage-parallel-schema-recovery=true` and + `--storage-recovery-thread-count=x` + Enable **parallel recovery** to speed up startup time after a crash by using + multiple cores. + +- `--query-execution-timeout-sec=x` + Set reasonable query timeouts to **avoid stuck queries** and prevent resource + exhaustion.` + +## Choose the right Memgraph storage mode + + +For mission-critical deployments: + +- Always use `IN_MEMORY_TRANSACTIONAL` mode. +- This mode provides **full ACID guarantees**, **WAL support**, and **snapshot + consistency**. + + +`IN_MEMORY_ANALYTICAL` is optimized for high-speed ingestion but does **not +provide transactional durability**. It is not recommended for mission-critical +workloads. + + +## Importing mechanisms + +Importing mechanisms are best described in the [guide for high-throughput +workloads](/deployment/workloads/memgraph-in-high-throughput-workloads). The +rule of thumb is to always setup the drivers to perform retries if you're doing +heavy amount of writes, in order to avoid read conflicts. The high throguhput +guide also outlines the need for idempotent queries, to ensure data consistency +if writes fail for any reason. + +## Enterprise features you might require + +For robust production environments, consider enabling: + +- [High availability clustering](/clustering/high-availability): Deploy multiple + Memgraph instances with automatic leader election and failover. + +- [Replication for resilience](/clustering/replication): Distribute replicas + geographically or across availability zones to minimize the risk of localized + outages. + +- [Role-based and label-based access + control](/database-management/authentication-and-authorization/role-based-access-control): + Protect sensitive graph data and ensure only authorized operations are + performed. + +- [Multi-tenancy](/database-management/multi-tenancy): Securely isolate data and + permissions between different teams, projects, or customers. + +## Backup and recovery mechanisms + +Data durability is critical in mission-critical environments. Memgraph supports: + +- [Snapshots](/fundamentals/data-durability#snapshots) + Automatically or manually triggered full-database snapshots. + +- [Write-ahead logging + (WALs)](/fundamentals/data-durability#write-ahead-logging) + Transaction logs that enable you to **replay changes** made after the last + snapshot. + +- **Manual backup and offloading** + Use external tools (like [`rclone`](https://rclone.org/)) to back up snapshots + and WALs to **cloud storage** or **remote servers** for additional redundancy. + + + +Memgraph currently does not automate backing up data to 3rd party locations, so +integrating a backup process into your system is highly recommended. + + + +Learn more about backup and restore [on our backup and restore documentation +page](/database-management/backup-and-restore). + + +## Queries that best suit your workload + +In mission-critical workloads: + +- Prefer **idempotent writes** (`MERGE`) to avoid inconsistent state during + retries. +- Optimize long-running queries and [profile](/querying/clauses/profile) them + regularly. + +- Avoid complex, unpredictable queries inside critical transactional paths. +- Use **schema constraints** and **indexes** wisely to enforce data integrity + without hurting performance. + +Example of safe, idempotent data ingestion: + +```cypher +MERGE (n:Customer {id: $id}) +SET n += $props; +``` + diff --git a/public/pages/memgraph-in-production/benchmarking-memgraph/benchgraph-snippet.png b/public/pages/deployment/benchmarking-memgraph/benchgraph-snippet.png similarity index 100% rename from public/pages/memgraph-in-production/benchmarking-memgraph/benchgraph-snippet.png rename to public/pages/deployment/benchmarking-memgraph/benchgraph-snippet.png diff --git a/public/pages/memgraph-in-production/benchmarking-memgraph/realistic-workload.png b/public/pages/deployment/benchmarking-memgraph/realistic-workload.png similarity index 100% rename from public/pages/memgraph-in-production/benchmarking-memgraph/realistic-workload.png rename to public/pages/deployment/benchmarking-memgraph/realistic-workload.png