Skip to content

Add session storage clustering support #3555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
woutslakhorst merged 14 commits into from
Nov 22, 2024
Merged

Add session storage clustering support #3555

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
acfd175
replaced dual session DB impl to go-cache impl
woutslakhorst Nov 11, 2024
877531b
added memcached impl
woutslakhorst Nov 11, 2024
39aa9aa
link memcached
woutslakhorst Nov 11, 2024
9272fa0
add e2e test with redis sentinel clustered session memory
woutslakhorst Nov 11, 2024
93aa329
added e2e-tests
woutslakhorst Nov 11, 2024
57d1f62
reuse oauth2.js
woutslakhorst Nov 12, 2024
e9f3eaa
remove unused files
woutslakhorst Nov 12, 2024
5a3013a
reduce added lines by removing comments from redis conf files
woutslakhorst Nov 18, 2024
ae2e0c6
add documentation on clustering
woutslakhorst Nov 18, 2024
1126341
some godoc
woutslakhorst Nov 18, 2024
bb6abcd
deduplicate configs
woutslakhorst Nov 19, 2024
2aa24c0
deduplicate configs
woutslakhorst Nov 19, 2024
5e7844d
add session DB to health check
woutslakhorst Nov 19, 2024
0d38af5
added comment on default memcached behaviour
woutslakhorst Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@ The following options can be configured on the server:
pki.maxupdatefailhours 4 Maximum number of hours that a denylist update can fail
pki.softfail true Do not reject certificates if their revocation status cannot be established when softfail is true
**Storage**
storage.session.memcached.address [] List of Memcached server addresses. These can be a simple 'host:port' or a Memcached connection URL with scheme, auth and other options.
storage.session.redis.address Redis session database server address. This can be a simple 'host:port' or a Redis connection URL with scheme, auth and other options. If not set it, defaults to an in-memory database.
storage.session.redis.database Redis session database name, which is used as prefix every key. Can be used to have multiple instances use the same Redis instance.
storage.session.redis.password Redis session database password. If set, it overrides the username in the connection URL.
Expand Down
1 change: 1 addition & 0 deletions docs/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Nuts documentation
pages/deployment/configuration.rst
pages/deployment/migration.rst
pages/deployment/recommended-deployment.rst
pages/deployment/clustering.rst
pages/deployment/certificates.rst
pages/deployment/docker.rst
pages/deployment/storage.rst
Expand Down
21 changes: 21 additions & 0 deletions docs/pages/deployment/clustering.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
.. _clustering:

Clustering
##########

Clustering is currently limited to nodes that have the ``did:nuts`` method disabled.
Copy link
Member

@reinkrul reinkrul Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe state what NOT to do (use default on-disk/in-memory stores for private keys and stuff)?

To enable clustering, you must support the following:

- A clustered SQL database (SQLite is not supported)
- A clustered session storage (Redis sentinel is recommended)
- A clustered private key storage (Hashicorp Vault or Azure Keyvault)
- Read only mounts for configuration, policy, discovery and JSON-LD context files.

It's recommended to use a level 4 load balancer to distribute the load across the nodes.
Each node should have a reverse proxy for TLS termination.

Clustering will not work if you use one of the following:

- The did:nuts method
- SQLite
- Disk based private key storage
1 change: 1 addition & 0 deletions docs/pages/deployment/server_options.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
pki.maxupdatefailhours 4 Maximum number of hours that a denylist update can fail
pki.softfail true Do not reject certificates if their revocation status cannot be established when softfail is true
**Storage**
storage.session.memcached.address [] List of Memcached server addresses. These can be a simple 'host:port' or a Memcached connection URL with scheme, auth and other options.
storage.session.redis.address Redis session database server address. This can be a simple 'host:port' or a Redis connection URL with scheme, auth and other options. If not set it, defaults to an in-memory database.
storage.session.redis.database Redis session database name, which is used as prefix every key. Can be used to have multiple instances use the same Redis instance.
storage.session.redis.password Redis session database password. If set, it overrides the username in the connection URL.
Expand Down
63 changes: 63 additions & 0 deletions docs/pages/deployment/storage.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,69 @@ Refer to the documentation of the driver for the database you are using for the
Usage of SQLite is not recommended for production environments.
Connections to a SQLite DB are restricted to 1, which will lead to severe performance reduction.

Session storage
***************

Session storage is used for storing access tokens, nonces and other volatile data.
Session data is volatile by nature. There are 3 supported session storage types:

- In-memory
- Memcached
- Redis (standalone, cluster, sentinel)

Local
=====

This is the default and will store data in-memory. Any restart will wipe all data.
Data is also not shared if you run multiple nodes.

Memcached
=========

Memcached can be enabled with the following config:

.. code-block:: yaml

storage.session.memcached.address:
- localhost:11211

You can add multiple memcached servers to the list.
memcached is not capable of clustering. Each piece of data is stored on a single instance.
If you want true HA, you'll need to use Redis.
For more information on Memcached connection strings, refer to the `Memcached documentation <https://docs.memcached.org/>`_.

Redis
=====

Redis is the only option if you want to run multiple nodes and the cache as HA.
Redis can be configured in standalone or sentinel mode.
Standalone:

.. code-block:: yaml

storage:
session:
redis:
address: localhost:6379
username: user
password: pass
db: 0

Sentinel:

.. code-block:: yaml

storage:
session:
redis:
sentinel:
master: mymaster
nodes:
- localhost:26379
- localhost:26380
- localhost:26381


Private Keys
************

Expand Down
65 changes: 65 additions & 0 deletions e2e-tests/clustering/memcached/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
services:
memcached:
image: memcached
command:
- --conn-limit=1024
- --memory-limit=64
- --threads=4
nodeA-backend:
image: "${IMAGE_NODE_A:-nutsfoundation/nuts-node:master}"
ports:
- "18081:8081"
environment:
NUTS_CONFIGFILE: /opt/nuts/nuts.yaml
NUTS_URL: "https://nodeA"
NUTS_DISCOVERY_SERVER_IDS: "e2e-test"
NUTS_STORAGE_SESSION_MEMCACHED_ADDRESS: "memcached:11211"
volumes:
- "../shared/nuts.yaml:/opt/nuts/nuts.yaml:ro"
- "../../tls-certs/nodeA-backend-certificate.pem:/opt/nuts/certificate-and-key.pem:ro"
- "../../tls-certs/truststore.pem:/opt/nuts/truststore.pem:ro"
# did:web resolver uses the OS CA bundle, but e2e tests use a self-signed CA which can be found in truststore.pem
# So we need to mount that file to the OS CA bundle location, otherwise did:web resolving will fail due to untrusted certs.
- "../../tls-certs/truststore.pem:/etc/ssl/certs/Nuts_RootCA.pem:ro"
- "../shared/presentationexchangemapping.json:/opt/nuts/policies/presentationexchangemapping.json:ro"
- "../shared/discovery:/nuts/discovery:ro"
healthcheck:
interval: 1s # Make test run quicker by checking health status more often
nodeA:
image: nginx:1.25.1
ports:
- "10443:443"
volumes:
- "../shared/node-A/nginx.conf:/etc/nginx/nginx.conf:ro"
- "../../tls-certs/nodeA-certificate.pem:/etc/nginx/ssl/server.pem:ro"
- "../../tls-certs/nodeA-certificate.pem:/etc/nginx/ssl/key.pem:ro"
- "../../tls-certs/truststore.pem:/etc/nginx/ssl/truststore.pem:ro"
- "../../scripts/oauth2.js:/etc/nginx/oauth2.js:ro"
nodeB-backend:
image: "${IMAGE_NODE_B:-nutsfoundation/nuts-node:master}"
ports:
- "28081:8081"
environment:
NUTS_CONFIGFILE: /opt/nuts/nuts.yaml
NUTS_URL: "https://nodeB"
volumes:
- "../shared/nuts.yaml:/opt/nuts/nuts.yaml:ro"
- "../../tls-certs/nodeB-certificate.pem:/opt/nuts/certificate-and-key.pem:ro"
- "../../tls-certs/truststore.pem:/opt/nuts/truststore.pem:ro"
- "../../tls-certs/truststore.pem:/etc/ssl/certs/truststore.pem:ro"
# did:web resolver uses the OS CA bundle, but e2e tests use a self-signed CA which can be found in truststore.pem
# So we need to mount that file to the OS CA bundle location, otherwise did:web resolving will fail due to untrusted certs.
- "../../tls-certs/truststore.pem:/etc/ssl/certs/Nuts_RootCA.pem:ro"
- "../shared/presentationexchangemapping.json:/opt/nuts/policies/presentationexchangemapping.json:ro"
- "../shared/discovery:/nuts/discovery:ro"
healthcheck:
interval: 1s # Make test run quicker by checking health status more often
nodeB:
image: nginx:1.25.1
ports:
- "20443:443"
volumes:
- "../../shared_config/nodeB-http-nginx.conf:/etc/nginx/conf.d/nuts-http.conf:ro"
- "../../tls-certs/nodeB-certificate.pem:/etc/nginx/ssl/server.pem:ro"
- "../../tls-certs/nodeB-certificate.pem:/etc/nginx/ssl/key.pem:ro"
- "../../tls-certs/truststore.pem:/etc/nginx/ssl/truststore.pem:ro"
Loading
Loading