From ad5ca79a2d585c91912c63656b93ebe513082d9d Mon Sep 17 00:00:00 2001 From: hschne Date: Mon, 9 Jun 2025 11:03:29 +0200 Subject: [PATCH 1/6] Update initializer and readme --- README.md | 24 ++++++-- .../litestream/templates/initializer.rb | 58 +++++++++++++++---- 2 files changed, 65 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index e50eb54..c4538fc 100644 --- a/README.md +++ b/README.md @@ -80,24 +80,27 @@ LITESTREAM_INSTALL_DIR=.bin You configure the Litestream executable through the [`config/litestream.yml` file](https://litestream.io/reference/config/), which is a standard Litestream configuration file as if Litestream was running in a traditional installation. -The gem streamlines the configuration process by providing a default configuration file for you. This configuration file will backup all SQLite databases defined in your `config/database.yml` file to one replication bucket. In order to ensure that no secrets are stored in plain-text in your repository, this configuration file leverages Litestream's support for environment variables. The default configuration file looks like this if you only have one SQLite database: +The gem streamlines the configuration process by providing a default configuration file for you. This configuration file will backup all SQLite databases defined in your `config/database.yml` file to one replication bucket. In order to ensure that no secrets are stored in plain-text in your repository, this configuration file leverages Litestream's support for environment variables. Inspect which environment variables are available run `litestream:env`. + +The default configuration file looks like this if you only have one SQLite database: ```yaml dbs: - path: storage/production.sqlite3 replicas: - type: s3 - bucket: $LITESTREAM_REPLICA_BUCKET path: storage/production.sqlite3 + bucket: $LITESTREAM_REPLICA_BUCKET access-key-id: $LITESTREAM_ACCESS_KEY_ID secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY ``` This is the default for Amazon S3. The full range of possible replica types (e.g. other S3-compatible object storage servers) are covered in Litestream's [replica guides](https://litestream.io/guides/#replica-guides). -The gem also provides a default initializer file at `config/initializers/litestream.rb` that allows you to configure these three environment variables referenced in the configuration file in Ruby. By providing a Ruby interface to these environment variables, you can use any method of storing secrets that you prefer. For example, the default generated file uses Rails' encrypted credentials to store your secrets: +The gem also provides a default initializer file at `config/initializers/litestream.rb` that allows you to configure various variables referenced in the configuration file in Ruby. By providing a Ruby interface to these environment variables, you can use your preferred method of storing secrets. For example, the default generated file uses Rails' encrypted credentials to store your secrets. ```ruby +# config/initializers/litestream.rb Rails.application.configure do litestream_credentials = Rails.application.credentials.litestream @@ -107,7 +110,17 @@ Rails.application.configure do end ``` -However, if you need manual control over the Litestream configuration, you can manually edit the `config/litestream.yml` file. The full range of possible configurations are covered in Litestream's [configuration reference](https://litestream.io/reference/config/). NB: If you configure a longer `sync-interval`, you may need to adjust `replication_sleep` when calling `Litestream.verify!`. +However, if you need manual control over the Litestream configuration, you can manually edit the `config/litestream.yml` file. The full range of possible configurations are covered in Litestream's [configuration reference](https://litestream.io/reference/config/). Outside of configuring Litestream, you may also configure various other aspects of `litestream-ruby` itself. + +```ruby +# config/initializers/litestream.rb +Rails.application.configure do + # Base controller used for Litestream dashboard + config.litestream.base_controller_class = "MyApplicationController" + # Set the location of the Litestream config + config.config_path = "config/litestream.yml" +end +``` ### Replication @@ -232,9 +245,10 @@ You can verify the integrity of your backed-up databases using the gem's provide ```ruby Litestream.verify! "storage/production.sqlite3" +Litestream.verify!(replication_sleep: 10) "storage/production.sqlite3" ``` -In order to verify that the backup for that database is both restorable and fresh, the method will add a new row to that database under the `_litestream_verification` table, which it will create if needed. It will then wait `replication_sleep` seconds (defaults to 10) to give the Litestream utility time to replicate that change to whatever storage providers you have configured. After that, it will download the latest backup from that storage provider and ensure that this verification row is present in the backup. If the verification row is _not_ present, the method will raise a `Litestream::VerificationFailure` exception. This check ensures that the restored database file: +In order to verify that the backup for that database is both restorable and fresh, the method will add a new row to that database under the `_litestream_verification` table, which it will create if needed. It will then wait `replication_sleep` seconds (defaults to 10) to give the Litestream utility time to replicate that change to whatever storage providers you have configured. Note that if you configure a longer `sync-interval`, you may need to adjust `replication_sleep` to a longer period. After that, it will download the latest backup from that storage provider and ensure that this verification row is present in the backup. If the verification row is _not_ present, the method will raise a `Litestream::VerificationFailure` exception. This check ensures that the restored database file: 1. exists, 2. can be opened by SQLite, and diff --git a/lib/litestream/generators/litestream/templates/initializer.rb b/lib/litestream/generators/litestream/templates/initializer.rb index 5b1a418..89f41f9 100644 --- a/lib/litestream/generators/litestream/templates/initializer.rb +++ b/lib/litestream/generators/litestream/templates/initializer.rb @@ -5,29 +5,63 @@ # or some other mechanism where the values are only available at runtime. Rails.application.configure do - # An example of using Rails encrypted credentials to configure Litestream. + # Configure Litestream through environment variables. Use Rails encrypted credentials for secrets. + # # litestream_credentials = Rails.application.credentials.litestream # Replica-specific bucket location. + # # This will be your bucket's URL without the `https://` prefix. # For example, if you used DigitalOcean Spaces, your bucket URL could look like: + # # https://myapp.fra1.digitaloceanspaces.com + # # And so you should set your `replica_bucket` to: + # # myapp.fra1.digitaloceanspaces.com - # Litestream supports Azure Blog Storage, Backblaze B2, DigitalOcean Spaces, - # Scaleway Object Storage, Google Cloud Storage, Linode Object Storage, and - # any SFTP server. - # In this example, we are using Rails encrypted credentials to store the URL to - # our storage provider bucket. + # # config.litestream.replica_bucket = litestream_credentials&.replica_bucket - - # Replica-specific authentication key. + # + # + # Replica-specific authentication key + # # Litestream needs authentication credentials to access your storage provider bucket. - # In this example, we are using Rails encrypted credentials to store the access key ID. + # # config.litestream.replica_key_id = litestream_credentials&.replica_key_id - - # Replica-specific secret key. + # + # + # Replica-specific secret key + # # Litestream needs authentication credentials to access your storage provider bucket. - # In this example, we are using Rails encrypted credentials to store the secret access key. + # # config.litestream.replica_access_key = litestream_credentials&.replica_access_key + # + # + # Replica-specific region + # + # Optionally specifies the bucket’s region. Only used for AWS S3 & Backblaze B2. + # + # config.litestream.replica_region = "us-east-1" + # + # + # Replica-specific endpoint + # + # Optionally specifies the endpoint URL of the S3-compatible service. Only required for non-AWS services. + # + # config.litestream.replica_endpoint = "endpoint.your-objectstorage.com" + + # Configure the default Litestream config path + # + # config.config_path = Rails.root.join("config", "litestream.yml") + + # Configure the Litestream dashboard + # + # Set the default base controller class + # + # config.litestream.base_controller_class = "MyApplicationController" + # + # Set authentication credentials for Litestream dashboard + # + # config.litestream.username = Rails.application.credentials.dig(:litestream, :username) + # config.litestream.password = Rails.application.credentials.dig(:litestream, :password) end From 300d04533de267f0a99884d7515f2d536a66462e Mon Sep 17 00:00:00 2001 From: hschne Date: Mon, 9 Jun 2025 20:11:27 +0200 Subject: [PATCH 2/6] Update readme and initializer --- README.md | 2 +- .../litestream/templates/initializer.rb | 32 +++---------------- 2 files changed, 6 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index c4538fc..4e81757 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ Rails.application.configure do # Base controller used for Litestream dashboard config.litestream.base_controller_class = "MyApplicationController" # Set the location of the Litestream config - config.config_path = "config/litestream.yml" + config.litestream.config_path = "config/litestream.yml" end ``` diff --git a/lib/litestream/generators/litestream/templates/initializer.rb b/lib/litestream/generators/litestream/templates/initializer.rb index 89f41f9..46b885a 100644 --- a/lib/litestream/generators/litestream/templates/initializer.rb +++ b/lib/litestream/generators/litestream/templates/initializer.rb @@ -6,12 +6,9 @@ Rails.application.configure do # Configure Litestream through environment variables. Use Rails encrypted credentials for secrets. - # # litestream_credentials = Rails.application.credentials.litestream - # Replica-specific bucket location. - # - # This will be your bucket's URL without the `https://` prefix. + # Replica-specific bucket location. This will be your bucket's URL without the `https://` prefix. # For example, if you used DigitalOcean Spaces, your bucket URL could look like: # # https://myapp.fra1.digitaloceanspaces.com @@ -22,46 +19,27 @@ # # config.litestream.replica_bucket = litestream_credentials&.replica_bucket # - # - # Replica-specific authentication key - # - # Litestream needs authentication credentials to access your storage provider bucket. - # + # Replica-specific authentication key. Litestream needs authentication credentials to access your storage provider bucket. # config.litestream.replica_key_id = litestream_credentials&.replica_key_id # - # - # Replica-specific secret key - # - # Litestream needs authentication credentials to access your storage provider bucket. - # + # Replica-specific secret key. Litestream needs authentication credentials to access your storage provider bucket. # config.litestream.replica_access_key = litestream_credentials&.replica_access_key # - # - # Replica-specific region - # - # Optionally specifies the bucket’s region. Only used for AWS S3 & Backblaze B2. - # + # Replica-specific region. Set the bucket’s region. Only used for AWS S3 & Backblaze B2. # config.litestream.replica_region = "us-east-1" # - # - # Replica-specific endpoint - # - # Optionally specifies the endpoint URL of the S3-compatible service. Only required for non-AWS services. - # + # Replica-specific endpoint. Set the endpoint URL of the S3-compatible service. Only required for non-AWS services. # config.litestream.replica_endpoint = "endpoint.your-objectstorage.com" # Configure the default Litestream config path - # # config.config_path = Rails.root.join("config", "litestream.yml") # Configure the Litestream dashboard # # Set the default base controller class - # # config.litestream.base_controller_class = "MyApplicationController" # # Set authentication credentials for Litestream dashboard - # # config.litestream.username = Rails.application.credentials.dig(:litestream, :username) # config.litestream.password = Rails.application.credentials.dig(:litestream, :password) end From da1c1bf12e1b5b53ed961271849a976a18b5ec36 Mon Sep 17 00:00:00 2001 From: Stephen Margheim Date: Tue, 10 Jun 2025 17:01:42 +0200 Subject: [PATCH 3/6] Fix one sentence --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4e81757..c5fcdeb 100644 --- a/README.md +++ b/README.md @@ -80,7 +80,7 @@ LITESTREAM_INSTALL_DIR=.bin You configure the Litestream executable through the [`config/litestream.yml` file](https://litestream.io/reference/config/), which is a standard Litestream configuration file as if Litestream was running in a traditional installation. -The gem streamlines the configuration process by providing a default configuration file for you. This configuration file will backup all SQLite databases defined in your `config/database.yml` file to one replication bucket. In order to ensure that no secrets are stored in plain-text in your repository, this configuration file leverages Litestream's support for environment variables. Inspect which environment variables are available run `litestream:env`. +The gem streamlines the configuration process by providing a default configuration file for you. This configuration file will backup all SQLite databases defined in your `config/database.yml` file to one replication bucket. In order to ensure that no secrets are stored in plain-text in your repository, this configuration file leverages Litestream's support for environment variables. Inspect which environment variables are available by running the `bin/rails litestream:env` command. The default configuration file looks like this if you only have one SQLite database: From 63e6f0083cb0ae23ac1c617e2bd9c989d88f39ff Mon Sep 17 00:00:00 2001 From: Stephen Margheim Date: Tue, 10 Jun 2025 17:04:51 +0200 Subject: [PATCH 4/6] Tweak wording for configuration section --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c5fcdeb..dfd40b4 100644 --- a/README.md +++ b/README.md @@ -110,11 +110,13 @@ Rails.application.configure do end ``` -However, if you need manual control over the Litestream configuration, you can manually edit the `config/litestream.yml` file. The full range of possible configurations are covered in Litestream's [configuration reference](https://litestream.io/reference/config/). Outside of configuring Litestream, you may also configure various other aspects of `litestream-ruby` itself. +Outside of configuring Litestream's replication, you may also configure various other aspects of `litestream-ruby` itself. ```ruby # config/initializers/litestream.rb Rails.application.configure do + # ... + # Base controller used for Litestream dashboard config.litestream.base_controller_class = "MyApplicationController" # Set the location of the Litestream config @@ -122,6 +124,8 @@ Rails.application.configure do end ``` +However, if you need manual control over the Litestream configuration, you can edit the `config/litestream.yml` file. The full range of possible configurations are covered in Litestream's [configuration reference](https://litestream.io/reference/config/). + ### Replication In order to stream changes to your configured replicas, you need to start the Litestream replication process. From fda9db5dcd45cd468685b1c140517670acbd40f8 Mon Sep 17 00:00:00 2001 From: Stephen Margheim Date: Tue, 10 Jun 2025 17:11:06 +0200 Subject: [PATCH 5/6] Tweak section on sync-interval --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index dfd40b4..07474bd 100644 --- a/README.md +++ b/README.md @@ -252,7 +252,7 @@ Litestream.verify! "storage/production.sqlite3" Litestream.verify!(replication_sleep: 10) "storage/production.sqlite3" ``` -In order to verify that the backup for that database is both restorable and fresh, the method will add a new row to that database under the `_litestream_verification` table, which it will create if needed. It will then wait `replication_sleep` seconds (defaults to 10) to give the Litestream utility time to replicate that change to whatever storage providers you have configured. Note that if you configure a longer `sync-interval`, you may need to adjust `replication_sleep` to a longer period. After that, it will download the latest backup from that storage provider and ensure that this verification row is present in the backup. If the verification row is _not_ present, the method will raise a `Litestream::VerificationFailure` exception. This check ensures that the restored database file: +In order to verify that the backup for that database is both restorable and fresh, the method will add a new row to that database under the `_litestream_verification` table, which it will create if needed. It will then wait `replication_sleep` seconds (defaults to 10) to give the Litestream utility time to replicate that change to whatever storage providers you have configured. After that, it will download the latest backup from that storage provider and ensure that this verification row is present in the backup. If the verification row is _not_ present, the method will raise a `Litestream::VerificationFailure` exception. This check ensures that the restored database file: 1. exists, 2. can be opened by SQLite, and @@ -260,6 +260,9 @@ In order to verify that the backup for that database is both restorable and fres After restoring the backup, the `Litestream.verify!` method will delete the restored database file. If you need the restored database file, use the `litestream:restore` rake task or `Litestream::Commands.restore` method instead. +> [!NOTE] +> If you configure Litestream's [`sync-interval`](https://litestream.io/reference/config/#replica-settings) to be longer than the default `replication_sleep` value of 10 seconds, you will need to adjust `replication_sleep` to a value larger than `sync-interval`; otherwise, `Litestream.verify!` may appear to fail where it actually simply didn't wait long enough for replication. + ### Dashboard The gem provides a web dashboard for monitoring the status of your Litestream replication. To mount the dashboard in your Rails application, add the following to your `config/routes.rb` file: From 9417c7960ae2fd18c905b71cfd1569f5ed081937 Mon Sep 17 00:00:00 2001 From: Stephen Margheim Date: Tue, 10 Jun 2025 17:12:20 +0200 Subject: [PATCH 6/6] Reuse litestream_credentials value for username and password --- lib/litestream/generators/litestream/templates/initializer.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/litestream/generators/litestream/templates/initializer.rb b/lib/litestream/generators/litestream/templates/initializer.rb index 46b885a..944607c 100644 --- a/lib/litestream/generators/litestream/templates/initializer.rb +++ b/lib/litestream/generators/litestream/templates/initializer.rb @@ -40,6 +40,6 @@ # config.litestream.base_controller_class = "MyApplicationController" # # Set authentication credentials for Litestream dashboard - # config.litestream.username = Rails.application.credentials.dig(:litestream, :username) - # config.litestream.password = Rails.application.credentials.dig(:litestream, :password) + # config.litestream.username = litestream_credentials&.username + # config.litestream.password = litestream_credentials&.password end