Create and initialize the database if it does not exist #55

tankyleo · 2025-08-15T07:39:59Z

    Create and initialize the database if it does not exist

    Also implement a versioning and migration scheme for future updates to
    the schema. It is an adaptation of the scheme used in CLN.

and

    Let the test suite setup the database

ldk-reviews-bot · 2025-08-15T07:40:02Z

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

tnull

Took a first look and left some higher-level comments. Have yet to review the actual database statements and operations.

rust/impls/Cargo.toml

rust/impls/src/postgres_store.rs

rust/server/src/main.rs

tankyleo · 2025-08-15T16:55:54Z

A possible todo I thought of: right now our TABLE_CHECK_STMT SELECT 1 FROM vss_db WHERE false merely checks whether a vss_db table exists in the database. We could have a deeper check of all the columns in the table etc...

tnull · 2025-08-18T12:40:32Z

rust/impls/src/postgres_store.rs

+	pub async fn new(postgres_endpoint: &str, db_name: &str, init_db: bool) -> Result<Self, Error> {
+		if init_db {
+			tokio::time::timeout(
+				tokio::time::Duration::from_secs(3),


Can we extract the timeout value(s) into appropriately named consts?

Yes thank you !

tnull · 2025-08-18T12:41:42Z

rust/impls/src/postgres_store.rs

+		Ok(())
+	}
+
+	async fn check_health(&self) -> Result<(), Error> {


Do we really need this? I think we should be able to trust that the CREATE TABLE statement does the right thing if it doesn't error out?

This mostly useful if the user does not set --init-db; in that case we want to make sure we can establish a connection to the database as part of startup checks before entering the main loop ?

If this passes, then we can log something like "Connected to PostgreSQL backend" in main as you suggested.

tnull · 2025-08-18T12:44:05Z

rust/server/src/main.rs

 		std::process::exit(1);
 	}
+	let init_db = args.iter().any(|arg| arg == "--init-db");


Why is this an optional flag to begin with?

When the flag is not set, I want the startup sequence to abort if the database is not present.

When the flag is set, I want the startup sequence to create the database if it is not present.

Right, just wondering when we'd ever want the first case?

If the intention is that we can initiate a VSS server instance even if we don't have / can establish a connection to the Postgres backend (which I don't fully buy), then we'd need to take additional steps. AFAIU, Pool::build(), which we call ~just after the init_db block would just as well fail if we can't connect to the pool:

"The Pool will not be returned until it has established its configured minimum number of connections, or it times out." (https://docs.rs/bb8/0.9.0/bb8/struct.Builder.html#method.build)

Right, just wondering when we'd ever want the first case?

If the user does not intend to initialize a new database (which is most cases), I don't think we should silently create a new database if we don't find the one we are looking for. Instead, I think we should complain loudly, and make sure the user actually intends to create a new one.

If the intention is that we can initiate a VSS server instance even if we don't have / can establish a connection to the Postgres backend (which I don't fully buy), then we'd need to take additional steps. AFAIU, Pool::build(), which we call ~just after the init_db block would just as well fail if we can't connect to the pool:

Actually Pool::build returns Ok(()) even if it cannot establish a connection to the Postgres backend - see #52 (comment) . On the main branch, we then enter the main loop even though we have established no connections to the Postgres backend.

If the user does not intend to initialize a new database (which is most cases), I don't think we should silently create a new database if we don't find the one we are looking for. Instead, I think we should complain loudly, and make sure the user actually intends to create a new one.

Hmm, I don't know. A binary that needs a special flag to init just on first start is a bit weird tbh. I can't think of another daemon that works that way.

Actually Pool::build returns Ok(()) even if it cannot establish a connection to the Postgres backend - see #52 (comment) . On the main branch, we then enter the main loop even though we have established no connections to the Postgres backend.

Huh, this API is very unexpected then. You'd expect it to return a TimedOut error if the connection attempts failed, just as get does. Might even be good to add a comment there in our code.

Huh, this API is very unexpected then. You'd expect it to return a TimedOut error if the connection attempts failed, just as get does. Might even be good to add a comment there in our code.

Sounds good will add the comment. Agreed this is unexpected.

Hmm, I don't know. A binary that needs a special flag to init just on first start is a bit weird tbh. I can't think of another daemon that works that way.

I agree it is weird. FWIW, CLN, LDK-node automatically generates a new wallet, but LND requires you to explicitly run lncli createwallet instead of lncli unlock (lncli unlock aborts startup if the wallet does not exist).

If we always create a new database if it does not exist, when a user misconfigures the DB connection string to an existing database, we'll create a new database and complete startup, when instead we should(?) have aborted startup. See #52 (comment)

cc @TheBlueMatt

I don't have a super strong feeling. I've seen it done both ways, though it is generally the case for binaries like this that they'll auto-create the table by default, I can see why that might be surprising.

tnull · 2025-08-18T12:45:15Z

rust/server/src/main.rs

 				.await
 				.unwrap(),
 		);
+		println!("Loaded postgres!");


If I read that message I'd be a bit confused what's meant exactly. Maybe "Connected to PostgreSQL backend with address .."?

Yes that's clearer I agree thank you

tnull · 2025-08-18T12:45:41Z

rust/server/src/main.rs

 		let rest_svc_listener =
 			TcpListener::bind(&addr).await.expect("Failed to bind listening port");
+		println!("Bound to {}", addr);


Maybe "Listening for incoming connections on .."

rust/server/src/util/config.rs

tnull

Let me know when this is ready for another round of review.

rust/server/src/util/config.rs

rust/impls/src/postgres_store.rs

tankyleo · 2025-08-19T14:53:14Z

Let me know when this is ready for another round of review.

Thanks for the review @tnull I've responded to your comments above would be good to see your responses before I write more code :)

TheBlueMatt · 2025-08-20T17:54:39Z

rust/impls/src/postgres_store.rs

+const DB_INIT_CMD: &str = "CREATE DATABASE";
+const TABLE_CHECK_STMT: &str = "SELECT 1 FROM vss_db WHERE false";
+const TABLE_INIT_STMT: &str = "
+	CREATE TABLE IF NOT EXISTS vss_db (


In addition to the main table itself, we really need a "config" table that at least just stores the version of the schema we're currently using. That way we can easily check that the schema in use is compatible with this version and, in the future, we can upgrade the schema as needed.

I think in Postgres migrations are usually done by sequentially applying the corresponding XXX_YYY.sql files. I agree it's probably good to track what version was last applied in a separate schema_version table or similar.

However, this probably also highlights that it would make sense to move the SQL statements to a dedicated migrations.rs, and rename TABLE_INIT_STMT in accordance to the .sql file to const V0_CREATE_VSS_DB, essentially to prepare for the eventual V1_.., V2_, .. migrations to follow.

FWIW, we could even consider to read in the sql files on startup. If we don't want to do that, we could consider dropping the sql files and just have the migrations.rs to avoid the risk of having them get out-of-sync at some point.

Also implement a versioning and migration scheme for future updates to the schema. It is an adaptation of the scheme used in CLN.

ldk-reviews-bot · 2025-08-30T00:01:02Z

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-09-01T00:01:45Z

🔔 2nd Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull

Some minor comments, but basically LGTM

tnull · 2025-09-01T08:30:49Z

rust/impls/src/postgres_store.rs

+const UPDATE_VERSION_STMT: &str = "UPDATE vss_db_version SET db_version=$1;";
+const LOG_MIGRATION_STMT: &str = "INSERT INTO vss_db_upgrades VALUES($1);";
+
+const MIGRATIONS: &[&str] = &[


Mind adding a comment here that these are meant to be append-only (i.e., should never been changed in-place) and need to be run in order, and only the needed migration should be run?

Also, would it make sense to split out the migrations and potential test code into a migrations.rs?

tnull · 2025-09-01T08:36:00Z

rust/impls/src/postgres_store.rs

+const MIGRATIONS: &[&str] = &[
+	"CREATE TABLE vss_db_version (db_version INTEGER);",
+	"INSERT INTO vss_db_version VALUES(1);",
+	"CREATE TABLE vss_db_upgrades (upgrade_from INTEGER);",


Could you expand on what would be stored in the vss_db_upgrades table? Why do we need it in addition to vss_db_version?

tnull · 2025-09-01T08:54:31Z

rust/impls/src/postgres_store.rs

+		Ok(postgres_backend)
+	}
+
+	async fn migrate_vss_database(&self) -> Result<(), Error> {


Would be cool to get some test coverage for this migration logic.

tankyleo requested a review from tnull August 15, 2025 07:42

tnull reviewed Aug 15, 2025

View reviewed changes

rust/impls/Cargo.toml Show resolved Hide resolved

rust/impls/src/postgres_store.rs Outdated Show resolved Hide resolved

rust/impls/src/postgres_store.rs Outdated Show resolved Hide resolved

rust/server/src/main.rs Outdated Show resolved Hide resolved

tankyleo force-pushed the init branch from 08ed0fc to 837b300 Compare August 15, 2025 17:03

tnull self-requested a review August 18, 2025 11:15

tnull reviewed Aug 18, 2025

View reviewed changes

tnull reviewed Aug 19, 2025

View reviewed changes

rust/server/src/util/config.rs Outdated Show resolved Hide resolved

rust/impls/src/postgres_store.rs Outdated Show resolved Hide resolved

TheBlueMatt reviewed Aug 20, 2025

View reviewed changes

tankyleo force-pushed the init branch from 054c880 to 7396c56 Compare August 27, 2025 20:59

tankyleo added 2 commits August 27, 2025 22:27

Create and initialize the database if it does not exist

d06d670

Also implement a versioning and migration scheme for future updates to the schema. It is an adaptation of the scheme used in CLN.

Let the test suite setup the database

eace62d

tankyleo force-pushed the init branch from 7396c56 to eace62d Compare August 27, 2025 22:28

tankyleo changed the title ~~init db and table, check health~~ Create and initialize the database if it does not exist Aug 27, 2025

tankyleo requested a review from tnull August 27, 2025 22:46

tankyleo added this to Weekly Goals Aug 28, 2025

tankyleo moved this to Goal: Merge in Weekly Goals Aug 28, 2025

tankyleo self-assigned this Aug 28, 2025

tnull reviewed Sep 1, 2025

View reviewed changes

Create and initialize the database if it does not exist #55

Are you sure you want to change the base?

Create and initialize the database if it does not exist #55

Uh oh!

Conversation

tankyleo commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldk-reviews-bot commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tankyleo commented Aug 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tankyleo Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tankyleo Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tankyleo commented Aug 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tnull Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldk-reviews-bot commented Aug 30, 2025

Uh oh!

ldk-reviews-bot commented Sep 1, 2025

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

tankyleo commented Aug 15, 2025 •

edited

Loading

ldk-reviews-bot commented Aug 15, 2025 •

edited

Loading

tankyleo Aug 19, 2025 •

edited

Loading

tankyleo Aug 20, 2025 •

edited

Loading

tnull Aug 21, 2025 •

edited

Loading