Improve database management documentation (#593)

* Improve database management documentation

* Link to Getting Started subsection

* Untwist description of change need

* Don't say please

* Add folder names to EDD steps

* Restore reference to finalization folders since that's been completed

* Additional content changes suggested from a prior PR

* Fix an oversight from #596

* Fix bungled merge from the wrong ref

Author: withinfocus

docs/contributing/database-migrations/_category_.yml

@@ -0,0 +1,2 @@
+label: "Database Migrations"
+position: 2

docs/contributing/database-migrations/edd.mdx

@@ -1,3 +1,7 @@
+---
+sidebar_position: 3
+---
+
 import Tabs from "@theme/Tabs";
 import TabItem from "@theme/TabItem";
 
@@ -130,6 +134,8 @@ actions.
 <Tabs>
 <TabItem value="first" label="Initial Migration" default>
 
+Located in `util/Migrator/DbScripts`.
+
 ```sql
 -- Add Column
 IF COL_LENGTH('[dbo].[Customer]', 'FirstName') IS NULL
@@ -177,6 +183,8 @@ END
 </TabItem>
 <TabItem value="data" label="Transition Migration">
 
+Located in `util/Migrator/DbScripts_transition`.
+
 ```sql
 UPDATE [dbo].Customer SET
     FirstName=FName
@@ -186,6 +194,8 @@ WHERE FirstName IS NULL
 </TabItem>
 <TabItem value="second" label="Finalization Migration">
 
+Located in `util/Migrator/DbScripts_finalization`.
+
 ```sql
 -- Remove Column
 IF COL_LENGTH('[dbo].[Customer]', 'FName') IS NOT NULL
@@ -239,7 +249,7 @@ There are some important constraints to the implementation of the process:
 
 The process to support all of these constraints is a complex one. Below is an image of a state
 machine that will hopefully help visualize the process and what it supports. It assumes that all
-database changes follow the standards that are laid out in [Migrations](./).
+database changes follow the standards that are laid out in [Migrations](./mssql.md).
 
 ---
 

docs/contributing/database-migrations/ef.md

@@ -0,0 +1,31 @@
+---
+sidebar_position: 2
+---
+
+# Entity Framework
+
+:::info
+
+For instructions on how to apply database migrations, please refer to the
+[Getting Started](../../getting-started/server/database/ef/index.mdx#updating-the-database)
+documentation.
+
+:::
+
+If you alter the database schema, you must create an EF migration script to ensure that EF databases
+keep pace with these changes. Developers must do this and include the migrations with their PR.
+
+To create these scripts, you must first update your data model in `Core/Entities` as desired. This
+will be used to generate the migrations for each of our EF targets. Additionally, for table changes
+it is strongly recommended to define or update an `IEntityTypeConfiguration<T>` to accurately
+represent any constraints needed on the data model.
+
+Once the model is updated, navigate to the `dev` directory in the `server` repo and execute the
+`ef_migrate.ps1` PowerShell command. You should provide a name for the migration as the only
+parameter:
+
+```bash
+pwsh ef_migrate.ps1 [NAME_OF_MIGRATION]
+```
+
+This will generate the migrations, which should then be included in your PR.

docs/contributing/database-migrations/index.md

@@ -0,0 +1,98 @@
+---
+sidebar_position: 1
+---
+
+# MSSQL
+
+:::info
+
+For instructions on how to apply database migrations, please refer to the
+[Getting Started](../../getting-started/server/database/mssql/index.md#updating-the-database)
+documentation.
+
+:::
+
+:::tip
+
+We recommend reading [T-SQL Code Style](../code-style/sql.md) since it has a major impact in how we
+write migrations.
+
+:::
+
+## SQL database project
+
+We use a
+[SDK-style SQL project](https://learn.microsoft.com/en-us/sql/azure-data-studio/extensions/sql-database-project-extension-sdk-style-projects?view=sql-server-ver16)
+(`.sqlproj`) to develop the database locally -- this means we have an up-to-date representation of
+the database in `src/Sql`, and any modifications need to be represented there as well. SDK-style SQL
+projects are available in Visual Studio Code that provides schema comparison and more. You may also
+modify the `.sql` files directly with any text editor.
+
+To make a database change, start by modifying the `.sql` files in `src/Sql/dbo`. These changes will
+also need to be applied in a migration script. Migration scripts are located in
+`util/Migrator/DbScripts`.
+
+You can either generate the migration scripts automatically using the _Schema Comparison_
+functionality or by manually writing them. Do note that the automatic method will only take you so
+far and it will need to be manually edited to adhere to the code styles.
+
+For added safeguards we have automated linting and validation to ensure the SQL project is always up
+to date with the migrations.
+
+The separate database definitions in `src/Sql/.../dbo` serve as a "master" reference for the
+intended and final state of the database at that time. This is crucial because the state of database
+definitions at the current moment may differ from when a migration was added in the past. These
+definitions act as a lint and validation step to ensure that migrations work as expected, and the
+separation helps maintain clarity and accuracy in database schema management and synchronization
+processes.
+
+## Modifying the database
+
+In accordance with the tenets of [Evolutionary Database Design](./edd.mdx) every change must be
+considered as split into two parts:
+
+1. A backwards-compatible transition migration
+2. A non-backwards-compatible final migration
+
+Most changes are entirely backwards-compatible in their final form. If this is the case, only one
+phase of changes is required. With the use of beta testing, partial roll-outs,
+[feature flags](../feature-flags.md), etc. the often-chosen path is to spread a change across
+several major releases with a calculated future state that can perform a "cleanup" migration that is
+backwards-compatible but still represents an overall-_incompatible_ change beyond the boundaries of
+what we need for individual release safety.
+
+### Backwards compatible migration
+
+1. Modify the source `.sql` files in `src/Sql/dbo`.
+2. Write a migration script, and place it in `util/Migrator/DbScripts`. Each script must be prefixed
+   with the current date.
+
+Tips to ensure backwards compatibility:
+
+- any existing stored procedure accepts the same input parameters and that new parameters have
+  nullable defaults
+- when a column is renamed the existing stored procedures first check (coalesce) the new location
+  before falling back to the old location
+- continued updating of the old data columns since in case of a rollback no data should be lost
+
+### Non-backwards compatible migration
+
+These changes should be written from the perspective of "all data has been migrated" and any old
+stored procedures that were kept around for backwards compatibility should be removed. Any logic for
+syncing old and new data should also be removed in this step.
+
+Since the `dbo` schema represents the current state we need to introduce a "future" state that we
+will call `dbo_finalization`.
+
+1. Copy the relevant `.sql` files from `src/Sql/dbo` to `src/Sql/dbo_finalization`.
+2. Remove the backwards compatibility logic that is no longer needed e.g. dual reads and writes to
+   columns.
+3. Write a new Migration and place it in `src/Migrator/DbScripts_finalization`. Name it
+   `YYYY-0M-FinalizationMigration.sql`.
+   - Typically migrations are designed to be run in sequence. However since the migrations in
+     `DbScripts_finalization` can be run out of order, care must be taken to ensure they remain
+     compatible with the changes to `DbScripts`. In order to achieve this we only keep a single
+     migration, which executes all backwards incompatible schema changes.
+
+Upon execution any finalization scripts will be [automatically moved](./edd.mdx#online-environments)
+for proper history.

docs/contributing/database-migrations/mssql.md

@@ -263,3 +263,8 @@ existing databases if you have not already. If these settings are not present at
 With connection strings applied to your projects: ensure your databases are all migrated using
 `pwsh server/dev/migrate.ps1 --all`. Then you can run EF tests from the
 `test/Infrastructure.IntegrationTest` folder using `dotnet test`.
+
+# Modifying the database
+
+The process for modifying the database is described in
+[Migrations](./../../../../contributing/database-migrations/ef.md).

docs/getting-started/server/database/ef/index.mdx

@@ -23,4 +23,4 @@ of your database.
 ## Modifying the database
 
 The process for modifying the database is described in
-[Migrations](./../../../../contributing/database-migrations/).
+[Migrations](./../../../../contributing/database-migrations/mssql.md).