diff --git a/doc/operator-guide/schema-migration-process.rst b/doc/operator-guide/schema-migration-process.rst index 1b85bc78..dba19398 100644 --- a/doc/operator-guide/schema-migration-process.rst +++ b/doc/operator-guide/schema-migration-process.rst @@ -2,56 +2,74 @@ Schema Migration Process ######################## -Add columns to sdm_schemas -========================== +Gather information +================== -First, add the requested database additions, justifications, and where they are generated to our `confluence entry table `__. -Then, create a ticket and edit the `repository `__ to apply your schema changes to any of the ``cdb_*.yml`` schemas. +Gather the following information in a Jira ticket: -If your sdm_schemas PR has issues, check that the schema conforms to Felis's data model and valid SQL tables can be created with `felis validate/create `__ -Alembic migrations should be automatically created by a git workflow after your sdm_schemas pull request completes. +- Names of columns to be added +- Source of information for the columns, which will help determine the tables to which the columns will be added +- If possible, descriptions, units, and UCDs for the columns, ideally via a PR to sdm_schemas +Each table in the ConsDB schema should be filled by a single data source (e.g. the ``hinfo`` service at the Summit or Rapid Analysis). +Each data source can write to multiple tables. +The name of each table can be related to the name of the data source, but it's best for it to be something meaningful to science users. -Create an Alembic Migration (manually) -====================================== +Get approvals +============= -`Alembic `__ keeps track of versioning by autogenerated migrations to sync the test stands and summit databases. -Versioning our database schema changes allows us to apply edits and move the database’s state forward or backward as needed. +- Get the ``sdm_schemas`` PR approved as to syntax, column naming, and description content by a member of the Data Engineering team. +- Get the ``sdm_schemas`` PR approved as to usefulness and non-redundancy by the ConsDB Product Owner or designate. +- Get the ``sdm_schemas`` PR approved as to table name, agreement with data source, and maintainability by a member of the ConsDB team. -1. Create an Alembic migration on your ConsDB ticket branch. -2. Use the script ``consdb/alembic-autogenerate.py`` to generate Alembic migrations. -3. Follow the directions in the header of the script, then run ``python alembic-autogenerate.py`` to create version files in respective database-named directories in ``consdb/alembic/``. -4. Manually edit the generated files in ``consdb/alembic//`` to: +If the data source is at the USDF (currently only the Transformed EFD), follow the process in Transformed EFD Contributor Guide (transformed-efd.rst). +If the data source is at the Summit (currently the Header Service or Rapid Analysis), follow this process. - - Remove the ``visit1`` and ``ccdvisit1`` views. - - Ensure constraints and renamed columns are correct. +Create the schema migration in lsst-dm/consdb +============================================= -Test alembic migration -====================== -Before merging your ConsDB migration PR or applying this migration to the Summit, you must test applying the migration in a test environment. -Test both applying the migration and any code that populates the new columns/tables at TTS/BTS if Summit schema is changing. +If there is more than one ``sdm_schema`` PR being requested, you can rebase them onto each other such that the last one contains all previous ones to simplify deployment (but you can also do each one as a separate migration). +Build an Alembic migration based on the desired PR branch. +The "Generate migration scripts" GitHub Action in ``lsst-dm/consdb`` can be used to generate this. +There are some manual steps that need to occur: -1. Update the deployment on the test stand: -------------------------------------------- +- Remove view drop/creates from both upgrade and downgrade if the ``exposure``/``ccdexposure`` tables are not being modified. On the other hand, make sure they're present if those tables are being modified. +- Ensure constraints and renamed columns are correct. +- Remove any migrations for schemas that are not actually changing. -1. Choose the appropriate test stand (TTS, BTS) -2. Create a branch in ``phalanx`` and edit the corresponding test stand environment file ``phalanx/applications/consdb/values-.yaml`` to point to your branch's built docker image (tickets-DM-###). -3. Coordinate and announce in the appropriate slack channel that you will begin testing your migrations. -4. Update the ConsDB deployment in ``/argo-cd`` to use your ``phalanx`` branch in the ``Target Revision``. Refresh and check pod logs. -5. Verify the tables that you will be upgrading exist using ``psql`` -6. From the ``consdb/`` directory, (where ``alembic.ini`` file is) use the alembic commands to upgrade the existing database tables: ``alembic upgrade head -n `` -7. Deploy new ConsDB software (``hinfo``, ``pqserver``) and check the initial logs. +Apply and test the migration at BTS +=================================== -2. Test with LATISS imaging in ATQueue: ---------------------------------------- +If the data source is at the Summit, apply the migration at the Base Test Stand (BTS) using this procedure: + +1. Verify the tables that you will be upgrading exist using ``psql`` or ``pgcli``. +2. From the ``consdb/`` directory, (where the ``alembic.ini`` file is) use the alembic commands to upgrade the existing database tables: ``alembic upgrade head -n `` + +If the data sources is at USDF, apply the migration to the ``dev`` database and test with the ``usdf-rsp-int`` environment. + +If you need to deploy a new ``hinfo`` container to write to the new schema: + +1. Create a branch in ``phalanx`` and edit the corresponding test stand environment file ``phalanx/applications/consdb/values-.yaml`` to point to your branch's built docker image (tickets-DM-###). +2. Coordinate and announce in the appropriate Slack channel that you will begin testing your migrations. +3. Update the ConsDB deployment in ``/argo-cd`` to use your ``phalanx`` branch in the ``Target Revision``. Refresh and check pod logs. + +You will need to restart the ``pqserver`` deployment in order to pick up the new schema. + +Test: + +- Hand-check the schema to make sure it looks correct. +- If ``hinfo`` changed, take an image and check that the right data is written to the database. +- For other data sources, try to do an integrated test with a new version of that data source. See `TTS Start Guide `__ for guidelines on using the test stands. -Access LOVE via ``/love`` and use the 1Password admin information to sign in, or your SLAC username and password. -Navigate to the ATQueue or Auxillary Telescope (AuxTel) Script Queue. +Access LOVE via ``/love`` and use the 1Password admin information to sign in, or your Summit username and password. +Navigate to the ATQueue or Auxiliary Telescope (AuxTel) Script Queue. + +.. note:: -- Before editing these scripts, note their starting configurations, as we will return the scripts to this configuration when we are done. + Before editing these scripts, note their starting configurations, as we will return the scripts to this configuration when we are done. Take a test/simulated picture with LATISS through the ATQueue using these three scripts: @@ -65,17 +83,39 @@ Address any errors and retest. Check the database by using ``psql`` commands like ``\dt`` to display the table names and maybe even ``SELECT * from cdb_latiss.exposure where day_obs == ;`` to view the most recent data. -Run set_summary_state to set ATHeaderService and ATCamera back to STANDBY, and return LATISS back to STANDBY. +Run ``set_summary_state`` to set ATHeaderService and ATCamera back to STANDBY, and return LATISS back to STANDBY. Then return these three scripts to their original configurations. If you have encountered errors in this process, do not proceed to the summit, but address those errors and retest them with your ``phalanx`` branch pointing to your ConsDB branch with the fix to these errors. -If tests are successful, create a pull request for the Alembic migration in ConsDB. Tag the release according to ``standards-practices`` guidelines. +If tests are successful, merge the ConsDB PR containing the Alembic migration. +Tag the release according to ``standards-practices`` guidelines. Update your existing ``phalanx`` branch to point the environment based deployments to this ConsDB tag. -You are able to retest on the test stand at this point, hopefully there were no changes to your ConsDB pull request so this step is trivial. +You are able to retest on the test stand at this point; hopefully there were no changes to your ConsDB pull request so this step is trivial. + +Migrate the Summit and update logical replication +================================================= + +Follow the process in https://df-ops.lsst.io/usdf-applications/qa/summit-db-replica/procedures.html#schema-updates-consdb to update the Summit and USDF databases. + +Restart pqserver +================ + +Restart ``pqserver`` in all environments to pick up the new schema. + +The new schema is now visible in the database and ``pqserver`` and can start to be filled in for new exposures as well as back-filled as needed. + +Update TAP Schema +================= + +Make sure that a new version of the TAP Schema container for the ``consdbtap`` service has been generated. +Deploy that to all environments. + +The new schema is now visible for user queries. -Deploy migration in synchrony at Summit (if necessary), USDF, and Prompt Release (if necessary) ------------------------------------------------------------------------------------------------ +Merge the sdm_schemas PR +======================== -See deployment page for specific environment deployment steps \ No newline at end of file +The final step is to merge the PR in ``sdm_schemas``. +This makes the new schema visible in the schema browser. diff --git a/setup.cfg b/setup.cfg index c75f99c1..3805fde2 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,7 +1,7 @@ [flake8] max-line-length = 110 max-doc-length = 79 -ignore = E133, E203, E226, E228, N802, N803, N806, N812, N813, N815, N816, W503 +ignore = E133, E203, E226, E228, F824, N802, N803, N806, N812, N813, N815, N816, W503 exclude = bin, doc,