Checking for pending migrations

[erez-rabih]

Is there a way to know if there are pending migrations which were not applied to the DB?

My purpose it to avoid running code which includes migrations that were not applied.
I'd like to run this task before deploying code to ensure that.

[weavejester]

It depends how you're storing your pending migrations.

You can get the applied migrations with ragtime.core/applied-migrations or ragtime.protocols/applied-migration-ids. You can then compare this to your list of migrations to find those that aren't applied.

If you're storing SQL migrations in a directory, for example, you can use ragtime.jdbc/load-directory to get a list of migrations. So for example:

(set/difference
 (set (map :id (jdbc/load-directory "migrations")))
 (set (p/applied-migration-ids (jdbc/sql-database db-spec))))

[erez-rabih]

Currently we run the following migrations on migrate: available (from filesystem) - applied (from db table)
If we have that list available, we can use count on it, and if the result is larger than 0 it means we have pending migrations (available but not applied)
I just couldn't find where this value is calculated.

[weavejester]

So does applied-migration-ids answer your question?

[erez-rabih]

It does, partially
I need to calculate all-migrations - applied-migrations
How do I get all-migrations?

[weavejester]

As I mentioned in my earlier comment, it depends on how you're storing your migrations. If you're storing them in a directory for example, you can use load-directory. If you're storing them as resources, you can use load-resources. If you're using some custom method, then it's up to you.

[erez-rabih]

Yes I just tested it and it works well
I was wondering - would you accept a PR to add this feature to the library? I mean it is a common use case to fail a deployment if there are unapplied pending migrations...

[weavejester]

Is it a common case? It's my experience that migrations are usually handled as part of deployment.

[erez-rabih]

It depends how you want to handle DB migrations:
I think it is best practice to run the migrations explicitly rather than running them automatically as part of the deployment to avoid running migrations in mistake.
If you go by this advice then running migrations must be explicit and might be forgotten. Then you end up with a code version that does not match the DB schema.
To avoid that the strategy is failing code deployments unless all migrations were applied.
This way you gain both advantages:

1. Explicit migrations
2. No code <-> DB schema mismatch

Ruby on Rails, for example, would fail some tasks if there are unapplied pending migrations.

[weavejester]

I think it is best practice to run the migrations explicitly rather than running them automatically as part of the deployment to avoid running migrations in mistake.

Why would it be a mistake to run migrations if the code requires them? If you're uncertain about your migrations, then test them in a staging environment first.

That said, I don't see any problem in theory with a function like:

(defn unapplied-migrations [store migrations]
  "Return a collection of migrations that have not been applied to the store."
  (set/difference (set (map :id migrations))
                  (set (p/applied-migration-ids store))))

In ragtime.core. Then (seq (unapplied-migrations ...)) or (empty? (unapplied-migrations ...)) could be used.

[erez-rabih]

Why would it be a mistake to run migrations if the code requires them? If you're uncertain about your migrations, then test them in a staging environment first.

As I see it, migrations are more dangerous than code deployments and running them in staging isn't always a good solution:

1. Some migrations might cause heavy DB load or even lock certain tables and you might want to run them at pre-defined maintenance windows. This kind of errors may not be found on staging where the amount of data and traffic does not match production.
2. Migrations that cause data corruption are far more dangerous then bad code deployments which might be reverted in a matter of seconds

I've seen a few cases in which a migration that accidentally "slipped" into a production deployment caused a service disruption or even downtime just because the developer forgot he even had one in the code.
Of course you could argue that the deployment itself should have been managed but in today's reality where developers deploy dozens of versions of different services a day it is very easy to forget about migrations.

To the PR point: I think you pretty much wrote all the needed code. Would you like me to add it, test it and open a PR?

[weavejester]

As I see it, migrations are more dangerous than code deployments and running them in staging isn't always a good solution

So you're advocating running migrations manually as a "double-check"? I think I'd consider working that into the CI system - for example, rejecting any update to the migrations without approval from another developer.

To the PR point: I think you pretty much wrote all the needed code. Would you like me to add it, test it and open a PR?

Sure. I haven't tested it manually or written a unit test for it. I've written a few contributing guidelines which I'm in the middle of adding to my repositories.