store id-mappings to resume partial migration

[senny]

while running a migration we have access to the "old <---> new" id-mapping table.

mapping 'Country' do |context, value|
  {:country_id => context.definition('Countries').new_id_of(value)}
end

It would be cool if these mappings were persisted to disk after the migration process. That way it would be possible to reuse a partially migrated state and continue migrating missing parts. This would also impact the dependency system since it needs to know, which definitions are already migrated.

[stmichael]

I started by collecting some ideas. Please take a look at them and tell me your opinion.

• all id mappings will be loaded before any definition has been run
• as soon as a definition has been run its id mappings will be stored to disk. This ensures that definitions that have been finished before a possible import failure will be completely persisted and won't have to be run again in the next attempt of the import.
• if you run the complete import all id mappings will be erased before the importer starts
• if you run a partial import only the id mappings of the definitions specified in the option :only will be erased
• in partial imports we also need a mechanism to truncate tables that we'd like to reimport

We should split these problems into two issues: the first one deals with the loading and storage of the id mappings and the second one deals with the partial import. Else this issue will get too big. So actually only the first three points are relevant for this issue.

[senny]

So the implementation of this ticket will allow us to do partial migrations only on tables, which have not been imported before and a future ticket will allow the re-import of existing tables?

Your ideas sound reasonable. I think we should make the act of partially migrating a separate "thing", which is not only decided by the :only option. Perhaps we can apply some kind of MigrationStrategy which could be either PartialMigration or FullMigration. The strategy can then execute the tasks above: erasing the right mapping-tables and so on.