Add script for compression + pickle deletion

[KaspariK]

Script for migrating Tron's DynamoDB state store to gzip-compressed binary. I've added 3 commands:

• status: Scans the table and reports how many keys need migration.
• compress: Gzip-compresses items/partitions using TransactWriteItems. Initially this was slow enough that I wanted to do it while Tron was running so I added some ConditionExpression guards to help prevent conflicting writes. I've since sped this up quite a bit so if we really want we can take Tron down while this runs and it shouldn't be too terrible, though the most heavily written jobs are already migrated so this is mostly just historic runs/less frequent jobs.
• delete-pickles: Removes pickle data (val, num_partitions). This will run once we've stopped writing pickles in #TODO

Testing

• Dev: https://fluffy.yelpcorp.com/i/gnnHl4fLZMwDNcGlCJpkZHCpQXtDjMX6.html
• Prod: https://fluffy.yelpcorp.com/i/J1B6tKLhsK4m9p448Hr26STtqqSf6fzz.html
• 8 workers seems like a sweet spot. I think this could be cut down a bit more by skipping some of the json validation but I'm not trying to set records or anything 🤷‍♂️.
• Peaks at around 30% of our write capacity, which is totally fine with our overall write usage.
• The failed keys are from throttling (the usual suspects). These actually throttle on literally the first Transaction. Since the write and deletes (also technically a write) are in the same transaction we are making enough traffic to hit the hot key issue. I can either manually convert these using --keys ahead of time a few at a time, or do so after. I don't think it makes a difference.
• Restarts before and after succeeded with no issues.
• Skipped keys are old failed deletes. Harmless cruft.

General plan:

• Compress
  • Via this script after it's merged
• Remove pickle reads (doing this first keeps things more revert safe since worst case we'd still be writing new runs)
  • U/kkasp/unlearn how to read pickles #1096
• Remove pickle writes
  • TODO
• Remove pickles
  • Via this script after it's merged

[nemacysts]

(i'm assuming this is mostly temporary, and the general skeleton looks fine to me!)

[nemacysts]

i was gonna ask if we should grab this from tron/serialize/runstate/dynamodb_state_store.py - but i assume we'll delete this once we're done so it doesn't really matter?

[KaspariK]

Yeah pretty much