Skip to content

Restore not working / dump file seems to be inconsistent #1

@ThomasAlxDmy

Description

@ThomasAlxDmy

Hey There!

I started to mess around with ETCD clusters few weeks ago, and I was looking for a good dump/restore tool for it. Yesterday I tried etcdump and I discover 2 (different?) issues.

  1. After a restore, the number of keys restored is completely random.
  2. No directories are being restored.

More information:
A) I created a simple script that populate an empty ETCD cluster with 10 000 keys and 10 000 empty directories at the root. Structure looks like that: https://gist.github.com/ThomasAlxDmy/43ec114953cbc1b600f4. Notice the format "key": "/message*" for regular keys and "key": "/message_dir*" for directories.

B) Then I created a dump file with the command etcdump --file /mnt/etcd/dump/etcdump.json --config /mnt/etcd/dump/etcdump-config.json dump where /mnt/etcd/dump/etcdump-config.json = https://gist.github.com/ThomasAlxDmy/791d0270228da563ef3e

The dump file contains 20 000 keys (correct number) and look like that: https://gist.github.com/ThomasAlxDmy/132a041410159439410c

The first thing that shocked me was the format of the directory keys:
{ "key": "/message_dir4524" }
Looks like it's missing "dir":true. I don't know if it's a mistake or by design but it would explain why no directory are created.

C) I removed all the keys in the cluster.
D) Restored the keys with the following command: etcdump --file /mnt/etcd/dump/etcdump.json --config /mnt/etcd/dump/etcdump-config.json restore And got the messagesuccessfuly restored`

E) I re-checked the data in the ETCD cluster and only got 0 directories and 1815 unique keys

I restarted those step several times and got different numbers for the last step but always 0 directories:

  • 0 directories and 3434 unique keys
  • 0 directories and 1159 unique keys

I dig into my cluster log a little bit and found that a lot of leader election happened during that time(https://gist.github.com/ThomasAlxDmy/ce75821fead70b9d240c). The ETCD restore and cluster reset were the only thing running at that time.

CPU, Network, and Memory usage on the instance are pretty low.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions