-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Hey There!
I started to mess around with ETCD clusters few weeks ago, and I was looking for a good dump/restore tool for it. Yesterday I tried etcdump and I discover 2 (different?) issues.
- After a restore, the number of keys restored is completely random.
- No directories are being restored.
More information:
A) I created a simple script that populate an empty ETCD cluster with 10 000 keys and 10 000 empty directories at the root. Structure looks like that: https://gist.github.com/ThomasAlxDmy/43ec114953cbc1b600f4. Notice the format "key": "/message*" for regular keys and "key": "/message_dir*" for directories.
B) Then I created a dump file with the command etcdump --file /mnt/etcd/dump/etcdump.json --config /mnt/etcd/dump/etcdump-config.json dump where /mnt/etcd/dump/etcdump-config.json = https://gist.github.com/ThomasAlxDmy/791d0270228da563ef3e
The dump file contains 20 000 keys (correct number) and look like that: https://gist.github.com/ThomasAlxDmy/132a041410159439410c
The first thing that shocked me was the format of the directory keys:
{ "key": "/message_dir4524" }
Looks like it's missing "dir":true. I don't know if it's a mistake or by design but it would explain why no directory are created.
C) I removed all the keys in the cluster.
D) Restored the keys with the following command: etcdump --file /mnt/etcd/dump/etcdump.json --config /mnt/etcd/dump/etcdump-config.json restore And got the messagesuccessfuly restored`
E) I re-checked the data in the ETCD cluster and only got 0 directories and 1815 unique keys
I restarted those step several times and got different numbers for the last step but always 0 directories:
- 0 directories and 3434 unique keys
- 0 directories and 1159 unique keys
I dig into my cluster log a little bit and found that a lot of leader election happened during that time(https://gist.github.com/ThomasAlxDmy/ce75821fead70b9d240c). The ETCD restore and cluster reset were the only thing running at that time.
CPU, Network, and Memory usage on the instance are pretty low.