When the journal directory is even, the journal data is unevenly distributed

[1559924775]

BUG REPORT

Describe the bug

When the journal directory is configured to be even, the ledger allocated to each journal will be uneven. The root cause is that almost all the ledger IDs generated by the LedgerIdGenerator class are even. The number of ZK temporary order nodes is determined by the cversion of the parent node. Whenever a child node is added or deleted, it will increase by 1. The LedgerIdGenerator class uses ZK's temporary order node to generate the ledgerid, and deletes the node immediately after generation. This way of use causes almost all cversions to be even, resulting in all ledgerids to be even.

To Reproduce

Steps to reproduce the behavior:

1. Configure in bk_server.conf: jouranlDirectorie=/tmp/bk-txn1,/tmp/bk-txn2
2. production data to bookie

Expected behavior

Almost all ledgerids are assigned to directory 1

Screenshots

ledgerIdGenerator class : ZKledgerIdGenerator

Changing the routing of ledgers to journal directories can fix this bug

[Vanlightly]

Changing the routing without some kind of migration will result in effective data loss. Plus an md5 hash is relatively expensive.

The issue here is that ledger ids are all even. I am seeing this myself which results in only even numbered read threads from being routed to. This problem of even ledger ids affects the selection of journals, ledger storage, read threads and write threads. It is a real problem.

@eolivelli This is a pretty serious problem as we're throwing away a bunch of scaling mechanisms.

[dlg99]

@Vanlightly @eolivelli I updated OrderedExecutor to better distribute data across the threads, added tests reproducing the issue. please take a look:

#3023

This does not address the fact that generated ledger ids are skewed towards even numbers.
This does not fix routing of the journals/data dirs.

[hezhangjian]

@dlg99 Can we close this issue or need other works? :)

[hezhangjian]

From Andrey:
I think it makes sense to split this into bugs for "fix routing of the journals/data dirs" - at least to conform that these aren't skewed towards a subset of directories

[jprieto-temporal]

I submitted a potential solution to this issue and verified it works.

#4709