Improve default shard rebalancing algorithm, #26012 (#26101)

* Improve default shard rebalancing algorithm, #26012

* Use rebalance-threshold=1 because it will give the best distribution,
  and previous default could result in too large difference between nodes
* Off by one error, difference > threshold vs >=
* Added more unit tests
* Note that in some cases it may still not be optimal, stopping more
  shards than necessary, but a different strategy that looks at more
  than most and least is out of scope for this issue. In practise
  those cases shouldn't matter much.
* Also note that the rebalance interval is by default 10 seconds,
  so typically shards will start up before next rebalance tick.
  It's intentionally a slow process to not cause instabilities by
  moving too much at the same time.
This commit is contained in:
Patrik Nordwall 2018-12-21 13:39:41 +01:00 committed by GitHub
parent 5f409c17e0
commit b48938f4a0
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 155 additions and 49 deletions

View file

@ -158,7 +158,7 @@ abstract class ClusterShardingSpecConfig(
number-of-entities = 1
}
least-shard-allocation-strategy {
rebalance-threshold = 2
rebalance-threshold = 1
max-simultaneous-rebalance = 1
}
distributed-data.durable.lmdb {