raboof/pekko
1
0
Fork 0
Code Issues Pull requests Projects Releases 6 Packages Wiki Activity Actions 11
pekko/akka-distributed-data/src/test/scala/akka/cluster/ddata/DeltaPropagationSelectorSpec.scala

230 lines
9.5 KiB
Scala
Raw Normal View History

Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
/**
Update copyright to 2018 (#24241)
2018-01-04 17:26:29 +00:00
* Copyright (C) 2017-2018 Lightbend Inc. <https://www.lightbend.com>
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
*/
package akka.cluster.ddata
import akka.actor.Address
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
import akka.cluster.UniqueAddress
import akka.cluster.ddata.Key.KeyId
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
import akka.cluster.ddata.Replicator.Internal.DataEnvelope
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
import akka.cluster.ddata.Replicator.Internal.Delta
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
import akka.cluster.ddata.Replicator.Internal.DeltaPropagation
discard large deltas, #23025 * to avoid OversizedPayloadException * some complex deltas grow for each update operation, e.g. when updating different keys in ORMap (PNCounterMap) * such large deltas can safely be discarded and disseminated as full state instead * added ReplicatedDeltaSize interface to be able to define the "size" and when that size exceeds configured threshold the delta is discarded
2017-05-30 12:09:56 +02:00
import akka.cluster.ddata.Replicator.Internal.DeltaPropagation.NoDeltaPlaceholder
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
import org.scalactic.TypeCheckedTripleEquals
import org.scalatest.Matchers
import org.scalatest.WordSpec
object DeltaPropagationSelectorSpec {
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
class TestSelector(
val selfUniqueAddress: UniqueAddress,
override val allNodes: Vector[Address]) extends DeltaPropagationSelector {
override val gossipIntervalDivisor = 5
override def createDeltaPropagation(deltas: Map[KeyId, (ReplicatedData, Long, Long)]): DeltaPropagation =
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
DeltaPropagation(selfUniqueAddress, false, deltas.mapValues {
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
case (d, fromSeqNr, toSeqNr) ⇒ Delta(DataEnvelope(d), fromSeqNr, toSeqNr)
})
discard large deltas, #23025 * to avoid OversizedPayloadException * some complex deltas grow for each update operation, e.g. when updating different keys in ORMap (PNCounterMap) * such large deltas can safely be discarded and disseminated as full state instead * added ReplicatedDeltaSize interface to be able to define the "size" and when that size exceeds configured threshold the delta is discarded
2017-05-30 12:09:56 +02:00
override def maxDeltaSize: Int = 10
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
}
val deltaA = GSet.empty[String] + "a"
val deltaB = GSet.empty[String] + "b"
val deltaC = GSet.empty[String] + "c"
}
class DeltaPropagationSelectorSpec extends WordSpec with Matchers with TypeCheckedTripleEquals {
import DeltaPropagationSelectorSpec._
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
val selfUniqueAddress = UniqueAddress(Address("akka", "Sys", "localhost", 4999), 1L)
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
val nodes = (2500 until 2600).map(n ⇒ Address("akka", "Sys", "localhost", n)).toVector
"DeltaPropagationSelector" must {
"collect none when no nodes" in {
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
val selector = new TestSelector(selfUniqueAddress, Vector.empty)
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.update("A", deltaA)
selector.collectPropagations() should ===(Map.empty[Address, DeltaPropagation])
selector.cleanupDeltaEntries()
selector.hasDeltaEntries("A") should ===(false)
}
"collect 1 when one node" in {
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
val selector = new TestSelector(selfUniqueAddress, nodes.take(1))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.update("A", deltaA)
selector.update("B", deltaB)
selector.cleanupDeltaEntries()
selector.hasDeltaEntries("A") should ===(true)
selector.hasDeltaEntries("B") should ===(true)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(deltaA), 1L, 1L),
"B" → Delta(DataEnvelope(deltaB), 1L, 1L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(0) → expected))
selector.collectPropagations() should ===(Map.empty[Address, DeltaPropagation])
selector.cleanupDeltaEntries()
selector.hasDeltaEntries("A") should ===(false)
selector.hasDeltaEntries("B") should ===(false)
}
"collect 2+1 when three nodes" in {
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
val selector = new TestSelector(selfUniqueAddress, nodes.take(3))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.update("A", deltaA)
selector.update("B", deltaB)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(deltaA), 1L, 1L),
"B" → Delta(DataEnvelope(deltaB), 1L, 1L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(0) → expected, nodes(1) → expected))
selector.cleanupDeltaEntries()
selector.hasDeltaEntries("A") should ===(true)
selector.hasDeltaEntries("B") should ===(true)
selector.collectPropagations() should ===(Map(nodes(2) → expected))
selector.collectPropagations() should ===(Map.empty[Address, DeltaPropagation])
selector.cleanupDeltaEntries()
selector.hasDeltaEntries("A") should ===(false)
selector.hasDeltaEntries("B") should ===(false)
}
"keep track of deltas per node" in {
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
val selector = new TestSelector(selfUniqueAddress, nodes.take(3))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.update("A", deltaA)
selector.update("B", deltaB)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected1 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(deltaA), 1L, 1L),
"B" → Delta(DataEnvelope(deltaB), 1L, 1L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(0) → expected1, nodes(1) → expected1))
// new update before previous was propagated to all nodes
selector.update("C", deltaC)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected2 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(deltaA), 1L, 1L),
"B" → Delta(DataEnvelope(deltaB), 1L, 1L),
"C" → Delta(DataEnvelope(deltaC), 1L, 1L)))
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected3 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"C" → Delta(DataEnvelope(deltaC), 1L, 1L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(2) → expected2, nodes(0) → expected3))
selector.cleanupDeltaEntries()
selector.hasDeltaEntries("A") should ===(false)
selector.hasDeltaEntries("B") should ===(false)
selector.hasDeltaEntries("C") should ===(true)
selector.collectPropagations() should ===(Map(nodes(1) → expected3))
selector.collectPropagations() should ===(Map.empty[Address, DeltaPropagation])
selector.cleanupDeltaEntries()
selector.hasDeltaEntries("C") should ===(false)
}
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"bump version for each update" in {
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
val delta1 = GSet.empty[String] + "a1"
val delta2 = GSet.empty[String] + "a2"
val delta3 = GSet.empty[String] + "a3"
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
val selector = new TestSelector(selfUniqueAddress, nodes.take(1))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.update("A", delta1)
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
selector.currentVersion("A") should ===(1L)
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.update("A", delta2)
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
selector.currentVersion("A") should ===(2L)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected1 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(delta1.merge(delta2)), 1L, 2L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(0) → expected1))
selector.update("A", delta3)
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
selector.currentVersion("A") should ===(3L)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected2 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(delta3), 3L, 3L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(0) → expected2))
selector.collectPropagations() should ===(Map.empty[Address, DeltaPropagation])
}
"merge deltas" in {
val delta1 = GSet.empty[String] + "a1"
val delta2 = GSet.empty[String] + "a2"
val delta3 = GSet.empty[String] + "a3"
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
val selector = new TestSelector(selfUniqueAddress, nodes.take(3)) {
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
override def nodesSliceSize(allNodesSize: Int): Int = 1
}
selector.update("A", delta1)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected1 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(delta1), 1L, 1L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(0) → expected1))
selector.update("A", delta2)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected2 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(delta1.merge(delta2)), 1L, 2L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(1) → expected2))
selector.update("A", delta3)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected3 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(delta1.merge(delta2).merge(delta3)), 1L, 3L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(2) → expected3))
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected4 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(delta2.merge(delta3)), 2L, 3L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(0) → expected4))
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val expected5 = DeltaPropagation(selfUniqueAddress, false, Map(
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
"A" → Delta(DataEnvelope(delta3), 3L, 3L)))
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.collectPropagations() should ===(Map(nodes(1) → expected5))
selector.collectPropagations() should ===(Map.empty[Address, DeltaPropagation])
}
discard large deltas, #23025 * to avoid OversizedPayloadException * some complex deltas grow for each update operation, e.g. when updating different keys in ORMap (PNCounterMap) * such large deltas can safely be discarded and disseminated as full state instead * added ReplicatedDeltaSize interface to be able to define the "size" and when that size exceeds configured threshold the delta is discarded
2017-05-30 12:09:56 +02:00
"discard too large deltas" in {
val selector = new TestSelector(selfUniqueAddress, nodes.take(3)) {
override def nodesSliceSize(allNodesSize: Int): Int = 1
}
var data = PNCounterMap.empty[String]
(1 to 1000).foreach { n ⇒
val d = data.resetDelta.increment(selfUniqueAddress, (n % 2).toString, 1)
selector.update("A", d.delta.get)
data = d
}
val expected = DeltaPropagation(selfUniqueAddress, false, Map(
"A" → Delta(DataEnvelope(NoDeltaPlaceholder), 1L, 1000L)))
selector.collectPropagations() should ===(Map(nodes(0) → expected))
}
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
"calcualte right slice size" in {
causal delivery of deltas, #22188 * keep track of delta interval versions and skip deltas that are not consequtive, i.e. when some delta message was lost * send the delta versions in the full state gossip to sync up the expected versions after dropped deltas * implementation of deltas for ORSet * refactoring of the delta types to allow for different type for the delta and the full state * extensive tests * mima filter * performance optimizations * simple pruning of deltas * Java API * update documentation * KeyId type alias * Use InternalApi annotation
2017-02-07 11:21:56 +01:00
val selector = new TestSelector(selfUniqueAddress, nodes)
Add support for delta-CRDT, #21875 * delta GCounter and PNCounter * first stab at delta propagation protocol * send delta in the direct write * possibility to turn off delta propagation * tests * protobuf serializer for DeltaPropagation * documentation
2017-01-04 15:25:12 +01:00
selector.nodesSliceSize(0) should ===(0)
selector.nodesSliceSize(1) should ===(1)
(2 to 9).foreach { n ⇒
withClue(s"n=$n") {
selector.nodesSliceSize(n) should ===(2)
}
}
(10 to 14).foreach { n ⇒
withClue(s"n=$n") {
selector.nodesSliceSize(n) should ===(3)
}
}
(15 to 19).foreach { n ⇒
withClue(s"n=$n") {
selector.nodesSliceSize(n) should ===(4)
}
}
(20 to 24).foreach { n ⇒
withClue(s"n=$n") {
selector.nodesSliceSize(n) should ===(5)
}
}
(25 to 29).foreach { n ⇒
withClue(s"n=$n") {
selector.nodesSliceSize(n) should ===(6)
}
}
(30 to 34).foreach { n ⇒
withClue(s"n=$n") {
selector.nodesSliceSize(n) should ===(7)
}
}
(35 to 39).foreach { n ⇒
withClue(s"n=$n") {
selector.nodesSliceSize(n) should ===(8)
}
}
(40 to 44).foreach { n ⇒
withClue(s"n=$n") {
selector.nodesSliceSize(n) should ===(9)
}
}
(45 to 200).foreach { n ⇒
withClue(s"n=$n") {
selector.nodesSliceSize(n) should ===(10)
}
}
}
}
}
Reference in a new issue Copy permalink