raboof/pekko
1
0
Fork 0
Code Issues Pull requests Projects Releases 6 Packages Wiki Activity Actions 10
pekko/akka-distributed-data/src/test/scala/akka/cluster/ddata/WriteAggregatorSpec.scala

489 lines
14 KiB
Scala
Raw Normal View History

update headers to regular comment (#25807)
2018-10-29 17:19:37 +08:00
/*
Update headers from 2018 to 2019 once for all. (#26165) * Add CopyrightHeader support for sbt-boilerplate plugin. * Add CopyrightHeader support for `*.proto` files. * Add regex match for both `–` and `-` for CopyrightHeader. * Add CopyrightHeader support for sbt build files. * Update copyright from 2018 to 2019.
2019-01-02 18:55:26 +08:00
* Copyright (C) 2009-2019 Lightbend Inc. <https://www.lightbend.com>
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
*/
package akka.cluster.ddata
import scala.concurrent.duration._
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
import akka.actor.Actor
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
import akka.actor.ActorRef
import akka.actor.ActorSelection
import akka.actor.ActorSystem
import akka.actor.Address
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
import akka.actor.Props
import akka.testkit._
import akka.cluster.ddata.Replicator.Internal._
import akka.cluster.ddata.Replicator._
More tests working on artery * non-multi-jvm tests from akka-cluster * akka-cluster-metrics * akka-cluster-tools * akka-cluster-sharding
2016-09-14 11:40:42 +02:00
import akka.remote.RARP
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
import scala.concurrent.Future
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
import akka.cluster.Cluster
import akka.cluster.UniqueAddress
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
object WriteAggregatorSpec {
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val KeyA = GSetKey[String]("A")
val KeyB = ORSetKey[String]("B")
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
def writeAggregatorProps(
data: GSet[String],
consistency: Replicator.WriteConsistency,
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
probes: Map[UniqueAddress, ActorRef],
selfUniqueAddress: UniqueAddress,
nodes: Set[UniqueAddress],
unreachable: Set[UniqueAddress],
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
replyTo: ActorRef,
durable: Boolean): Props =
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
Props(
new TestWriteAggregator(
KeyA,
data,
None,
consistency,
probes,
selfUniqueAddress,
nodes,
unreachable,
replyTo,
durable))
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
def writeAggregatorPropsWithDelta(
data: ORSet[String],
delta: Delta,
consistency: Replicator.WriteConsistency,
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
probes: Map[UniqueAddress, ActorRef],
selfUniqueAddress: UniqueAddress,
nodes: Set[UniqueAddress],
unreachable: Set[UniqueAddress],
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
replyTo: ActorRef,
durable: Boolean): Props =
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
Props(
new TestWriteAggregator(
KeyB,
data,
Some(delta),
consistency,
probes,
selfUniqueAddress,
nodes,
unreachable,
replyTo,
durable))
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
class TestWriteAggregator(
key: Key.KeyR,
data: ReplicatedData,
delta: Option[Delta],
consistency: Replicator.WriteConsistency,
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
probes: Map[UniqueAddress, ActorRef],
selfUniqueAddress: UniqueAddress,
nodes: Set[UniqueAddress],
unreachable: Set[UniqueAddress],
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
replyTo: ActorRef,
durable: Boolean)
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
extends WriteAggregator(
key,
DataEnvelope(data),
delta,
consistency,
None,
selfUniqueAddress,
nodes,
unreachable,
replyTo,
durable) {
override def replica(address: UniqueAddress): ActorSelection =
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
context.actorSelection(probes(address).path)
override def senderAddress(): Address =
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
probes.find { case (_, r) => r == sender() }.get._1.address
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
}
def writeAckAdapterProps(replica: ActorRef): Props =
Props(new WriteAckAdapter(replica))
class WriteAckAdapter(replica: ActorRef) extends Actor {
var replicator: Option[ActorRef] = None
def receive = {
replace unicode arrows * ⇒, →, ← * because we don't want to show them in documentation snippets and then it's complicated to avoid that when snippets are located in src/test/scala in individual modules * dont replace object `→` in FSM.scala and PersistentFSM.scala
2019-02-09 15:25:39 +01:00
case WriteAck =>
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
replicator.foreach(_ ! WriteAck)
replace unicode arrows * ⇒, →, ← * because we don't want to show them in documentation snippets and then it's complicated to avoid that when snippets are located in src/test/scala in individual modules * dont replace object `→` in FSM.scala and PersistentFSM.scala
2019-02-09 15:25:39 +01:00
case WriteNack =>
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
replicator.foreach(_ ! WriteNack)
replace unicode arrows * ⇒, →, ← * because we don't want to show them in documentation snippets and then it's complicated to avoid that when snippets are located in src/test/scala in individual modules * dont replace object `→` in FSM.scala and PersistentFSM.scala
2019-02-09 15:25:39 +01:00
case DeltaNack =>
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
replicator.foreach(_ ! DeltaNack)
replace unicode arrows * ⇒, →, ← * because we don't want to show them in documentation snippets and then it's complicated to avoid that when snippets are located in src/test/scala in individual modules * dont replace object `→` in FSM.scala and PersistentFSM.scala
2019-02-09 15:25:39 +01:00
case msg =>
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
replicator = Some(sender())
replica ! msg
}
}
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
object TestMock {
def apply()(implicit system: ActorSystem) = new TestMock(system)
}
class TestMock(_application: ActorSystem) extends TestProbe(_application) {
val writeAckAdapter = system.actorOf(WriteAggregatorSpec.writeAckAdapterProps(this.ref))
}
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
}
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
class WriteAggregatorSpec extends AkkaSpec(s"""
A simpler ActorRefProvider config #20649 (#20767) * Provide shorter aliases for the ActorRefProviders #20649 * Use the new actorefprovider aliases throughout code and docs * Cleaner alias replacement logic
2016-06-10 15:04:13 +02:00
akka.actor.provider = "cluster"
Make artery the default remoting (#26772)
2019-05-01 08:12:09 +01:00
akka.remote.classic.netty.tcp.port = 0
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
akka.remote.artery.canonical.port = 0
akka.cluster.distributed-data.durable.lmdb {
dir = target/WriteAggregatorSpec-${System.currentTimeMillis}-ddata
map-size = 10 MiB
}
format source with scalafmt
2019-03-11 10:38:24 +01:00
""") with ImplicitSender {
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
import WriteAggregatorSpec._
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
More tests working on artery * non-multi-jvm tests from akka-cluster * akka-cluster-metrics * akka-cluster-tools * akka-cluster-sharding
2016-09-14 11:40:42 +02:00
val protocol =
if (RARP(system).provider.remoteSettings.Artery.Enabled) "akka"
else "akka.tcp"
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
val nodeA = UniqueAddress(Address(protocol, "Sys", "a", 2552), 17L)
val nodeB = UniqueAddress(Address(protocol, "Sys", "b", 2552), 17L)
val nodeC = UniqueAddress(Address(protocol, "Sys", "c", 2552), 17L)
val nodeD = UniqueAddress(Address(protocol, "Sys", "d", 2552), 17L)
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
// 4 replicas + the local => 5
val nodes = Set(nodeA, nodeB, nodeC, nodeD)
val data = GSet.empty + "A" + "B"
val timeout = 3.seconds.dilated
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
val writeThree = WriteTo(3, timeout)
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
val writeMajority = WriteMajority(timeout)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val writeAll = WriteAll(timeout)
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
val selfUniqueAddress: UniqueAddress = Cluster(system).selfUniqueAddress
def probes(probe: ActorRef): Map[UniqueAddress, ActorRef] =
replace unicode arrows * ⇒, →, ← * because we don't want to show them in documentation snippets and then it's complicated to avoid that when snippets are located in src/test/scala in individual modules * dont replace object `→` in FSM.scala and PersistentFSM.scala
2019-02-09 15:25:39 +01:00
nodes.toSeq.map(_ -> system.actorOf(WriteAggregatorSpec.writeAckAdapterProps(probe))).toMap
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
/**
* Create a tuple for each node with the WriteAckAdapter and the TestProbe
*/
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
def probes(): Map[UniqueAddress, TestMock] = {
replace unicode arrows * ⇒, →, ← * because we don't want to show them in documentation snippets and then it's complicated to avoid that when snippets are located in src/test/scala in individual modules * dont replace object `→` in FSM.scala and PersistentFSM.scala
2019-02-09 15:25:39 +01:00
nodes.toSeq.map(_ -> TestMock()).toMap
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
}
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
"WriteAggregator" must {
"send to at least N/2+1 replicas when WriteMajority" in {
val probe = TestProbe()
format source with scalafmt
2019-03-11 10:38:24 +01:00
val aggr = system.actorOf(
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
WriteAggregatorSpec.writeAggregatorProps(
data,
writeMajority,
probes(probe.ref),
selfUniqueAddress,
nodes,
Set.empty,
testActor,
durable = false))
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
expectMsg(UpdateSuccess(WriteAggregatorSpec.KeyA, None))
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
watch(aggr)
expectTerminated(aggr)
}
"send to more when no immediate reply" in {
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
val testProbes = probes()
replace unicode arrows * ⇒, →, ← * because we don't want to show them in documentation snippets and then it's complicated to avoid that when snippets are located in src/test/scala in individual modules * dont replace object `→` in FSM.scala and PersistentFSM.scala
2019-02-09 15:25:39 +01:00
val testProbeRefs = testProbes.map { case (a, tm) => a -> tm.writeAckAdapter }
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
val aggr = system.actorOf(
WriteAggregatorSpec.writeAggregatorProps(
data,
writeMajority,
testProbeRefs,
selfUniqueAddress,
nodes,
Set(nodeC, nodeD),
testActor,
durable = false))
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
testProbes(nodeA).expectMsgType[Write]
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
// no reply
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
testProbes(nodeB).expectMsgType[Write]
testProbes(nodeB).lastSender ! WriteAck
// Make sure that unreachable nodes do not get a message until 1/5 of the time the reachable nodes did not answer
val t = timeout / 5 - 50.milliseconds.dilated
import system.dispatcher
Future.sequence {
akka-distributed-data compiler warnings #26088
2019-03-19 08:12:51 +01:00
Seq(Future { testProbes(nodeC).expectNoMessage(t) }, Future { testProbes(nodeD).expectNoMessage(t) })
#21648 Prefer reachable nodes in consistency writes/reads
2016-12-23 10:59:24 +01:00
}.futureValue
testProbes(nodeC).expectMsgType[Write]
testProbes(nodeC).lastSender ! WriteAck
testProbes(nodeD).expectMsgType[Write]
testProbes(nodeD).lastSender ! WriteAck
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
expectMsg(UpdateSuccess(WriteAggregatorSpec.KeyA, None))
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
watch(aggr)
expectTerminated(aggr)
}
"timeout when less than required acks" in {
val probe = TestProbe()
format source with scalafmt
2019-03-11 10:38:24 +01:00
val aggr = system.actorOf(
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
WriteAggregatorSpec.writeAggregatorProps(
data,
writeMajority,
probes(probe.ref),
selfUniqueAddress,
nodes,
Set.empty,
testActor,
durable = false))
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
probe.expectMsgType[Write]
// no reply
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
probe.expectMsgType[Write]
// no reply
probe.expectMsgType[Write]
// no reply
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
expectMsg(UpdateTimeout(WriteAggregatorSpec.KeyA, None))
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
watch(aggr)
expectTerminated(aggr)
}
Add minimum cap for ReadMajority/WriteMajority #21618
2017-01-18 14:13:40 +02:00
"calculate majority with minCap" in {
val minCap = 5
import ReadWriteAggregator._
stash messages while Replicator is loading durable data, #22150
2017-01-19 12:55:28 +01:00
calculateMajorityWithMinCap(minCap, 3) should be(3)
calculateMajorityWithMinCap(minCap, 4) should be(4)
calculateMajorityWithMinCap(minCap, 5) should be(5)
calculateMajorityWithMinCap(minCap, 6) should be(5)
calculateMajorityWithMinCap(minCap, 7) should be(5)
calculateMajorityWithMinCap(minCap, 8) should be(5)
calculateMajorityWithMinCap(minCap, 9) should be(5)
calculateMajorityWithMinCap(minCap, 10) should be(6)
calculateMajorityWithMinCap(minCap, 11) should be(6)
calculateMajorityWithMinCap(minCap, 12) should be(7)
Add minimum cap for ReadMajority/WriteMajority #21618
2017-01-18 14:13:40 +02:00
}
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
}
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
"WriteAggregator with delta" must {
akka-distributed-data compiler warnings #26088
2019-03-19 08:12:51 +01:00
implicit val node = DistributedData(system).selfUniqueAddress
val fullState1 = ORSet.empty[String] :+ "a" :+ "b"
val fullState2 = fullState1.resetDelta :+ "c"
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
val delta = Delta(DataEnvelope(fullState2.delta.get), 2L, 2L)
"send deltas first" in {
val probe = TestProbe()
format source with scalafmt
2019-03-11 10:38:24 +01:00
val aggr = system.actorOf(
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
WriteAggregatorSpec.writeAggregatorPropsWithDelta(
fullState2,
delta,
writeMajority,
probes(probe.ref),
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
selfUniqueAddress,
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
nodes,
Set.empty,
testActor,
durable = false))
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
probe.expectMsgType[DeltaPropagation]
probe.lastSender ! WriteAck
probe.expectMsgType[DeltaPropagation]
probe.lastSender ! WriteAck
expectMsg(UpdateSuccess(WriteAggregatorSpec.KeyB, None))
watch(aggr)
expectTerminated(aggr)
}
"retry with full state when no immediate reply or nack" in {
val testProbes = probes()
replace unicode arrows * ⇒, →, ← * because we don't want to show them in documentation snippets and then it's complicated to avoid that when snippets are located in src/test/scala in individual modules * dont replace object `→` in FSM.scala and PersistentFSM.scala
2019-02-09 15:25:39 +01:00
val testProbeRefs = testProbes.map { case (a, tm) => a -> tm.writeAckAdapter }
format source with scalafmt
2019-03-11 10:38:24 +01:00
val aggr = system.actorOf(
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
WriteAggregatorSpec.writeAggregatorPropsWithDelta(
fullState2,
delta,
writeAll,
testProbeRefs,
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
selfUniqueAddress,
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
nodes,
Set.empty,
testActor,
durable = false))
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
testProbes(nodeA).expectMsgType[DeltaPropagation]
// no reply
testProbes(nodeB).expectMsgType[DeltaPropagation]
testProbes(nodeB).lastSender ! WriteAck
testProbes(nodeC).expectMsgType[DeltaPropagation]
testProbes(nodeC).lastSender ! WriteAck
testProbes(nodeD).expectMsgType[DeltaPropagation]
testProbes(nodeD).lastSender ! DeltaNack
// here is the second round
testProbes(nodeA).expectMsgType[Write]
testProbes(nodeA).lastSender ! WriteAck
testProbes(nodeD).expectMsgType[Write]
testProbes(nodeD).lastSender ! WriteAck
akka-distributed-data compiler warnings #26088
2019-03-19 08:12:51 +01:00
testProbes(nodeB).expectNoMessage(100.millis)
testProbes(nodeC).expectNoMessage(100.millis)
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
expectMsg(UpdateSuccess(WriteAggregatorSpec.KeyB, None))
watch(aggr)
expectTerminated(aggr)
}
"timeout when less than required acks" in {
val probe = TestProbe()
format source with scalafmt
2019-03-11 10:38:24 +01:00
val aggr = system.actorOf(
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
WriteAggregatorSpec.writeAggregatorPropsWithDelta(
fullState2,
delta,
writeAll,
probes(probe.ref),
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
selfUniqueAddress,
format source with scalafmt, #26511
2019-03-13 10:56:20 +01:00
nodes,
Set.empty,
testActor,
durable = false))
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
probe.expectMsgType[DeltaPropagation]
// no reply
probe.expectMsgType[DeltaPropagation]
probe.lastSender ! WriteAck
probe.expectMsgType[DeltaPropagation]
fix race condition in WriteAggregatorSpec, #25581 (#28242) * sometimes failed: "WriteAggregator with delta must timeout when less than required acks" * the DeltaNack will trigger immedediate send of Write * seems to be a race condition of when the probe is receiving the Write from the DeltaNack vs the last expected DeltaPropagation
2019-11-25 17:06:27 +01:00
// no reply
probe.expectMsgType[DeltaPropagation]
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
// nack
probe.lastSender ! DeltaNack
fix race condition in WriteAggregatorSpec, #25581 (#28242) * sometimes failed: "WriteAggregator with delta must timeout when less than required acks" * the DeltaNack will trigger immedediate send of Write * seems to be a race condition of when the probe is receiving the Write from the DeltaNack vs the last expected DeltaPropagation
2019-11-25 17:06:27 +01:00
// the nack will triggger an immediate Write
probe.expectMsgType[Write]
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
// no reply
// only 1 ack so we expect 3 full state Write
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
probe.expectMsgType[Write]
probe.expectMsgType[Write]
// still not enough acks
expectMsg(UpdateTimeout(WriteAggregatorSpec.KeyB, None))
watch(aggr)
expectTerminated(aggr)
}
}
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
"Durable WriteAggregator" must {
"not reply before local confirmation" in {
val probe = TestProbe()
format source with scalafmt
2019-03-11 10:38:24 +01:00
val aggr = system.actorOf(
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
WriteAggregatorSpec.writeAggregatorProps(
data,
writeThree,
probes(probe.ref),
selfUniqueAddress,
nodes,
Set.empty,
testActor,
durable = true))
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
akka-distributed-data compiler warnings #26088
2019-03-19 08:12:51 +01:00
expectNoMessage(200.millis)
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
// the local write
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
aggr ! UpdateSuccess(WriteAggregatorSpec.KeyA, None)
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
expectMsg(UpdateSuccess(WriteAggregatorSpec.KeyA, None))
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
watch(aggr)
expectTerminated(aggr)
}
"tolerate WriteNack if enough WriteAck" in {
val probe = TestProbe()
format source with scalafmt
2019-03-11 10:38:24 +01:00
val aggr = system.actorOf(
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
WriteAggregatorSpec.writeAggregatorProps(
data,
writeThree,
probes(probe.ref),
selfUniqueAddress,
nodes,
Set.empty,
testActor,
durable = true))
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
aggr ! UpdateSuccess(WriteAggregatorSpec.KeyA, None) // the local write
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
probe.expectMsgType[Write]
probe.lastSender ! WriteNack
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
expectMsg(UpdateSuccess(WriteAggregatorSpec.KeyA, None))
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
watch(aggr)
expectTerminated(aggr)
}
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
"reply with StoreFailure when too many nacks" in {
val probe = TestProbe()
format source with scalafmt
2019-03-11 10:38:24 +01:00
val aggr = system.actorOf(
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
WriteAggregatorSpec.writeAggregatorProps(
data,
writeMajority,
probes(probe.ref),
selfUniqueAddress,
nodes,
Set.empty,
testActor,
durable = true))
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
probe.expectMsgType[Write]
probe.lastSender ! WriteNack
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
aggr ! UpdateSuccess(WriteAggregatorSpec.KeyA, None) // the local write
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
probe.expectMsgType[Write]
probe.lastSender ! WriteNack
probe.expectMsgType[Write]
probe.lastSender ! WriteNack
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
expectMsg(StoreFailure(WriteAggregatorSpec.KeyA, None))
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
watch(aggr)
expectTerminated(aggr)
}
"timeout when less than required acks" in {
val probe = TestProbe()
format source with scalafmt
2019-03-11 10:38:24 +01:00
val aggr = system.actorOf(
Avoid false removals in ClusterReceptionist, #26284 * The scenario was (probably) that a node was restarted with same host:port and then didn't join the same cluster. The DData Replicator in the original cluster would continue sending messages to the new incarnation resulting in false removals. * The fix is that DData Replicator includes the system uid of the sending or target system in messages and if recipient gets a message that is from/to unknown it will discard it and thereby not spreading information across different clusters. * Reproduced in ClusterReceptionistSpec * Much hardening of other things in ClusterReceptionistSpec * There are also some improvements to ClusterReceptionist to not leak Listing with refs of removed nodes. * use ClusterShuttingDown * The reason for using sender system uid instead of target uid in messages like Read and Write is that then the optimization for sending same message to many destinations can remain.
2019-02-21 09:09:20 +01:00
WriteAggregatorSpec.writeAggregatorProps(
data,
writeMajority,
probes(probe.ref),
selfUniqueAddress,
nodes,
Set.empty,
testActor,
durable = true))
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
probe.expectMsgType[Write]
// no reply
probe.expectMsgType[Write]
probe.lastSender ! WriteAck
probe.expectMsgType[Write]
probe.lastSender ! WriteNack
probe.expectMsgType[Write]
probe.lastSender ! WriteNack
Use delta in direct write also, #22188 * Follow up on the causal delivery of deltas. * The first implementation used full state for the direct Write messages, i.e. updates with WriteConsistency != LocalWrite * This is an optimization so that delatas are tried first and if they can't be applied it falls back to full state. * For simultanious updates the messages may be reordered because we create separate WriteAggregator actor and such, but normally they will be sent in order so the deltas will typically be received in order, otherwise we fall back to retrying with full state in the second round in the WriteAggregator.
2017-02-23 12:12:29 +01:00
expectMsg(UpdateTimeout(WriteAggregatorSpec.KeyA, None))
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
watch(aggr)
expectTerminated(aggr)
}
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
}
Add support for durable storage of distributed data, #21645 * using lmdbjava libarary
2016-10-10 20:00:24 +02:00
=cdd #18768 Cache serialization of read and write msg The WriteAggregator and ReadAggregator typically send the same message to several replicas and by caching the serialized bytes we avoid doing the same thing for each message and add a test for WriteAggregator
2015-10-18 09:36:32 +02:00
}
Reference in a new issue Copy permalink