* we already had it in the ShardRegion
* it's possible to see it from the actor path but that might not be
obvious and many forget to configure their logback to show the akkaSource
* fix test
* when SBR downs the reachable side (minority) it's important
to quickly inform everybody to shutdown
* send gossip directly to downed node, STONITH signal
* gossip to a few random immediatly when self is downed, which
is always the last from the SBR downing
* enable gossip speedup when there are downed members
* adjust StressSpect to normal again
* adjust TransitionSpect to the new behavior
* Config for when to move to WeaklyUp
* noticed when I was testing with the StressSpec that it's often moving nodes to WeaklyUp
in normal joining scenarios (also seen in Kubernetes testing)
* better to wait some longer since the WeaklyUp will require a new convergence round
and making the full joining -> up take longer time
* changed existing config property to be a duration
* default 7s, previously it was 3s
* on => 7s
* Since DeathWatchNotification is sent over the control channel it may overtake
other messages that have been sent from the same actor before it stopped.
* It can be confusing that Terminated can't be used as an end-of-conversation marker.
* In classic Remoting we didn't have this problem because all messages were sent over
the same connection.
* don't send DeathWatchNotification when system is terminating
* when using Cluster we can rely on that the other side will publish AddressTerminated
when the member has been removed
* it's actually already a race condition that often will result in that the DeathWatchNotification
from the terminating side
* in DeathWatch.scala it will remove the watchedBy when receiving AddressTerminated, and that
may (sometimes) happen before tellWatchersWeDied
* same for Unwatch
* to avoid sending many Unwatch messages when watcher's ActorSystem is terminated
* same race exists for Unwatch as for DeathWatchNotification, if RemoteWatcher publishAddressTerminated
before the watcher is terminated
* config for the flush timeout, and possibility to disable
* adjust default minimum for down-all-when-unstable
* when down-all-when-unstable=on it will be >= 4 seconds
* in case stable-after is tweaked to low value such as 5 seconds
* Update aeron-client, aeron-driver to 1.30.0
* Upgrade to agrona 1.7.2, to keep in line with aeron
Co-authored-by: Christopher Batey <christopher.batey@gmail.com>
