* Expanding the LogMarker to optionally include Map of additional
properties
* The name of the marker shows up as `tags` in Kibana
* The properties of the LogMarker are included as MDC entries, which
Logstash encoder automatically understands
* Implemented with classic eventStream logging so that it can be used
from all places without dependencies to Slf4j
* also means that it's possible to subscribe to LogEventWithMarker
to consume these events via the eventStream
* move marker definitions to RemoteLogMarker and ClusterLogMarker
* marker for dead letters
* marker for leader detained/allowed
* marker for member status changed
* markers for shard allocated and started
* test LogMarker with properties in Slf4jLoggerSpec
* doc note of the LogMarker definitions
* moved to cluster tests, in new package akka.cluster.testkit
* changed config in tests
* migration guide
* documentation clarificiations for Downing and Leaving
* update warnings in Singleton and Sharding
* Fix singleton issue when leaving several nodes, #27487
* When leaving several nodes at about the same time the new singleton
could be started before previous had been completely stopped.
* Found two possible ways this could happen.
* Acting on MemberRemoved that is emitted when the self
cluster node is shutting down.
* The HandOverDone confirmation when in Younger state,
but that node is also Leaving so could be seen as Exiting
from a third node that is the next singleton.
* keep track of all previous oldest, not only the latest
* Option => List
* Otherwise in BecomingOldest it could transition to Oldest
when the previous oldest was removed even though the previous-previous wasn't removed yet
* fix failure in ClusterSingletonRestart2Spec
* OldestChanged was not emitted when Exiting member was removed
* The initial membersByAge must also contain Leaving, Exiting members
(cherry picked from commit ee188565b9f3cf2257ebda218cec6af5a4777439)
* previous `schedule` method is trying to maintain a fixed average frequency
over time, but that can result in undesired bursts of scheduled tasks after a long
GC or if the JVM process has been suspended, same with all other periodic
scheduled message sending via various Timer APIs
* most of the time "fixed delay" is more desirable
* we can't just change because it's too big behavioral change and some might
depend on previous behavior
* deprecate the old `schedule` and introduce new `scheduleWithFixedDelay`
and `scheduleAtFixedRate`, when fixing the deprecation warning users should
make a concious decision of which behavior to use (scheduleWithFixedDelay in
most cases)
* Streams
* SchedulerSpec
* test both fixed delay and fixed rate
* TimerSpec
* FSM and PersistentFSM
* mima
* runnable as second parameter list, also in typed.Scheduler
* IllegalStateException vs SchedulerException
* deprecated annotations
* api and reference docs, all places
* migration guide
* Allow for dispatcher aliases and define a internal dispatcher
* Test checking dispatcher name
* MiMa for Dispatchers
* Migration guide entry
* No need to have custom dispatcher lookup logic in streams anymore
* Default dispatcher size and migration note about that
* Test checking exact config values...
* Typed receptionist on internal dispatcher
* All internal usages of system.dispatcher gone through
* lease api
* Cluster singleton manager with lease
* Refactor OldestData to use option for actor reference
* Sharding with lease
* Docs for singleton and sharding lease + config for sharding lease
* Have ddata shard wait until lease is acquired before getting state
* ⇒, →, ←
* because we don't want to show them in documentation snippets and
then it's complicated to avoid that when snippets are
located in src/test/scala in individual modules
* dont replace object `→` in FSM.scala and PersistentFSM.scala
* Stop singleton when self MemberDowned, #26336
* It's safer to stop singleton instance early in case of downing.
* Instead of waiting for MemberRemoved and trying to hand over.
* Stop ShardRegion when self MemberDowned, #26336
* Upper bound when waiting for seen in shutdownSelfWhenDown, #26336
* Add CopyrightHeader support for sbt-boilerplate plugin.
* Add CopyrightHeader support for `*.proto` files.
* Add regex match for both `–` and `-` for CopyrightHeader.
* Add CopyrightHeader support for sbt build files.
* Update copyright from 2018 to 2019.
* Testing of singleton leaving
* gossip optimization, exiting change to two oldest per role
* hardening ClusterSingletonManagerIsStuck restart, increase ClusterSingletonManagerIsStuck
There exists a race where a cluter node that is being downed seens its
self as the oldest node (as it has had the other nodes removed) and it
takes over the singleton manager sending the real oldest node to go into
the End state meaning that cluster singletons never work again.
This fix simply prevents Member events being given to the Cluster
Manager FSM during a shut down, instread relying on SelfExiting.
This also hardens the test by not downing the node that the current
sharding coordinator is running on as well as fixing a bug in the
probes.
* CoordinatedShutdown that can run tasks for configured phases in order (DAG)
* coordinate handover/shutdown of singleton with cluster exiting/shutdown
* phase config obj with depends-on list
* integrate graceful leaving of sharding in coordinated shutdown
* add timeout and recover
* add some missing artery ports to tests
* leave via CoordinatedShutdown.run
* optionally exit-jvm in last phase
* run via jvm shutdown hook
* send ExitingConfirmed to leader before shutdown of Exiting
to not have to wait for failure detector to mark it as
unreachable before removing
* the unreachable signal is still kept as a safe guard if
message is lost or leader dies
* PhaseClusterExiting vs MemberExited in ClusterSingletonManager
* terminate ActorSystem when cluster shutdown (via Down)
* add more predefined and custom phases
* reference documentation
* migration guide
* problem when the leader order was sys2, sys1, sys3,
then sys3 could not perform it's duties and move Leving sys1 to
Exiting because it was observing sys1 as unreachable
* exclude Leaving with exitingConfirmed from convergence condidtion
* send terminationMessage to singleton when leaving last, #21592
* When leaving last node, i.e. no newOldestOption, the manager was
just stopped. The change is to send the terminationMessage also
in this case and wait until the singleton actor is terminated
before stopping the manager.
* Also changed so that the singleton is stopped immediately when
cluster has been terminated when last node is leaving, i.e.
no newOldestOption. Previously it retried until maxTakeOverRetries
before stopping.
* More comprehensive test of this scenario in ClusterSingletonManagerLeaveSpec
* increase test timeout
* When the test fails the node is removed from the membership
twice, which triggers two OldestChanged cycles, but in
the 2.4.9 change https://github.com/akka/akka/pull/21152/files#diff-f0ae95c926a050aecf45dba3e08d1c77L669
the singleton manager always goes to End (stop) when it has been Oldest
* This fix restores the previous behavior for this scenario
* track nodes by UniqueAddress in Cluster Singleton, #20942
* reply with HandOverDone from new incarnation, #20942
* confirm as terminated immediately when new incarnation joins, #20942 instead of waiting for failure detector to mark it as unreachable this will speed-up removal when restarting cluster node with same hostname:port
* the reported issue is fixed by the immediate leaderActions
(moving to Up) when joining the first node to itself
* the other changes are precautions just in case
* In 2.4 we derive the number of hand-over/take-over retries from
the removal margin, but we decided to set that to 0 by default, since
it is intended for network partition scenarios. maxTakeOverRetries
became 1. So there must be also be a min number of retries property.
* The test failed for the leaving scenario because the singleton
instance was stopped hard without sending the terminationMessage when
the maxTakeOverRetries was exceeded.
