- capture and save snapshots of processor state
- start processor recovery from saved snapshots
- snapshot storage on local filesystem
- snapshot store completely isolated from journal
- LevelDB journal modularized (and completely re-rwritten)
- In-memory journal removed
* Replace (deprecate) akka.cluster.auto-down config setting with
akka.cluster.auto-down-unreachable-after
* AutoDown actor that keeps track of unreachable members
and performs down from the leader node when they have been
unreachable for the specified duration
* Migration guide
- introduce around life cycle hooks for symmetry with aroundReceive
- no custom processor-specific life cycle hooks needed any more
- preStart and preRestart can be overridden with empty implementation
(interceptors ensure that super.preXxx calls are still executed)
- all around life cycle hooks can be final
- standard life cycle hooks are non-final to preserve composability with existing traits (FSM, ...)
- use overridden processor and channel ids
- no need anymore to wait for processor instances to stop
- unrelated: fix wrong artifact names in documentation
The most prominent changes compared to eventsourced are:
- No central processor and channel registry any more
- Auto-recovery of processors on start and restart (can be disabled)
- Recovery of processor networks doesn't require coordination
- Explicit channel activation not needed any more
- Message sequence numbers generated per processor (no gaps)
- Sender references are journaled along with messages
- Processors can determine their recovery status
- No custom API on extension object, only messages
- Journal created by extension from config, not by application
- Applications only interact with processors and channels via messages
- Internal design prepared for having processor-specific journal actors (for later optimization possibilities)
Further additions and changes during review:
- Allow processor implementation classes to use inherited stash
- Channel support to resolve (potentially invalid) sender references
- Logical intead of physical deletion of messages
- Pinned dispatcher for LevelDB journal
- Processor can handle failures during recovery
- Message renamed to Persistent
This prototype has the following limitations:
- Serialization of persistent messages and their payload via JavaSerializer only (will be configurable later)
- The LevelDB journal implementation based on a LevelDB Java port, not the native LevelDB (will be configurable later)
The following features will be added later using separate tickets:
- Snapshot-based recovery
- Reliable channels
- Journal plugin API
- Optimizations
- ...
* Replace unreachable Set with Reachability table
* Unreachable members stay in member Set
* Downing a live member was moved it to the unreachable Set,
and then removed from there by the leader. That will not
work when flipping back to reachable, so a Down member must
be detected as unreachable before beeing removed. Similar
to Exiting. Member shuts down itself if it sees itself as
Down.
* Flip back to reachable when failure detector monitors it as
available again
* ReachableMember event
* Can't ignore gossip from aggregated unreachable (see SurviveNetworkInstabilitySpec)
* Make use of ReachableMember event in cluster router
* End heartbeat when acknowledged, EndHeartbeatAck
* Remove nr-of-end-heartbeats from conf
* Full reachability info in JMX cluster status
* Don't use interval after unreachable for AccrualFailureDetector history
* Add QuarantinedEvent to remoting, used for Reachability.Terminated
* Prune reachability table when all reachable
* Update documentation
* Performance testing and optimizations
* Based on discussion on akka-user, Sept 6-9 2013. It seems appropriate
to mention that Agents are not remoteable, so that naive users (like me)
know to design for that.
* This change has not been locally built with Sphinx, but seems to be
small and safe.
* This can't go into 2.2.x since ScalaTest 1.9.2-SNAP2 has source incompatible changes and the dependecy in akka-multi-node-node-testkit would force people to upgrade.
