From 7535c241640569b089277d0f76004bc33e553a3e Mon Sep 17 00:00:00 2001 From: Roland Date: Mon, 17 Dec 2012 17:19:43 +0100 Subject: [PATCH] discuss message delivery guarantees in more detail, see #2783 - also add PeekMailbox extension for demonstration of that principle --- akka-contrib/docs/index.rst | 1 + akka-contrib/docs/peek-mailbox.rst | 55 +++ akka-contrib/docs/reliable-proxy.rst | 4 +- .../akka/contrib/mailbox/PeekMailbox.scala | 89 +++++ .../contrib/mailbox/PeekMailboxSpec.scala | 128 +++++++ akka-docs/rst/general/index.rst | 4 +- .../general/message-delivery-guarantees.rst | 359 ++++++++++++++++++ .../rst/general/message-send-semantics.rst | 116 ------ 8 files changed, 637 insertions(+), 119 deletions(-) create mode 100644 akka-contrib/docs/peek-mailbox.rst create mode 100644 akka-contrib/src/main/scala/akka/contrib/mailbox/PeekMailbox.scala create mode 100644 akka-contrib/src/test/scala/akka/contrib/mailbox/PeekMailboxSpec.scala create mode 100644 akka-docs/rst/general/message-delivery-guarantees.rst delete mode 100644 akka-docs/rst/general/message-send-semantics.rst diff --git a/akka-contrib/docs/index.rst b/akka-contrib/docs/index.rst index e26b847827..f0d3245187 100644 --- a/akka-contrib/docs/index.rst +++ b/akka-contrib/docs/index.rst @@ -31,6 +31,7 @@ The Current List of Modules reliable-proxy throttle jul + peek-mailbox Suggested Way of Using these Contributions ------------------------------------------ diff --git a/akka-contrib/docs/peek-mailbox.rst b/akka-contrib/docs/peek-mailbox.rst new file mode 100644 index 0000000000..3c7a50ec0d --- /dev/null +++ b/akka-contrib/docs/peek-mailbox.rst @@ -0,0 +1,55 @@ +.. _mailbox-acking: + +Mailbox with Explicit Acknowledgement +===================================== + +When an Akka actor is processing a message and an exception occurs, the normal +behavior is for the actor to drop that message, and then continue with the next +message after it has been restarted. This is in some cases not the desired +solution, e.g. when using failure and supervision to manage a connection to an +unreliable resource; the actor could after the restart go into a buffering mode +(i.e. change its behavior) and retry the real processing later, when the +unreliable resource is back online. + +One way to do this is by sending all messages through the supervisor and +buffering them there, acknowledging successful processing in the child; another +way is to build an explicit acknowledgement mechanism into the mailbox. The +idea with the latter is that a message is reprocessed in case of failure until +the mailbox is told that processing was successful. + +The pattern is implemented `here +<@github@/akka-contrib/src/main/scala/akka/contrib/mailbox/PeekMailbox.scala>`_. +A demonstration of how to use it (although for brevity not a perfect example) +is the following: + +.. includecode:: @contribSrc@/src/test/scala/akka/contrib/mailbox/PeekMailboxSpec.scala + :include: demo + :exclude: business-logic-elided + +Running this application (try it in the Akka sources by saying +``sbt akka-contrib/test:run``) may produce the following output (note the +processing of “World” on lines 2 and 16): + +.. code-block:: none + + Hello + World + [ERROR] [12/17/2012 16:28:36.581] [MySystem-peek-dispatcher-5] [akka://MySystem/user/myActor] DONTWANNA + java.lang.Exception: DONTWANNA + at akka.contrib.mailbox.MyActor.doStuff(PeekMailbox.scala:105) + at akka.contrib.mailbox.MyActor$$anonfun$receive$1.applyOrElse(PeekMailbox.scala:98) + at akka.actor.ActorCell.receiveMessage(ActorCell.scala:425) + at akka.actor.ActorCell.invoke(ActorCell.scala:386) + at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230) + at akka.dispatch.Mailbox.run(Mailbox.scala:212) + at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:502) + at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262) + at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) + at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478) + at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) + World + +Normally one would want to make processing idempotent (i.e. it does not matter +if a message is processed twice) or ``context.become`` a different behavior +upon restart; the above example included the ``println(msg)`` call just to +demonstrate the re-processing. diff --git a/akka-contrib/docs/reliable-proxy.rst b/akka-contrib/docs/reliable-proxy.rst index 74470aaf3f..4bba12f9eb 100644 --- a/akka-contrib/docs/reliable-proxy.rst +++ b/akka-contrib/docs/reliable-proxy.rst @@ -1,7 +1,9 @@ +.. _reliable-proxy: + Reliable Proxy Pattern ====================== -Looking at :ref:`message-send-semantics` one might come to the conclusion that +Looking at :ref:`message-delivery-guarantees` one might come to the conclusion that Akka actors are made for blue-sky scenarios: sending messages is the only way for actors to communicate, and then that is not even guaranteed to work. Is the whole paradigm built on sand? Of course the answer is an emphatic “No!”. diff --git a/akka-contrib/src/main/scala/akka/contrib/mailbox/PeekMailbox.scala b/akka-contrib/src/main/scala/akka/contrib/mailbox/PeekMailbox.scala new file mode 100644 index 0000000000..bb580bccb9 --- /dev/null +++ b/akka-contrib/src/main/scala/akka/contrib/mailbox/PeekMailbox.scala @@ -0,0 +1,89 @@ +package akka.contrib.mailbox + +import java.util.concurrent.{ ConcurrentHashMap, ConcurrentLinkedQueue } + +import com.typesafe.config.Config + +import akka.actor.{ ActorContext, ActorRef, ActorSystem, ExtendedActorSystem, Extension, ExtensionId, ExtensionIdProvider } +import akka.dispatch.{ Envelope, MailboxType, MessageQueue, QueueBasedMessageQueue, UnboundedMessageQueueSemantics } + +object PeekMailboxExtension extends ExtensionId[PeekMailboxExtension] with ExtensionIdProvider { + def lookup = this + def createExtension(s: ExtendedActorSystem) = new PeekMailboxExtension(s) + + def ack()(implicit context: ActorContext): Unit = PeekMailboxExtension(context.system).ack() +} + +class PeekMailboxExtension(val system: ExtendedActorSystem) extends Extension { + private val mailboxes = new ConcurrentHashMap[ActorRef, PeekMailbox] + + def register(actorRef: ActorRef, mailbox: PeekMailbox): Unit = + mailboxes.put(actorRef, mailbox) + + def unregister(actorRef: ActorRef): Unit = mailboxes.remove(actorRef) + + def ack()(implicit context: ActorContext): Unit = + mailboxes.get(context.self) match { + case null ⇒ throw new IllegalArgumentException("Mailbox not registered for: " + context.self) + case mailbox ⇒ mailbox.ack() + } +} + +/** + * configure the mailbox via dispatcher configuration: + * {{{ + * peek-dispatcher { + * mailbox-type = "example.PeekMailboxType" + * } + * }}} + */ +class PeekMailboxType(settings: ActorSystem.Settings, config: Config) extends MailboxType { + override def create(owner: Option[ActorRef], system: Option[ActorSystem]) = (owner, system) match { + case (Some(o), Some(s)) ⇒ + val tries = config.getInt("max-tries") ensuring (_ >= 1, "max-tries must be at least 1") + val mailbox = new PeekMailbox(o, s, tries) + PeekMailboxExtension(s).register(o, mailbox) + mailbox + case _ ⇒ throw new Exception("no mailbox owner or system given") + } +} + +class PeekMailbox(owner: ActorRef, system: ActorSystem, maxTries: Int) + extends QueueBasedMessageQueue with UnboundedMessageQueueSemantics { + final val queue = new ConcurrentLinkedQueue[Envelope]() + + /* + * Since the queue itself is used to determine when to schedule the actor + * (see Mailbox.hasMessages), we cannot poll() on the first try and then + * continue handing back out that same message until ACKed, peek() must be + * used. The retry limit logic is then formulated in terms of the `tries` + * field, which holds + * 0 if clean slate (i.e. last dequeue was ack()ed) + * 1..maxTries if not yet ack()ed + * Marker if last try was done (at which point we had to poll()) + * -1 during cleanUp (in order to disable the ack() requirement) + */ + // the mutable state is only ever accessed by the actor (i.e. dequeue() side) + var tries = 0 + val Marker = maxTries + 1 + + override def dequeue(): Envelope = tries match { + case -1 ⇒ queue.poll() + case 0 | Marker ⇒ tries = 1; queue.peek() + case `maxTries` ⇒ tries = Marker; queue.poll() + case n ⇒ tries = n + 1; queue.peek() + } + + def ack(): Unit = { + // do not dequeue for real if double-ack() or ack() on last try + if (tries != 0 && tries != Marker) queue.poll() + tries = 0 + } + + override def cleanUp(owner: ActorRef, deadLetters: MessageQueue): Unit = { + tries = -1 // put the queue into auto-ack mode + super.cleanUp(owner, deadLetters) + PeekMailboxExtension(system).unregister(owner) + } +} + diff --git a/akka-contrib/src/test/scala/akka/contrib/mailbox/PeekMailboxSpec.scala b/akka-contrib/src/test/scala/akka/contrib/mailbox/PeekMailboxSpec.scala new file mode 100644 index 0000000000..1c27ce4828 --- /dev/null +++ b/akka-contrib/src/test/scala/akka/contrib/mailbox/PeekMailboxSpec.scala @@ -0,0 +1,128 @@ +package akka.contrib.mailbox + +import com.typesafe.config.ConfigFactory + +import akka.actor.{ Actor, ActorSystem, DeadLetter, PoisonPill, Props, Terminated, actorRef2Scala } +import akka.testkit.{ AkkaSpec, EventFilter, ImplicitSender } + +object PeekMailboxSpec { + case object Check + case object DoubleAck + class PeekActor(tries: Int) extends Actor { + var togo = tries + def receive = { + case Check ⇒ + sender ! Check + PeekMailboxExtension.ack() + case DoubleAck ⇒ + PeekMailboxExtension.ack() + PeekMailboxExtension.ack() + case msg ⇒ + sender ! msg + if (togo == 0) throw new RuntimeException("DONTWANNA") + togo -= 1 + PeekMailboxExtension.ack() + } + override def preRestart(cause: Throwable, msg: Option[Any]) { + for (m ← msg if m == "DIE") context stop self // for testing the case of mailbox.cleanUp + } + } +} + +class PeekMailboxSpec extends AkkaSpec(""" + peek-dispatcher { + mailbox-type = "akka.contrib.mailbox.PeekMailboxType" + max-tries = 3 + } + """) with ImplicitSender { + + import PeekMailboxSpec._ + + "A PeekMailbox" must { + + "retry messages" in { + val a = system.actorOf(Props(new PeekActor(1)).withDispatcher("peek-dispatcher")) + a ! "hello" + expectMsg("hello") + EventFilter[RuntimeException]("DONTWANNA", occurrences = 1) intercept { + a ! "world" + } + expectMsg("world") + expectMsg("world") + a ! Check + expectMsg(Check) + } + + "put a bound on retries" in { + val a = system.actorOf(Props(new PeekActor(0)).withDispatcher("peek-dispatcher")) + EventFilter[RuntimeException]("DONTWANNA", occurrences = 3) intercept { + a ! "hello" + } + a ! Check + expectMsg("hello") + expectMsg("hello") + expectMsg("hello") + expectMsg(Check) + } + + "not waste messages on double-ack()" in { + val a = system.actorOf(Props(new PeekActor(0)).withDispatcher("peek-dispatcher")) + a ! DoubleAck + a ! Check + expectMsg(Check) + } + + "support cleanup" in { + system.eventStream.subscribe(testActor, classOf[DeadLetter]) + val a = system.actorOf(Props(new PeekActor(0)).withDispatcher("peek-dispatcher")) + watch(a) + EventFilter[RuntimeException]("DONTWANNA", occurrences = 1) intercept { + a ! "DIE" // stays in the mailbox + } + expectMsg("DIE") + expectMsgType[DeadLetter].message must be("DIE") + expectMsgType[Terminated].actor must be(a) + } + + } + +} + +//#demo +class MyActor extends Actor { + def receive = { + case msg ⇒ + println(msg) + doStuff(msg) // may fail + PeekMailboxExtension.ack() + } + + //#business-logic-elided + var i = 0 + def doStuff(m: Any) { + if (i == 1) throw new Exception("DONTWANNA") + i += 1 + } + + override def postStop() { + context.system.shutdown() + } + //#business-logic-elided +} + +object MyApp extends App { + val system = ActorSystem("MySystem", ConfigFactory.parseString(""" + peek-dispatcher { + mailbox-type = "akka.contrib.mailbox.PeekMailboxType" + max-tries = 2 + } + """)) + + val myActor = system.actorOf(Props[MyActor].withDispatcher("peek-dispatcher"), + name = "myActor") + + myActor ! "Hello" + myActor ! "World" + myActor ! PoisonPill +} +//#demo diff --git a/akka-docs/rst/general/index.rst b/akka-docs/rst/general/index.rst index e76a81a80c..ca50cae55d 100644 --- a/akka-docs/rst/general/index.rst +++ b/akka-docs/rst/general/index.rst @@ -10,5 +10,5 @@ General addressing remoting jmm - message-send-semantics - configuration \ No newline at end of file + message-delivery-guarantees + configuration diff --git a/akka-docs/rst/general/message-delivery-guarantees.rst b/akka-docs/rst/general/message-delivery-guarantees.rst new file mode 100644 index 0000000000..86b0716c90 --- /dev/null +++ b/akka-docs/rst/general/message-delivery-guarantees.rst @@ -0,0 +1,359 @@ + +.. _message-delivery-guarantees: + +########################### +Message Delivery Guarantees +########################### + +Akka helps you build reliable applications which make use of multiple processor +cores in one machine (“scaling up”) or distributed across a computer network +(“scaling out”). The key abstraction to make this work is that all interactions +between your code units—actors—happen via message passing, which is why the +precise semantics of how messages are passed between actors deserve their own +chapter. + +In order to give some context to the discussion below, consider an application +which spans multiple network hosts. The basic mechanism for communication is +the same whether sending to an actor on the local JVM or to a remote actor, but +of course there will be observable differences in the latency of delivery +(possibly also depending on the bandwidth of the network link and the message +size) and the reliability. In case of a remote message send there are obviously +more steps involved which means that more can go wrong. Another aspect is that +local sending will just pass a reference to the message inside the same JVM, +without any restrictions on the underlying object which is sent, whereas a +remote transport will place a limit on the message size. + +Writing your actors such that every interaction could possibly be remote is the +safe, pessimistic bet. It means to only rely on those properties which are +always guaranteed and which are discussed in detail below. This has of course +some overhead in the actor’s implementation. If you are willing to sacrifice full +location transparency—for example in case of a group of closely collaborating +actors—you can place them always on the same JVM and enjoy stricter guarantees +on message delivery. The details of this trade-off are discussed further below. + +As a supplementary part we give a few pointers at how to build stronger +guarantees on top of the built-in ones. The chapter closes by discussing the +role of the “Dead Letter Office”. + +The General Rules +================= + +These are the rules for message sends (i.e. the ``tell`` or ``!`` method, which +also underlies the ``ask`` pattern): + +* **at-most-once delivery**, i.e. no guaranteed delivery +* **message ordering per sender–receiver pair** + +The first rule is typically found also in other actor implementations while the +second is specific to Akka. + +Discussion: What does “at-most-once” mean? +------------------------------------------ + +When it comes to describing the semantics of a delivery mechanism, there are +three basic categories: + +* **at-most-once** delivery means that for each message handed to the + mechanism, that message is delivered zero or one times; in more casual terms + it means that messages may be lost. + +* **at-least-once** delivery means that for each message handed to the + mechanism potentially multiple attempts are made at delivering it, such that + at least one succeeds; again, in more casual terms this means that messages + may be duplicated but not lost. + +* **exactly-once** delivery means that for each message handed to the mechanism + exactly one delivery is made to the recipient; the message can neither be + lost nor duplicated. + +The first one is the cheapest—highest performance, least implementation +overhead—because it can be done in a fire-and-forget fashion without keeping +state at the sending end or in the transport mechanism. The second one requires +retries to counter transport losses, which means keeping state at the sending +end and having an acknowledgement mechanism at the receiving end. The third is +most expensive—and has consequently worst performance—because in addition to +the second it requires state to be kept at the receiving end in order to filter +out duplicate deliveries. + +Discussion: Why No Guaranteed Delivery? +--------------------------------------- + +At the core of the problem lies the question what exactly this guarantee shall +mean: + +1. The message is sent out on the network? +2. The message is received by the other host? +3. The message is put into the target actor's mailbox? +4. The message is starting to be processed by the target actor? +5. The message is processed successfully by the target actor? + +Each one of these have different challenges and costs, and it is obvious that +there are conditions under which any message passing library would be unable to +comply; think for example about configurable mailbox types and how a bounded +mailbox would interact with the third point, or even what it would mean to +decide upon the “successfully” part of point five. + +Along those same lines goes the reasoning in `Nobody Needs Reliable +Messaging`_. The only meaningful way for a sender to know whether an +interaction was successful is by receiving a business-level acknowledgement +message, which is not something Akka could make up on its own (neither are we +writing a “do what I mean” framework nor would you want us to). + +Akka embraces distributed computing and makes the fallibility of communication +explicit through message passing, therefore it does not try to lie and emulate +a leaky abstraction. This is a model that has been used with great success in +Erlang and requires the users to design their applications around it. You can +read more about this approach in the `Erlang documentation`_ (section 10.9 and +10.10), Akka follows it closely. + +Another angle on this issue is that by providing only basic guarantees those +use cases which do not need stricter guarantees do not pay the cost of their +implementation; it is always possible to add stricter guarantees on top of +basic ones, but it is not possible to retro-actively remove guarantees in order +to gain more performance. + +Discussion: Message Ordering +---------------------------- + +The rule more specifically is that *for a given pair of actors, messages sent +from the first to the second will not be received out-of-order.* This is +illustrated in the following: + + Actor ``A1`` sends messages ``M1``, ``M2``, ``M3`` to ``A2`` + + Actor ``A3`` sends messages ``M4``, ``M5``, ``M6`` to ``A2`` + + This means that: + 1) If ``M1`` is delivered it must be delivered before ``M2`` and ``M3`` + 2) If ``M2`` is delivered it must be delivered before ``M3`` + 3) If ``M4`` is delivered it must be delivered before ``M5`` and ``M6`` + 4) If ``M5`` is delivered it must be delivered before ``M6`` + 5) ``A2`` can see messages from ``A1`` interleaved with messages from ``A3`` + 6) Since there is no guaranteed delivery, any of the messages may be dropped, i.e. not arrive at ``A2`` + +.. note:: + + It is important to note that Akka’s guarantee applies to the order in which + messages are enqueued into the recipient’s mailbox. If the mailbox + implementation does not respect FIFO order (e.g. a :class:`PriorityMailbox`), + then the order of processing by the actor can deviate from the enqueueing + order. + +Please note that this rule is **not transitive**: + + Actor ``A`` sends message ``M1`` to actor ``C`` + + Actor ``A`` then sends message ``M2`` to actor ``B`` + + Actor ``B`` forwards message ``M2`` to actor ``C`` + + Actor ``C`` may receive ``M1`` and ``M2`` in any order + +Causal transitive ordering would imply that ``M2`` is never received before +``M1`` at actor ``C`` (though any of them might be lost). This ordering can be +violated due to different message delivery latencies when ``A``, ``B`` and +``C`` reside on different network hosts, see more below. + +The Rules for In-JVM (Local) Message Sends +========================================== + +Be careful what you do with this section! +----------------------------------------- + +Relying on the stronger guarantees in this section is not recommended since it +will bind your application to local-only deployment: an application may have to +be designed differently (as opposed to just employing some message exchange +patterns local to some actors) in order to be fit for running on a cluster of +machines. Our credo is “design once, deploy any way you wish”, and to achieve +this you should only rely on `The General Rules`_. + +Reliability of Local Message Sends +---------------------------------- + +The Akka test suite relies on not losing messages in the local context (and for +non-error condition tests also for remote deployment), meaning that the we +actually do apply the best effort to keep our tests stable. A local ``tell`` +operation can however fail for the same reasons as a normal method call can on +the JVM: + +- :class:`StackOverflowError` +- :class:`OutOfMemoryError` +- other :class:`VirtualMachineError` + +In addition, local sends can fail in Akka-specific ways: + +- if the mailbox does not accept the message (e.g. full BoundedMailbox) +- if the receiving actor fails while processing the message or is already + terminated + +While the first is clearly a matter of configuration the second deserves some +thought: the sender of a message does not get feedback if there was an +exception while processing, that notification goes to the supervisor instead. +This is in general not distinguishable from a lost message for an outside +observer. + +Ordering of Local Message Sends +------------------------------- + +Assuming strict FIFO mailboxes the abovementioned caveat of non-transitivity of +the message ordering guarantee is eliminated under certain conditions. As you +will note, these are quite subtle as it stands, and it is even possible that +future performance optimizations will invalidate this whole paragraph. The +possibly non-exhaustive list of counter-indications is: + +- Before receiving the first reply from a top-level actor, there is a lock + which protects an internal interim queue, and this lock is not fair; the + implication is that enqueue requests from different senders which arrive + during the actor’s construction (figuratively, the details are more involved) + may be reordered depending on low-level thread scheduling. Since completely + fair locks do not exist on the JVM this is unfixable. + +- The same mechanism is used during the construction of a Router, more + precisely the routed ActorRef, hence the same problem exists for actors + deployed with Routers. + +- As mentioned above, the problem occurs anywhere a lock is involved during + enqueueing, which may also apply to custom mailboxes (or durable mailboxes). + +This list has been compiled carefully, but other problematic scenarios may have +escaped our analysis. + +How does Local Ordering relate to Network Ordering +-------------------------------------------------- + +As explained in the previous paragraph local message sends obey transitive +causal ordering under certain conditions. If the remote message transport would +respect this ordering as well, that would translate to transitive causal +ordering across one network link, i.e. if exactly two network hosts are +involved. Involving multiple links, e.g. the three actors on three different +nodes mentioned above, then no guarantees can be made. + +The current remote transport does **not** support this (again this is caused by +non-FIFO wake-up order of a lock, this time serializing connection +establishment). + +As a speculative view into the future it might be possible to support this +ordering guarantee by re-implementing the remote transport layer based +completely on actors; at the same time we are looking into providing other +low-level transport protocols like UDP or SCTP which would enable higher +throughput or lower latency by removing this guarantee again, which would mean +that choosing between different implementations would allow trading guarantees +versus performance. + +Building On Top Of Akka +======================= + +The philosophy of Akka is to provide a small and consistent tool set which is +well suited for building powerful abstractions on top. + +Messaging Patterns +------------------ + +As discussed above a straight-forward answer to the requirement of guaranteed +delivery is an explicit ACK–RETRY protocol. In its simplest form this requires + +- a way to identify individual messages to correlate message with + acknowledgement +- a retry mechanism which will resend messages if not acknowledged in time +- a way for the receiver to detect and discard duplicates + +The third becomes necessary by virtue of the acknowledgements not being +guaranteed to arrive either. An example of implementing all three requirements +is shown at :ref:`reliable-proxy`. Another way of implementing the third part +would be to make processing the messages idempotent at the receiving end on the +level of the business logic; this is convenient if it arises naturally and +otherwise implemented by keeping track of processed message IDs. + +Event Sourcing +-------------- + +Event sourcing (and sharding) is what makes large websites scale to +billions of users, and the idea is quite simple: when a component (think actor) +processes a command it will generate a list of events representing the effect +of the command. These events are stored in addition to being applied to the +component’s state. The nice thing about this scheme is that events only ever +are appended to the storage, nothing is ever mutated; this enables perfect +replication and scaling of consumers of this event stream (i.e. other +components may consume the event stream as a means to replicate the component’s +state on a different continent or to react to changes). If the component’s +state is lost—due to a machine failure or by being pushed out of a cache—it can +easily be reconstructed by replaying the event stream (usually employing +snapshots to speed up the process). Read a lot more about `Event Sourcing`_. + +Martin Krasser has written an implementation of event sourcing principles on +top of Akka called `eventsourced`_, including support for guaranteed delivery +semantics as described in the previous section. + +.. _Event Sourcing: http://martinfowler.com/eaaDev/EventSourcing.html +.. _eventsourced: https://github.com/eligosource/eventsourced + +Mailbox with Explicit Acknowledgement +------------------------------------- + +By implementing a custom mailbox type it is possible retry message processing +at the receiving actor’s end in order to handle temporary failures. This +pattern is mostly useful in the local communication context where delivery +guarantees are otherwise sufficient to fulfill the application’s requirements. + +Please note that the caveats for `The Rules for In-JVM (Local) Message Sends`_ +do apply. + +An example implementation of this pattern is shown at :ref:`mailbox-acking`. + +.. _deadletters: + +Dead Letters +============ + +Messages which cannot be delivered (and for which this can be ascertained) will +be delivered to a synthetic actor called ``/deadLetters``. This delivery +happens on a best-effort basis; it may fail even within the local JVM (e.g. +during actor termination). Messages sent via unreliable network transports will +be lost without turning up as dead letters. + +What Should I Use Dead Letters For? +----------------------------------- + +The main use of this facility is for debugging, especially if an actor send +does not arrive consistently (where usually inspecting the dead letters will +tell you that the sender or recipient was set wrong somewhere along the way). +In order to be useful for this purpose it is good practice to avoid sending to +deadLetters where possible, i.e. run your application with a suitable dead +letter logger (see more below) from time to time and clean up the log output. +This exercise—like all else—requires judicious application of common sense: it +may well be that avoiding to send to a terminated actor complicates the +sender’s code more than is gained in debug output clarity. + +The dead letter service follows the same rules with respect to delivery +guarantees as all other message sends, hence it cannot be used to implement +guaranteed delivery. + +How do I Receive Dead Letters? +------------------------------ + +An actor can subscribe to class :class:`akka.actor.DeadLetter` on the event +stream, see :ref:`event-stream-java` (Java) or :ref:`event-stream-scala` +(Scala) for how to do that. The subscribed actor will then receive all dead +letters published in the (local) system from that point onwards. Dead letters +are not propagated over the network, if you want to collect them in one place +you will have to subscribe one actor per network node and forward them +manually. Also consider that dead letters are generated at that node which can +determine that a send operation is failed, which for a remote send can be the +local system (if no network connection can be established) or the remote one +(if the actor you are sending to does not exist at that point in time). + +Dead Letters Which are (Usually) not Worrisome +---------------------------------------------- + +Every time an actor does not terminate by its own decision, there is a chance +that some messages which it sends to itself are lost. There is one which +happens quite easily in complex shutdown scenarios that is usually benign: +seeing a :class:`akka.dispatch.Terminate` message dropped means that two +termination requests were given, but of course only one can succeed. In the +same vein, you might see :class:`akka.actor.Terminated` messages from children +while stopping a hierarchy of actors turning up in dead letters if the parent +is still watching the child when the parent terminates. + +.. _Erlang documentation: http://www.erlang.org/faq/academic.html +.. _Nobody Needs Reliable Messaging: http://www.infoq.com/articles/no-reliable-messaging + diff --git a/akka-docs/rst/general/message-send-semantics.rst b/akka-docs/rst/general/message-send-semantics.rst deleted file mode 100644 index 2a83c38532..0000000000 --- a/akka-docs/rst/general/message-send-semantics.rst +++ /dev/null @@ -1,116 +0,0 @@ - -.. _message-send-semantics: - -####################### - Message send semantics -####################### - - - -Guaranteed Delivery? -==================== - -Akka does *not* support guaranteed delivery. - -First it is close to impossible to actually give guarantees like that, -second it is extremely costly trying to do so. -The network is inherently unreliable and there is no such thing as 100% -guarantee delivery, so it can never be guaranteed. - -The question is what to guarantee. That: - -1. The message is sent out on the network? -2. The message is received by the other host? -3. The message is put on the target actor's mailbox? -4. The message is applied to the target actor? -5. The message is starting to be executed by the target actor? -6. The message is finished executing by the target actor? - -Each one of this have different challenges and costs. - -Akka embraces distributed computing and the network and makes it explicit -through message passing, therefore it does not try to lie and emulate a -leaky abstraction. This is a model that have been used with great success -in Erlang and requires the user to model his application around. You can -read more about this approach in the `Erlang documentation`_ (section -10.9 and 10.10), Akka follows it closely. - -Bottom line: you as a developer know what guarantees you need in your -application and can solve it fastest and most reliable by explicit ``ACK`` and -``RETRY`` (if you really need it, most often you don't). Using Akka's Durable -Mailboxes could help with this. - -Delivery semantics -================== - -At-most-once ------------- - -Actual transports may provide stronger semantics, -but at-most-once is the semantics you should expect. -The alternatives would be once-and-only-once, which is extremely costly, -or at-least-once which essentially requires idempotency of message processing, -which is a user-level concern. - -Ordering is preserved on a per-sender basis -------------------------------------------- - -Actor ``A1`` sends messages ``M1``, ``M2``, ``M3`` to ``A2`` -Actor ``A3`` sends messages ``M4``, ``M5``, ``M6`` to ``A2`` - -This means that: - 1) If ``M1`` is delivered it must be delivered before ``M2`` and ``M3`` - 2) If ``M2`` is delivered it must be delivered before ``M3`` - 3) If ``M4`` is delivered it must be delivered before ``M5`` and ``M6`` - 4) If ``M5`` is delivered it must be delivered before ``M6`` - 5) ``A2`` can see messages from ``A1`` interleaved with messages from ``A3`` - 6) Since there is no guaranteed delivery, none, some or all of the messages may arrive to ``A2`` - -.. _deadletters: - -Dead Letters -============ - -Messages which cannot be delivered (and for which this can be ascertained) will -be delivered to a synthetic actor called ``/deadLetters``. This delivery -happens on a best-effort basis; it may fail even within the local JVM (e.g. -during actor termination). Messages sent via unreliable network transports will -be lost without turning up as dead letters. - -How do I Receive Dead Letters? ------------------------------- - -An actor can subscribe to class :class:`akka.actor.DeadLetter` on the event -stream, see :ref:`event-stream-java` (Java) or :ref:`event-stream-scala` -(Scala) for how to do that. The subscribed actor will then receive all dead -letters published in the (local) system from that point onwards. Dead letters -are not propagated over the network, if you want to collect them in one place -you will have to subscribe one actor per network node and forward them -manually. Also consider that dead letters are generated at that node which can -determine that a send operation is failed, which for a remote send can be the -local system (if no network connection can be established) or the remote one -(if the actor you are sending to does not exist at that point in time). - -What Should I Use Dead Letters For? ------------------------------------ - -The dead letter service follows the same rules with respect to delivery -guarantees as all other message sends, hence it cannot be used to implement -guaranteed delivery. The main use is for debugging, especially if an actor send -does not arrive consistently (where usually inspecting the dead letters will -tell you that the sender or recipient was set wrong somewhere along the way). - -Dead Letters Which are (Usually) not Worrisome ----------------------------------------------- - -Every time an actor does not terminate by its own decision, there is a chance -that some messages which it sends to itself are lost. There is one which -happens quite easily in complex shutdown scenarios that is usually benign: -seeing a :class:`akka.dispatch.Terminate` message dropped means that two -termination requests were given, but of course only one can succeed. In the -same vein, you might see :class:`akka.actor.Terminated` messages from children -while stopping a hierarchy of actors turning up in dead letters if the parent -is still watching the child when the parent terminates. - -.. _Erlang documentation: http://www.erlang.org/faq/academic.html -