diff --git a/.gitignore b/.gitignore index 02f7ff993d..7dbbb248f7 100755 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ *# +*.md= *.log *.orig *.jfr diff --git a/akka-docs/src/main/paradox/additional/books.md b/akka-docs/src/main/paradox/additional/books.md index 5737625a7c..4bbd95359b 100644 --- a/akka-docs/src/main/paradox/additional/books.md +++ b/akka-docs/src/main/paradox/additional/books.md @@ -1,17 +1,16 @@ -Books -===== -* `Mastering Akka `_, by Christian Baxter, PACKT Publishing, ISBN: 9781786465023, October 2016 -* `Learning Akka `_, by Jason Goodwin, PACKT Publishing, ISBN: 9781784393007, December 2015 -* `Akka in Action `_, by Raymond Roestenburg and Rob Bakker, Manning Publications Co., ISBN: 9781617291012, September 2016 -* `Reactive Messaging Patterns with the Actor Model `_, by Vaughn Vernon, Addison-Wesley Professional, ISBN: 0133846830, August 2015 -* `Developing an Akka Edge `_, by Thomas Lockney and Raymond Tay, Bleeding Edge Press, ISBN: 9781939902054, April 2014 -* `Effective Akka `_, by Jamie Allen, O'Reilly Media, ISBN: 1449360076, August 2013 -* `Akka Concurrency `_, by Derek Wyatt, artima developer, ISBN: 0981531660, May 2013 -* `Akka Essentials `_, by Munish K. Gupta, PACKT Publishing, ISBN: 1849518289, October 2012 +# Books -Videos -====== + * [Mastering Akka](https://www.packtpub.com/application-development/mastering-akka), by Christian Baxter, PACKT Publishing, ISBN: 9781786465023, October 2016 + * [Learning Akka](https://www.packtpub.com/application-development/learning-akka), by Jason Goodwin, PACKT Publishing, ISBN: 9781784393007, December 2015 + * [Akka in Action](http://www.lightbend.com/resources/e-book/akka-in-action), by Raymond Roestenburg and Rob Bakker, Manning Publications Co., ISBN: 9781617291012, September 2016 + * [Reactive Messaging Patterns with the Actor Model](http://www.informit.com/store/reactive-messaging-patterns-with-the-actor-model-applications-9780133846836), by Vaughn Vernon, Addison-Wesley Professional, ISBN: 0133846830, August 2015 + * [Developing an Akka Edge](http://bleedingedgepress.com/our-books/developing-an-akka-edge/), by Thomas Lockney and Raymond Tay, Bleeding Edge Press, ISBN: 9781939902054, April 2014 + * [Effective Akka](http://shop.oreilly.com/product/0636920028789.do), by Jamie Allen, O'Reilly Media, ISBN: 1449360076, August 2013 + * [Akka Concurrency](http://www.artima.com/shop/akka_concurrency), by Derek Wyatt, artima developer, ISBN: 0981531660, May 2013 + * [Akka Essentials](https://www.packtpub.com/application-development/akka-essentials), by Munish K. Gupta, PACKT Publishing, ISBN: 1849518289, October 2012 -* `Learning Akka Videos `_, by Salma Khater, PACKT Publishing, ISBN: 9781784391836, January 2016 -* `Building Microservice with AKKA HTTP (Video) `_, by Tomasz Lelek, PACKT Publishing, ISBN: 9781788298582, March 2017 +# Videos + + * [Learning Akka Videos](https://www.packtpub.com/application-development/learning-akka-video), by Salma Khater, PACKT Publishing, ISBN: 9781784391836, January 2016 + * [Building Microservice with AKKA HTTP (Video)](https://www.packtpub.com/application-development/building-microservice-akka-http-video), by Tomasz Lelek, PACKT Publishing, ISBN: 9781788298582, March 2017 \ No newline at end of file diff --git a/akka-docs/src/main/paradox/additional/faq.md b/akka-docs/src/main/paradox/additional/faq.md index 0af3dfefd8..b7f68eb2d5 100644 --- a/akka-docs/src/main/paradox/additional/faq.md +++ b/akka-docs/src/main/paradox/additional/faq.md @@ -1,13 +1,10 @@ -Frequently Asked Questions -========================== +# Frequently Asked Questions -Akka Project -^^^^^^^^^^^^ +## Akka Project -Where does the name Akka come from? ------------------------------------ +### Where does the name Akka come from? -It is the name of a beautiful Swedish `mountain `_ +It is the name of a beautiful Swedish [mountain](https://lh4.googleusercontent.com/-z28mTALX90E/UCOsd249TdI/AAAAAAAAAB0/zGyNNZla-zY/w442-h331/akka-beautiful-panorama.jpg) up in the northern part of Sweden called Laponia. The mountain is also sometimes called 'The Queen of Laponia'. @@ -19,26 +16,24 @@ Also, the name AKKA is the a palindrome of letters A and K as in Actor Kernel. Akka is also: -* the name of the goose that Nils traveled across Sweden on in `The Wonderful Adventures of Nils `_ by the Swedish writer Selma Lagerlöf. -* the Finnish word for 'nasty elderly woman' and the word for 'elder sister' in the Indian languages Tamil, Telugu, Kannada and Marathi. -* a `font `_ -* a town in Morocco -* a near-earth asteroid + * the name of the goose that Nils traveled across Sweden on in [The Wonderful Adventures of Nils](http://en.wikipedia.org/wiki/The_Wonderful_Adventures_of_Nils) by the Swedish writer Selma Lagerlöf. + * the Finnish word for 'nasty elderly woman' and the word for 'elder sister' in the Indian languages Tamil, Telugu, Kannada and Marathi. + * a [font](http://www.dafont.com/akka.font) + * a town in Morocco + * a near-earth asteroid -Resources with Explicit Lifecycle -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## Resources with Explicit Lifecycle Actors, ActorSystems, ActorMaterializers (for streams), all these types of objects bind resources that must be released explicitly. The reason is that Actors are meant to have a life of their own, existing independently of whether messages are currently en route to them. Therefore you should always make sure that for every creation of such an object -you have a matching ``stop``, ``terminate``, or ``shutdown`` call implemented. +you have a matching `stop`, `terminate`, or `shutdown` call implemented. In particular you typically want to bind such values to immutable references, i.e. -``final ActorSystem system`` in Java or ``val system: ActorSystem`` in Scala. +`final ActorSystem system` in Java or `val system: ActorSystem` in Scala. -JVM application or Scala REPL “hanging” ---------------------------------------- +### JVM application or Scala REPL “hanging” Due to an ActorSystem’s explicit lifecycle the JVM will not exit until it is stopped. Therefore it is necessary to shutdown all ActorSystems within a running application or @@ -47,11 +42,9 @@ Scala REPL session in order to allow these processes to terminate. Shutting down an ActorSystem will properly terminate all Actors and ActorMaterializers that were created within it. -Actors in General -^^^^^^^^^^^^^^^^^ +## Actors in General -sender()/getSender() disappears when I use Future in my Actor, why? -------------------------------------------------------------------- +### sender()/getSender() disappears when I use Future in my Actor, why? When using future callbacks, inside actors you need to carefully avoid closing over the containing actor’s reference, i.e. do not call methods or access mutable state @@ -60,10 +53,9 @@ and may introduce synchronization bugs and race conditions because the callback be scheduled concurrently to the enclosing actor. Unfortunately there is not yet a way to detect these illegal accesses at compile time. -Read more about it in the docs for :ref:`jmm-shared-state`. +Read more about it in the docs for @ref:[Actors and shared mutable state](../general/jmm.md#jmm-shared-state). -Why OutOfMemoryError? ---------------------- +### Why OutOfMemoryError? It can be many reasons for OutOfMemoryError. For example, in a pure push based system with message consumers that are potentially slower than corresponding message producers you must @@ -72,14 +64,12 @@ mailboxes and thereby filling up the heap memory. Some articles for inspiration: -* `Balancing Workload across Nodes with Akka 2 `_. -* `Work Pulling Pattern to prevent mailbox overflow, throttle and distribute work `_ + * [Balancing Workload across Nodes with Akka 2](http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2). + * [Work Pulling Pattern to prevent mailbox overflow, throttle and distribute work](http://www.michaelpollmeier.com/akka-work-pulling-pattern) -Actors Scala API -^^^^^^^^^^^^^^^^ +## Actors Scala API -How can I get compile time errors for missing messages in `receive`? --------------------------------------------------------------------- +### How can I get compile time errors for missing messages in *receive*? One solution to help you get a compile time warning for not handling a message that you should be handling is to define your actors input and output messages @@ -89,13 +79,11 @@ exhaustiveness. Here is an example where the compiler will warn you that the match in receive isn't exhaustive: -.. includecode:: code/docs/faq/Faq.scala#exhaustiveness-check +@@snip [Faq.scala](code/docs/faq/Faq.scala) { #exhaustiveness-check } -Remoting -^^^^^^^^ +## Remoting -I want to send to a remote system but it does not do anything -------------------------------------------------------------- +### I want to send to a remote system but it does not do anything Make sure that you have remoting enabled on both ends: client and server. Both do need hostname and port configured, and you will need to know the port of the @@ -105,86 +93,81 @@ be different If you still do not see anything, look at what the logging of remote life-cycle events tells you (normally logged at INFO level) or switch on -:ref:`logging-remote-java` +@ref:[Auxiliary remote logging options](../java/logging.md#logging-remote-java) to see all sent and received messages (logged at DEBUG level). -Which options shall I enable when debugging remoting issues? ------------------------------------------------------------- +### Which options shall I enable when debugging remoting issues? -Have a look at the :ref:`remote-configuration-java`, the typical candidates are: +Have a look at the @ref:[Remote Configuration](../java/remoting.md#remote-configuration-java), the typical candidates are: -* `akka.remote.log-sent-messages` -* `akka.remote.log-received-messages` -* `akka.remote.log-remote-lifecycle-events` (this also includes deserialization errors) + * *akka.remote.log-sent-messages* + * *akka.remote.log-received-messages* + * *akka.remote.log-remote-lifecycle-events* (this also includes deserialization errors) -What is the name of a remote actor? ------------------------------------ +### What is the name of a remote actor? When you want to send messages to an actor on a remote host, you need to know -its :ref:`full path `, which is of the form:: +its @ref:[full path](../general/addressing.md), which is of the form: - akka.protocol://system@host:1234/user/my/actor/hierarchy/path +``` +akka.protocol://system@host:1234/user/my/actor/hierarchy/path +``` Observe all the parts you need here: -* ``protocol`` is the protocol to be used to communicate with the remote system. - Most of the cases this is `tcp`. + * + `protocol` + is the protocol to be used to communicate with the remote system. + : Most of the cases this is *tcp*. + + * `system` is the remote system’s name (must match exactly, case-sensitive!) + * `host` is the remote system’s IP address or DNS name, and it must match that +system’s configuration (i.e. *akka.remote.netty.tcp.hostname*) + * `1234` is the port number on which the remote system is listening for +connections and receiving messages + * `/user/my/actor/hierarchy/path` is the absolute path of the remote actor in +the remote system’s supervision hierarchy, including the system’s guardian +(i.e. `/user`, there are others e.g. `/system` which hosts loggers, `/temp` +which keeps temporary actor refs used with `ask`, `/remote` which enables +remote deployment, etc.); this matches how the actor prints its own `self` +reference on the remote host, e.g. in log output. -* ``system`` is the remote system’s name (must match exactly, case-sensitive!) - -* ``host`` is the remote system’s IP address or DNS name, and it must match that - system’s configuration (i.e. `akka.remote.netty.tcp.hostname`) - -* ``1234`` is the port number on which the remote system is listening for - connections and receiving messages - -* ``/user/my/actor/hierarchy/path`` is the absolute path of the remote actor in - the remote system’s supervision hierarchy, including the system’s guardian - (i.e. ``/user``, there are others e.g. ``/system`` which hosts loggers, ``/temp`` - which keeps temporary actor refs used with :meth:`ask`, ``/remote`` which enables - remote deployment, etc.); this matches how the actor prints its own ``self`` - reference on the remote host, e.g. in log output. - -Why are replies not received from a remote actor? -------------------------------------------------- +### Why are replies not received from a remote actor? The most common reason is that the local system’s name (i.e. the -``system@host:1234`` part in the answer above) is not reachable from the remote -system’s network location, e.g. because ``host`` was configured to be ``0.0.0.0``, -``localhost`` or a NAT’ed IP address. +`system@host:1234` part in the answer above) is not reachable from the remote +system’s network location, e.g. because `host` was configured to be `0.0.0.0`, +`localhost` or a NAT’ed IP address. If you are running an ActorSystem under a NAT or inside a docker container, make sure to -set `akka.remote.netty.tcp.hostname` and `akka.remote.netty.tcp.port` to the address +set *akka.remote.netty.tcp.hostname* and *akka.remote.netty.tcp.port* to the address it is reachable at from other ActorSystems. If you need to bind your network interface -to a different address - use `akka.remote.netty.tcp.bind-hostname` and -`akka.remote.netty.tcp.bind-port` settings. Also make sure your network is configured +to a different address - use *akka.remote.netty.tcp.bind-hostname* and +*akka.remote.netty.tcp.bind-port* settings. Also make sure your network is configured to translate from the address your ActorSystem is reachable at to the address your ActorSystem network interface is bound to. -How reliable is the message delivery? -------------------------------------- +### How reliable is the message delivery? The general rule is **at-most-once delivery**, i.e. no guaranteed delivery. Stronger reliability can be built on top, and Akka provides tools to do so. -Read more in :ref:`message-delivery-reliability`. +Read more in @ref:[Message Delivery Reliability](../general/message-delivery-reliability.md). -Debugging -^^^^^^^^^ +## Debugging -How do I turn on debug logging? -------------------------------- +### How do I turn on debug logging? -To turn on debug logging in your actor system add the following to your configuration:: +To turn on debug logging in your actor system add the following to your configuration: - akka.loglevel = DEBUG +``` +akka.loglevel = DEBUG +``` To enable different types of debug logging add the following to your configuration: -* ``akka.actor.debug.receive`` will log all messages sent to an actor if that actors `receive` method is a ``LoggingReceive`` + * `akka.actor.debug.receive` will log all messages sent to an actor if that actors *receive* method is a `LoggingReceive` + * `akka.actor.debug.autoreceive` will log all *special* messages like `Kill`, `PoisonPill` e.t.c. sent to all actors + * `akka.actor.debug.lifecycle` will log all actor lifecycle events of all actors -* ``akka.actor.debug.autoreceive`` will log all *special* messages like ``Kill``, ``PoisonPill`` e.t.c. sent to all actors - -* ``akka.actor.debug.lifecycle`` will log all actor lifecycle events of all actors - -Read more about it in the docs for :ref:`logging-java` and :ref:`actor.logging-scala`. +Read more about it in the docs for @ref:[Logging](../java/logging.md) and @ref:[actor.logging-scala](../scala/testing.md#actor-logging-scala). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/additional/index.md b/akka-docs/src/main/paradox/additional/index.md index d5a1e1062b..bed69f2023 100644 --- a/akka-docs/src/main/paradox/additional/index.md +++ b/akka-docs/src/main/paradox/additional/index.md @@ -1,11 +1,13 @@ -Additional Information -====================== +# Additional Information -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - ../common/binary-compatibility-rules - ../common/may-change - faq - books - osgi +@@@ index + +* [../common/binary-compatibility-rules](../common/binary-compatibility-rules.md) +* [../common/may-change](../common/may-change.md) +* [faq](faq.md) +* [books](books.md) +* [osgi](osgi.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/additional/osgi.md b/akka-docs/src/main/paradox/additional/osgi.md index 221ac9a4e3..933f02030d 100644 --- a/akka-docs/src/main/paradox/additional/osgi.md +++ b/akka-docs/src/main/paradox/additional/osgi.md @@ -1,17 +1,13 @@ -Akka in OSGi -============ +# Akka in OSGi -Background ----------- +## Background -OSGi_ is a mature packaging and deployment standard for component-based systems. It +[OSGi](http://www.osgi.org/developer) is a mature packaging and deployment standard for component-based systems. It has similar capabilities as Project Jigsaw (originally scheduled for JDK 1.8), but has far stronger facilities to support legacy Java code. This is to say that while Jigsaw-ready modules require significant changes to most source files and on occasion to the structure of the overall application, OSGi can be used to modularize almost any Java code as far back as JDK 1.2, usually with no changes at all to the binaries. -.. _OSGI: http://www.osgi.org/developer - These legacy capabilities are OSGi's major strength and its major weakness. The creators of OSGi realized early on that implementors would be unlikely to rush to support OSGi metadata in existing JARs. There were already a handful of new concepts to learn in the JRE and the added value to teams that were managing well with straight J2EE was not obvious. @@ -20,49 +16,41 @@ situations. An application of the "80/20 Rule" here would have that "80% of the but it was enough to give OSGi a reputation that has stuck with it to this day. This document aims to the productivity basics folks need to use it with Akka, the 20% that users need to get 80% of what they want. -For more information than is provided here, `OSGi In Action`_ is worth exploring. +For more information than is provided here, [OSGi In Action](https://www.manning.com/books/osgi-in-action) is worth exploring. -.. _OSGi In Action: https://www.manning.com/books/osgi-in-action +## Core Components and Structure of OSGi Applications -Core Components and Structure of OSGi Applications --------------------------------------------------- - -The fundamental unit of deployment in OSGi is the ``Bundle``. A bundle is a Java JAR with `additional -entries ` in ``MANIFEST.MF`` that minimally expose the name and version +The fundamental unit of deployment in OSGi is the `Bundle`. A bundle is a Java JAR with *additional +entries * in `MANIFEST.MF` that minimally expose the name and version of the bundle and packages for import and export. Since these manifest entries are ignored outside OSGi deployments, a bundle can interchangeably be used as a JAR in the JRE. -When a bundle is loaded, a specialized implementation of the Java ``ClassLoader`` is instantiated for each bundle. Each -classloader reads the manifest entries and publishes both capabilities (in the form of the ``Bundle-Exports``) and -requirements (as ``Bundle-Imports``) in a container singleton for discovery by other bundles. The process of matching imports to +When a bundle is loaded, a specialized implementation of the Java `ClassLoader` is instantiated for each bundle. Each +classloader reads the manifest entries and publishes both capabilities (in the form of the `Bundle-Exports`) and +requirements (as `Bundle-Imports`) in a container singleton for discovery by other bundles. The process of matching imports to exports across bundles through these classloaders is the process of resolution, one of six discrete steps in the lifecycle FSM of a bundle in an OSGi container: -1. INSTALLED: A bundle that is installed has been loaded from disk and a classloader instantiated with its capabilities. - Bundles are iteratively installed manually or through container-specific descriptors. For those familiar with legacy packging - such as EJB, the modular nature of OSGi means that bundles may be used by multiple applications with overlapping dependencies. - By resolving them individually from repositories, these overlaps can be de-duplicated across multiple deployemnts to - the same container. - -2. RESOLVED: A bundle that has been resolved is one that has had its requirements (imports) satisfied. Resolution does - mean that a bundle can be started. - -3. STARTING: A bundle that is started can be used by other bundles. For an otherwise complete application closure of - resolved bundles, the implication here is they must be started in the order directed by a depth-first search for all to - be started. When a bundle is starting, any exposed lifecycle interfaces in the bundle are called, giving the bundle - the opportunity to start its own service endpoints and threads. - -4. ACTIVE: Once a bundle's lifecycle interfaces return without error, a bundle is marked as active. - -5. STOPPING: A bundle that is stopping is in the process of calling the bundle's stop lifecycle and transitions back to - the RESOLVED state when complete. Any long running services or threads that were created while STARTING should be shut - down when the bundle's stop lifecycle is called. - -6. UNINSTALLED: A bundle can only transition to this state from the INSTALLED state, meaning it cannot be uninstalled - before it is stopped. + 1. INSTALLED: A bundle that is installed has been loaded from disk and a classloader instantiated with its capabilities. +Bundles are iteratively installed manually or through container-specific descriptors. For those familiar with legacy packging +such as EJB, the modular nature of OSGi means that bundles may be used by multiple applications with overlapping dependencies. +By resolving them individually from repositories, these overlaps can be de-duplicated across multiple deployemnts to +the same container. + 2. RESOLVED: A bundle that has been resolved is one that has had its requirements (imports) satisfied. Resolution does +mean that a bundle can be started. + 3. STARTING: A bundle that is started can be used by other bundles. For an otherwise complete application closure of +resolved bundles, the implication here is they must be started in the order directed by a depth-first search for all to +be started. When a bundle is starting, any exposed lifecycle interfaces in the bundle are called, giving the bundle +the opportunity to start its own service endpoints and threads. + 4. ACTIVE: Once a bundle's lifecycle interfaces return without error, a bundle is marked as active. + 5. STOPPING: A bundle that is stopping is in the process of calling the bundle's stop lifecycle and transitions back to +the RESOLVED state when complete. Any long running services or threads that were created while STARTING should be shut +down when the bundle's stop lifecycle is called. + 6. UNINSTALLED: A bundle can only transition to this state from the INSTALLED state, meaning it cannot be uninstalled +before it is stopped. Note the dependency in this FSM on lifecycle interfaces. While there is no requirement that a bundle publishes these -interfaces or accepts such callbacks, the lifecycle interfaces provide the semantics of a ``main()`` method and allow +interfaces or accepts such callbacks, the lifecycle interfaces provide the semantics of a `main()` method and allow the bundle to start and stop long-running services such as REST web services, ActorSystems, Clusters, etc. Secondly, note when considering requirements and capabilities, it's a common misconception to equate these with repository @@ -71,69 +59,59 @@ parallel type of dependency (such as Blueprint Services) that cannot be easily m the core specification leaves these facilities up to the container in use. In turn, some containers have tooling to generate application load descriptors from repository metadata. -Notable Behavior Changes ------------------------- +## Notable Behavior Changes Combined with understanding the bundle lifecycle, the OSGi developer must pay attention to sometimes unexpected behaviors that are introduced. These are generally within the JVM specification, but are unexpected and can lead to frustration. -* Bundles should not export overlapping package spaces. It is not uncommon for legacy JVM frameworks to expect plugins - in an application composed of multiple JARs to reside under a single package name. For example, a frontend application - might scan all classes from ``com.example.plugins`` for specific service implementations with that package existing in - several contributed JARs. + * + Bundles should not export overlapping package spaces. It is not uncommon for legacy JVM frameworks to expect plugins +in an application composed of multiple JARs to reside under a single package name. For example, a frontend application +might scan all classes from `com.example.plugins` for specific service implementations with that package existing in +several contributed JARs. + While it is possible to support overlapping packages with complex manifest headers, it's much better to use non-overlapping +package spaces and facilities such as [Akka Cluster](@github@/akka-docs/rst/scala/code/docs/akka/current/common/cluster.html) +for service discovery. Stylistically, many organizations opt to use the root package path as the name of the bundle +distribution file. - While it is possible to support overlapping packages with complex manifest headers, it's much better to use non-overlapping - package spaces and facilities such as `Akka Cluster`_ - for service discovery. Stylistically, many organizations opt to use the root package path as the name of the bundle - distribution file. + * Resources are not shared across bundles unless they are explicitly exported, as with classes. The common +case of this is expecting that `getClass().getClassLoader().getResources("foo")` will return all files on the classpath +named `foo`. The `getResources()` method only returns resources from the current classloader, and since there are +separate classloaders for every bundle, resource files such as configurations are no longer searchable in this manner. -.. _Akka Cluster: @github@/akka-docs/rst/scala/code/docs/akka/current/common/cluster.html +## Configuring the OSGi Framework -* Resources are not shared across bundles unless they are explicitly exported, as with classes. The common - case of this is expecting that ``getClass().getClassLoader().getResources("foo")`` will return all files on the classpath - named ``foo``. The ``getResources()`` method only returns resources from the current classloader, and since there are - separate classloaders for every bundle, resource files such as configurations are no longer searchable in this manner. +To use Akka in an OSGi environment, the container must be configured such that the `org.osgi.framework.bootdelegation` +property delegates the `sun.misc` package to the boot classloader instead of resolving it through the normal OSGi class space. -Configuring the OSGi Framework ------------------------------- - -To use Akka in an OSGi environment, the container must be configured such that the ``org.osgi.framework.bootdelegation`` -property delegates the ``sun.misc`` package to the boot classloader instead of resolving it through the normal OSGi class space. - -Intended Use ------------- +## Intended Use Akka only supports the usage of an ActorSystem strictly confined to a single OSGi bundle, where that bundle contains or imports all of the actor system's requirements. This means that the approach of offering an ActorSystem as a service to which Actors can be deployed dynamically via other bundles is not recommended — an ActorSystem and its contained actors are not meant to be dynamic in this way. ActorRefs may safely be exposed to other bundles. -Activator ---------- +## Activator -To bootstrap Akka inside an OSGi environment, you can use the ``akka.osgi.ActorSystemActivator`` class +To bootstrap Akka inside an OSGi environment, you can use the `akka.osgi.ActorSystemActivator` class to conveniently set up the ActorSystem. -.. includecode:: ../../../akka-osgi/src/test/scala/docs/osgi/Activator.scala#Activator +@@snip [Activator.scala]../../../../../akka-osgi/src/test/scala/docs/osgi/Activator.scala) { #Activator } -The goal here is to map the OSGi lifecycle more directly to the Akka lifecycle. The ``ActorSystemActivator`` creates -the actor system with a class loader that finds resources (``application.conf`` and ``reference.conf`` files) and classes +The goal here is to map the OSGi lifecycle more directly to the Akka lifecycle. The `ActorSystemActivator` creates +the actor system with a class loader that finds resources (`application.conf` and `reference.conf` files) and classes from the application bundle and all transitive dependencies. +The `ActorSystemActivator` class is included in the `akka-osgi` artifact: +``` + + com.typesafe.akka + akka-osgi_@binVersion@ + @version@ + +``` -The ``ActorSystemActivator`` class is included in the ``akka-osgi`` artifact:: +## Sample - - com.typesafe.akka - akka-osgi_@binVersion@ - @version@ - - - -Sample ------- - -A complete sample project is provided in `akka-sample-osgi-dining-hakkers`_ - -.. _akka-sample-osgi-dining-hakkers: @samples@/tree/master/akka-sample-osgi-dining-hakkers +A complete sample project is provided in [akka-sample-osgi-dining-hakkers](@samples@/tree/master/akka-sample-osgi-dining-hakkers) \ No newline at end of file diff --git a/akka-docs/src/main/paradox/common/binary-compatibility-rules.md b/akka-docs/src/main/paradox/common/binary-compatibility-rules.md index af0a19283a..46949757d2 100644 --- a/akka-docs/src/main/paradox/common/binary-compatibility-rules.md +++ b/akka-docs/src/main/paradox/common/binary-compatibility-rules.md @@ -1,7 +1,4 @@ -.. _BinCompatRules: - -Binary Compatibility Rules -########################## +# Binary Compatibility Rules Akka maintains and verifies *backwards binary compatibility* across versions of modules. @@ -11,83 +8,89 @@ In the rest of this document whenever *binary compatibility* is mentioned "*back This means that the new JARs are a drop-in replacement for the old one (but not the other way around) as long as your build does not enable the inliner (Scala-only restriction). -Binary compatibility rules explained -==================================== +## Binary compatibility rules explained + Binary compatibility is maintained between: -- **minor** and **patch** versions - please note that the meaning of "minor" has shifted to be more restrictive with Akka ``2.4.0``, read :ref:`24versioningChange` for details. + * **minor** and **patch** versions - please note that the meaning of "minor" has shifted to be more restrictive with Akka `2.4.0`, read [24versioningChange](#24versioningchange) for details. Binary compatibility is **NOT** maintained between: -- **major** versions -- any versions of **may change** modules – read :ref:`may-change` for details -- a few notable exclusions explained below + * **major** versions + * any versions of **may change** modules – read @ref:[Modules marked "May Change"](may-change.md) for details + * a few notable exclusions explained below -Specific examples (please read :ref:`24versioningChange` to understand the difference in "before 2.4 era" and "after 2.4 era"):: +Specific examples (please read [24versioningChange](#24versioningchange) to understand the difference in "before 2.4 era" and "after 2.4 era"): - # [epoch.major.minor] era - OK: 2.2.0 --> 2.2.1 --> ... --> 2.2.x - NO: 2.2.y --x 2.3.y - OK: 2.3.0 --> 2.3.1 --> ... --> 2.3.x - OK: 2.3.x --> 2.4.x (special case, migration to new versioning scheme) - # [major.minor.path] era - OK: 2.4.0 --> 2.5.x - OK: 2.5.0 --> 2.6.x - NO: 2.x.y --x 3.x.y - OK: 3.0.0 --> 3.0.1 --> ... --> 3.0.n - OK: 3.0.n --> 3.1.0 --> ... --> 3.1.n - OK: 3.1.n --> 3.2.0 ... - ... +``` +# [epoch.major.minor] era +OK: 2.2.0 --> 2.2.1 --> ... --> 2.2.x +NO: 2.2.y --x 2.3.y +OK: 2.3.0 --> 2.3.1 --> ... --> 2.3.x +OK: 2.3.x --> 2.4.x (special case, migration to new versioning scheme) +# [major.minor.path] era +OK: 2.4.0 --> 2.5.x +OK: 2.5.0 --> 2.6.x +NO: 2.x.y --x 3.x.y +OK: 3.0.0 --> 3.0.1 --> ... --> 3.0.n +OK: 3.0.n --> 3.1.0 --> ... --> 3.1.n +OK: 3.1.n --> 3.2.0 ... + ... +``` + +### Cases where binary compatibility is not retained -Cases where binary compatibility is not retained ------------------------------------------------- Some modules are excluded from the binary compatibility guarantees, such as: - - ``*-testkit`` modules - since these are to be used only in tests, which usually are re-compiled and run on demand - - ``*-tck`` modules - since they may want to add new tests (or force configuring something), in order to discover possible - failures in an existing implementation that the TCK is supposed to be testing. - Compatibility here is not *guaranteed*, however it is attempted to make the upgrade prosess as smooth as possible. - - all :ref:`may change ` modules - which by definition are subject to rapid iteration and change. Read more about that in :ref:`may-change` +> + * `*-testkit` modules - since these are to be used only in tests, which usually are re-compiled and run on demand + * + `*-tck` + modules - since they may want to add new tests (or force configuring something), in order to discover possible + : failures in an existing implementation that the TCK is supposed to be testing. +Compatibility here is not *guaranteed*, however it is attempted to make the upgrade prosess as smooth as possible. + + * all @ref:[may change](may-change.md) modules - which by definition are subject to rapid iteration and change. Read more about that in @ref:[Modules marked "May Change"](may-change.md) -.. _24versioningChange: + +## Change in versioning scheme, stronger compatibility since 2.4 -Change in versioning scheme, stronger compatibility since 2.4 -============================================================= -Since the release of Akka ``2.4.0`` a new versioning scheme is in effect. +Since the release of Akka `2.4.0` a new versioning scheme is in effect. Historically, Akka has been following the Java or Scala style of versioning where as the first number would mean "**epoch**", -the second one would mean **major**, and third be the **minor**, thus: ``epoch.major.minor`` (versioning scheme followed until and during ``2.3.x``). +the second one would mean **major**, and third be the **minor**, thus: `epoch.major.minor` (versioning scheme followed until and during `2.3.x`). -**Currently**, since Akka ``2.4.0``, the new versioning applies which is closer to semantic versioning many have come to expect, -in which the version number is deciphered as ``major.minor.patch``. This also means that Akka ``2.5.x`` is binary compatible with the ``2.4`` series releases (with the exception of "may change" APIs of course). +**Currently**, since Akka `2.4.0`, the new versioning applies which is closer to semantic versioning many have come to expect, +in which the version number is deciphered as `major.minor.patch`. This also means that Akka `2.5.x` is binary compatible with the `2.4` series releases (with the exception of "may change" APIs of course). -In addition to that, Akka ``2.4.x`` has been made binary compatible with the ``2.3.x`` series, +In addition to that, Akka `2.4.x` has been made binary compatible with the `2.3.x` series, so there is no reason to remain on Akka 2.3.x, since upgrading is completely compatible (and many issues have been fixed ever since). -Mixed versioning is not allowed -=============================== +## Mixed versioning is not allowed Modules that are released together under the Akka project are intended to be upgraded together. -For example, it is not legal to mix Akka Actor ``2.4.2`` with Akka Cluster ``2.4.5`` even though -"Akka ``2.4.2``" and "Akka ``2.4.5``" *are* binary compatible. +For example, it is not legal to mix Akka Actor `2.4.2` with Akka Cluster `2.4.5` even though +"Akka `2.4.2`" and "Akka `2.4.5`" *are* binary compatible. This is because modules may assume internals changes across module boundaries, for example some feature in Clustering may have required an internals change in Actor, however it is not public API, thus such change is considered safe. -.. note:: - We recommend keeping an ``akkaVersion`` variable in your build file, and re-use it for all - included modules, so when you upgrade you can simply change it in this one place. +@@@ note + +We recommend keeping an `akkaVersion` variable in your build file, and re-use it for all +included modules, so when you upgrade you can simply change it in this one place. + +@@@ + +## The meaning of "may change" -The meaning of "may change" -=========================== **May change** is used in module descriptions and docs in order to signify that the API that they contain is subject to change without any prior warning and is not covered by the binary compatibility promise. -Read more in :ref:`may-change`. +Read more in @ref:[Modules marked "May Change"](may-change.md). -API stability annotations and comments -====================================== +## API stability annotations and comments Akka gives a very strong binary compatibility promise to end-users. However some parts of Akka are excluded from these rules, for example internal or known evolving APIs may be marked as such and shipped as part of @@ -95,51 +98,49 @@ an overall stable module. As general rule any breakage is avoided and handled vi however certain APIs which are known to not yet be fully frozen (or are fully internal) are marked as such and subject to change at any time (even if best-effort is taken to keep them compatible). -The INTERNAL API and `@InternalAPI` marker ------------------------------------------- +### The INTERNAL API and *@InternalAPI* marker + When browsing the source code and/or looking for methods available to be called, especially from Java which does not have as rich of an access protection system as Scala has, you may sometimes find methods or classes annotated with -the ``/** INTERNAL API */`` comment or the ``@akka.annotation.InternalApi`` annotation. +the `/** INTERNAL API */` comment or the `@akka.annotation.InternalApi` annotation. No compatibility guarantees are given about these classes. They may change or even dissappear in minor versions, and user code is not supposed to call them. -Side-note on JVM representation details of the Scala ``private[akka]`` pattern that Akka is using extensively in +Side-note on JVM representation details of the Scala `private[akka]` pattern that Akka is using extensively in it's internals: Such methods or classes, which act as "accessible only from the given package" in Scala, are compiled -down to ``public`` (!) in raw Java bytecode. The access restriction, that Scala understands is carried along +down to `public` (!) in raw Java bytecode. The access restriction, that Scala understands is carried along as metadata stored in the classfile. Thus, such methods are safely guarded from being accessed from Scala, -however Java users will not be warned about this fact by the ``javac`` compiler. Please be aware of this and do not call +however Java users will not be warned about this fact by the `javac` compiler. Please be aware of this and do not call into Internal APIs, as they are subject to change without any warning. -The ``@DoNotInherit`` and ``@ApiMayChange`` markers ---------------------------------------------------- +### The `@DoNotInherit` and `@ApiMayChange` markers In addition to the special internal API marker two annotations exist in Akka and specifically address the following use cases: -- ``@ApiMayChange`` – which marks APIs which are known to be not fully stable yet. Read more in :ref:`may-change` -- ``@DoNotInherit`` – which marks APIs that are designed under a closed-world assumption, and thus must not be - extended outside Akka itself (or such code will risk facing binary incompatibilities). E.g. an interface may be - marked using this annotation, and while the type is public, it is not meant for extension by user-code. This allows - adding new methods to these interfaces without risking to break client code. Examples of such API are the ``FlowOps`` - trait or the Akka HTTP domain model. + * `@ApiMayChange` – which marks APIs which are known to be not fully stable yet. Read more in @ref:[Modules marked "May Change"](may-change.md) + * `@DoNotInherit` – which marks APIs that are designed under a closed-world assumption, and thus must not be +extended outside Akka itself (or such code will risk facing binary incompatibilities). E.g. an interface may be +marked using this annotation, and while the type is public, it is not meant for extension by user-code. This allows +adding new methods to these interfaces without risking to break client code. Examples of such API are the `FlowOps` +trait or the Akka HTTP domain model. Please note that a best-effort approach is always taken when having to change APIs and breakage is avoided as much as possible, however these markers allow to experiment, gather feedback and stabilize the best possible APIs we could build. -Binary Compatibility Checking Toolchain -======================================= -Akka uses the Lightbend maintained `Migration Manager `_, -called ``MiMa`` for short, for enforcing binary compatibility is kept where it was promised. +## Binary Compatibility Checking Toolchain + +Akka uses the Lightbend maintained [Migration Manager](https://github.com/typesafehub/migration-manager), +called `MiMa` for short, for enforcing binary compatibility is kept where it was promised. All Pull Requests must pass MiMa validation (which happens automatically), and if failures are detected, manual exception overrides may be put in place if the change happened to be in an Internal API for example. -Serialization compatibility across Scala versions -================================================= +## Serialization compatibility across Scala versions Scala does not maintain serialization compatibility across major versions. This means that if Java serialization is used there is no guarantee objects can be cleanly deserialized if serialized with a different version of Scala. -The internal Akka Protobuf serializers that can be enabled explicitly with ``enable-additional-serialization-bindings`` -or implicitly with ``akka.actor.allow-java-serialization = off`` (which is preferable from a security standpoint) -does not suffer from this problem. +The internal Akka Protobuf serializers that can be enabled explicitly with `enable-additional-serialization-bindings` +or implicitly with `akka.actor.allow-java-serialization = off` (which is preferable from a security standpoint) +does not suffer from this problem. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/common/circuitbreaker.md b/akka-docs/src/main/paradox/common/circuitbreaker.md index c33808836c..41973b057a 100644 --- a/akka-docs/src/main/paradox/common/circuitbreaker.md +++ b/akka-docs/src/main/paradox/common/circuitbreaker.md @@ -1,12 +1,7 @@ -.. _circuit-breaker: +# Circuit Breaker -############### -Circuit Breaker -############### +## Why are they used? -================== -Why are they used? -================== A circuit breaker is used to provide stability and prevent cascading failures in distributed systems. These should be used in conjunction with judicious timeouts at the interfaces between remote systems to prevent the failure of a single component from bringing down all components. @@ -30,162 +25,154 @@ the site that use the functionality unavailable, or perhaps show some cached con appropriate while the breaker is open. The Akka library provides an implementation of a circuit breaker called -:class:`akka.pattern.CircuitBreaker` which has the behavior described below. +`akka.pattern.CircuitBreaker` which has the behavior described below. -================ -What do they do? -================ -* During normal operation, a circuit breaker is in the `Closed` state: - * Exceptions or calls exceeding the configured `callTimeout` increment a failure counter - * Successes reset the failure count to zero - * When the failure counter reaches a `maxFailures` count, the breaker is tripped into `Open` state -* While in `Open` state: - * All calls fail-fast with a :class:`CircuitBreakerOpenException` - * After the configured `resetTimeout`, the circuit breaker enters a `Half-Open` state -* In `Half-Open` state: - * The first call attempted is allowed through without failing fast - * All other calls fail-fast with an exception just as in `Open` state - * If the first call succeeds, the breaker is reset back to `Closed` state and the `resetTimeout` is reset - * If the first call fails, the breaker is tripped again into the `Open` state (as for exponential backoff circuit breaker, the `resetTimeout` is multiplied by the exponential backoff factor) -* State transition listeners: - * Callbacks can be provided for every state entry via `onOpen`, `onClose`, and `onHalfOpen` - * These are executed in the :class:`ExecutionContext` provided. -* Calls result listeners: +## What do they do? + + * + During normal operation, a circuit breaker is in the + *Closed* + state: + : + * Exceptions or calls exceeding the configured *callTimeout* increment a failure counter + * Successes reset the failure count to zero + * When the failure counter reaches a *maxFailures* count, the breaker is tripped into *Open* state + + * + While in + *Open* + state: + : + * All calls fail-fast with a `CircuitBreakerOpenException` + * After the configured *resetTimeout*, the circuit breaker enters a *Half-Open* state + + * + In + *Half-Open* + state: + : + * The first call attempted is allowed through without failing fast + * All other calls fail-fast with an exception just as in *Open* state + * If the first call succeeds, the breaker is reset back to *Closed* state and the *resetTimeout* is reset + * If the first call fails, the breaker is tripped again into the *Open* state (as for exponential backoff circuit breaker, the *resetTimeout* is multiplied by the exponential backoff factor) + + * + State transition listeners: + : + * Callbacks can be provided for every state entry via *onOpen*, *onClose*, and *onHalfOpen* + * These are executed in the `ExecutionContext` provided. + + * + Calls result listeners: + : * Callbacks can be used eg. to collect statistics about all invocations or to react on specific call results like success, failures or timeouts. - * Supported callbacks are: `onCallSuccess`, `onCallFailure`, `onCallTimeout`, `onCallBreakerOpen`. - * These are executed in the :class:`ExecutionContext` provided. + * Supported callbacks are: *onCallSuccess*, *onCallFailure*, *onCallTimeout*, *onCallBreakerOpen*. + * These are executed in the `ExecutionContext` provided. + -.. image:: ../images/circuit-breaker-states.png +![circuit-breaker-states.png](../images/circuit-breaker-states.png) + +## Examples + +### Initialization + +Here's how a :class: +*CircuitBreaker* + would be configured for: +: + * 5 maximum failures + * a call timeout of 10 seconds + * a reset timeout of 1 minute -======== -Examples -======== +#### Scala --------------- -Initialization --------------- +@@snip [CircuitBreakerDocSpec.scala](code/docs/circuitbreaker/CircuitBreakerDocSpec.scala) { #imports1 #circuit-breaker-initialization } -Here's how a :class:`CircuitBreaker` would be configured for: - * 5 maximum failures - * a call timeout of 10 seconds - * a reset timeout of 1 minute +#### Java -~~~~~ -Scala -~~~~~ +@@snip [DangerousJavaActor.java](code/docs/circuitbreaker/DangerousJavaActor.java) { #imports1 #circuit-breaker-initialization } -.. includecode:: code/docs/circuitbreaker/CircuitBreakerDocSpec.scala - :include: imports1,circuit-breaker-initialization +### Future & Synchronous based API -~~~~ -Java -~~~~ +Once a circuit breaker actor has been intialized, interacting with that actor is done by either using the Future based API or the synchronous API. Both of these APIs are considered `Call Protection` because whether synchronously or asynchronously, the purpose of the circuit breaker is to protect your system from cascading failures while making a call to another service. In the future based API, we use the `withCircuitBreaker` which takes an asynchronous method (some method wrapped in a `Future`), for instance a call to retrieve data from a database, and we pipe the result back to the sender. If for some reason the database in this example isn't responding, or there is another issue, the circuit breaker will open and stop trying to hit the database again and again until the timeout is over. -.. includecode:: code/docs/circuitbreaker/DangerousJavaActor.java - :include: imports1,circuit-breaker-initialization +The Synchronous API would also wrap your call with the circuit breaker logic, however, it uses the `withSyncCircuitBreaker` and receives a method that is not wrapped in a `Future`. ------------------------------- -Future & Synchronous based API ------------------------------- +#### Scala -Once a circuit breaker actor has been intialized, interacting with that actor is done by either using the Future based API or the synchronous API. Both of these APIs are considered ``Call Protection`` because whether synchronously or asynchronously, the purpose of the circuit breaker is to protect your system from cascading failures while making a call to another service. In the future based API, we use the :meth:`withCircuitBreaker` which takes an asynchronous method (some method wrapped in a :class:`Future`), for instance a call to retrieve data from a database, and we pipe the result back to the sender. If for some reason the database in this example isn't responding, or there is another issue, the circuit breaker will open and stop trying to hit the database again and again until the timeout is over. +@@snip [CircuitBreakerDocSpec.scala](code/docs/circuitbreaker/CircuitBreakerDocSpec.scala) { #circuit-breaker-usage } -The Synchronous API would also wrap your call with the circuit breaker logic, however, it uses the :meth:`withSyncCircuitBreaker` and receives a method that is not wrapped in a :class:`Future`. +#### Java -~~~~~ -Scala -~~~~~ +@@snip [DangerousJavaActor.java](code/docs/circuitbreaker/DangerousJavaActor.java) { #circuit-breaker-usage } -.. includecode:: code/docs/circuitbreaker/CircuitBreakerDocSpec.scala - :include: circuit-breaker-usage +@@@ note -~~~~ -Java -~~~~ +Using the `CircuitBreaker` companion object's *apply* or *create* methods +will return a `CircuitBreaker` where callbacks are executed in the caller's thread. +This can be useful if the asynchronous `Future` behavior is unnecessary, for +example invoking a synchronous-only API. -.. includecode:: code/docs/circuitbreaker/DangerousJavaActor.java - :include: circuit-breaker-usage +@@@ -.. note:: +@@@ note - Using the :class:`CircuitBreaker` companion object's `apply` or `create` methods - will return a :class:`CircuitBreaker` where callbacks are executed in the caller's thread. - This can be useful if the asynchronous :class:`Future` behavior is unnecessary, for - example invoking a synchronous-only API. +There is also a `CircuitBreakerProxy` actor that you can use, which is an alternative implementation of the pattern. +The main difference is that it is intended to be used only for request-reply interactions with another actor. See circuit-breaker-proxy -.. note:: - - There is also a :class:`CircuitBreakerProxy` actor that you can use, which is an alternative implementation of the pattern. - The main difference is that it is intended to be used only for request-reply interactions with another actor. See :ref:`Circuit Breaker Actor ` +@@@ --------------------------------- -Control failure count explicitly --------------------------------- +### Control failure count explicitly -By default, the circuit breaker treat :class:`Exception` as failure in synchronized API, or failed :class:`Future` as failure in future based API. -Failure will increment failure count, when failure count reach the `maxFailures`, circuit breaker will be opened. +By default, the circuit breaker treat `Exception` as failure in synchronized API, or failed `Future` as failure in future based API. +Failure will increment failure count, when failure count reach the *maxFailures*, circuit breaker will be opened. However, some applications might requires certain exception to not increase failure count, or vice versa, sometime we want to increase the failure count even if the call succeeded. Akka circuit breaker provides a way to achieve such use case: - * `withCircuitBreaker` - * `withSyncCircuitBreaker` +> + * *withCircuitBreaker* + * *withSyncCircuitBreaker* + * *callWithCircuitBreaker* + * *callWithCircuitBreakerCS* + * *callWithSyncCircuitBreaker* - * `callWithCircuitBreaker` - * `callWithCircuitBreakerCS` - * `callWithSyncCircuitBreaker` +All methods above accepts an argument `defineFailureFn` -All methods above accepts an argument ``defineFailureFn`` +#### Scala -~~~~~ -Scala -~~~~~ +Type of `defineFailureFn`: `Try[T] ⇒ Boolean` -Type of ``defineFailureFn``: ``Try[T] ⇒ Boolean`` +This is a function which takes in a `Try[T]` and return a `Boolean`. The `Try[T]` correspond to the `Future[T]` of the protected call. This function should return `true` if the call should increase failure count, else false. -This is a function which takes in a :class:`Try[T]` and return a :class:`Boolean`. The :class:`Try[T]` correspond to the :class:`Future[T]` of the protected call. This function should return ``true`` if the call should increase failure count, else false. +@@snip [CircuitBreakerDocSpec.scala](code/docs/circuitbreaker/CircuitBreakerDocSpec.scala) { #even-no-as-failure } -.. includecode:: code/docs/circuitbreaker/CircuitBreakerDocSpec.scala - :include: even-no-as-failure +#### Java -~~~~ -Java -~~~~ +Type of `defineFailureFn`: `BiFunction[Optional[T], Optional[Throwable], java.lang.Boolean]` -Type of ``defineFailureFn``: :class:`BiFunction[Optional[T], Optional[Throwable], java.lang.Boolean]` +For Java Api, the signature is a bit different as there's no `Try` in Java, so the response of protected call is modelled using `Optional[T]` for succeeded return value and `Optional[Throwable]` for exception, and the rules of return type is the same. +Ie. this function should return `true` if the call should increase failure count, else false. -For Java Api, the signature is a bit different as there's no :class:`Try` in Java, so the response of protected call is modelled using :class:`Optional[T]` for succeeded return value and :class:`Optional[Throwable]` for exception, and the rules of return type is the same. -Ie. this function should return ``true`` if the call should increase failure count, else false. +@@snip [EvenNoFailureJavaExample.java](code/docs/circuitbreaker/EvenNoFailureJavaExample.java) { #even-no-as-failure } -.. includecode:: code/docs/circuitbreaker/EvenNoFailureJavaExample.java - :include: even-no-as-failure +### Low level API -------------- -Low level API -------------- +The low-level API allows you to describe the behaviour of the CircuitBreaker in detail, including deciding what to return to the calling `Actor` in case of success or failure. This is especially useful when expecting the remote call to send a reply. CircuitBreaker doesn't support `Tell Protection` (protecting against calls that expect a reply) natively at the moment, so you need to use the low-level power-user APIs, `succeed` and `fail` methods, as well as `isClose`, `isOpen`, `isHalfOpen` to implement it. -The low-level API allows you to describe the behaviour of the CircuitBreaker in detail, including deciding what to return to the calling ``Actor`` in case of success or failure. This is especially useful when expecting the remote call to send a reply. CircuitBreaker doesn't support ``Tell Protection`` (protecting against calls that expect a reply) natively at the moment, so you need to use the low-level power-user APIs, ``succeed`` and ``fail`` methods, as well as ``isClose``, ``isOpen``, ``isHalfOpen`` to implement it. +As can be seen in the examples below, a `Tell Protection` pattern could be implemented by using the `succeed` and `fail` methods, which would count towards the `CircuitBreaker` counts. In the example, a call is made to the remote service if the `breaker.isClosed`, and once a response is received, the `succeed` method is invoked, which tells the `CircuitBreaker` to keep the breaker closed. If on the other hand an error or timeout is received, we trigger a `fail` and the breaker accrues this failure towards its count for opening the breaker. -As can be seen in the examples below, a ``Tell Protection`` pattern could be implemented by using the ``succeed`` and ``fail`` methods, which would count towards the :class:`CircuitBreaker` counts. In the example, a call is made to the remote service if the ``breaker.isClosed``, and once a response is received, the ``succeed`` method is invoked, which tells the :class:`CircuitBreaker` to keep the breaker closed. If on the other hand an error or timeout is received, we trigger a ``fail`` and the breaker accrues this failure towards its count for opening the breaker. +@@@ note -.. note:: +The below examples doesn't make a remote call when the state is *HalfOpen*. Using the power-user APIs, it is your responsibility to judge when to make remote calls in *HalfOpen*. - The below examples doesn't make a remote call when the state is `HalfOpen`. Using the power-user APIs, it is your responsibility to judge when to make remote calls in `HalfOpen`. +@@@ +#### Scala -~~~~~~ -Scala -~~~~~~ +@@snip [CircuitBreakerDocSpec.scala](code/docs/circuitbreaker/CircuitBreakerDocSpec.scala) { #circuit-breaker-tell-pattern } -.. includecode:: code/docs/circuitbreaker/CircuitBreakerDocSpec.scala - :include: circuit-breaker-tell-pattern - - -~~~~ -Java -~~~~ - -.. includecode:: code/docs/circuitbreaker/TellPatternJavaActor.java - :include: circuit-breaker-tell-pattern +#### Java +@@snip [TellPatternJavaActor.java](code/docs/circuitbreaker/TellPatternJavaActor.java) { #circuit-breaker-tell-pattern } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/common/cluster.md b/akka-docs/src/main/paradox/common/cluster.md index 89e64d6ced..c72f8074f3 100644 --- a/akka-docs/src/main/paradox/common/cluster.md +++ b/akka-docs/src/main/paradox/common/cluster.md @@ -1,87 +1,70 @@ -.. _cluster: +# Cluster Specification -##################### -Cluster Specification -##################### +@@@ note -.. note:: This document describes the design concepts of the clustering. +This document describes the design concepts of the clustering. -Intro -===== +@@@ + +## Intro Akka Cluster provides a fault-tolerant decentralized peer-to-peer based cluster -`membership`_ service with no single point of failure or single point of bottleneck. -It does this using `gossip`_ protocols and an automatic `failure detector`_. +[membership](#membership) service with no single point of failure or single point of bottleneck. +It does this using [gossip](#gossip) protocols and an automatic [failure detector](#failure-detector). - -Terms -===== +## Terms **node** - A logical member of a cluster. There could be multiple nodes on a physical - machine. Defined by a `hostname:port:uid` tuple. +: A logical member of a cluster. There could be multiple nodes on a physical +machine. Defined by a *hostname:port:uid* tuple. **cluster** - A set of nodes joined together through the `membership`_ service. +: A set of nodes joined together through the [membership](#membership) service. **leader** - A single node in the cluster that acts as the leader. Managing cluster convergence - and membership state transitions. +: A single node in the cluster that acts as the leader. Managing cluster convergence +and membership state transitions. -Membership -========== +## Membership A cluster is made up of a set of member nodes. The identifier for each node is a -``hostname:port:uid`` tuple. An Akka application can be distributed over a cluster with +`hostname:port:uid` tuple. An Akka application can be distributed over a cluster with each node hosting some part of the application. Cluster membership and the actors running on that node of the application are decoupled. A node could be a member of a cluster without hosting any actors. Joining a cluster is initiated -by issuing a ``Join`` command to one of the nodes in the cluster to join. +by issuing a `Join` command to one of the nodes in the cluster to join. The node identifier internally also contains a UID that uniquely identifies this -actor system instance at that ``hostname:port``. Akka uses the UID to be able to +actor system instance at that `hostname:port`. Akka uses the UID to be able to reliably trigger remote death watch. This means that the same actor system can never join a cluster again once it's been removed from that cluster. To re-join an actor -system with the same ``hostname:port`` to a cluster you have to stop the actor system -and start a new one with the same ``hostname:port`` which will then receive a different +system with the same `hostname:port` to a cluster you have to stop the actor system +and start a new one with the same `hostname:port` which will then receive a different UID. -The cluster membership state is a specialized `CRDT`_, which means that it has a monotonic +The cluster membership state is a specialized [CRDT](http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf), which means that it has a monotonic merge function. When concurrent changes occur on different nodes the updates can always be merged and converge to the same end result. -.. _CRDT: http://hal.upmc.fr/docs/00/55/55/88/PDF/techreport.pdf +### Gossip -Gossip ------- - -The cluster membership used in Akka is based on Amazon's `Dynamo`_ system and -particularly the approach taken in Basho's' `Riak`_ distributed database. -Cluster membership is communicated using a `Gossip Protocol`_, where the current +The cluster membership used in Akka is based on Amazon's [Dynamo](http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) system and +particularly the approach taken in Basho's' [Riak](http://basho.com/technology/architecture/) distributed database. +Cluster membership is communicated using a [Gossip Protocol](http://en.wikipedia.org/wiki/Gossip_protocol), where the current state of the cluster is gossiped randomly through the cluster, with preference to members that have not seen the latest version. -.. _Gossip Protocol: http://en.wikipedia.org/wiki/Gossip_protocol -.. _Dynamo: http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf -.. _Riak: http://basho.com/technology/architecture/ +#### Vector Clocks - -Vector Clocks -^^^^^^^^^^^^^ - -`Vector clocks`_ are a type of data structure and algorithm for generating a partial +[Vector clocks](http://en.wikipedia.org/wiki/Vector_clock) are a type of data structure and algorithm for generating a partial ordering of events in a distributed system and detecting causality violations. We use vector clocks to reconcile and merge differences in cluster state during gossiping. A vector clock is a set of (node, counter) pairs. Each update to the cluster state has an accompanying update to the vector clock. -.. _Vector Clocks: http://en.wikipedia.org/wiki/Vector_clock - - -Gossip Convergence -^^^^^^^^^^^^^^^^^^ +#### Gossip Convergence Information about the cluster converges locally at a node at certain points in time. This is when a node can prove that the cluster state he is observing has been observed @@ -90,22 +73,20 @@ that have seen current state version during gossip. This information is referred seen set in the gossip overview. When all nodes are included in the seen set there is convergence. -Gossip convergence cannot occur while any nodes are ``unreachable``. The nodes need -to become ``reachable`` again, or moved to the ``down`` and ``removed`` states -(see the `Membership Lifecycle`_ section below). This only blocks the leader +Gossip convergence cannot occur while any nodes are `unreachable`. The nodes need +to become `reachable` again, or moved to the `down` and `removed` states +(see the [Membership Lifecycle](#membership-lifecycle) section below). This only blocks the leader from performing its cluster membership management and does not influence the application running on top of the cluster. For example this means that during a network partition it is not possible to add more nodes to the cluster. The nodes can join, but they -will not be moved to the ``up`` state until the partition has healed or the unreachable +will not be moved to the `up` state until the partition has healed or the unreachable nodes have been downed. - -Failure Detector -^^^^^^^^^^^^^^^^ +#### Failure Detector The failure detector is responsible for trying to detect if a node is -``unreachable`` from the rest of the cluster. For this we are using an -implementation of `The Phi Accrual Failure Detector`_ by Hayashibara et al. +`unreachable` from the rest of the cluster. For this we are using an +implementation of [The Phi Accrual Failure Detector](http://www.jaist.ac.jp/~defago/files/pdf/IS_RR_2004_010.pdf) by Hayashibara et al. An accrual failure detector decouple monitoring and interpretation. That makes them applicable to a wider area of scenarios and more adequate to build generic @@ -114,21 +95,21 @@ statistics, calculated from heartbeats received from other nodes, and is trying to do educated guesses by taking multiple factors, and how they accumulate over time, into account in order to come up with a better guess if a specific node is up or down. Rather than just answering "yes" or "no" to the -question "is the node down?" it returns a ``phi`` value representing the +question "is the node down?" it returns a `phi` value representing the likelihood that the node is down. -The ``threshold`` that is the basis for the calculation is configurable by the -user. A low ``threshold`` is prone to generate many wrong suspicions but ensures -a quick detection in the event of a real crash. Conversely, a high ``threshold`` +The `threshold` that is the basis for the calculation is configurable by the +user. A low `threshold` is prone to generate many wrong suspicions but ensures +a quick detection in the event of a real crash. Conversely, a high `threshold` generates fewer mistakes but needs more time to detect actual crashes. The -default ``threshold`` is 8 and is appropriate for most situations. However in +default `threshold` is 8 and is appropriate for most situations. However in cloud environments, such as Amazon EC2, the value could be increased to 12 in order to account for network issues that sometimes occur on such platforms. In a cluster each node is monitored by a few (default maximum 5) other nodes, and when -any of these detects the node as ``unreachable`` that information will spread to +any of these detects the node as `unreachable` that information will spread to the rest of the cluster through the gossip. In other words, only one node needs to -mark a node ``unreachable`` to have the rest of the cluster mark that node ``unreachable``. +mark a node `unreachable` to have the rest of the cluster mark that node `unreachable`. The nodes to monitor are picked out of neighbors in a hashed ordered node ring. This is to increase the likelihood to monitor across racks and data centers, but the order @@ -137,44 +118,38 @@ is the same on all nodes, which ensures full coverage. Heartbeats are sent out every second and every heartbeat is performed in a request/reply handshake with the replies used as input to the failure detector. -The failure detector will also detect if the node becomes ``reachable`` again. When -all nodes that monitored the ``unreachable`` node detects it as ``reachable`` again -the cluster, after gossip dissemination, will consider it as ``reachable``. +The failure detector will also detect if the node becomes `reachable` again. When +all nodes that monitored the `unreachable` node detects it as `reachable` again +the cluster, after gossip dissemination, will consider it as `reachable`. If system messages cannot be delivered to a node it will be quarantined and then it -cannot come back from ``unreachable``. This can happen if the there are too many +cannot come back from `unreachable`. This can happen if the there are too many unacknowledged system messages (e.g. watch, Terminated, remote actor deployment, failures of actors supervised by remote parent). Then the node needs to be moved -to the ``down`` or ``removed`` states (see the `Membership Lifecycle`_ section below) +to the `down` or `removed` states (see the [Membership Lifecycle](#membership-lifecycle) section below) and the actor system must be restarted before it can join the cluster again. -.. _The Phi Accrual Failure Detector: http://www.jaist.ac.jp/~defago/files/pdf/IS_RR_2004_010.pdf +#### Leader - -Leader -^^^^^^ - -After gossip convergence a ``leader`` for the cluster can be determined. There is no -``leader`` election process, the ``leader`` can always be recognised deterministically +After gossip convergence a `leader` for the cluster can be determined. There is no +`leader` election process, the `leader` can always be recognised deterministically by any node whenever there is gossip convergence. The leader is just a role, any node can be the leader and it can change between convergence rounds. -The ``leader`` is simply the first node in sorted order that is able to take the leadership role, -where the preferred member states for a ``leader`` are ``up`` and ``leaving`` -(see the `Membership Lifecycle`_ section below for more information about member states). +The `leader` is simply the first node in sorted order that is able to take the leadership role, +where the preferred member states for a `leader` are `up` and `leaving` +(see the [Membership Lifecycle](#membership-lifecycle) section below for more information about member states). -The role of the ``leader`` is to shift members in and out of the cluster, changing -``joining`` members to the ``up`` state or ``exiting`` members to the ``removed`` -state. Currently ``leader`` actions are only triggered by receiving a new cluster +The role of the `leader` is to shift members in and out of the cluster, changing +`joining` members to the `up` state or `exiting` members to the `removed` +state. Currently `leader` actions are only triggered by receiving a new cluster state with gossip convergence. -The ``leader`` also has the power, if configured so, to "auto-down" a node that -according to the `Failure Detector`_ is considered ``unreachable``. This means setting -the ``unreachable`` node status to ``down`` automatically after a configured time +The `leader` also has the power, if configured so, to "auto-down" a node that +according to the [Failure Detector](#failure-detector) is considered `unreachable`. This means setting +the `unreachable` node status to `down` automatically after a configured time of unreachability. - -Seed Nodes -^^^^^^^^^^ +#### Seed Nodes The seed nodes are configured contact points for new nodes joining the cluster. When a new node is started it sends a message to all seed nodes and then sends @@ -185,9 +160,7 @@ cluster itself, it is only relevant for new nodes joining the cluster as it helps them to find contact points to send the join command to; a new member can send this command to any current member of the cluster, not only to the seed nodes. - -Gossip Protocol -^^^^^^^^^^^^^^^ +#### Gossip Protocol A variation of *push-pull gossip* is used to reduce the amount of gossip information sent around the cluster. In push-pull gossip a digest is sent @@ -225,14 +198,12 @@ then it goes back to biased gossip again. The recipient of the gossip state or the gossip status can use the gossip version (vector clock) to determine whether: -#. it has a newer version of the gossip state, in which case it sends that back - to the gossiper - -#. it has an outdated version of the state, in which case the recipient requests - the current state from the gossiper by sending back its version of the gossip state - -#. it has conflicting gossip versions, in which case the different versions are merged - and sent back + 1. it has a newer version of the gossip state, in which case it sends that back +to the gossiper + 2. it has an outdated version of the state, in which case the recipient requests +the current state from the gossiper by sending back its version of the gossip state + 3. it has conflicting gossip versions, in which case the different versions are merged +and sent back If the recipient and the gossip have the same version then the gossip state is not sent or requested. @@ -241,118 +212,124 @@ The periodic nature of the gossip has a nice batching effect of state changes, e.g. joining several nodes quickly after each other to one node will result in only one state change to be spread to other members in the cluster. -The gossip messages are serialized with `protobuf`_ and also gzipped to reduce payload +The gossip messages are serialized with [protobuf](https://code.google.com/p/protobuf/) and also gzipped to reduce payload size. -.. _protobuf: https://code.google.com/p/protobuf/ +### Membership Lifecycle -Membership Lifecycle --------------------- - -A node begins in the ``joining`` state. Once all nodes have seen that the new -node is joining (through gossip convergence) the ``leader`` will set the member -state to ``up``. +A node begins in the `joining` state. Once all nodes have seen that the new +node is joining (through gossip convergence) the `leader` will set the member +state to `up`. If a node is leaving the cluster in a safe, expected manner then it switches to -the ``leaving`` state. Once the leader sees the convergence on the node in the -``leaving`` state, the leader will then move it to ``exiting``. Once all nodes -have seen the exiting state (convergence) the ``leader`` will remove the node -from the cluster, marking it as ``removed``. +the `leaving` state. Once the leader sees the convergence on the node in the +`leaving` state, the leader will then move it to `exiting`. Once all nodes +have seen the exiting state (convergence) the `leader` will remove the node +from the cluster, marking it as `removed`. -If a node is ``unreachable`` then gossip convergence is not possible and therefore -any ``leader`` actions are also not possible (for instance, allowing a node to +If a node is `unreachable` then gossip convergence is not possible and therefore +any `leader` actions are also not possible (for instance, allowing a node to become a part of the cluster). To be able to move forward the state of the -``unreachable`` nodes must be changed. It must become ``reachable`` again or marked -as ``down``. If the node is to join the cluster again the actor system must be +`unreachable` nodes must be changed. It must become `reachable` again or marked +as `down`. If the node is to join the cluster again the actor system must be restarted and go through the joining process again. The cluster can, through the leader, also *auto-down* a node after a configured time of unreachability. If new incarnation of unreachable node tries to rejoin the cluster old incarnation will be -marked as ``down`` and new incarnation can rejoin the cluster without manual intervention. +marked as `down` and new incarnation can rejoin the cluster without manual intervention. -.. note:: If you have *auto-down* enabled and the failure detector triggers, you - can over time end up with a lot of single node clusters if you don't put - measures in place to shut down nodes that have become ``unreachable``. This - follows from the fact that the ``unreachable`` node will likely see the rest of - the cluster as ``unreachable``, become its own leader and form its own cluster. +@@@ note -As mentioned before, if a node is ``unreachable`` then gossip convergence is not -possible and therefore any ``leader`` actions are also not possible. By enabling -``akka.cluster.allow-weakly-up-members`` (enabled by default) it is possible to +If you have *auto-down* enabled and the failure detector triggers, you +can over time end up with a lot of single node clusters if you don't put +measures in place to shut down nodes that have become `unreachable`. This +follows from the fact that the `unreachable` node will likely see the rest of +the cluster as `unreachable`, become its own leader and form its own cluster. + +@@@ + +As mentioned before, if a node is `unreachable` then gossip convergence is not +possible and therefore any `leader` actions are also not possible. By enabling +`akka.cluster.allow-weakly-up-members` (enabled by default) it is possible to let new joining nodes be promoted while convergence is not yet reached. These -``Joining`` nodes will be promoted as ``WeaklyUp``. Once gossip convergence is -reached, the leader will move ``WeaklyUp`` members to ``Up``. +`Joining` nodes will be promoted as `WeaklyUp`. Once gossip convergence is +reached, the leader will move `WeaklyUp` members to `Up`. Note that members on the other side of a network partition have no knowledge about -the existence of the new members. You should for example not count ``WeaklyUp`` +the existence of the new members. You should for example not count `WeaklyUp` members in quorum decisions. -State Diagram for the Member States (``akka.cluster.allow-weakly-up-members=off``) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### State Diagram for the Member States (`akka.cluster.allow-weakly-up-members=off`) -.. image:: ../images/member-states.png +![member-states.png](../images/member-states.png) -State Diagram for the Member States (``akka.cluster.allow-weakly-up-members=on``) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### State Diagram for the Member States (`akka.cluster.allow-weakly-up-members=on`) -.. image:: ../images/member-states-weakly-up.png +![member-states-weakly-up.png](../images/member-states-weakly-up.png) -Member States -^^^^^^^^^^^^^ +#### Member States -- **joining** - transient state when joining a cluster + * + **joining** + : transient state when joining a cluster + + * + **weakly up** + : transient state while network split (only if `akka.cluster.allow-weakly-up-members=on`) + + * + **up** + : normal operating state + + * + **leaving** + / + **exiting** + : states during graceful removal + + * + **down** + : marked as down (no longer part of cluster decisions) + + * + **removed** + : tombstone state (no longer a member) + -- **weakly up** - transient state while network split (only if ``akka.cluster.allow-weakly-up-members=on``) +#### User Actions -- **up** - normal operating state + * + **join** + : join a single node to a cluster - can be explicit or automatic on +startup if a node to join have been specified in the configuration + + * + **leave** + : tell a node to leave the cluster gracefully + + * + **down** + : mark a node as down + -- **leaving** / **exiting** - states during graceful removal +#### Leader Actions -- **down** - marked as down (no longer part of cluster decisions) +The `leader` has the following duties: -- **removed** - tombstone state (no longer a member) + * shifting members in and out of the cluster + * joining -> up + * exiting -> removed +#### Failure Detection and Unreachability -User Actions -^^^^^^^^^^^^ - -- **join** - join a single node to a cluster - can be explicit or automatic on - startup if a node to join have been specified in the configuration - -- **leave** - tell a node to leave the cluster gracefully - -- **down** - mark a node as down - - -Leader Actions -^^^^^^^^^^^^^^ - -The ``leader`` has the following duties: - -- shifting members in and out of the cluster - - - joining -> up - - - exiting -> removed - - -Failure Detection and Unreachability -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- fd* - the failure detector of one of the monitoring nodes has triggered - causing the monitored node to be marked as unreachable - -- unreachable* - unreachable is not a real member states but more of a flag in addition - to the state signaling that the cluster is unable to talk to this node, - after being unreachable the failure detector may detect it as reachable - again and thereby remove the flag + * + fd* + : the failure detector of one of the monitoring nodes has triggered +causing the monitored node to be marked as unreachable + + * + unreachable* + : unreachable is not a real member states but more of a flag in addition +to the state signaling that the cluster is unable to talk to this node, +after being unreachable the failure detector may detect it as reachable +again and thereby remove the flag + \ No newline at end of file diff --git a/akka-docs/src/main/paradox/common/duration.md b/akka-docs/src/main/paradox/common/duration.md index 97136d48b3..febc38405b 100644 --- a/akka-docs/src/main/paradox/common/duration.md +++ b/akka-docs/src/main/paradox/common/duration.md @@ -1,60 +1,55 @@ -.. _Duration: - -######## -Duration -######## +# Duration Durations are used throughout the Akka library, wherefore this concept is -represented by a special data type, :class:`scala.concurrent.duration.Duration`. -Values of this type may represent infinite (:obj:`Duration.Inf`, -:obj:`Duration.MinusInf`) or finite durations, or be :obj:`Duration.Undefined`. +represented by a special data type, `scala.concurrent.duration.Duration`. +Values of this type may represent infinite (`Duration.Inf`, +`Duration.MinusInf`) or finite durations, or be `Duration.Undefined`. -Finite vs. Infinite -=================== +## Finite vs. Infinite Since trying to convert an infinite duration into a concrete time unit like seconds will throw an exception, there are different types available for distinguishing the two kinds at compile time: -* :class:`FiniteDuration` is guaranteed to be finite, calling :meth:`toNanos` - and friends is safe -* :class:`Duration` can be finite or infinite, so this type should only be used - when finite-ness does not matter; this is a supertype of :class:`FiniteDuration` + * `FiniteDuration` is guaranteed to be finite, calling `toNanos` +and friends is safe + * `Duration` can be finite or infinite, so this type should only be used +when finite-ness does not matter; this is a supertype of `FiniteDuration` -Scala -===== +## Scala In Scala durations are constructable using a mini-DSL and support all expected arithmetic operations: -.. includecode:: code/docs/duration/Sample.scala#dsl +@@snip [Sample.scala](code/docs/duration/Sample.scala) { #dsl } -.. note:: +@@@ note - You may leave out the dot if the expression is clearly delimited (e.g. - within parentheses or in an argument list), but it is recommended to use it - if the time unit is the last token on a line, otherwise semi-colon inference - might go wrong, depending on what starts the next line. +You may leave out the dot if the expression is clearly delimited (e.g. +within parentheses or in an argument list), but it is recommended to use it +if the time unit is the last token on a line, otherwise semi-colon inference +might go wrong, depending on what starts the next line. -Java -==== +@@@ + +## Java Java provides less syntactic sugar, so you have to spell out the operations as method calls instead: -.. includecode:: code/docs/duration/Java.java#import -.. includecode:: code/docs/duration/Java.java#dsl +@@snip [Java.java](code/docs/duration/Java.java) { #import } -Deadline -======== +@@snip [Java.java](code/docs/duration/Java.java) { #dsl } -Durations have a brother named :class:`Deadline`, which is a class holding a representation +## Deadline + +Durations have a brother named `Deadline`, which is a class holding a representation of an absolute point in time, and support deriving a duration from this by calculating the difference between now and the deadline. This is useful when you want to keep one overall deadline without having to take care of the book-keeping wrt. the passing of time yourself: -.. includecode:: code/docs/duration/Sample.scala#deadline +@@snip [Sample.scala](code/docs/duration/Sample.scala) { #deadline } In Java you create these from durations: -.. includecode:: code/docs/duration/Java.java#deadline +@@snip [Java.java](code/docs/duration/Java.java) { #deadline } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/common/may-change.md b/akka-docs/src/main/paradox/common/may-change.md index c049d59e6f..69a3c347c9 100644 --- a/akka-docs/src/main/paradox/common/may-change.md +++ b/akka-docs/src/main/paradox/common/may-change.md @@ -1,24 +1,19 @@ - -.. _may-change: - -########################### -Modules marked "May Change" -########################### +# Modules marked "May Change" To be able to introduce new modules and APIs without freezing them the moment they -are released because of our :ref:`binary compatibility guarantees ` we have introduced +are released because of our @ref:[binary compatibility guarantees](binary-compatibility-rules.md) we have introduced the term **may change**. Concretely **may change** means that an API or module is in early access mode and that it: -- is not covered by Lightbend's commercial support (unless specifically stated otherwise) -- is not guaranteed to be binary compatible in minor releases -- may have its API change in breaking ways in minor releases -- may be entirely dropped from Akka in a minor release + * is not covered by Lightbend's commercial support (unless specifically stated otherwise) + * is not guaranteed to be binary compatible in minor releases + * may have its API change in breaking ways in minor releases + * may be entirely dropped from Akka in a minor release Complete modules can be marked as **may change**, this will can be found in their module description and in the docs. -Individual public APIs can be annotated with ``akka.api.annotation.ApiMayChange`` to signal that it has less +Individual public APIs can be annotated with `akka.api.annotation.ApiMayChange` to signal that it has less guarantees than the rest of the module it lives in. For example, when while introducing "new" Java 8 APIs into existing stable modules, these APIs may be marked with this annotation to signal that they are not frozen yet. Please use such methods and classes with care, however if you see such APIs that is the best point in time to try them @@ -33,17 +28,12 @@ that the module or API wasn't useful. These are the current complete modules marked as **may change**: -.. toctree:: - :maxdepth: 1 +@@toc { depth=1 } - ../dev/multi-node-testing - ../scala/typed +@@@ index -Another reason for marking a module as "may change" is that it's too early -to tell if the module has a maintainer that can take the responsibility -of the module over time. These modules live in the ``akka-contrib`` subproject: +* [../dev/multi-node-testing](../dev/multi-node-testing.md) +* [../scala/typed](../scala/typed.md) -.. toctree:: - :maxdepth: 1 +@@@ - ../contrib/index \ No newline at end of file diff --git a/akka-docs/src/main/paradox/dev/building-akka.md b/akka-docs/src/main/paradox/dev/building-akka.md index fafbaab5a7..2ad2e357bb 100644 --- a/akka-docs/src/main/paradox/dev/building-akka.md +++ b/akka-docs/src/main/paradox/dev/building-akka.md @@ -1,157 +1,145 @@ - -.. highlightlang:: none - -.. _building-akka: - -############### - Building Akka -############### +# Building Akka This page describes how to build and run Akka from the latest source code. +## Get the Source Code -Get the Source Code -=================== - -Akka uses `Git`_ and is hosted at `Github`_. - -.. _Git: http://git-scm.com -.. _Github: http://github.com +Akka uses [Git](http://git-scm.com) and is hosted at [Github](http://github.com). You first need Git installed on your machine. You can then clone the source -repository from http://github.com/akka/akka. +repository from [http://github.com/akka/akka](http://github.com/akka/akka). -For example:: +For example: - git clone git://github.com/akka/akka.git +``` +git clone git://github.com/akka/akka.git +``` If you have already cloned the repository previously then you can update the -code with ``git pull``:: +code with `git pull`: - git pull origin master +``` +git pull origin master +``` +## sbt -sbt -=== - -Akka is using the excellent `sbt`_ build system. So the first thing you have to +Akka is using the excellent [sbt](https://github.com/sbt/sbt) build system. So the first thing you have to do is to download and install sbt. You can read more about how to do that in the -`sbt setup`_ documentation. - -.. _sbt: https://github.com/sbt/sbt -.. _sbt setup: http://www.scala-sbt.org/0.13/tutorial/index.html +[sbt setup](http://www.scala-sbt.org/0.13/tutorial/index.html) documentation. The sbt commands that you'll need to build Akka are all included below. If you want to find out more about sbt and using it for your own projects do read the -`sbt documentation`_. +[sbt documentation](http://www.scala-sbt.org/documentation.html). -.. _sbt documentation: http://www.scala-sbt.org/documentation.html - -The main Akka sbt build file is ``project/AkkaBuild.scala``, with a `build.sbt` in +The main Akka sbt build file is `project/AkkaBuild.scala`, with a *build.sbt* in each subproject’s directory. It is advisable to allocate at least 2GB of heap size to the JVM that runs sbt, otherwise you may experience some spurious failures when running the tests. -Building Akka -============= +## Building Akka -First make sure that you are in the akka code directory:: +First make sure that you are in the akka code directory: - cd akka +``` +cd akka +``` +### Building -Building --------- +To compile all the Akka core modules use the `compile` command: -To compile all the Akka core modules use the ``compile`` command:: +``` +sbt compile +``` - sbt compile +You can run all tests with the `test` command: -You can run all tests with the ``test`` command:: - - sbt test +``` +sbt test +``` If compiling and testing are successful then you have everything working for the latest Akka development version. - -Parallel Execution ------------------- +### Parallel Execution By default the tests are executed sequentially. They can be executed in parallel to reduce build times, if hardware can handle the increased memory and cpu usage. Add the following system property to sbt -launch script to activate parallel execution:: +launch script to activate parallel execution: - -Dakka.parallelExecution=true +``` +-Dakka.parallelExecution=true +``` -Long Running and Time Sensitive Tests -------------------------------------- +### Long Running and Time Sensitive Tests By default are the long running tests (mainly cluster tests) and time sensitive tests (dependent on the -performance of the machine it is running on) disabled. You can enable them by adding one of the flags:: +performance of the machine it is running on) disabled. You can enable them by adding one of the flags: - -Dakka.test.tags.include=long-running - -Dakka.test.tags.include=timing +``` +-Dakka.test.tags.include=long-running +-Dakka.test.tags.include=timing +``` -Or if you need to enable them both:: +Or if you need to enable them both: - -Dakka.test.tags.include=long-running,timing +``` +-Dakka.test.tags.include=long-running,timing +``` -Publish to Local Ivy Repository -------------------------------- +### Publish to Local Ivy Repository If you want to deploy the artifacts to your local Ivy repository (for example, -to use from an sbt project) use the ``publish-local`` command:: +to use from an sbt project) use the `publish-local` command: - sbt publish-local +``` +sbt publish-local +``` +### sbt Interactive Mode -sbt Interactive Mode --------------------- - -Note that in the examples above we are calling ``sbt compile`` and ``sbt test`` -and so on, but sbt also has an interactive mode. If you just run ``sbt`` you +Note that in the examples above we are calling `sbt compile` and `sbt test` +and so on, but sbt also has an interactive mode. If you just run `sbt` you enter the interactive sbt prompt and can enter the commands directly. This saves starting up a new JVM instance for each command and can be much faster and more convenient. -For example, building Akka as above is more commonly done like this:: +For example, building Akka as above is more commonly done like this: - % sbt - [info] Set current project to default (in build file:/.../akka/project/plugins/) - [info] Set current project to akka (in build file:/.../akka/) - > compile - ... - > test - ... +``` +% sbt +[info] Set current project to default (in build file:/.../akka/project/plugins/) +[info] Set current project to akka (in build file:/.../akka/) +> compile +... +> test +... +``` - -sbt Batch Mode --------------- +### sbt Batch Mode It's also possible to combine commands in a single call. For example, testing, -and publishing Akka to the local Ivy repository can be done with:: +and publishing Akka to the local Ivy repository can be done with: - sbt test publish-local +``` +sbt test publish-local +``` - -.. _dependencies: - -Dependencies -============ + +## Dependencies You can look at the Ivy dependency resolution information that is created on -``sbt update`` and found in ``~/.ivy2/cache``. For example, the -``~/.ivy2/cache/com.typesafe.akka-akka-remote-compile.xml`` file contains +`sbt update` and found in `~/.ivy2/cache`. For example, the +`~/.ivy2/cache/com.typesafe.akka-akka-remote-compile.xml` file contains the resolution information for the akka-remote module compile dependencies. If you open this file in a web browser you will get an easy to navigate view of dependencies. -Scaladoc Dependencies -===================== +## Scaladoc Dependencies Akka generates class diagrams for the API documentation using ScalaDoc. This -needs the ``dot`` command from the Graphviz software package to be installed to +needs the `dot` command from the Graphviz software package to be installed to avoid errors. You can disable the diagram generation by adding the flag -``-Dakka.scaladoc.diagrams=false``. After installing Graphviz, make sure you add -the toolset to the PATH (definitely on Windows). +`-Dakka.scaladoc.diagrams=false`. After installing Graphviz, make sure you add +the toolset to the PATH (definitely on Windows). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/dev/developer-guidelines.md b/akka-docs/src/main/paradox/dev/developer-guidelines.md index efb672fe77..adeef1764a 100644 --- a/akka-docs/src/main/paradox/dev/developer-guidelines.md +++ b/akka-docs/src/main/paradox/dev/developer-guidelines.md @@ -1,69 +1,59 @@ -.. _developer_guidelines: +# Developer Guidelines -Developer Guidelines -==================== +@@@ note -.. note:: +First read [The Akka Contributor Guidelines](https://github.com/akka/akka/blob/master/CONTRIBUTING.md). - First read `The Akka Contributor Guidelines`_. +@@@ -Code Style ----------- +## Code Style -The Akka code style follows the `Scala Style Guide `_ . The only exception is the +The Akka code style follows the [Scala Style Guide](http://docs.scala-lang.org/style/) . The only exception is the style of block comments: -.. code-block:: scala +```scala +/** + * Style mandated by "Scala Style Guide" + */ - /** - * Style mandated by "Scala Style Guide" - */ +/** + * Style adopted in the Akka codebase + */ +``` - /** - * Style adopted in the Akka codebase - */ +Akka is using `Scalariform` to format the source code as part of the build. So just hack away and then run `sbt compile` and it will reformat the code according to Akka standards. -Akka is using ``Scalariform`` to format the source code as part of the build. So just hack away and then run ``sbt compile`` and it will reformat the code according to Akka standards. +## Process -Process -------- +The full process is described in [The Akka Contributor Guidelines](https://github.com/akka/akka/blob/master/CONTRIBUTING.md). In summary: -The full process is described in `The Akka Contributor Guidelines`_. In summary: + * Make sure you have signed the Akka CLA, if not, [sign it online](http://www.lightbend.com/contribute/cla). + * Pick a ticket, if there is no ticket for your work then create one first. + * Fork [akka/akka](https://github.com/akka/akka). Start working in a feature branch. + * When you are done, create a GitHub Pull-Request towards the targeted branch. + * When there's consensus on the review, someone from the Akka Core Team will merge it. -* Make sure you have signed the Akka CLA, if not, `sign it online `_. -* Pick a ticket, if there is no ticket for your work then create one first. -* Fork `akka/akka `_. Start working in a feature branch. -* When you are done, create a GitHub Pull-Request towards the targeted branch. -* When there's consensus on the review, someone from the Akka Core Team will merge it. +## Commit messages -Commit messages ---------------- +Please follow the conventions described in [The Akka Contributor Guidelines](https://github.com/akka/akka/blob/master/CONTRIBUTING.md) when creating public commits and writing commit messages. -Please follow the conventions described in `The Akka Contributor Guidelines`_ when creating public commits and writing commit messages. +## Testing -Testing -------- +All code that is checked in **should** have tests. All testing is done with `ScalaTest` and `ScalaCheck`. -All code that is checked in **should** have tests. All testing is done with ``ScalaTest`` and ``ScalaCheck``. + * Name tests as **Test.scala** if they do not depend on any external stuff. That keeps surefire happy. + * Name tests as **Spec.scala** if they have external dependencies. -* Name tests as **Test.scala** if they do not depend on any external stuff. That keeps surefire happy. -* Name tests as **Spec.scala** if they have external dependencies. +### Actor TestKit -Actor TestKit -^^^^^^^^^^^^^ +There is a useful test kit for testing actors: [akka.util.TestKit](@github@/akka-testkit/src/main/scala/akka/testkit/TestKit.scala). It enables assertions concerning replies received and their timing, there is more documentation in the akka-testkit module. -There is a useful test kit for testing actors: `akka.util.TestKit <@github@/akka-testkit/src/main/scala/akka/testkit/TestKit.scala>`_. It enables assertions concerning replies received and their timing, there is more documentation in the :ref:`akka-testkit` module. - -Multi-JVM Testing -^^^^^^^^^^^^^^^^^ +### Multi-JVM Testing Included in the example is an sbt trait for multi-JVM testing which will fork JVMs for multi-node testing. There is support for running applications (objects with main methods) and running ScalaTest tests. -NetworkFailureTest -^^^^^^^^^^^^^^^^^^ +### NetworkFailureTest -You can use the 'NetworkFailureTest' trait to test network failure. - -.. _The Akka Contributor Guidelines: https://github.com/akka/akka/blob/master/CONTRIBUTING.md +You can use the 'NetworkFailureTest' trait to test network failure. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/dev/documentation.md b/akka-docs/src/main/paradox/dev/documentation.md index e493d92ef3..243093b4e9 100644 --- a/akka-docs/src/main/paradox/dev/documentation.md +++ b/akka-docs/src/main/paradox/dev/documentation.md @@ -1,215 +1,202 @@ -.. highlightlang:: rest +# Documentation Guidelines -.. _documentation: +The Akka documentation uses [reStructuredText](http://docutils.sourceforge.net/rst.html) as its markup language and is +built using [Sphinx](http://sphinx.pocoo.org). -######################### - Documentation Guidelines -######################### +## Sphinx -The Akka documentation uses `reStructuredText`_ as its markup language and is -built using `Sphinx`_. +For more details see [The Sphinx Documentation](http://sphinx.pocoo.org/contents.html) -.. _reStructuredText: http://docutils.sourceforge.net/rst.html -.. _sphinx: http://sphinx.pocoo.org +## reStructuredText +For more details see [The reST Quickref](http://docutils.sourceforge.net/docs/user/rst/quickref.html) -Sphinx -====== - -For more details see `The Sphinx Documentation `_ - -reStructuredText -================ - -For more details see `The reST Quickref `_ - -Sections --------- +### Sections Section headings are very flexible in reST. We use the following convention in the Akka documentation: -* ``#`` (over and under) for module headings -* ``=`` for sections -* ``-`` for subsections -* ``^`` for subsubsections -* ``~`` for subsubsubsections + * `#` (over and under) for module headings + * `=` for sections + * `-` for subsections + * `^` for subsubsections + * `~` for subsubsubsections - -Cross-referencing ------------------ +### Cross-referencing Sections that may be cross-referenced across the documentation should be marked -with a reference. To mark a section use ``.. _ref-name:`` before the section -heading. The section can then be linked with ``:ref:`ref-name```. These are +with a reference. To mark a section use `.. _ref-name:` before the section +heading. The section can then be linked with `:ref:`ref-name``. These are unique references across the entire documentation. -For example:: +For example: - .. _akka-module: +``` +.. _akka-module: - ############# - Akka Module - ############# +############# + Akka Module +############# - This is the module documentation. +This is the module documentation. - .. _akka-section: +.. _akka-section: - Akka Section - ============ +Akka Section +============ - Akka Subsection - --------------- +Akka Subsection +--------------- - Here is a reference to "akka section": :ref:`akka-section` which will have the - name "Akka Section". +Here is a reference to "akka section": :ref:`akka-section` which will have the +name "Akka Section". +``` -Build the documentation -======================= +## Build the documentation -First install `Sphinx`_. See below. +First install [Sphinx](http://sphinx.pocoo.org). See below. -Building --------- +### Building -For the html version of the docs:: +For the html version of the docs: - sbt sphinx:generateHtml +``` +sbt sphinx:generateHtml - open /akka-docs/target/sphinx/html/index.html +open /akka-docs/target/sphinx/html/index.html +``` -For the pdf version of the docs:: +For the pdf version of the docs: - sbt sphinx:generatePdf +``` +sbt sphinx:generatePdf - open /akka-docs/target/sphinx/latex/AkkaJava.pdf - or - open /akka-docs/target/sphinx/latex/AkkaScala.pdf +open /akka-docs/target/sphinx/latex/AkkaJava.pdf +or +open /akka-docs/target/sphinx/latex/AkkaScala.pdf +``` -Installing Sphinx on OS X -------------------------- +### Installing Sphinx on OS X -Install `Homebrew `_ +Install [Homebrew](https://github.com/mxcl/homebrew) Install Python with Homebrew: -:: - - brew install python +``` +brew install python +``` Homebrew will automatically add Python executable to your $PATH and pip is a part of the default Python installation with Homebrew. More information in case of trouble: -https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python +[https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python](https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python) Install sphinx: -:: - - pip install sphinx +``` +pip install sphinx +``` Install BasicTeX package from: -http://www.tug.org/mactex/morepackages.html +[http://www.tug.org/mactex/morepackages.html](http://www.tug.org/mactex/morepackages.html) Add texlive bin to $PATH: -:: - - export TEXLIVE_PATH=/usr/local/texlive/2016basic/bin/universal-darwin - export PATH=$TEXLIVE_PATH:$PATH +``` +export TEXLIVE_PATH=/usr/local/texlive/2016basic/bin/universal-darwin +export PATH=$TEXLIVE_PATH:$PATH +``` Add missing tex packages: -:: - - sudo tlmgr update --self - sudo tlmgr install titlesec - sudo tlmgr install framed - sudo tlmgr install threeparttable - sudo tlmgr install wrapfig - sudo tlmgr install helvetic - sudo tlmgr install courier - sudo tlmgr install multirow - sudo tlmgr install capt-of - sudo tlmgr install needspace - sudo tlmgr install eqparbox - sudo tlmgr install environ - sudo tlmgr install trimspaces +``` +sudo tlmgr update --self +sudo tlmgr install titlesec +sudo tlmgr install framed +sudo tlmgr install threeparttable +sudo tlmgr install wrapfig +sudo tlmgr install helvetic +sudo tlmgr install courier +sudo tlmgr install multirow +sudo tlmgr install capt-of +sudo tlmgr install needspace +sudo tlmgr install eqparbox +sudo tlmgr install environ +sudo tlmgr install trimspaces +``` If you get the error "unknown locale: UTF-8" when generating the documentation the solution is to define the following environment variables: -:: +``` +export LANG=en_US.UTF-8 +export LC_ALL=en_US.UTF-8 +``` - export LANG=en_US.UTF-8 - export LC_ALL=en_US.UTF-8 +### Installing Sphinx on Linux -Installing Sphinx on Linux --------------------------- Install Python with your package manager: -:: +``` +apt-get install python # for Debian based systems +yum install python # for CentOS/RHEL systems +``` - apt-get install python # for Debian based systems - yum install python # for CentOS/RHEL systems - -This will automatically add Python executable to your $PATH and pip is a part of the default Python installation. Remember you need `sudo` rights to run this command. +This will automatically add Python executable to your $PATH and pip is a part of the default Python installation. Remember you need *sudo* rights to run this command. More information in case of trouble: -https://packaging.python.org/install_requirements_linux/ +[https://packaging.python.org/install_requirements_linux](https://packaging.python.org/install_requirements_linux)/ Install Sphinx: -:: - - apt-get install python-sphinx # for Debian based systems - #alternatively - pip install sphinx +``` +apt-get install python-sphinx # for Debian based systems +#alternatively +pip install sphinx +``` For other Linux systems please check Sphinx website: -http://www.sphinx-doc.org/en/stable/install.html#other-linux-distributions +[http://www.sphinx-doc.org/en/stable/install.html#other-linux-distributions](http://www.sphinx-doc.org/en/stable/install.html#other-linux-distributions) Install TextLive: -:: +``` +apt-get install texlive-latex-base texlive-latex-extra texlive-latex-recommended +# additionally you may need xzdec +apt-get install xzdec +``` - apt-get install texlive-latex-base texlive-latex-extra texlive-latex-recommended - # additionally you may need xzdec - apt-get install xzdec - In case you get the following error: - - - Unknown directive ...containerchecksum c59200574a316416a23695c258edf3a32531fbda43ccdc09360ee105c3f07f9fb77df17c4ba4c2ea4f3a5ea6667e064b51e3d8c2fe6c984ba3e71b4e32716955... , please fix it! at /usr/share/texlive/tlpkg/TeXLive/TLPOBJ.pm line 210, <$retfh> line 5579. +> +Unknown directive ...containerchecksum c59200574a316416a23695c258edf3a32531fbda43ccdc09360ee105c3f07f9fb77df17c4ba4c2ea4f3a5ea6667e064b51e3d8c2fe6c984ba3e71b4e32716955... , please fix it! at /usr/share/texlive/tlpkg/TeXLive/TLPOBJ.pm line 210, <$retfh> line 5579. you need to specify you want to continue using the 2015 version: -:: - - tlmgr option repository ftp://tug.org/historic/systems/texlive/2015/tlnet-final +``` +tlmgr option repository ftp://tug.org/historic/systems/texlive/2015/tlnet-final +``` Add missing tex packages: -:: - - sudo tlmgr update --self - sudo tlmgr install titlesec - sudo tlmgr install framed - sudo tlmgr install threeparttable - sudo tlmgr install wrapfig - sudo tlmgr install helvetic - sudo tlmgr install courier - sudo tlmgr install multirow - sudo tlmgr install capt-of - sudo tlmgr install needspace - sudo tlmgr install eqparbox - sudo tlmgr install environ - sudo tlmgr install trimspaces +``` +sudo tlmgr update --self +sudo tlmgr install titlesec +sudo tlmgr install framed +sudo tlmgr install threeparttable +sudo tlmgr install wrapfig +sudo tlmgr install helvetic +sudo tlmgr install courier +sudo tlmgr install multirow +sudo tlmgr install capt-of +sudo tlmgr install needspace +sudo tlmgr install eqparbox +sudo tlmgr install environ +sudo tlmgr install trimspaces +``` If you get the error "unknown locale: UTF-8" when generating the documentation the solution is to define the following environment variables: -:: - - export LANG=en_US.UTF-8 - export LC_ALL=en_US.UTF-8 +``` +export LANG=en_US.UTF-8 +export LC_ALL=en_US.UTF-8 +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/dev/index.md b/akka-docs/src/main/paradox/dev/index.md index 3ebc1996a2..6f767b9c50 100644 --- a/akka-docs/src/main/paradox/dev/index.md +++ b/akka-docs/src/main/paradox/dev/index.md @@ -1,12 +1,13 @@ -Information for Akka Developers -=============================== +# Information for Akka Developers -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - building-akka - multi-jvm-testing - io-layer - developer-guidelines - documentation +@@@ index +* [building-akka](building-akka.md) +* [multi-jvm-testing](multi-jvm-testing.md) +* [io-layer](io-layer.md) +* [developer-guidelines](developer-guidelines.md) +* [documentation](documentation.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/dev/io-layer.md b/akka-docs/src/main/paradox/dev/io-layer.md index 9fc955b73a..eae8eb7ca0 100644 --- a/akka-docs/src/main/paradox/dev/io-layer.md +++ b/akka-docs/src/main/paradox/dev/io-layer.md @@ -1,50 +1,39 @@ -.. _io-layer: +# I/O Layer Design -################ -I/O Layer Design -################ - -The ``akka.io`` package has been developed in collaboration between the Akka -and `spray.io`_ teams. Its design incorporates the experiences with the -``spray-io`` module along with improvements that were jointly developed for +The `akka.io` package has been developed in collaboration between the Akka +and [spray.io](http://spray.io) teams. Its design incorporates the experiences with the +`spray-io` module along with improvements that were jointly developed for more general consumption as an actor-based service. -Requirements -============ +## Requirements In order to form a general and extensible IO layer basis for a wide range of applications, with Akka remoting and spray HTTP being the initial ones, the following requirements were established as key drivers for the design: -* scalability to millions of concurrent connections + * scalability to millions of concurrent connections + * lowest possible latency in getting data from an input channel into the +target actor’s mailbox + * maximal throughput + * optional back-pressure in both directions (i.e. throttling local senders as +well as allowing local readers to throttle remote senders, where allowed by +the protocol) + * a purely actor-based API with immutable data representation + * extensibility for integrating new transports by way of a very lean SPI; the +goal is to not force I/O mechanisms into a lowest common denominator but +instead allow completely protocol-specific user-level APIs. -* lowest possible latency in getting data from an input channel into the - target actor’s mailbox - -* maximal throughput - -* optional back-pressure in both directions (i.e. throttling local senders as - well as allowing local readers to throttle remote senders, where allowed by - the protocol) - -* a purely actor-based API with immutable data representation - -* extensibility for integrating new transports by way of a very lean SPI; the - goal is to not force I/O mechanisms into a lowest common denominator but - instead allow completely protocol-specific user-level APIs. - -Basic Architecture -================== +## Basic Architecture Each transport implementation will be made available as a separate Akka -extension, offering an :class:`ActorRef` representing the initial point of +extension, offering an `ActorRef` representing the initial point of contact for client code. This "manager" accepts requests for establishing a communications channel (e.g. connect or listen on a TCP socket). Each communications channel is represented by one dedicated actor, which is exposed to client code for all interaction with this channel over its entire lifetime. The central element of the implementation is the transport-specific “selector” -actor; in the case of TCP this would wrap a :class:`java.nio.channels.Selector`. +actor; in the case of TCP this would wrap a `java.nio.channels.Selector`. The channel actors register their interest in readability or writability of their channel by sending corresponding messages to their assigned selector actor. However, the actual channel reading and writing is performed by the @@ -67,7 +56,7 @@ levels, with the management actor at the top-, the channel actors at the leaf- and the selector actors at the mid-level. Back-pressure for output is enabled by allowing the user to specify within its -:class:`Write` messages whether it wants to receive an acknowledgement for +`Write` messages whether it wants to receive an acknowledgement for enqueuing that write to the O/S kernel. Back-pressure for input is enabled by sending the channel actor a message which temporarily disables read interest for the channel until reading is re-enabled with a corresponding resume command. @@ -76,8 +65,7 @@ consuming data at the receiving end (thereby causing them to remain in the kernels read buffers) is propagated back to the sender, linking these two mechanisms across the network. -Design Benefits -=============== +## Design Benefits Staying within the actor model for the whole implementation allows us to remove the need for explicit thread handling logic, and it also means that there are @@ -96,13 +84,12 @@ actors to notice the demise of their user-level handler actors and terminate in an orderly fashion in that case as well; this naturally reduces the chances of leaking open channels. -The choice of using :class:`ActorRef` for exposing all functionality entails +The choice of using `ActorRef` for exposing all functionality entails that these references can be distributed or delegated freely and in general handled as the user sees fit, including the use of remoting and life-cycle monitoring (just to name two). -How to go about Adding a New Transport -====================================== +## How to go about Adding a New Transport The best start is to study the TCP reference implementation to get a good grip on the basic working principle and then design an implementation, which is @@ -110,7 +97,4 @@ similar in spirit, but adapted to the new protocol in question. There are vast differences between I/O mechanisms (e.g. compare file I/O to a message broker) and the goal of this I/O layer is explicitly **not** to shoehorn all of them into a uniform API, which is why only the basic architecture ideas are -documented here. - -.. _spray.io: http://spray.io - +documented here. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/dev/multi-jvm-testing.md b/akka-docs/src/main/paradox/dev/multi-jvm-testing.md index 2a837e9afb..1c6eadbee8 100644 --- a/akka-docs/src/main/paradox/dev/multi-jvm-testing.md +++ b/akka-docs/src/main/paradox/dev/multi-jvm-testing.md @@ -1,207 +1,200 @@ - -.. _multi-jvm-testing: - -################### - Multi JVM Testing -################### +# Multi JVM Testing Supports running applications (objects with main methods) and ScalaTest tests in multiple JVMs at the same time. Useful for integration testing where multiple systems communicate with each other. -Setup -===== +## Setup -The multi-JVM testing is an sbt plugin that you can find at ``_. +The multi-JVM testing is an sbt plugin that you can find at [http://github.com/sbt/sbt-multi-jvm](http://github.com/sbt/sbt-multi-jvm). You can add it as a plugin by adding the following to your project/plugins.sbt: -.. includecode:: ../../../project/plugins.sbt#sbt-multi-jvm +@@snip [plugins.sbt]../../../../../project/plugins.sbt) { #sbt-multi-jvm } -You can then add multi-JVM testing to ``build.sbt`` or ``project/Build.scala`` by including the ``MultiJvm`` -settings and config. Please note that MultiJvm test sources are located in ``src/multi-jvm/...``, -and not in ``src/test/...``. +You can then add multi-JVM testing to `build.sbt` or `project/Build.scala` by including the `MultiJvm` +settings and config. Please note that MultiJvm test sources are located in `src/multi-jvm/...`, +and not in `src/test/...`. -You can specify JVM options for the forked JVMs:: +You can specify JVM options for the forked JVMs: - jvmOptions in MultiJvm := Seq("-Xmx256M") +``` +jvmOptions in MultiJvm := Seq("-Xmx256M") +``` -Here is an example of a `sample project`_ that uses the ``sbt-multi-jvm`` plugin. +Here is an example of a [sample project](@samples@/tree/master/akka-sample-multi-node-scala) that uses the `sbt-multi-jvm` plugin. -Running tests -============= +## Running tests -The multi-JVM tasks are similar to the normal tasks: ``test``, ``test-only``, -and ``run``, but are under the ``multi-jvm`` configuration. +The multi-JVM tasks are similar to the normal tasks: `test`, `test-only`, +and `run`, but are under the `multi-jvm` configuration. So in Akka, to run all the multi-JVM tests in the akka-remote project use (at the sbt prompt): -.. code-block:: none +```none +akka-remote-tests/multi-jvm:test +``` - akka-remote-tests/multi-jvm:test - -Or one can change to the ``akka-remote-tests`` project first, and then run the +Or one can change to the `akka-remote-tests` project first, and then run the tests: -.. code-block:: none +```none +project akka-remote-tests +multi-jvm:test +``` - project akka-remote-tests - multi-jvm:test +To run individual tests use `test-only`: -To run individual tests use ``test-only``: - -.. code-block:: none - - multi-jvm:test-only akka.remote.RandomRoutedRemoteActor +```none +multi-jvm:test-only akka.remote.RandomRoutedRemoteActor +``` More than one test name can be listed to run multiple specific tests. Tab-completion in sbt makes it easy to complete the test names. -It's also possible to specify JVM options with ``test-only`` by including those -options after the test names and ``--``. For example: +It's also possible to specify JVM options with `test-only` by including those +options after the test names and `--`. For example: -.. code-block:: none +```none +multi-jvm:test-only akka.remote.RandomRoutedRemoteActor -- -Dsome.option=something +``` - multi-jvm:test-only akka.remote.RandomRoutedRemoteActor -- -Dsome.option=something - - -Creating application tests -========================== +## Creating application tests The tests are discovered, and combined, through a naming convention. MultiJvm test sources -are located in ``src/multi-jvm/...``. A test is named with the following pattern: +are located in `src/multi-jvm/...`. A test is named with the following pattern: -.. code-block:: none +```none +{TestName}MultiJvm{NodeName} +``` - {TestName}MultiJvm{NodeName} - -That is, each test has ``MultiJvm`` in the middle of its name. The part before -it groups together tests/applications under a single ``TestName`` that will run -together. The part after, the ``NodeName``, is a distinguishing name for each +That is, each test has `MultiJvm` in the middle of its name. The part before +it groups together tests/applications under a single `TestName` that will run +together. The part after, the `NodeName`, is a distinguishing name for each forked JVM. -So to create a 3-node test called ``Sample``, you can create three applications -like the following:: +So to create a 3-node test called `Sample`, you can create three applications +like the following: - package sample +``` +package sample - object SampleMultiJvmNode1 { - def main(args: Array[String]) { - println("Hello from node 1") - } - } +object SampleMultiJvmNode1 { + def main(args: Array[String]) { + println("Hello from node 1") + } +} - object SampleMultiJvmNode2 { - def main(args: Array[String]) { - println("Hello from node 2") - } - } +object SampleMultiJvmNode2 { + def main(args: Array[String]) { + println("Hello from node 2") + } +} - object SampleMultiJvmNode3 { - def main(args: Array[String]) { - println("Hello from node 3") - } - } +object SampleMultiJvmNode3 { + def main(args: Array[String]) { + println("Hello from node 3") + } +} +``` -When you call ``multi-jvm:run sample.Sample`` at the sbt prompt, three JVMs will be +When you call `multi-jvm:run sample.Sample` at the sbt prompt, three JVMs will be spawned, one for each node. It will look like this: -.. code-block:: none +```none +> multi-jvm:run sample.Sample +... +[info] * sample.Sample +[JVM-1] Hello from node 1 +[JVM-2] Hello from node 2 +[JVM-3] Hello from node 3 +[success] Total time: ... +``` - > multi-jvm:run sample.Sample - ... - [info] * sample.Sample - [JVM-1] Hello from node 1 - [JVM-2] Hello from node 2 - [JVM-3] Hello from node 3 - [success] Total time: ... - - -Changing Defaults -================= +## Changing Defaults You can change the name of the multi-JVM test source directory by adding the following configuration to your project: -.. code-block:: none +```none +unmanagedSourceDirectories in MultiJvm <<= + Seq(baseDirectory(_ / "src/some_directory_here")).join +``` - unmanagedSourceDirectories in MultiJvm <<= - Seq(baseDirectory(_ / "src/some_directory_here")).join +You can change what the `MultiJvm` identifier is. For example, to change it to +`ClusterTest` use the `multiJvmMarker` setting: +```none +multiJvmMarker in MultiJvm := "ClusterTest" +``` -You can change what the ``MultiJvm`` identifier is. For example, to change it to -``ClusterTest`` use the ``multiJvmMarker`` setting: +Your tests should now be named `{TestName}ClusterTest{NodeName}`. -.. code-block:: none - - multiJvmMarker in MultiJvm := "ClusterTest" - - -Your tests should now be named ``{TestName}ClusterTest{NodeName}``. - - -Configuration of the JVM instances -================================== +## Configuration of the JVM instances You can define specific JVM options for each of the spawned JVMs. You do that by creating -a file named after the node in the test with suffix ``.opts`` and put them in the same +a file named after the node in the test with suffix `.opts` and put them in the same directory as the test. -For example, to feed the JVM options ``-Dakka.remote.port=9991`` and ``-Xmx256m`` to the ``SampleMultiJvmNode1`` -let's create three ``*.opts`` files and add the options to them. Separate multiple options with +For example, to feed the JVM options `-Dakka.remote.port=9991` and `-Xmx256m` to the `SampleMultiJvmNode1` +let's create three `*.opts` files and add the options to them. Separate multiple options with space. -``SampleMultiJvmNode1.opts``:: +`SampleMultiJvmNode1.opts`: - -Dakka.remote.port=9991 -Xmx256m +``` +-Dakka.remote.port=9991 -Xmx256m +``` -``SampleMultiJvmNode2.opts``:: +`SampleMultiJvmNode2.opts`: - -Dakka.remote.port=9992 -Xmx256m +``` +-Dakka.remote.port=9992 -Xmx256m +``` -``SampleMultiJvmNode3.opts``:: +`SampleMultiJvmNode3.opts`: - -Dakka.remote.port=9993 -Xmx256m +``` +-Dakka.remote.port=9993 -Xmx256m +``` -ScalaTest -========= +## ScalaTest There is also support for creating ScalaTest tests rather than applications. To do this use the same naming convention as above, but create ScalaTest suites rather than objects with main methods. You need to have ScalaTest on the -classpath. Here is a similar example to the one above but using ScalaTest:: +classpath. Here is a similar example to the one above but using ScalaTest: - package sample +``` +package sample - import org.scalatest.WordSpec - import org.scalatest.matchers.MustMatchers +import org.scalatest.WordSpec +import org.scalatest.matchers.MustMatchers - class SpecMultiJvmNode1 extends WordSpec with MustMatchers { - "A node" should { - "be able to say hello" in { - val message = "Hello from node 1" - message must be("Hello from node 1") - } - } +class SpecMultiJvmNode1 extends WordSpec with MustMatchers { + "A node" should { + "be able to say hello" in { + val message = "Hello from node 1" + message must be("Hello from node 1") } + } +} - class SpecMultiJvmNode2 extends WordSpec with MustMatchers { - "A node" should { - "be able to say hello" in { - val message = "Hello from node 2" - message must be("Hello from node 2") - } - } +class SpecMultiJvmNode2 extends WordSpec with MustMatchers { + "A node" should { + "be able to say hello" in { + val message = "Hello from node 2" + message must be("Hello from node 2") } + } +} +``` -To run just these tests you would call ``multi-jvm:test-only sample.Spec`` at +To run just these tests you would call `multi-jvm:test-only sample.Spec` at the sbt prompt. -Multi Node Additions -==================== +## Multi Node Additions -There has also been some additions made to the ``SbtMultiJvm`` plugin to accommodate the -:ref:`may change ` module :ref:`multi node testing `, -described in that section. - -.. _sample project: @samples@/tree/master/akka-sample-multi-node-scala \ No newline at end of file +There has also been some additions made to the `SbtMultiJvm` plugin to accommodate the +@ref:[may change](../common/may-change.md) module @ref:[multi node testing](multi-node-testing.md), +described in that section. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/dev/multi-node-testing.md b/akka-docs/src/main/paradox/dev/multi-node-testing.md index cfd2c3b758..65810f1a3f 100644 --- a/akka-docs/src/main/paradox/dev/multi-node-testing.md +++ b/akka-docs/src/main/paradox/dev/multi-node-testing.md @@ -1,226 +1,201 @@ -.. _multi-node-testing: +# Multi Node Testing -################### - Multi Node Testing -################### - -Multi Node Testing Concepts -=========================== +## Multi Node Testing Concepts When we talk about multi node testing in Akka we mean the process of running coordinated tests on multiple actor systems in different JVMs. The multi node testing kit consist of three main parts. -* `The Test Conductor`_. that coordinates and controls the nodes under test. -* `The Multi Node Spec`_. that is a convenience wrapper for starting the ``TestConductor`` and letting all - nodes connect to it. -* `The SbtMultiJvm Plugin`_. that starts tests in multiple JVMs possibly on multiple machines. + * [The Test Conductor](#the-test-conductor). that coordinates and controls the nodes under test. + * [The Multi Node Spec](#the-multi-node-spec). that is a convenience wrapper for starting the `TestConductor` and letting all +nodes connect to it. + * [The SbtMultiJvm Plugin](#the-sbtmultijvm-plugin). that starts tests in multiple JVMs possibly on multiple machines. -The Test Conductor -================== +## The Test Conductor -The basis for the multi node testing is the ``TestConductor``. It is an Akka Extension that plugs in to the +The basis for the multi node testing is the `TestConductor`. It is an Akka Extension that plugs in to the network stack and it is used to coordinate the nodes participating in the test and provides several features including: -* Node Address Lookup: Finding out the full path to another test node (No need to share configuration between - test nodes) -* Node Barrier Coordination: Waiting for other nodes at named barriers. -* Network Failure Injection: Throttling traffic, dropping packets, unplugging and plugging nodes back in. + * Node Address Lookup: Finding out the full path to another test node (No need to share configuration between +test nodes) + * Node Barrier Coordination: Waiting for other nodes at named barriers. + * Network Failure Injection: Throttling traffic, dropping packets, unplugging and plugging nodes back in. This is a schematic overview of the test conductor. -.. image:: ../images/akka-remote-testconductor.png +![akka-remote-testconductor.png](../images/akka-remote-testconductor.png) The test conductor server is responsible for coordinating barriers and sending commands to the test conductor clients that act upon them, e.g. throttling network traffic to/from another client. More information on the -possible operations is available in the ``akka.remote.testconductor.Conductor`` API documentation. +possible operations is available in the `akka.remote.testconductor.Conductor` API documentation. -The Multi Node Spec -=================== +## The Multi Node Spec -The Multi Node Spec consists of two parts. The ``MultiNodeConfig`` that is responsible for common -configuration and enumerating and naming the nodes under test. The ``MultiNodeSpec`` that contains a number +The Multi Node Spec consists of two parts. The `MultiNodeConfig` that is responsible for common +configuration and enumerating and naming the nodes under test. The `MultiNodeSpec` that contains a number of convenience functions for making the test nodes interact with each other. More information on the possible -operations is available in the ``akka.remote.testkit.MultiNodeSpec`` API documentation. +operations is available in the `akka.remote.testkit.MultiNodeSpec` API documentation. -The setup of the ``MultiNodeSpec`` is configured through java system properties that you set on all JVMs that's going to run a -node under test. These can easily be set on the JVM command line with ``-Dproperty=value``. +The setup of the `MultiNodeSpec` is configured through java system properties that you set on all JVMs that's going to run a +node under test. These can easily be set on the JVM command line with `-Dproperty=value`. These are the available properties: - * ``multinode.max-nodes`` +: + * + `multinode.max-nodes` + The maximum number of nodes that a test can have. + * + `multinode.host` + The host name or IP for this node. Must be resolvable using InetAddress.getByName. + * + `multinode.port` + The port number for this node. Defaults to 0 which will use a random port. + * + `multinode.server-host` + The host name or IP for the server node. Must be resolvable using InetAddress.getByName. + * + `multinode.server-port` + The port number for the server node. Defaults to 4711. + * + `multinode.index` + The index of this node in the sequence of roles defined for the test. The index 0 is special and that machine +will be the server. All failure injection and throttling must be done from this node. - The maximum number of nodes that a test can have. - * ``multinode.host`` +## The SbtMultiJvm Plugin - The host name or IP for this node. Must be resolvable using InetAddress.getByName. - - * ``multinode.port`` - - The port number for this node. Defaults to 0 which will use a random port. - - * ``multinode.server-host`` - - The host name or IP for the server node. Must be resolvable using InetAddress.getByName. - - * ``multinode.server-port`` - - The port number for the server node. Defaults to 4711. - - * ``multinode.index`` - - The index of this node in the sequence of roles defined for the test. The index 0 is special and that machine - will be the server. All failure injection and throttling must be done from this node. - -The SbtMultiJvm Plugin -====================== - -The :ref:`SbtMultiJvm Plugin ` has been updated to be able to run multi node tests, by -automatically generating the relevant ``multinode.*`` properties. This means that you can easily run multi node tests +The @ref:[SbtMultiJvm Plugin](multi-jvm-testing.md) has been updated to be able to run multi node tests, by +automatically generating the relevant `multinode.*` properties. This means that you can easily run multi node tests on a single machine without any special configuration by just running them as normal multi-jvm tests. These tests can then be run distributed over multiple machines without any changes simply by using the multi-node additions to the plugin. -Multi Node Specific Additions -+++++++++++++++++++++++++++++ +### Multi Node Specific Additions -The plugin also has a number of new ``multi-node-*`` sbt tasks and settings to support running tests on multiple +The plugin also has a number of new `multi-node-*` sbt tasks and settings to support running tests on multiple machines. The necessary test classes and dependencies are packaged for distribution to other machines with -`SbtAssembly `_ into a jar file with a name on the format -``_--multi-jvm-assembly.jar`` +[SbtAssembly](https://github.com/sbt/sbt-assembly) into a jar file with a name on the format +`_--multi-jvm-assembly.jar` -.. note:: +@@@ note - To be able to distribute and kick off the tests on multiple machines, it is assumed that both host and target - systems are POSIX like systems with ``ssh`` and ``rsync`` available. +To be able to distribute and kick off the tests on multiple machines, it is assumed that both host and target +systems are POSIX like systems with `ssh` and `rsync` available. + +@@@ These are the available sbt multi-node settings: - * ``multiNodeHosts`` - - A sequence of hosts to use for running the test, on the form ``user@host:java`` where host is the only required - part. Will override settings from file. - - * ``multiNodeHostsFileName`` - - A file to use for reading in the hosts to use for running the test. One per line on the same format as above. - Defaults to ``multi-node-test.hosts`` in the base project directory. - - * ``multiNodeTargetDirName`` - - A name for the directory on the target machine, where to copy the jar file. Defaults to ``multi-node-test`` in - the base directory of the ssh user used to rsync the jar file. - - * ``multiNodeJavaName`` - - The name of the default Java executable on the target machines. Defaults to ``java``. +: + * + `multiNodeHosts` + A sequence of hosts to use for running the test, on the form `user@host:java` where host is the only required +part. Will override settings from file. + * + `multiNodeHostsFileName` + A file to use for reading in the hosts to use for running the test. One per line on the same format as above. +Defaults to `multi-node-test.hosts` in the base project directory. + * + `multiNodeTargetDirName` + A name for the directory on the target machine, where to copy the jar file. Defaults to `multi-node-test` in +the base directory of the ssh user used to rsync the jar file. + * + `multiNodeJavaName` + The name of the default Java executable on the target machines. Defaults to `java`. Here are some examples of how you define hosts: - * ``localhost`` +: + * + `localhost` + The current user on localhost using the default java. + * + `user1@host1` + User `user1` on host `host1` with the default java. + * + `user2@host2:/usr/lib/jvm/java-7-openjdk-amd64/bin/java` + User `user2` on host `host2` using java 7. + * + `host3:/usr/lib/jvm/java-6-openjdk-amd64/bin/java` + The current user on host `host3` using java 6. - The current user on localhost using the default java. - * ``user1@host1`` - - User ``user1`` on host ``host1`` with the default java. - - * ``user2@host2:/usr/lib/jvm/java-7-openjdk-amd64/bin/java`` - - User ``user2`` on host ``host2`` using java 7. - - * ``host3:/usr/lib/jvm/java-6-openjdk-amd64/bin/java`` - - The current user on host ``host3`` using java 6. - -Running the Multi Node Tests -++++++++++++++++++++++++++++ +### Running the Multi Node Tests To run all the multi node test in multi-node mode (i.e. distributing the jar files and kicking off the tests -remotely) from inside sbt, use the ``multi-node-test`` task: +remotely) from inside sbt, use the `multi-node-test` task: -.. code-block:: none - - multi-node-test +```none +multi-node-test +``` To run all of them in multi-jvm mode (i.e. all JVMs on the local machine) do: -.. code-block:: none +```none +multi-jvm:test +``` - multi-jvm:test +To run individual tests use the `multi-node-test-only` task: -To run individual tests use the ``multi-node-test-only`` task: - -.. code-block:: none - - multi-node-test-only your.MultiNodeTest +```none +multi-node-test-only your.MultiNodeTest +``` To run individual tests in the multi-jvm mode do: -.. code-block:: none - - multi-jvm:test-only your.MultiNodeTest +```none +multi-jvm:test-only your.MultiNodeTest +``` More than one test name can be listed to run multiple specific tests. Tab completion in sbt makes it easy to complete the test names. -Preparing Your Project for Multi Node Testing -============================================= +## Preparing Your Project for Multi Node Testing The multi node testing kit is a separate jar file. Make sure that you have the following dependency in your project: -.. parsed-literal:: - - "com.typesafe.akka" %% "akka-multi-node-testkit" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-multi-node-testkit" % "@version@" @crossString@``` If you are using the latest nightly build you should pick a timestamped Akka version from -``_. -We recommend against using ``SNAPSHOT`` in order to obtain stable builds. +[http://repo.akka.io/snapshots/com/typesafe/akka/akka-multi-node-testkit_@binVersion@/](http://repo.akka.io/snapshots/com/typesafe/akka/akka-multi-node-testkit_@binVersion@/). +We recommend against using `SNAPSHOT` in order to obtain stable builds. -A Multi Node Testing Example -============================ +## A Multi Node Testing Example -First we need some scaffolding to hook up the ``MultiNodeSpec`` with your favorite test framework. Lets define a trait -``STMultiNodeSpec`` that uses ScalaTest to start and stop ``MultiNodeSpec``. +First we need some scaffolding to hook up the `MultiNodeSpec` with your favorite test framework. Lets define a trait +`STMultiNodeSpec` that uses ScalaTest to start and stop `MultiNodeSpec`. -.. includecode:: ../../../akka-remote-tests/src/test/scala/akka/remote/testkit/STMultiNodeSpec.scala#example +@@snip [STMultiNodeSpec.scala]../../../../../akka-remote-tests/src/test/scala/akka/remote/testkit/STMultiNodeSpec.scala) { #example } -Then we need to define a configuration. Lets use two nodes ``"node1`` and ``"node2"`` and call it -``MultiNodeSampleConfig``. +Then we need to define a configuration. Lets use two nodes `"node1` and `"node2"` and call it +`MultiNodeSampleConfig`. -.. includecode:: ../../../akka-remote-tests/src/multi-jvm/scala/akka/remote/sample/MultiNodeSample.scala - :include: package,config +@@snip [MultiNodeSample.scala]../../../../../akka-remote-tests/src/multi-jvm/scala/akka/remote/sample/MultiNodeSample.scala) { #package #config } And then finally to the node test code. That starts the two nodes, and demonstrates a barrier, and a remote actor message send/receive. -.. includecode:: ../../../akka-remote-tests/src/multi-jvm/scala/akka/remote/sample/MultiNodeSample.scala - :include: package,spec +@@snip [MultiNodeSample.scala]../../../../../akka-remote-tests/src/multi-jvm/scala/akka/remote/sample/MultiNodeSample.scala) { #package #spec } The easiest way to run this example yourself is to download the ready to run -`Akka Multi-Node Testing Sample with Scala <@exampleCodeService@/akka-samples-multi-node-scala>`_ -together with the tutorial. The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-multi-node-scala>`_. +[Akka Multi-Node Testing Sample with Scala](@exampleCodeService@/akka-samples-multi-node-scala) +together with the tutorial. The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-multi-node-scala). -Things to Keep in Mind -====================== +## Things to Keep in Mind There are a couple of things to keep in mind when writing multi node tests or else your tests might behave in surprising ways. - * Don't issue a shutdown of the first node. The first node is the controller and if it shuts down your test will break. +> + * Don't issue a shutdown of the first node. The first node is the controller and if it shuts down your test will break. + * To be able to use `blackhole`, `passThrough`, and `throttle` you must activate the failure injector and +throttler transport adapters by specifying `testTransport(on = true)` in your MultiNodeConfig. + * Throttling, shutdown and other failure injections can only be done from the first node, which again is the controller. + * Don't ask for the address of a node using `node(address)` after the node has been shut down. Grab the address before +shutting down the node. + * Don't use MultiNodeSpec methods like address lookup, barrier entry et.c. from other threads than the main test +thread. This also means that you shouldn't use them from inside an actor, a future, or a scheduled task. - * To be able to use ``blackhole``, ``passThrough``, and ``throttle`` you must activate the failure injector and - throttler transport adapters by specifying ``testTransport(on = true)`` in your MultiNodeConfig. - - * Throttling, shutdown and other failure injections can only be done from the first node, which again is the controller. - - * Don't ask for the address of a node using ``node(address)`` after the node has been shut down. Grab the address before - shutting down the node. - - * Don't use MultiNodeSpec methods like address lookup, barrier entry et.c. from other threads than the main test - thread. This also means that you shouldn't use them from inside an actor, a future, or a scheduled task. - -Configuration -============= +## Configuration There are several configuration properties for the Multi-Node Testing module, please refer -to the :ref:`reference configuration `. - +to the @ref:[reference configuration](../general/configuration.md#config-akka-multi-node-testkit). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/experimental/index-java.md b/akka-docs/src/main/paradox/experimental/index-java.md index c3f1433c59..90baded126 100644 --- a/akka-docs/src/main/paradox/experimental/index-java.md +++ b/akka-docs/src/main/paradox/experimental/index-java.md @@ -1,8 +1,4 @@ -.. _experimental-java: - -#################### -Experimental Modules -#################### +# Experimental Modules The label experimental caused confusion and discomfort and has therefore been replaced with "May Change" -please see :ref:`may-change`. +please see @ref:[Modules marked "May Change"](../common/may-change.md). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/experimental/index.md b/akka-docs/src/main/paradox/experimental/index.md index 04cd5f47c6..90baded126 100644 --- a/akka-docs/src/main/paradox/experimental/index.md +++ b/akka-docs/src/main/paradox/experimental/index.md @@ -1,8 +1,4 @@ -.. _experimental: - -#################### -Experimental Modules -#################### +# Experimental Modules The label experimental caused confusion and discomfort and has therefore been replaced with "May Change" -please see :ref:`may-change`. +please see @ref:[Modules marked "May Change"](../common/may-change.md). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/actor-systems.md b/akka-docs/src/main/paradox/general/actor-systems.md index cf5b4577fc..9f5d464bec 100644 --- a/akka-docs/src/main/paradox/general/actor-systems.md +++ b/akka-docs/src/main/paradox/general/actor-systems.md @@ -1,7 +1,4 @@ -.. _actor-systems: - -Actor Systems -============= +# Actor Systems Actors are objects which encapsulate state and behavior, they communicate exclusively by exchanging messages which are placed into the recipient’s @@ -14,19 +11,20 @@ which means that we need not concern ourselves with their emotional state or moral issues). The result can then serve as a mental scaffolding for building the software implementation. -.. note:: +@@@ note - An ActorSystem is a heavyweight structure that will allocate 1…N Threads, - so create one per logical application. +An ActorSystem is a heavyweight structure that will allocate 1…N Threads, +so create one per logical application. -Hierarchical Structure ----------------------- +@@@ + +## Hierarchical Structure Like in an economic organization, actors naturally form hierarchies. One actor, which is to oversee a certain function in the program might want to split up its task into smaller, more manageable pieces. For this purpose it starts child actors which it supervises. While the details of supervision are explained -:ref:`here `, we shall concentrate on the underlying concepts in +@ref:[here](supervision.md), we shall concentrate on the underlying concepts in this section. The only prerequisite is to know that each actor has exactly one supervisor, which is the actor that created it. @@ -48,31 +46,29 @@ Now, the difficulty in designing such a system is how to decide who should supervise what. There is of course no single best solution, but there are a few guidelines which might be helpful: - - If one actor manages the work another actor is doing, e.g. by passing on - sub-tasks, then the manager should supervise the child. The reason is that - the manager knows which kind of failures are expected and how to handle - them. - - - If one actor carries very important data (i.e. its state shall not be lost - if avoidable), this actor should source out any possibly dangerous sub-tasks - to children it supervises and handle failures of these children as - appropriate. Depending on the nature of the requests, it may be best to - create a new child for each request, which simplifies state management for - collecting the replies. This is known as the “Error Kernel Pattern” from - Erlang. - - - If one actor depends on another actor for carrying out its duty, it should - watch that other actor’s liveness and act upon receiving a termination - notice. This is different from supervision, as the watching party has no - influence on the supervisor strategy, and it should be noted that a - functional dependency alone is not a criterion for deciding where to place a - certain child actor in the hierarchy. +> + * If one actor manages the work another actor is doing, e.g. by passing on +sub-tasks, then the manager should supervise the child. The reason is that +the manager knows which kind of failures are expected and how to handle +them. + * If one actor carries very important data (i.e. its state shall not be lost +if avoidable), this actor should source out any possibly dangerous sub-tasks +to children it supervises and handle failures of these children as +appropriate. Depending on the nature of the requests, it may be best to +create a new child for each request, which simplifies state management for +collecting the replies. This is known as the “Error Kernel Pattern” from +Erlang. + * If one actor depends on another actor for carrying out its duty, it should +watch that other actor’s liveness and act upon receiving a termination +notice. This is different from supervision, as the watching party has no +influence on the supervisor strategy, and it should be noted that a +functional dependency alone is not a criterion for deciding where to place a +certain child actor in the hierarchy. There are of course always exceptions to these rules, but no matter whether you follow the rules or break them, you should always have a reason. -Configuration Container ------------------------ +## Configuration Container The actor system as a collaborating ensemble of actors is the natural unit for managing shared facilities like scheduling services, configuration, logging, @@ -82,43 +78,38 @@ Couple this with the transparent communication between actor systems—within on node or across a network connection—to see that actor systems themselves can be used as building blocks in a functional hierarchy. -Actor Best Practices --------------------- +## Actor Best Practices -#. Actors should be like nice co-workers: do their job efficiently without - bothering everyone else needlessly and avoid hogging resources. Translated - to programming this means to process events and generate responses (or more - requests) in an event-driven manner. Actors should not block (i.e. passively - wait while occupying a Thread) on some external entity—which might be a - lock, a network socket, etc.—unless it is unavoidable; in the latter case - see below. + 1. Actors should be like nice co-workers: do their job efficiently without +bothering everyone else needlessly and avoid hogging resources. Translated +to programming this means to process events and generate responses (or more +requests) in an event-driven manner. Actors should not block (i.e. passively +wait while occupying a Thread) on some external entity—which might be a +lock, a network socket, etc.—unless it is unavoidable; in the latter case +see below. + 2. Do not pass mutable objects between actors. In order to ensure that, prefer +immutable messages. If the encapsulation of actors is broken by exposing +their mutable state to the outside, you are back in normal Java concurrency +land with all the drawbacks. + 3. Actors are made to be containers for behavior and state, embracing this +means to not routinely send behavior within messages (which may be tempting +using Scala closures). One of the risks is to accidentally share mutable +state between actors, and this violation of the actor model unfortunately +breaks all the properties which make programming in actors such a nice +experience. + 4. Top-level actors are the innermost part of your Error Kernel, so create them +sparingly and prefer truly hierarchical systems. This has benefits with +respect to fault-handling (both considering the granularity of configuration +and the performance) and it also reduces the strain on the guardian actor, +which is a single point of contention if over-used. -#. Do not pass mutable objects between actors. In order to ensure that, prefer - immutable messages. If the encapsulation of actors is broken by exposing - their mutable state to the outside, you are back in normal Java concurrency - land with all the drawbacks. - -#. Actors are made to be containers for behavior and state, embracing this - means to not routinely send behavior within messages (which may be tempting - using Scala closures). One of the risks is to accidentally share mutable - state between actors, and this violation of the actor model unfortunately - breaks all the properties which make programming in actors such a nice - experience. - -#. Top-level actors are the innermost part of your Error Kernel, so create them - sparingly and prefer truly hierarchical systems. This has benefits with - respect to fault-handling (both considering the granularity of configuration - and the performance) and it also reduces the strain on the guardian actor, - which is a single point of contention if over-used. - -Blocking Needs Careful Management ---------------------------------- +## Blocking Needs Careful Management In some cases it is unavoidable to do blocking operations, i.e. to put a thread to sleep for an indeterminate time, waiting for an external event to occur. Examples are legacy RDBMS drivers or messaging APIs, and the underlying reason is typically that (network) I/O occurs under the covers. When facing this, you -may be tempted to just wrap the blocking call inside a :class:`Future` and work +may be tempted to just wrap the blocking call inside a `Future` and work with that instead, but this strategy is too simple: you are quite likely to find bottlenecks or run out of memory or threads when the application runs under increased load. @@ -126,22 +117,20 @@ under increased load. The non-exhaustive list of adequate solutions to the “blocking problem” includes the following suggestions: - - Do the blocking call within an actor (or a set of actors managed by a router - [:ref:`Java `, :ref:`Scala `]), making sure to - configure a thread pool which is either dedicated for this purpose or - sufficiently sized. - - - Do the blocking call within a :class:`Future`, ensuring an upper bound on - the number of such calls at any point in time (submitting an unbounded - number of tasks of this nature will exhaust your memory or thread limits). - - - Do the blocking call within a :class:`Future`, providing a thread pool with - an upper limit on the number of threads which is appropriate for the - hardware on which the application runs. - - - Dedicate a single thread to manage a set of blocking resources (e.g. a NIO - selector driving multiple channels) and dispatch events as they occur as - actor messages. +> + * Do the blocking call within an actor (or a set of actors managed by a router +[@ref:[Java](../java/routing.md), @ref:[Scala](../scala/routing.md)]), making sure to +configure a thread pool which is either dedicated for this purpose or +sufficiently sized. + * Do the blocking call within a `Future`, ensuring an upper bound on +the number of such calls at any point in time (submitting an unbounded +number of tasks of this nature will exhaust your memory or thread limits). + * Do the blocking call within a `Future`, providing a thread pool with +an upper limit on the number of threads which is appropriate for the +hardware on which the application runs. + * Dedicate a single thread to manage a set of blocking resources (e.g. a NIO +selector driving multiple channels) and dispatch events as they occur as +actor messages. The first possibility is especially well-suited for resources which are single-threaded in nature, like database handles which traditionally can only @@ -151,15 +140,16 @@ wraps a single DB connection and handles queries as sent to the router. The number N must then be tuned for maximum throughput, which will vary depending on which DBMS is deployed on what hardware. -.. note:: +@@@ note - Configuring thread pools is a task best delegated to Akka, simply configure - in the ``application.conf`` and instantiate through an :class:`ActorSystem` - [:ref:`Java `, :ref:`Scala - `] +Configuring thread pools is a task best delegated to Akka, simply configure +in the `application.conf` and instantiate through an `ActorSystem` +[@ref:[Java](../java/dispatchers.md#dispatcher-lookup-java), @ref:[Scala +](../scala/dispatchers.md#dispatcher-lookup-scala)] -What you should not concern yourself with ------------------------------------------ +@@@ + +## What you should not concern yourself with An actor system manages the resources it is configured to use in order to run the actors which it contains. There may be millions of actors within one such @@ -169,13 +159,12 @@ in which messages are processed in large systems is not controllable by the application author, but this is also not intended. Take a step back and relax while Akka does the heavy lifting under the hood. -Terminating ActorSystem ------------------------ +## Terminating ActorSystem When you know everything is done for your application, you can call the -``terminate`` method of ``ActorSystem``. That will stop the guardian +`terminate` method of `ActorSystem`. That will stop the guardian actor, which in turn will recursively stop all its child actors, the system guardian. -If you want to execute some operations while terminating ``ActorSystem``, -look at ``CoordinatedShutdown`` [:ref:`Java `, :ref:`Scala `] \ No newline at end of file +If you want to execute some operations while terminating `ActorSystem`, +look at `CoordinatedShutdown` [@ref:[Java](../java/actors.md#coordinated-shutdown-java), @ref:[Scala](../scala/actors.md#coordinated-shutdown-scala)] \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/actors.md b/akka-docs/src/main/paradox/general/actors.md index e31feeb17d..4b1fc0ed16 100644 --- a/akka-docs/src/main/paradox/general/actors.md +++ b/akka-docs/src/main/paradox/general/actors.md @@ -1,25 +1,21 @@ -.. _actors-general: +# What is an Actor? -What is an Actor? -================= - -The previous section about :ref:`actor-systems` explained how actors form +The previous section about @ref:[Actor Systems](actor-systems.md) explained how actors form hierarchies and are the smallest unit when building an application. This section looks at one such actor in isolation, explaining the concepts you encounter while implementing it. For a more in depth reference with all the details please refer to -:ref:`Actors (Scala) ` and :ref:`Actors (Java) `. +@ref:[Actors (Scala)](../scala/actors.md) and @ref:[Actors (Java)](../java/actors.md). -An actor is a container for `State`_, `Behavior`_, a `Mailbox`_, `Child Actors`_ -and a `Supervisor Strategy`_. All of this is encapsulated behind an `Actor -Reference`_. One noteworthy aspect is that actors have an explicit lifecycle, +An actor is a container for [State](#state), [Behavior](#behavior), a [Mailbox](#mailbox), [Child Actors](#child-actors) +and a [Supervisor Strategy](#supervisor-strategy). All of this is encapsulated behind an [Actor +Reference](#actor-reference). One noteworthy aspect is that actors have an explicit lifecycle, they are not automatically destroyed when no longer referenced; after having created one, it is your responsibility to make sure that it will eventually be terminated as well—which also gives you control over how resources are released -`When an Actor Terminates`_. +[When an Actor Terminates](#when-an-actor-terminates). -Actor Reference ---------------- +## Actor Reference As detailed below, an actor object needs to be shielded from the outside in order to benefit from the actor model. Therefore, actors are represented to the @@ -32,12 +28,11 @@ But the most important aspect is that it is not possible to look inside an actor and get hold of its state from the outside, unless the actor unwisely publishes this information itself. -State ------ +## State Actor objects will typically contain some variables which reflect possible states the actor may be in. This can be an explicit state machine (e.g. using -the :ref:`fsm-scala` module), or it could be a counter, set of listeners, +the @ref:[FSM](../scala/fsm.md) module), or it could be a counter, set of listeners, pending requests, etc. These data are what make an actor valuable, and they must be protected from corruption by other actors. The good news is that Akka actors conceptually each have their own light-weight thread, which is @@ -58,10 +53,9 @@ the actor. This is to enable the ability of self-healing of the system. Optionally, an actor's state can be automatically recovered to the state before a restart by persisting received messages and replaying them after -restart (see :ref:`persistence-scala`). +restart (see @ref:[Persistence](../scala/persistence.md)). -Behavior --------- +## Behavior Every time a message is processed, it is matched against the current behavior of the actor. Behavior means a function which defines the actions to be taken @@ -71,12 +65,11 @@ e.g. because different clients obtain authorization over time, or because the actor may go into an “out-of-service” mode and later come back. These changes are achieved by either encoding them in state variables which are read from the behavior logic, or the function itself may be swapped out at runtime, see the -``become`` and ``unbecome`` operations. However, the initial behavior defined +`become` and `unbecome` operations. However, the initial behavior defined during construction of the actor object is special in the sense that a restart of the actor will reset its behavior to this initial one. -Mailbox -------- +## Mailbox An actor’s purpose is the processing of messages, and these messages were sent to the actor from other actors (or from outside the actor system). The piece @@ -103,23 +96,21 @@ dequeued message, there is no scanning the mailbox for the next matching one. Failure to handle a message will typically be treated as a failure, unless this behavior is overridden. -Child Actors ------------- +## Child Actors Each actor is potentially a supervisor: if it creates children for delegating sub-tasks, it will automatically supervise them. The list of children is maintained within the actor’s context and the actor has access to it. -Modifications to the list are done by creating (``context.actorOf(...)``) or -stopping (``context.stop(child)``) children and these actions are reflected +Modifications to the list are done by creating (`context.actorOf(...)`) or +stopping (`context.stop(child)`) children and these actions are reflected immediately. The actual creation and termination actions happen behind the scenes in an asynchronous way, so they do not “block” their supervisor. -Supervisor Strategy -------------------- +## Supervisor Strategy The final piece of an actor is its strategy for handling faults of its children. Fault handling is then done transparently by Akka, applying one -of the strategies described in :ref:`supervision` for each incoming failure. +of the strategies described in @ref:[Supervision and Monitoring](supervision.md) for each incoming failure. As this strategy is fundamental to how an actor system is structured, it cannot be changed once an actor has been created. @@ -129,8 +120,7 @@ children should be grouped beneath intermediate supervisors with matching strategies, preferring once more the structuring of actor systems according to the splitting of tasks into sub-tasks. -When an Actor Terminates ------------------------- +## When an Actor Terminates Once an actor terminates, i.e. fails in a way which is not handled by a restart, stops itself or is stopped by its supervisor, it will free up its @@ -145,6 +135,4 @@ The reason for not just silently dumping the messages was inspired by our tests: we register the TestEventListener on the event bus to which the dead letters are forwarded, and that will log a warning for every dead letter received—this has been very helpful for deciphering test failures more quickly. -It is conceivable that this feature may also be of use for other purposes. - - +It is conceivable that this feature may also be of use for other purposes. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/addressing.md b/akka-docs/src/main/paradox/general/addressing.md index 9edbe6297d..864726ee68 100644 --- a/akka-docs/src/main/paradox/general/addressing.md +++ b/akka-docs/src/main/paradox/general/addressing.md @@ -1,75 +1,67 @@ -.. _addressing: - -Actor References, Paths and Addresses -===================================== +# Actor References, Paths and Addresses This chapter describes how actors are identified and located within a possibly distributed actor system. It ties into the central idea that -:ref:`actor-systems` form intrinsic supervision hierarchies as well as that +@ref:[Actor Systems](actor-systems.md) form intrinsic supervision hierarchies as well as that communication between actors is transparent with respect to their placement across multiple network nodes. -.. image:: ActorPath.png +![ActorPath.png](ActorPath.png) The above image displays the relationship between the most important entities within an actor system, please read on for the details. -What is an Actor Reference? ---------------------------- +## What is an Actor Reference? -An actor reference is a subtype of :class:`ActorRef`, whose foremost purpose is +An actor reference is a subtype of `ActorRef`, whose foremost purpose is to support sending messages to the actor it represents. Each actor has access -to its canonical (local) reference through the :meth:`self` field; this +to its canonical (local) reference through the `self` field; this reference is also included as sender reference by default for all messages sent to other actors. Conversely, during message processing the actor has access to a reference representing the sender of the current message through the -:meth:`sender()` method. +`sender()` method. There are several different types of actor references that are supported depending on the configuration of the actor system: -- Purely local actor references are used by actor systems which are not - configured to support networking functions. These actor references will not - function if sent across a network connection to a remote JVM. -- Local actor references when remoting is enabled are used by actor systems - which support networking functions for those references which represent - actors within the same JVM. In order to also be reachable when sent to - other network nodes, these references include protocol and remote addressing - information. -- There is a subtype of local actor references which is used for routers (i.e. - actors mixing in the :class:`Router` trait). Its logical structure is the - same as for the aforementioned local references, but sending a message to - them dispatches to one of their children directly instead. -- Remote actor references represent actors which are reachable using remote - communication, i.e. sending messages to them will serialize the messages - transparently and send them to the remote JVM. -- There are several special types of actor references which behave like local - actor references for all practical purposes: + * Purely local actor references are used by actor systems which are not +configured to support networking functions. These actor references will not +function if sent across a network connection to a remote JVM. + * Local actor references when remoting is enabled are used by actor systems +which support networking functions for those references which represent +actors within the same JVM. In order to also be reachable when sent to +other network nodes, these references include protocol and remote addressing +information. + * There is a subtype of local actor references which is used for routers (i.e. +actors mixing in the `Router` trait). Its logical structure is the +same as for the aforementioned local references, but sending a message to +them dispatches to one of their children directly instead. + * Remote actor references represent actors which are reachable using remote +communication, i.e. sending messages to them will serialize the messages +transparently and send them to the remote JVM. + * There are several special types of actor references which behave like local +actor references for all practical purposes: + * `PromiseActorRef` is the special representation of a `Promise` +for the purpose of being completed by the response from an actor. +`akka.pattern.ask` creates this actor reference. + * `DeadLetterActorRef` is the default implementation of the dead +letters service to which Akka routes all messages whose destinations +are shut down or non-existent. + * `EmptyLocalActorRef` is what Akka returns when looking up a +non-existent local actor path: it is equivalent to a +`DeadLetterActorRef`, but it retains its path so that Akka can send +it over the network and compare it to other existing actor references for +that path, some of which might have been obtained before the actor died. + * And then there are some one-off internal implementations which you should +never really see: + * There is an actor reference which does not represent an actor but acts only +as a pseudo-supervisor for the root guardian, we call it “the one who walks +the bubbles of space-time”. + * The first logging service started before actually firing up actor creation +facilities is a fake actor reference which accepts log events and prints +them directly to standard output; it is `Logging.StandardOutLogger`. - - :class:`PromiseActorRef` is the special representation of a :meth:`Promise` - for the purpose of being completed by the response from an actor. - :meth:`akka.pattern.ask` creates this actor reference. - - :class:`DeadLetterActorRef` is the default implementation of the dead - letters service to which Akka routes all messages whose destinations - are shut down or non-existent. - - :class:`EmptyLocalActorRef` is what Akka returns when looking up a - non-existent local actor path: it is equivalent to a - :class:`DeadLetterActorRef`, but it retains its path so that Akka can send - it over the network and compare it to other existing actor references for - that path, some of which might have been obtained before the actor died. - -- And then there are some one-off internal implementations which you should - never really see: - - - There is an actor reference which does not represent an actor but acts only - as a pseudo-supervisor for the root guardian, we call it “the one who walks - the bubbles of space-time”. - - The first logging service started before actually firing up actor creation - facilities is a fake actor reference which accepts log events and prints - them directly to standard output; it is :class:`Logging.StandardOutLogger`. - -What is an Actor Path? ----------------------- +## What is an Actor Path? Since actors are created in a strictly hierarchical fashion, there exists a unique sequence of actor names given by recursively following the supervision @@ -82,8 +74,7 @@ followed by the concatenation of the path elements, from root guardian to the designated actor; the path elements are the names of the traversed actors and are separated by slashes. -What is the Difference Between Actor Reference and Path? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### What is the Difference Between Actor Reference and Path? An actor reference designates a single actor and the life-cycle of the reference matches that actor’s life-cycle; an actor path represents a name which may or @@ -97,23 +88,23 @@ the same actor. An actor reference to the old incarnation is not valid for the n incarnation. Messages sent to the old actor reference will not be delivered to the new incarnation even though they have the same path. -Actor Path Anchors -^^^^^^^^^^^^^^^^^^ +### Actor Path Anchors Each actor path has an address component, describing the protocol and location by which the corresponding actor is reachable, followed by the names of the -actors in the hierarchy from the root up. Examples are:: +actors in the hierarchy from the root up. Examples are: - "akka://my-sys/user/service-a/worker1" // purely local - "akka.tcp://my-sys@host.example.com:5678/user/service-b" // remote +``` +"akka://my-sys/user/service-a/worker1" // purely local +"akka.tcp://my-sys@host.example.com:5678/user/service-b" // remote +``` -Here, ``akka.tcp`` is the default remote transport for the 2.4 release; other transports +Here, `akka.tcp` is the default remote transport for the 2.4 release; other transports are pluggable. -The interpretation of the host and port part (i.e. ``host.example.com:5678`` in the example) +The interpretation of the host and port part (i.e. `host.example.com:5678` in the example) depends on the transport mechanism used, but it must abide by the URI structural rules. -Logical Actor Paths -^^^^^^^^^^^^^^^^^^^ +### Logical Actor Paths The unique path obtained by following the parental supervision links towards the root guardian is called the logical actor path. This path matches exactly @@ -121,8 +112,7 @@ the creation ancestry of an actor, so it is completely deterministic as soon as the actor system’s remoting configuration (and with it the address component of the path) is set. -Physical Actor Paths -^^^^^^^^^^^^^^^^^^^^ +### Physical Actor Paths While the logical actor path describes the functional location within one actor system, configuration-based remote deployment means that an actor may be @@ -139,9 +129,8 @@ systems or JVMs. This means that the logical path (supervision hierarchy) and the physical path (actor deployment) of an actor may diverge if one of its ancestors is remotely supervised. +### Actor path alias or symbolic link? -Actor path alias or symbolic link? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As in some real file-systems you might think of a “path alias” or “symbolic link” for an actor, i.e. one actor may be reachable using more than one path. However, you should note that actor hierarchy is different from file system hierarchy. @@ -150,75 +139,73 @@ As described in the above logical and physical actor path sections, an actor path must be either logical path which represents supervision hierarchy, or physical path which represents actor deployment. - -How are Actor References obtained? ----------------------------------- +## How are Actor References obtained? There are two general categories to how actor references may be obtained: by creating actors or by looking them up, where the latter functionality comes in the two flavours of creating actor references from concrete actor paths and querying the logical actor hierarchy. -Creating Actors -^^^^^^^^^^^^^^^ +### Creating Actors An actor system is typically started by creating actors beneath the guardian -actor using the :meth:`ActorSystem.actorOf` method and then using -:meth:`ActorContext.actorOf` from within the created actors to spawn the actor +actor using the `ActorSystem.actorOf` method and then using +`ActorContext.actorOf` from within the created actors to spawn the actor tree. These methods return a reference to the newly created actor. Each actor -has direct access (through its ``ActorContext``) to references for its parent, +has direct access (through its `ActorContext`) to references for its parent, itself and its children. These references may be sent within messages to other actors, enabling those to reply directly. -Looking up Actors by Concrete Path -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Looking up Actors by Concrete Path In addition, actor references may be looked up using the -:meth:`ActorSystem.actorSelection` method. The selection can be used for +`ActorSystem.actorSelection` method. The selection can be used for communicating with said actor and the actor corresponding to the selection is looked up when delivering each message. -To acquire an :class:`ActorRef` that is bound to the life-cycle of a specific actor -you need to send a message, such as the built-in :class:`Identify` message, to the actor -and use the ``sender()`` reference of a reply from the actor. +To acquire an `ActorRef` that is bound to the life-cycle of a specific actor +you need to send a message, such as the built-in `Identify` message, to the actor +and use the `sender()` reference of a reply from the actor. -Absolute vs. Relative Paths -``````````````````````````` +#### Absolute vs. Relative Paths -In addition to :meth:`ActorSystem.actorSelection` there is also -:meth:`ActorContext.actorSelection`, which is available inside any actor as -``context.actorSelection``. This yields an actor selection much like its twin on -:class:`ActorSystem`, but instead of looking up the path starting from the root +In addition to `ActorSystem.actorSelection` there is also +`ActorContext.actorSelection`, which is available inside any actor as +`context.actorSelection`. This yields an actor selection much like its twin on +`ActorSystem`, but instead of looking up the path starting from the root of the actor tree it starts out on the current actor. Path elements consisting -of two dots (``".."``) may be used to access the parent actor. You can for -example send a message to a specific sibling:: +of two dots (`".."`) may be used to access the parent actor. You can for +example send a message to a specific sibling: - context.actorSelection("../brother") ! msg +``` +context.actorSelection("../brother") ! msg +``` -Absolute paths may of course also be looked up on `context` in the usual way, i.e. +Absolute paths may of course also be looked up on *context* in the usual way, i.e. -.. code-block:: scala - - context.actorSelection("/user/serviceA") ! msg +```scala +context.actorSelection("/user/serviceA") ! msg +``` will work as expected. -Querying the Logical Actor Hierarchy -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Querying the Logical Actor Hierarchy Since the actor system forms a file-system like hierarchy, matching on paths is possible in the same way as supported by Unix shells: you may replace (parts -of) path element names with wildcards (`«*»` and `«?»`) to formulate a +of) path element names with wildcards (*«*»* and *«?»*) to formulate a selection which may match zero or more actual actors. Because the result is not -a single actor reference, it has a different type :class:`ActorSelection` and -does not support the full set of operations an :class:`ActorRef` does. -Selections may be formulated using the :meth:`ActorSystem.actorSelection` and -:meth:`ActorContext.actorSelection` methods and do support sending messages:: +a single actor reference, it has a different type `ActorSelection` and +does not support the full set of operations an `ActorRef` does. +Selections may be formulated using the `ActorSystem.actorSelection` and +`ActorContext.actorSelection` methods and do support sending messages: - context.actorSelection("../*") ! msg +``` +context.actorSelection("../*") ! msg +``` -will send `msg` to all siblings including the current actor. As for references -obtained using `actorSelection`, a traversal of the supervision hierarchy is done in +will send *msg* to all siblings including the current actor. As for references +obtained using *actorSelection*, a traversal of the supervision hierarchy is done in order to perform the message send. As the exact set of actors which match a selection may change even while a message is making its way to the recipients, it is not possible to watch a selection for liveliness changes. In order to do @@ -227,41 +214,38 @@ extracting the sender references, and then watch all discovered concrete actors. This scheme of resolving a selection may be improved upon in a future release. -.. _actorOf-vs-actorSelection: + +### Summary: `actorOf` vs. `actorSelection` -Summary: ``actorOf`` vs. ``actorSelection`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@@ note -.. note:: +What the above sections described in some detail can be summarized and +memorized easily as follows: - What the above sections described in some detail can be summarized and - memorized easily as follows: + * `actorOf` only ever creates a new actor, and it creates it as a direct +child of the context on which this method is invoked (which may be any +actor or actor system). + * `actorSelection` only ever looks up existing actors when messages are +delivered, i.e. does not create actors, or verify existence of actors +when the selection is created. - - ``actorOf`` only ever creates a new actor, and it creates it as a direct - child of the context on which this method is invoked (which may be any - actor or actor system). +@@@ - - ``actorSelection`` only ever looks up existing actors when messages are - delivered, i.e. does not create actors, or verify existence of actors - when the selection is created. +## Actor Reference and Path Equality -Actor Reference and Path Equality ---------------------------------- - -Equality of ``ActorRef`` match the intention that an ``ActorRef`` corresponds to +Equality of `ActorRef` match the intention that an `ActorRef` corresponds to the target actor incarnation. Two actor references are compared equal when they have the same path and point to the same actor incarnation. A reference pointing to a terminated actor does not compare equal to a reference pointing to another (re-created) actor with the same path. Note that a restart of an actor caused by a failure still means that it is the same actor incarnation, i.e. a restart is not visible for the -consumer of the ``ActorRef``. +consumer of the `ActorRef`. If you need to keep track of actor references in a collection and do not care about -the exact actor incarnation you can use the ``ActorPath`` as key, because the identifier +the exact actor incarnation you can use the `ActorPath` as key, because the identifier of the target actor is not taken into account when comparing actor paths. -Reusing Actor Paths -------------------- +## Reusing Actor Paths When an actor is terminated, its reference will point to the dead letter mailbox, DeathWatch will publish its final transition and in general it is not expected @@ -269,7 +253,7 @@ to come back to life again (since the actor life cycle does not allow this). While it is possible to create an actor at a later time with an identical path—simply due to it being impossible to enforce the opposite without keeping the set of all actors ever created available—this is not good practice: -messages sent with ``actorSelection`` to an actor which “died” suddenly start to work +messages sent with `actorSelection` to an actor which “died” suddenly start to work again, but without any guarantee of ordering between this transition and any other event, hence the new inhabitant of the path may receive messages which were destined for the previous tenant. @@ -284,8 +268,7 @@ instantiated at a specific path. In that case it is best to mock its supervisor so that it will forward the Terminated message to the appropriate point in the test procedure, enabling the latter to await proper deregistration of the name. -The Interplay with Remote Deployment ------------------------------------- +## The Interplay with Remote Deployment When an actor creates a child, the actor system’s deployer will decide whether the new actor resides in the same JVM or on another node. In the second case, @@ -294,16 +277,15 @@ different JVM and consequently within a different actor system. The remote system will place the new actor below a special path reserved for this purpose and the supervisor of the new actor will be a remote actor reference (representing that actor which triggered its creation). In this case, -:meth:`context.parent` (the supervisor reference) and -:meth:`context.path.parent` (the parent node in the actor’s path) do not +`context.parent` (the supervisor reference) and +`context.path.parent` (the parent node in the actor’s path) do not represent the same actor. However, looking up the child’s name within the supervisor will find it on the remote node, preserving logical structure e.g. when sending to an unresolved actor reference. -.. image:: RemoteDeployment.png +![RemoteDeployment.png](RemoteDeployment.png) -What is the Address part used for? ----------------------------------- +## What is the Address part used for? When sending an actor reference across the network, it is represented by its path. Hence, the path must fully encode all information necessary to send @@ -314,27 +296,25 @@ the address of this actor system, in which case it will be resolved to the actor’s local reference. Otherwise, it will be represented by a remote actor reference. -.. _toplevel-paths: - -Top-Level Scopes for Actor Paths --------------------------------- + +## Top-Level Scopes for Actor Paths At the root of the path hierarchy resides the root guardian above which all -other actors are found; its name is ``"/"``. The next level consists of the +other actors are found; its name is `"/"`. The next level consists of the following: -- ``"/user"`` is the guardian actor for all user-created top-level actors; - actors created using :meth:`ActorSystem.actorOf` are found below this one. -- ``"/system"`` is the guardian actor for all system-created top-level actors, - e.g. logging listeners or actors automatically deployed by configuration at - the start of the actor system. -- ``"/deadLetters"`` is the dead letter actor, which is where all messages sent to - stopped or non-existing actors are re-routed (on a best-effort basis: messages - may be lost even within the local JVM). -- ``"/temp"`` is the guardian for all short-lived system-created actors, e.g. - those which are used in the implementation of :meth:`ActorRef.ask`. -- ``"/remote"`` is an artificial path below which all actors reside whose - supervisors are remote actor references + * `"/user"` is the guardian actor for all user-created top-level actors; +actors created using `ActorSystem.actorOf` are found below this one. + * `"/system"` is the guardian actor for all system-created top-level actors, +e.g. logging listeners or actors automatically deployed by configuration at +the start of the actor system. + * `"/deadLetters"` is the dead letter actor, which is where all messages sent to +stopped or non-existing actors are re-routed (on a best-effort basis: messages +may be lost even within the local JVM). + * `"/temp"` is the guardian for all short-lived system-created actors, e.g. +those which are used in the implementation of `ActorRef.ask`. + * `"/remote"` is an artificial path below which all actors reside whose +supervisors are remote actor references The need to structure the name space for actors like this arises from a central and very simple design goal: everything in the hierarchy is an actor, and all @@ -345,5 +325,4 @@ there are no quirks to remember, it makes the whole system more uniform and consistent. If you want to read more about the top-level structure of an actor system, have -a look at :ref:`toplevel-supervisors`. - +a look at @ref:[The Top-Level Supervisors](supervision.md#toplevel-supervisors). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/configuration.md b/akka-docs/src/main/paradox/general/configuration.md index cf9fca7fa9..205a753d3f 100644 --- a/akka-docs/src/main/paradox/general/configuration.md +++ b/akka-docs/src/main/paradox/general/configuration.md @@ -1,540 +1,490 @@ -.. _configuration: - -Configuration -============= +# Configuration You can start using Akka without defining any configuration, since sensible default values are provided. Later on you might need to amend the settings to change the default behavior or adapt for specific runtime environments. Typical examples of settings that you might amend: -* log level and logger backend -* enable remoting -* message serializers -* definition of routers -* tuning of dispatchers + * log level and logger backend + * enable remoting + * message serializers + * definition of routers + * tuning of dispatchers -Akka uses the `Typesafe Config Library -`_, which might also be a good choice +Akka uses the [Typesafe Config Library](https://github.com/typesafehub/config), which might also be a good choice for the configuration of your own application or library built with or without Akka. This library is implemented in Java with no external dependencies; you -should have a look at its documentation (in particular about `ConfigFactory -`_), +should have a look at its documentation (in particular about [ConfigFactory](http://typesafehub.github.io/config/v1.2.0/com/typesafe/config/ConfigFactory.html)), which is only summarized in the following. -.. warning:: +@@@ warning - If you use Akka from the Scala REPL from the 2.9.x series, - and you do not provide your own ClassLoader to the ActorSystem, - start the REPL with "-Yrepl-sync" to work around a deficiency in - the REPLs provided Context ClassLoader. +If you use Akka from the Scala REPL from the 2.9.x series, +and you do not provide your own ClassLoader to the ActorSystem, +start the REPL with "-Yrepl-sync" to work around a deficiency in +the REPLs provided Context ClassLoader. +@@@ -Where configuration is read from --------------------------------- +## Where configuration is read from -All configuration for Akka is held within instances of :class:`ActorSystem`, or -put differently, as viewed from the outside, :class:`ActorSystem` is the only +All configuration for Akka is held within instances of `ActorSystem`, or +put differently, as viewed from the outside, `ActorSystem` is the only consumer of configuration information. While constructing an actor system, you -can either pass in a :class:`Config` object or not, where the second case is -equivalent to passing ``ConfigFactory.load()`` (with the right class loader). -This means roughly that the default is to parse all ``application.conf``, -``application.json`` and ``application.properties`` found at the root of the +can either pass in a `Config` object or not, where the second case is +equivalent to passing `ConfigFactory.load()` (with the right class loader). +This means roughly that the default is to parse all `application.conf`, +`application.json` and `application.properties` found at the root of the class path—please refer to the aforementioned documentation for details. The -actor system then merges in all ``reference.conf`` resources found at the root +actor system then merges in all `reference.conf` resources found at the root of the class path to form the fallback configuration, i.e. it internally uses -.. code-block:: scala - - appConfig.withFallback(ConfigFactory.defaultReference(classLoader)) +```scala +appConfig.withFallback(ConfigFactory.defaultReference(classLoader)) +``` The philosophy is that code never contains default values, but instead relies -upon their presence in the ``reference.conf`` supplied with the library in +upon their presence in the `reference.conf` supplied with the library in question. -Highest precedence is given to overrides given as system properties, see `the -HOCON specification -`_ (near the +Highest precedence is given to overrides given as system properties, see [the +HOCON specification](https://github.com/typesafehub/config/blob/master/HOCON.md) (near the bottom). Also noteworthy is that the application configuration—which defaults -to ``application``—may be overridden using the ``config.resource`` property -(there are more, please refer to the `Config docs -`_). +to `application`—may be overridden using the `config.resource` property +(there are more, please refer to the [Config docs](https://github.com/typesafehub/config/blob/master/README.md)). -.. note:: +@@@ note - If you are writing an Akka application, keep you configuration in - ``application.conf`` at the root of the class path. If you are writing an - Akka-based library, keep its configuration in ``reference.conf`` at the root - of the JAR file. +If you are writing an Akka application, keep you configuration in +`application.conf` at the root of the class path. If you are writing an +Akka-based library, keep its configuration in `reference.conf` at the root +of the JAR file. +@@@ -When using JarJar, OneJar, Assembly or any jar-bundler ------------------------------------------------------- +## When using JarJar, OneJar, Assembly or any jar-bundler -.. warning:: +@@@ warning - Akka's configuration approach relies heavily on the notion of every - module/jar having its own reference.conf file, all of these will be - discovered by the configuration and loaded. Unfortunately this also means - that if you put/merge multiple jars into the same jar, you need to merge all the - reference.confs as well. Otherwise all defaults will be lost and Akka will not function. +Akka's configuration approach relies heavily on the notion of every +module/jar having its own reference.conf file, all of these will be +discovered by the configuration and loaded. Unfortunately this also means +that if you put/merge multiple jars into the same jar, you need to merge all the +reference.confs as well. Otherwise all defaults will be lost and Akka will not function. +@@@ If you are using Maven to package your application, you can also make use of -the `Apache Maven Shade Plugin -`_ support for `Resource -Transformers -`_ +the [Apache Maven Shade Plugin](http://maven.apache.org/plugins/maven-shade-plugin) support for [Resource +Transformers](http://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#AppendingTransformer) to merge all the reference.confs on the build classpath into one. -The plugin configuration might look like this:: +The plugin configuration might look like this: - - org.apache.maven.plugins - maven-shade-plugin - 1.5 - - - package - - shade - - - true - allinone - - - *:* - - - - - reference.conf - - - - akka.Main - - - - - - - +``` + + org.apache.maven.plugins + maven-shade-plugin + 1.5 + + + package + + shade + + + true + allinone + + + *:* + + + + + reference.conf + + + + akka.Main + + + + + + + +``` -Custom application.conf ------------------------ +## Custom application.conf -A custom ``application.conf`` might look like this:: +A custom `application.conf` might look like this: - # In this file you can override any option defined in the reference files. - # Copy in parts of the reference files and modify as you please. +``` +# In this file you can override any option defined in the reference files. +# Copy in parts of the reference files and modify as you please. - akka { +akka { - # Loggers to register at boot time (akka.event.Logging$DefaultLogger logs - # to STDOUT) - loggers = ["akka.event.slf4j.Slf4jLogger"] + # Loggers to register at boot time (akka.event.Logging$DefaultLogger logs + # to STDOUT) + loggers = ["akka.event.slf4j.Slf4jLogger"] - # Log level used by the configured loggers (see "loggers") as soon - # as they have been started; before that, see "stdout-loglevel" - # Options: OFF, ERROR, WARNING, INFO, DEBUG - loglevel = "DEBUG" + # Log level used by the configured loggers (see "loggers") as soon + # as they have been started; before that, see "stdout-loglevel" + # Options: OFF, ERROR, WARNING, INFO, DEBUG + loglevel = "DEBUG" - # Log level for the very basic logger activated during ActorSystem startup. - # This logger prints the log messages to stdout (System.out). - # Options: OFF, ERROR, WARNING, INFO, DEBUG - stdout-loglevel = "DEBUG" - - # Filter of log events that is used by the LoggingAdapter before - # publishing log events to the eventStream. - logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" + # Log level for the very basic logger activated during ActorSystem startup. + # This logger prints the log messages to stdout (System.out). + # Options: OFF, ERROR, WARNING, INFO, DEBUG + stdout-loglevel = "DEBUG" - actor { - provider = "cluster" - - default-dispatcher { - # Throughput for default Dispatcher, set to 1 for as fair as possible - throughput = 10 - } - } + # Filter of log events that is used by the LoggingAdapter before + # publishing log events to the eventStream. + logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" - remote { - # The port clients should connect to. Default is 2552. - netty.tcp.port = 4711 + actor { + provider = "cluster" + + default-dispatcher { + # Throughput for default Dispatcher, set to 1 for as fair as possible + throughput = 10 } } + remote { + # The port clients should connect to. Default is 2552. + netty.tcp.port = 4711 + } +} +``` -Including files ---------------- +## Including files -Sometimes it can be useful to include another configuration file, for example if you have one ``application.conf`` with all +Sometimes it can be useful to include another configuration file, for example if you have one `application.conf` with all environment independent settings and then override some settings for specific environments. -Specifying system property with ``-Dconfig.resource=/dev.conf`` will load the ``dev.conf`` file, which includes the ``application.conf`` +Specifying system property with `-Dconfig.resource=/dev.conf` will load the `dev.conf` file, which includes the `application.conf` dev.conf: -:: +``` +include "application" - include "application" +akka { + loglevel = "DEBUG" +} +``` - akka { - loglevel = "DEBUG" - } - -More advanced include and substitution mechanisms are explained in the `HOCON `_ +More advanced include and substitution mechanisms are explained in the [HOCON](https://github.com/typesafehub/config/blob/master/HOCON.md) specification. + +## Logging of Configuration -.. _-Dakka.log-config-on-start: - -Logging of Configuration ------------------------- - -If the system or config property ``akka.log-config-on-start`` is set to ``on``, then the +If the system or config property `akka.log-config-on-start` is set to `on`, then the complete configuration is logged at INFO level when the actor system is started. This is useful when you are uncertain of what configuration is used. If in doubt, you can also easily and nicely inspect configuration objects before or after using them to construct an actor system: -.. parsed-literal:: +``` +Welcome to Scala version @scalaVersion@ (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0). +Type in expressions to have them evaluated. +Type :help for more information. - Welcome to Scala version @scalaVersion@ (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0). - Type in expressions to have them evaluated. - Type :help for more information. +scala> import com.typesafe.config._ +import com.typesafe.config._ - scala> import com.typesafe.config._ - import com.typesafe.config._ +scala> ConfigFactory.parseString("a.b=12") +res0: com.typesafe.config.Config = Config(SimpleConfigObject({"a" : {"b" : 12}})) - scala> ConfigFactory.parseString("a.b=12") - res0: com.typesafe.config.Config = Config(SimpleConfigObject({"a" : {"b" : 12}})) - - scala> res0.root.render - res1: java.lang.String = - { - # String: 1 - "a" : { - # String: 1 - "b" : 12 - } - } +scala> res0.root.render +res1: java.lang.String = +{ + # String: 1 + "a" : { + # String: 1 + "b" : 12 + } +}``` The comments preceding every item give detailed information about the origin of the setting (file & line number) plus possible comments which were present, e.g. in the reference configuration. The settings as merged with the reference and parsed by the actor system can be displayed like this: -.. code-block:: java +```java +final ActorSystem system = ActorSystem.create(); +System.out.println(system.settings()); +// this is a shortcut for system.settings().config().root().render() +``` - final ActorSystem system = ActorSystem.create(); - System.out.println(system.settings()); - // this is a shortcut for system.settings().config().root().render() - -A Word About ClassLoaders -------------------------- +## A Word About ClassLoaders In several places of the configuration file it is possible to specify the fully-qualified class name of something to be instantiated by Akka. This is -done using Java reflection, which in turn uses a :class:`ClassLoader`. Getting +done using Java reflection, which in turn uses a `ClassLoader`. Getting the right one in challenging environments like application containers or OSGi bundles is not always trivial, the current approach of Akka is that each -:class:`ActorSystem` implementation stores the current thread’s context class +`ActorSystem` implementation stores the current thread’s context class loader (if available, otherwise just its own loader as in -``this.getClass.getClassLoader``) and uses that for all reflective accesses. +`this.getClass.getClassLoader`) and uses that for all reflective accesses. This implies that putting Akka on the boot class path will yield -:class:`NullPointerException` from strange places: this is simply not +`NullPointerException` from strange places: this is simply not supported. -Application specific settings ------------------------------ +## Application specific settings The configuration can also be used for application specific settings. A good practice is to place those settings in an Extension, as described in: - * Scala API: :ref:`extending-akka-scala.settings` - * Java API: :ref:`extending-akka-java.settings` +> + * Scala API: @ref:[extending-akka-scala.settings](../scala/extending-akka.md#extending-akka-scala-settings) + * Java API: @ref:[extending-akka-java.settings](../java/extending-akka.md#extending-akka-java-settings) -Configuring multiple ActorSystem --------------------------------- +## Configuring multiple ActorSystem -If you have more than one ``ActorSystem`` (or you're writing a -library and have an ``ActorSystem`` that may be separate from the +If you have more than one `ActorSystem` (or you're writing a +library and have an `ActorSystem` that may be separate from the application's) you may want to separate the configuration for each system. -Given that ``ConfigFactory.load()`` merges all resources with matching name +Given that `ConfigFactory.load()` merges all resources with matching name from the whole class path, it is easiest to utilize that functionality and -differentiate actor systems within the hierarchy of the configuration:: +differentiate actor systems within the hierarchy of the configuration: - myapp1 { - akka.loglevel = "WARNING" - my.own.setting = 43 - } - myapp2 { - akka.loglevel = "ERROR" - app2.setting = "appname" - } - my.own.setting = 42 - my.other.setting = "hello" +``` +myapp1 { + akka.loglevel = "WARNING" + my.own.setting = 43 +} +myapp2 { + akka.loglevel = "ERROR" + app2.setting = "appname" +} +my.own.setting = 42 +my.other.setting = "hello" +``` -.. code-block:: scala - - val config = ConfigFactory.load() - val app1 = ActorSystem("MyApp1", config.getConfig("myapp1").withFallback(config)) - val app2 = ActorSystem("MyApp2", - config.getConfig("myapp2").withOnlyPath("akka").withFallback(config)) +```scala +val config = ConfigFactory.load() +val app1 = ActorSystem("MyApp1", config.getConfig("myapp1").withFallback(config)) +val app2 = ActorSystem("MyApp2", + config.getConfig("myapp2").withOnlyPath("akka").withFallback(config)) +``` These two samples demonstrate different variations of the “lift-a-subtree” trick: in the first case, the configuration accessible from within the actor system is this -.. code-block:: ruby - - akka.loglevel = "WARNING" - my.own.setting = 43 - my.other.setting = "hello" - // plus myapp1 and myapp2 subtrees +```ruby +akka.loglevel = "WARNING" +my.own.setting = 43 +my.other.setting = "hello" +// plus myapp1 and myapp2 subtrees +``` while in the second one, only the “akka” subtree is lifted, with the following result -.. code-block:: ruby +```ruby +akka.loglevel = "ERROR" +my.own.setting = 42 +my.other.setting = "hello" +// plus myapp1 and myapp2 subtrees +``` - akka.loglevel = "ERROR" - my.own.setting = 42 - my.other.setting = "hello" - // plus myapp1 and myapp2 subtrees +@@@ note -.. note:: +The configuration library is really powerful, explaining all features exceeds +the scope affordable here. In particular not covered are how to include other +configuration files within other files (see a small example at [Including +files](#including-files)) and copying parts of the configuration tree by way of path +substitutions. - The configuration library is really powerful, explaining all features exceeds - the scope affordable here. In particular not covered are how to include other - configuration files within other files (see a small example at `Including - files`_) and copying parts of the configuration tree by way of path - substitutions. +@@@ You may also specify and parse the configuration programmatically in other ways when instantiating -the ``ActorSystem``. +the `ActorSystem`. -.. includecode:: code/docs/config/ConfigDocSpec.scala - :include: imports,custom-config +@@snip [ConfigDocSpec.scala](code/docs/config/ConfigDocSpec.scala) { #imports #custom-config } -Reading configuration from a custom location --------------------------------------------- +## Reading configuration from a custom location -You can replace or supplement ``application.conf`` either in code +You can replace or supplement `application.conf` either in code or using system properties. -If you're using ``ConfigFactory.load()`` (which Akka does by -default) you can replace ``application.conf`` by defining -``-Dconfig.resource=whatever``, ``-Dconfig.file=whatever``, or -``-Dconfig.url=whatever``. +If you're using `ConfigFactory.load()` (which Akka does by +default) you can replace `application.conf` by defining +`-Dconfig.resource=whatever`, `-Dconfig.file=whatever`, or +`-Dconfig.url=whatever`. From inside your replacement file specified with -``-Dconfig.resource`` and friends, you can ``include -"application"`` if you still want to use -``application.{conf,json,properties}`` as well. Settings -specified before ``include "application"`` would be overridden by +`-Dconfig.resource` and friends, you can `include +"application"` if you still want to use +`application.{conf,json,properties}` as well. Settings +specified before `include "application"` would be overridden by the included file, while those after would override the included file. In code, there are many customization options. -There are several overloads of ``ConfigFactory.load()``; these +There are several overloads of `ConfigFactory.load()`; these allow you to specify something to be sandwiched between system properties (which override) and the defaults (from -``reference.conf``), replacing the usual -``application.{conf,json,properties}`` and replacing -``-Dconfig.file`` and friends. +`reference.conf`), replacing the usual +`application.{conf,json,properties}` and replacing +`-Dconfig.file` and friends. -The simplest variant of ``ConfigFactory.load()`` takes a resource -basename (instead of ``application``); ``myname.conf``, -``myname.json``, and ``myname.properties`` would then be used -instead of ``application.{conf,json,properties}``. +The simplest variant of `ConfigFactory.load()` takes a resource +basename (instead of `application`); `myname.conf`, +`myname.json`, and `myname.properties` would then be used +instead of `application.{conf,json,properties}`. -The most flexible variant takes a ``Config`` object, which -you can load using any method in ``ConfigFactory``. For example +The most flexible variant takes a `Config` object, which +you can load using any method in `ConfigFactory`. For example you could put a config string in code using -``ConfigFactory.parseString()`` or you could make a map and -``ConfigFactory.parseMap()``, or you could load a file. +`ConfigFactory.parseString()` or you could make a map and +`ConfigFactory.parseMap()`, or you could load a file. You can also combine your custom config with the usual config, that might look like: -.. includecode:: code/docs/config/ConfigDoc.java - :include: java-custom-config +@@snip [ConfigDoc.java](code/docs/config/ConfigDoc.java) { #java-custom-config } -When working with ``Config`` objects, keep in mind that there are +When working with `Config` objects, keep in mind that there are three "layers" in the cake: - - ``ConfigFactory.defaultOverrides()`` (system properties) - - the app's settings - - ``ConfigFactory.defaultReference()`` (reference.conf) +> + * `ConfigFactory.defaultOverrides()` (system properties) + * the app's settings + * `ConfigFactory.defaultReference()` (reference.conf) The normal goal is to customize the middle layer while leaving the other two alone. - - ``ConfigFactory.load()`` loads the whole stack - - the overloads of ``ConfigFactory.load()`` let you specify a - different middle layer - - the ``ConfigFactory.parse()`` variations load single files or resources +> + * `ConfigFactory.load()` loads the whole stack + * the overloads of `ConfigFactory.load()` let you specify a +different middle layer + * the `ConfigFactory.parse()` variations load single files or resources -To stack two layers, use ``override.withFallback(fallback)``; try -to keep system props (``defaultOverrides()``) on top and -``reference.conf`` (``defaultReference()``) on the bottom. +To stack two layers, use `override.withFallback(fallback)`; try +to keep system props (`defaultOverrides()`) on top and +`reference.conf` (`defaultReference()`) on the bottom. -Do keep in mind, you can often just add another ``include`` -statement in ``application.conf`` rather than writing code. -Includes at the top of ``application.conf`` will be overridden by -the rest of ``application.conf``, while those at the bottom will +Do keep in mind, you can often just add another `include` +statement in `application.conf` rather than writing code. +Includes at the top of `application.conf` will be overridden by +the rest of `application.conf`, while those at the bottom will override the earlier stuff. -Actor Deployment Configuration ------------------------------- +## Actor Deployment Configuration -Deployment settings for specific actors can be defined in the ``akka.actor.deployment`` +Deployment settings for specific actors can be defined in the `akka.actor.deployment` section of the configuration. In the deployment section it is possible to define things like dispatcher, mailbox, router settings, and remote deployment. Configuration of these features are described in the chapters detailing corresponding topics. An example may look like this: -.. includecode:: code/docs/config/ConfigDocSpec.scala#deployment-section +@@snip [ConfigDocSpec.scala](code/docs/config/ConfigDocSpec.scala) { #deployment-section } +@@@ note -.. note:: +The deployment section for a specific actor is identified by the +path of the actor relative to `/user`. - The deployment section for a specific actor is identified by the - path of the actor relative to ``/user``. +@@@ You can use asterisks as wildcard matches for the actor path sections, so you could specify: -``/*/sampleActor`` and that would match all ``sampleActor`` on that level in the hierarchy. +`/*/sampleActor` and that would match all `sampleActor` on that level in the hierarchy. In addition, please note: - - you can also use wildcards in the last position to match all actors at a certain level: ``/someParent/*`` - - you can use double-wildcards in the last position to match all child actors and their children - recursively: ``/someParent/**`` - - non-wildcard matches always have higher priority to match than wildcards, and single wildcard matches - have higher priority than double-wildcards, so: ``/foo/bar`` is considered **more specific** than - ``/foo/*``, which is considered **more specific** than ``/foo/**``. Only the highest priority match is used - - wildcards **cannot** be used to partially match section, like this: ``/foo*/bar``, ``/f*o/bar`` etc. +> + * you can also use wildcards in the last position to match all actors at a certain level: `/someParent/*` + * you can use double-wildcards in the last position to match all child actors and their children +recursively: `/someParent/**` + * non-wildcard matches always have higher priority to match than wildcards, and single wildcard matches +have higher priority than double-wildcards, so: `/foo/bar` is considered **more specific** than +`/foo/*`, which is considered **more specific** than `/foo/**`. Only the highest priority match is used + * wildcards **cannot** be used to partially match section, like this: `/foo*/bar`, `/f*o/bar` etc. -.. note:: - Double-wildcards can only be placed in the last position. +@@@ note -Listing of the Reference Configuration --------------------------------------- +Double-wildcards can only be placed in the last position. + +@@@ + +## Listing of the Reference Configuration Each Akka module has a reference configuration file with the default values. -.. _config-akka-actor: + +### akka-actor -akka-actor -~~~~~~~~~~ +@@snip [reference.conf]../../../../../akka-actor/src/main/resources/reference.conf) -.. literalinclude:: ../../../akka-actor/src/main/resources/reference.conf - :language: none + +### akka-agent -.. _config-akka-agent: +@@snip [reference.conf]../../../../../akka-agent/src/main/resources/reference.conf) -akka-agent -~~~~~~~~~~ + +### akka-camel -.. literalinclude:: ../../../akka-agent/src/main/resources/reference.conf - :language: none +@@snip [reference.conf]../../../../../akka-camel/src/main/resources/reference.conf) -.. _config-akka-camel: + +### akka-cluster -akka-camel -~~~~~~~~~~ +@@snip [reference.conf]../../../../../akka-cluster/src/main/resources/reference.conf) -.. literalinclude:: ../../../akka-camel/src/main/resources/reference.conf - :language: none + +### akka-multi-node-testkit -.. _config-akka-cluster: +@@snip [reference.conf]../../../../../akka-multi-node-testkit/src/main/resources/reference.conf) -akka-cluster -~~~~~~~~~~~~ + +### akka-persistence -.. literalinclude:: ../../../akka-cluster/src/main/resources/reference.conf - :language: none +@@snip [reference.conf]../../../../../akka-persistence/src/main/resources/reference.conf) -.. _config-akka-multi-node-testkit: + +### akka-remote -akka-multi-node-testkit -~~~~~~~~~~~~~~~~~~~~~~~ +@@snip [reference.conf]../../../../../akka-remote/src/main/resources/reference.conf) { #shared #classic type=none } -.. literalinclude:: ../../../akka-multi-node-testkit/src/main/resources/reference.conf - :language: none + +### akka-remote (artery) -.. _config-akka-persistence: +@@snip [reference.conf]../../../../../akka-remote/src/main/resources/reference.conf) { #shared #artery type=none } -akka-persistence -~~~~~~~~~~~~~~~~ + +### akka-testkit -.. literalinclude:: ../../../akka-persistence/src/main/resources/reference.conf - :language: none +@@snip [reference.conf]../../../../../akka-testkit/src/main/resources/reference.conf) -.. _config-akka-remote: + +### akka-cluster-metrics -akka-remote -~~~~~~~~~~~ +@@snip [reference.conf]../../../../../akka-cluster-metrics/src/main/resources/reference.conf) -.. includecode:: ../../../akka-remote/src/main/resources/reference.conf - :include: shared,classic - :language: none + +### akka-cluster-tools +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) -.. _config-akka-remote-artery: + +### akka-cluster-sharding -akka-remote (artery) -~~~~~~~~~~~~~~~~~~~~ +@@snip [reference.conf]../../../../../akka-cluster-sharding/src/main/resources/reference.conf) -.. includecode:: ../../../akka-remote/src/main/resources/reference.conf - :include: shared,artery - :language: none - -.. _config-akka-testkit: - -akka-testkit -~~~~~~~~~~~~ - -.. literalinclude:: ../../../akka-testkit/src/main/resources/reference.conf - :language: none - -.. _config-cluster-metrics: - -akka-cluster-metrics -~~~~~~~~~~~~~~~~~~~~ - -.. literalinclude:: ../../../akka-cluster-metrics/src/main/resources/reference.conf - :language: none - -.. _config-cluster-tools: - -akka-cluster-tools -~~~~~~~~~~~~~~~~~~ - -.. literalinclude:: ../../../akka-cluster-tools/src/main/resources/reference.conf - :language: none - -.. _config-cluster-sharding: - -akka-cluster-sharding -~~~~~~~~~~~~~~~~~~~~~ - -.. literalinclude:: ../../../akka-cluster-sharding/src/main/resources/reference.conf - :language: none - -.. _config-distributed-data: - -akka-distributed-data -~~~~~~~~~~~~~~~~~~~~~ - -.. literalinclude:: ../../../akka-distributed-data/src/main/resources/reference.conf - :language: none + +### akka-distributed-data +@@snip [reference.conf]../../../../../akka-distributed-data/src/main/resources/reference.conf) \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/index.md b/akka-docs/src/main/paradox/general/index.md index ceda05f75f..83e7d48b06 100644 --- a/akka-docs/src/main/paradox/general/index.md +++ b/akka-docs/src/main/paradox/general/index.md @@ -1,15 +1,17 @@ -General -======= +# General -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - terminology - actor-systems - actors - supervision - addressing - remoting - jmm - message-delivery-reliability - configuration +@@@ index + +* [terminology](terminology.md) +* [actor-systems](actor-systems.md) +* [actors](actors.md) +* [supervision](supervision.md) +* [addressing](addressing.md) +* [remoting](remoting.md) +* [jmm](jmm.md) +* [message-delivery-reliability](message-delivery-reliability.md) +* [configuration](configuration.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/jmm.md b/akka-docs/src/main/paradox/general/jmm.md index 774f8e28a6..56c986358b 100644 --- a/akka-docs/src/main/paradox/general/jmm.md +++ b/akka-docs/src/main/paradox/general/jmm.md @@ -1,58 +1,55 @@ -.. _jmm: - -Akka and the Java Memory Model -================================ +# Akka and the Java Memory Model A major benefit of using the Lightbend Platform, including Scala and Akka, is that it simplifies the process of writing concurrent software. This article discusses how the Lightbend Platform, and Akka in particular, approaches shared memory in concurrent applications. -The Java Memory Model ---------------------- +## The Java Memory Model + Prior to Java 5, the Java Memory Model (JMM) was ill defined. It was possible to get all kinds of strange results when shared memory was accessed by multiple threads, such as: -* a thread not seeing values written by other threads: a visibility problem -* a thread observing 'impossible' behavior of other threads, caused by - instructions not being executed in the order expected: an instruction - reordering problem. + * a thread not seeing values written by other threads: a visibility problem + * a thread observing 'impossible' behavior of other threads, caused by +instructions not being executed in the order expected: an instruction +reordering problem. With the implementation of JSR 133 in Java 5, a lot of these issues have been resolved. The JMM is a set of rules based on the "happens-before" relation, which constrain when one memory access must happen before another, and conversely, when they are allowed to happen out of order. Two examples of these rules are: -* **The monitor lock rule:** a release of a lock happens before every subsequent acquire of the same lock. -* **The volatile variable rule:** a write of a volatile variable happens before every subsequent read of the same volatile variable + * **The monitor lock rule:** a release of a lock happens before every subsequent acquire of the same lock. + * **The volatile variable rule:** a write of a volatile variable happens before every subsequent read of the same volatile variable Although the JMM can seem complicated, the specification tries to find a balance between ease of use and the ability to write performant and scalable concurrent data structures. -Actors and the Java Memory Model --------------------------------- +## Actors and the Java Memory Model + With the Actors implementation in Akka, there are two ways multiple threads can execute actions on shared memory: -* if a message is sent to an actor (e.g. by another actor). In most cases messages are immutable, but if that message - is not a properly constructed immutable object, without a "happens before" rule, it would be possible for the receiver - to see partially initialized data structures and possibly even values out of thin air (longs/doubles). -* if an actor makes changes to its internal state while processing a message, and accesses that state while processing - another message moments later. It is important to realize that with the actor model you don't get any guarantee that - the same thread will be executing the same actor for different messages. + * if a message is sent to an actor (e.g. by another actor). In most cases messages are immutable, but if that message +is not a properly constructed immutable object, without a "happens before" rule, it would be possible for the receiver +to see partially initialized data structures and possibly even values out of thin air (longs/doubles). + * if an actor makes changes to its internal state while processing a message, and accesses that state while processing +another message moments later. It is important to realize that with the actor model you don't get any guarantee that +the same thread will be executing the same actor for different messages. To prevent visibility and reordering problems on actors, Akka guarantees the following two "happens before" rules: -* **The actor send rule:** the send of the message to an actor happens before the receive of that message by the same actor. -* **The actor subsequent processing rule:** processing of one message happens before processing of the next message by the same actor. + * **The actor send rule:** the send of the message to an actor happens before the receive of that message by the same actor. + * **The actor subsequent processing rule:** processing of one message happens before processing of the next message by the same actor. -.. note:: +@@@ note - In layman's terms this means that changes to internal fields of the actor are visible when the next message - is processed by that actor. So fields in your actor need not be volatile or equivalent. +In layman's terms this means that changes to internal fields of the actor are visible when the next message +is processed by that actor. So fields in your actor need not be volatile or equivalent. +@@@ Both rules only apply for the same actor instance and are not valid if different actors are used. -Futures and the Java Memory Model ---------------------------------- +## Futures and the Java Memory Model The completion of a Future "happens before" the invocation of any callbacks registered to it are executed. @@ -63,15 +60,13 @@ If you close over a reference, you must also ensure that the instance that is re We highly recommend staying away from objects that use locking, since it can introduce performance problems and in the worst case, deadlocks. Such are the perils of synchronized. -.. _jmm-shared-state: - -Actors and shared mutable state -------------------------------- + +## Actors and shared mutable state Since Akka runs on the JVM there are still some rules to be followed. -* Closing over internal Actor state and exposing it to other threads + * Closing over internal Actor state and exposing it to other threads -.. includecode:: ../scala/code/docs/actor/SharedMutableStateDocSpec.scala#mutable-state +@@snip [SharedMutableStateDocSpec.scala](../scala/code/docs/actor/SharedMutableStateDocSpec.scala) { #mutable-state } -* Messages **should** be immutable, this is to avoid the shared mutable state trap. + * Messages **should** be immutable, this is to avoid the shared mutable state trap. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/message-delivery-reliability.md b/akka-docs/src/main/paradox/general/message-delivery-reliability.md index 2839c495f0..de6ec58cec 100644 --- a/akka-docs/src/main/paradox/general/message-delivery-reliability.md +++ b/akka-docs/src/main/paradox/general/message-delivery-reliability.md @@ -1,8 +1,4 @@ -.. _message-delivery-reliability: - -############################ -Message Delivery Reliability -############################ +# Message Delivery Reliability Akka helps you build reliable applications which make use of multiple processor cores in one machine (“scaling up”) or distributed across a computer network @@ -34,36 +30,32 @@ As a supplementary part we give a few pointers at how to build stronger reliability on top of the built-in ones. The chapter closes by discussing the role of the “Dead Letter Office”. -The General Rules -================= +## The General Rules -These are the rules for message sends (i.e. the ``tell`` or ``!`` method, which -also underlies the ``ask`` pattern): +These are the rules for message sends (i.e. the `tell` or `!` method, which +also underlies the `ask` pattern): -* **at-most-once delivery**, i.e. no guaranteed delivery -* **message ordering per sender–receiver pair** + * **at-most-once delivery**, i.e. no guaranteed delivery + * **message ordering per sender–receiver pair** The first rule is typically found also in other actor implementations while the second is specific to Akka. -Discussion: What does “at-most-once” mean? ------------------------------------------- +### Discussion: What does “at-most-once” mean? When it comes to describing the semantics of a delivery mechanism, there are three basic categories: -* **at-most-once** delivery means that for each message handed to the - mechanism, that message is delivered zero or one times; in more casual terms - it means that messages may be lost. - -* **at-least-once** delivery means that for each message handed to the - mechanism potentially multiple attempts are made at delivering it, such that - at least one succeeds; again, in more casual terms this means that messages - may be duplicated but not lost. - -* **exactly-once** delivery means that for each message handed to the mechanism - exactly one delivery is made to the recipient; the message can neither be - lost nor duplicated. + * **at-most-once** delivery means that for each message handed to the +mechanism, that message is delivered zero or one times; in more casual terms +it means that messages may be lost. + * **at-least-once** delivery means that for each message handed to the +mechanism potentially multiple attempts are made at delivering it, such that +at least one succeeds; again, in more casual terms this means that messages +may be duplicated but not lost. + * **exactly-once** delivery means that for each message handed to the mechanism +exactly one delivery is made to the recipient; the message can neither be +lost nor duplicated. The first one is the cheapest—highest performance, least implementation overhead—because it can be done in a fire-and-forget fashion without keeping @@ -74,17 +66,16 @@ most expensive—and has consequently worst performance—because in addition to the second it requires state to be kept at the receiving end in order to filter out duplicate deliveries. -Discussion: Why No Guaranteed Delivery? ---------------------------------------- +### Discussion: Why No Guaranteed Delivery? At the core of the problem lies the question what exactly this guarantee shall mean: -1. The message is sent out on the network? -2. The message is received by the other host? -3. The message is put into the target actor's mailbox? -4. The message is starting to be processed by the target actor? -5. The message is processed successfully by the target actor? + 1. The message is sent out on the network? + 2. The message is received by the other host? + 3. The message is put into the target actor's mailbox? + 4. The message is starting to be processed by the target actor? + 5. The message is processed successfully by the target actor? Each one of these have different challenges and costs, and it is obvious that there are conditions under which any message passing library would be unable to @@ -92,8 +83,8 @@ comply; think for example about configurable mailbox types and how a bounded mailbox would interact with the third point, or even what it would mean to decide upon the “successfully” part of point five. -Along those same lines goes the reasoning in `Nobody Needs Reliable -Messaging`_. The only meaningful way for a sender to know whether an +Along those same lines goes the reasoning in [Nobody Needs Reliable +Messaging](http://www.infoq.com/articles/no-reliable-messaging). The only meaningful way for a sender to know whether an interaction was successful is by receiving a business-level acknowledgement message, which is not something Akka could make up on its own (neither are we writing a “do what I mean” framework nor would you want us to). @@ -102,7 +93,7 @@ Akka embraces distributed computing and makes the fallibility of communication explicit through message passing, therefore it does not try to lie and emulate a leaky abstraction. This is a model that has been used with great success in Erlang and requires the users to design their applications around it. You can -read more about this approach in the `Erlang documentation`_ (section 10.9 and +read more about this approach in the [Erlang documentation](http://www.erlang.org/faq/academic.html) (section 10.9 and 10.10), Akka follows it closely. Another angle on this issue is that by providing only basic guarantees those @@ -111,112 +102,115 @@ implementation; it is always possible to add stronger reliability on top of basic ones, but it is not possible to retro-actively remove reliability in order to gain more performance. -.. _message-ordering: - -Discussion: Message Ordering ----------------------------- + +### Discussion: Message Ordering The rule more specifically is that *for a given pair of actors, messages sent directly from the first to the second will not be received out-of-order.* The word *directly* emphasizes that this guarantee only applies when sending with -the `tell` operator to the final destination, not when employing mediators or +the *tell* operator to the final destination, not when employing mediators or other message dissemination features (unless stated otherwise). The guarantee is illustrated in the following: - Actor ``A1`` sends messages ``M1``, ``M2``, ``M3`` to ``A2`` +> +Actor `A1` sends messages `M1`, `M2`, `M3` to `A2` +> +Actor `A3` sends messages `M4`, `M5`, `M6` to `A2` +> +This means that: +: + 1. If `M1` is delivered it must be delivered before `M2` and `M3` + 2. If `M2` is delivered it must be delivered before `M3` + 3. If `M4` is delivered it must be delivered before `M5` and `M6` + 4. If `M5` is delivered it must be delivered before `M6` + 5. `A2` can see messages from `A1` interleaved with messages from `A3` + 6. Since there is no guaranteed delivery, any of the messages may be dropped, i.e. not arrive at `A2` - Actor ``A3`` sends messages ``M4``, ``M5``, ``M6`` to ``A2`` - - This means that: - 1) If ``M1`` is delivered it must be delivered before ``M2`` and ``M3`` - 2) If ``M2`` is delivered it must be delivered before ``M3`` - 3) If ``M4`` is delivered it must be delivered before ``M5`` and ``M6`` - 4) If ``M5`` is delivered it must be delivered before ``M6`` - 5) ``A2`` can see messages from ``A1`` interleaved with messages from ``A3`` - 6) Since there is no guaranteed delivery, any of the messages may be dropped, i.e. not arrive at ``A2`` -.. note:: +@@@ note - It is important to note that Akka’s guarantee applies to the order in which - messages are enqueued into the recipient’s mailbox. If the mailbox - implementation does not respect FIFO order (e.g. a :class:`PriorityMailbox`), - then the order of processing by the actor can deviate from the enqueueing - order. +It is important to note that Akka’s guarantee applies to the order in which +messages are enqueued into the recipient’s mailbox. If the mailbox +implementation does not respect FIFO order (e.g. a `PriorityMailbox`), +then the order of processing by the actor can deviate from the enqueueing +order. + +@@@ Please note that this rule is **not transitive**: - Actor ``A`` sends message ``M1`` to actor ``C`` +> +Actor `A` sends message `M1` to actor `C` +> +Actor `A` then sends message `M2` to actor `B` +> +Actor `B` forwards message `M2` to actor `C` +> +Actor `C` may receive `M1` and `M2` in any order - Actor ``A`` then sends message ``M2`` to actor ``B`` +Causal transitive ordering would imply that `M2` is never received before +`M1` at actor `C` (though any of them might be lost). This ordering can be +violated due to different message delivery latencies when `A`, `B` and +`C` reside on different network hosts, see more below. - Actor ``B`` forwards message ``M2`` to actor ``C`` +@@@ note - Actor ``C`` may receive ``M1`` and ``M2`` in any order +Actor creation is treated as a message sent from the parent to the child, +with the same semantics as discussed above. Sending a message to an actor in +a way which could be reordered with this initial creation message means that +the message might not arrive because the actor does not exist yet. An example +where the message might arrive too early would be to create a remote-deployed +actor R1, send its reference to another remote actor R2 and have R2 send a +message to R1. An example of well-defined ordering is a parent which creates +an actor and immediately sends a message to it. -Causal transitive ordering would imply that ``M2`` is never received before -``M1`` at actor ``C`` (though any of them might be lost). This ordering can be -violated due to different message delivery latencies when ``A``, ``B`` and -``C`` reside on different network hosts, see more below. +@@@ -.. note:: - - Actor creation is treated as a message sent from the parent to the child, - with the same semantics as discussed above. Sending a message to an actor in - a way which could be reordered with this initial creation message means that - the message might not arrive because the actor does not exist yet. An example - where the message might arrive too early would be to create a remote-deployed - actor R1, send its reference to another remote actor R2 and have R2 send a - message to R1. An example of well-defined ordering is a parent which creates - an actor and immediately sends a message to it. - -Communication of failure -........................ +#### Communication of failure Please note, that the ordering guarantees discussed above only hold for user messages between actors. Failure of a child of an actor is communicated by special system messages that are not ordered relative to ordinary user messages. In particular: - Child actor ``C`` sends message ``M`` to its parent ``P`` - - Child actor fails with failure ``F`` - - Parent actor ``P`` might receive the two events either in order ``M``, ``F`` or ``F``, ``M`` +> +Child actor `C` sends message `M` to its parent `P` +> +Child actor fails with failure `F` +> +Parent actor `P` might receive the two events either in order `M`, `F` or `F`, `M` The reason for this is that internal system messages has their own mailboxes therefore the ordering of enqueue calls of a user and system message cannot guarantee the ordering of their dequeue times. -The Rules for In-JVM (Local) Message Sends -========================================== +## The Rules for In-JVM (Local) Message Sends -Be careful what you do with this section! ------------------------------------------ +### Be careful what you do with this section! Relying on the stronger reliability in this section is not recommended since it will bind your application to local-only deployment: an application may have to be designed differently (as opposed to just employing some message exchange patterns local to some actors) in order to be fit for running on a cluster of machines. Our credo is “design once, deploy any way you wish”, and to achieve -this you should only rely on `The General Rules`_. +this you should only rely on [The General Rules](#the-general-rules). -Reliability of Local Message Sends ----------------------------------- +### Reliability of Local Message Sends The Akka test suite relies on not losing messages in the local context (and for non-error condition tests also for remote deployment), meaning that we -actually do apply the best effort to keep our tests stable. A local ``tell`` +actually do apply the best effort to keep our tests stable. A local `tell` operation can however fail for the same reasons as a normal method call can on the JVM: -- :class:`StackOverflowError` -- :class:`OutOfMemoryError` -- other :class:`VirtualMachineError` + * `StackOverflowError` + * `OutOfMemoryError` + * other `VirtualMachineError` In addition, local sends can fail in Akka-specific ways: -- if the mailbox does not accept the message (e.g. full BoundedMailbox) -- if the receiving actor fails while processing the message or is already - terminated + * if the mailbox does not accept the message (e.g. full BoundedMailbox) + * if the receiving actor fails while processing the message or is already +terminated While the first is clearly a matter of configuration the second deserves some thought: the sender of a message does not get feedback if there was an @@ -224,8 +218,7 @@ exception while processing, that notification goes to the supervisor instead. This is in general not distinguishable from a lost message for an outside observer. -Ordering of Local Message Sends -------------------------------- +### Ordering of Local Message Sends Assuming strict FIFO mailboxes the aforementioned caveat of non-transitivity of the message ordering guarantee is eliminated under certain conditions. As you @@ -233,25 +226,22 @@ will note, these are quite subtle as it stands, and it is even possible that future performance optimizations will invalidate this whole paragraph. The possibly non-exhaustive list of counter-indications is: -- Before receiving the first reply from a top-level actor, there is a lock - which protects an internal interim queue, and this lock is not fair; the - implication is that enqueue requests from different senders which arrive - during the actor’s construction (figuratively, the details are more involved) - may be reordered depending on low-level thread scheduling. Since completely - fair locks do not exist on the JVM this is unfixable. - -- The same mechanism is used during the construction of a Router, more - precisely the routed ActorRef, hence the same problem exists for actors - deployed with Routers. - -- As mentioned above, the problem occurs anywhere a lock is involved during - enqueueing, which may also apply to custom mailboxes. + * Before receiving the first reply from a top-level actor, there is a lock +which protects an internal interim queue, and this lock is not fair; the +implication is that enqueue requests from different senders which arrive +during the actor’s construction (figuratively, the details are more involved) +may be reordered depending on low-level thread scheduling. Since completely +fair locks do not exist on the JVM this is unfixable. + * The same mechanism is used during the construction of a Router, more +precisely the routed ActorRef, hence the same problem exists for actors +deployed with Routers. + * As mentioned above, the problem occurs anywhere a lock is involved during +enqueueing, which may also apply to custom mailboxes. This list has been compiled carefully, but other problematic scenarios may have escaped our analysis. -How does Local Ordering relate to Network Ordering --------------------------------------------------- +### How does Local Ordering relate to Network Ordering The rule that *for a given pair of actors, messages sent directly from the first to the second will not be received out-of-order* holds for messages sent over the @@ -261,46 +251,44 @@ As explained in the previous section local message sends obey transitive causal ordering under certain conditions. This ordering can be violated due to different message delivery latencies. For example: - Actor ``A`` on node-1 sends message ``M1`` to actor ``C`` on node-3 +> +Actor `A` on node-1 sends message `M1` to actor `C` on node-3 +> +Actor `A` on node-1 then sends message `M2` to actor `B` on node-2 +> +Actor `B` on node-2 forwards message `M2` to actor `C` on node-3 +> +Actor `C` may receive `M1` and `M2` in any order - Actor ``A`` on node-1 then sends message ``M2`` to actor ``B`` on node-2 +It might take longer time for `M1` to "travel" to node-3 than it takes +for `M2` to "travel" to node-3 via node-2. - Actor ``B`` on node-2 forwards message ``M2`` to actor ``C`` on node-3 - - Actor ``C`` may receive ``M1`` and ``M2`` in any order - -It might take longer time for ``M1`` to "travel" to node-3 than it takes -for ``M2`` to "travel" to node-3 via node-2. - -Higher-level abstractions -========================= +## Higher-level abstractions Based on a small and consistent tool set in Akka's core, Akka also provides powerful, higher-level abstractions on top it. -Messaging Patterns ------------------- +### Messaging Patterns As discussed above a straight-forward answer to the requirement of reliable delivery is an explicit ACK–RETRY protocol. In its simplest form this requires -- a way to identify individual messages to correlate message with - acknowledgement -- a retry mechanism which will resend messages if not acknowledged in time -- a way for the receiver to detect and discard duplicates + * a way to identify individual messages to correlate message with +acknowledgement + * a retry mechanism which will resend messages if not acknowledged in time + * a way for the receiver to detect and discard duplicates The third becomes necessary by virtue of the acknowledgements not being guaranteed to arrive either. An ACK-RETRY protocol with business-level acknowledgements is -supported by :ref:`at-least-once-delivery-scala` of the Akka Persistence module. Duplicates can be -detected by tracking the identifiers of messages sent via :ref:`at-least-once-delivery-scala`. +supported by @ref:[At-Least-Once Delivery](../scala/persistence.md#at-least-once-delivery-scala) of the Akka Persistence module. Duplicates can be +detected by tracking the identifiers of messages sent via @ref:[At-Least-Once Delivery](../scala/persistence.md#at-least-once-delivery-scala). Another way of implementing the third part would be to make processing the messages idempotent on the level of the business logic. Another example of implementing all three requirements is shown at -:ref:`reliable-proxy` (which is now superseded by :ref:`at-least-once-delivery-scala`). + reliable-proxy (which is now superseded by @ref:[At-Least-Once Delivery](../scala/persistence.md#at-least-once-delivery-scala)). -Event Sourcing --------------- +### Event Sourcing Event sourcing (and sharding) is what makes large websites scale to billions of users, and the idea is quite simple: when a component (think actor) @@ -313,35 +301,31 @@ components may consume the event stream as a means to replicate the component’ state on a different continent or to react to changes). If the component’s state is lost—due to a machine failure or by being pushed out of a cache—it can easily be reconstructed by replaying the event stream (usually employing -snapshots to speed up the process). :ref:`event-sourcing-scala` is supported by +snapshots to speed up the process). @ref:[Event sourcing](../scala/persistence.md#event-sourcing-scala) is supported by Akka Persistence. -Mailbox with Explicit Acknowledgement -------------------------------------- +### Mailbox with Explicit Acknowledgement By implementing a custom mailbox type it is possible to retry message processing at the receiving actor’s end in order to handle temporary failures. This pattern is mostly useful in the local communication context where delivery guarantees are otherwise sufficient to fulfill the application’s requirements. -Please note that the caveats for `The Rules for In-JVM (Local) Message Sends`_ +Please note that the caveats for [The Rules for In-JVM (Local) Message Sends](#the-rules-for-in-jvm-local-message-sends) do apply. -An example implementation of this pattern is shown at :ref:`mailbox-acking`. +An example implementation of this pattern is shown at mailbox-acking. -.. _deadletters: - -Dead Letters -============ + +## Dead Letters Messages which cannot be delivered (and for which this can be ascertained) will -be delivered to a synthetic actor called ``/deadLetters``. This delivery +be delivered to a synthetic actor called `/deadLetters`. This delivery happens on a best-effort basis; it may fail even within the local JVM (e.g. during actor termination). Messages sent via unreliable network transports will be lost without turning up as dead letters. -What Should I Use Dead Letters For? ------------------------------------ +### What Should I Use Dead Letters For? The main use of this facility is for debugging, especially if an actor send does not arrive consistently (where usually inspecting the dead letters will @@ -357,11 +341,10 @@ The dead letter service follows the same rules with respect to delivery guarantees as all other message sends, hence it cannot be used to implement guaranteed delivery. -How do I Receive Dead Letters? ------------------------------- +### How do I Receive Dead Letters? -An actor can subscribe to class :class:`akka.actor.DeadLetter` on the event -stream, see :ref:`event-stream-java` (Java) or :ref:`event-stream-scala` +An actor can subscribe to class `akka.actor.DeadLetter` on the event +stream, see @ref:[Event Stream](../java/event-bus.md#event-stream-java) (Java) or @ref:[Event Stream](../scala/event-bus.md#event-stream-scala) (Scala) for how to do that. The subscribed actor will then receive all dead letters published in the (local) system from that point onwards. Dead letters are not propagated over the network, if you want to collect them in one place @@ -371,18 +354,13 @@ determine that a send operation is failed, which for a remote send can be the local system (if no network connection can be established) or the remote one (if the actor you are sending to does not exist at that point in time). -Dead Letters Which are (Usually) not Worrisome ----------------------------------------------- +### Dead Letters Which are (Usually) not Worrisome Every time an actor does not terminate by its own decision, there is a chance that some messages which it sends to itself are lost. There is one which happens quite easily in complex shutdown scenarios that is usually benign: -seeing a :class:`akka.dispatch.Terminate` message dropped means that two +seeing a `akka.dispatch.Terminate` message dropped means that two termination requests were given, but of course only one can succeed. In the -same vein, you might see :class:`akka.actor.Terminated` messages from children +same vein, you might see `akka.actor.Terminated` messages from children while stopping a hierarchy of actors turning up in dead letters if the parent -is still watching the child when the parent terminates. - -.. _Erlang documentation: http://www.erlang.org/faq/academic.html -.. _Nobody Needs Reliable Messaging: http://www.infoq.com/articles/no-reliable-messaging - +is still watching the child when the parent terminates. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/remoting.md b/akka-docs/src/main/paradox/general/remoting.md index b0b3da9029..5bdda971a8 100644 --- a/akka-docs/src/main/paradox/general/remoting.md +++ b/akka-docs/src/main/paradox/general/remoting.md @@ -1,15 +1,11 @@ -.. _remoting: - -Location Transparency -===================== +# Location Transparency The previous section describes how actor paths are used to enable location transparency. This special feature deserves some extra explanation, because the related term “transparent remoting” was used quite differently in the context of programming languages, platforms and technologies. -Distributed by Default ----------------------- +## Distributed by Default Everything in Akka is designed to work in a distributed setting: all interactions of actors use purely message passing and everything is @@ -17,18 +13,16 @@ asynchronous. This effort has been undertaken to ensure that all functions are available equally when running within a single JVM or on a cluster of hundreds of machines. The key for enabling this is to go from remote to local by way of optimization instead of trying to go from local to remote by way of -generalization. See `this classic paper -`_ +generalization. See [this classic paper](http://doc.akka.io/docs/misc/smli_tr-94-29.pdf) for a detailed discussion on why the second approach is bound to fail. -Ways in which Transparency is Broken ------------------------------------- +## Ways in which Transparency is Broken What is true of Akka need not be true of the application which uses it, since designing for distributed execution poses some restrictions on what is possible. The most obvious one is that all messages sent over the wire must be serializable. While being a little less obvious this includes closures which -are used as actor factories (i.e. within :class:`Props`) if the actor is to be +are used as actor factories (i.e. within `Props`) if the actor is to be created on a remote node. Another consequence is that everything needs to be aware of all interactions @@ -38,8 +32,7 @@ configuration). It also means that the probability for a message to be lost is much higher than within one JVM, where it is close to zero (still: no hard guarantee!). -How is Remoting Used? ---------------------- +## How is Remoting Used? We took the idea of transparency to the limit in that there is nearly no API for the remoting layer of Akka: it is purely driven by configuration. Just @@ -47,25 +40,23 @@ write your application according to the principles outlined in the previous sections, then specify remote deployment of actor sub-trees in the configuration file. This way, your application can be scaled out without having to touch the code. The only piece of the API which allows programmatic -influence on remote deployment is that :class:`Props` contain a field which may -be set to a specific :class:`Deploy` instance; this has the same effect as +influence on remote deployment is that `Props` contain a field which may +be set to a specific `Deploy` instance; this has the same effect as putting an equivalent deployment into the configuration file (if both are given, configuration file wins). -.. _symmetric-communication: - -Peer-to-Peer vs. Client-Server ------------------------------- + +## Peer-to-Peer vs. Client-Server Akka Remoting is a communication module for connecting actor systems in a peer-to-peer fashion, and it is the foundation for Akka Clustering. The design of remoting is driven by two (related) design decisions: -#. Communication between involved systems is symmetric: if a system A can connect to a system B - then system B must also be able to connect to system A independently. -#. The role of the communicating systems are symmetric in regards to connection patterns: there - is no system that only accepts connections, and there is no system that only initiates connections. - + 1. Communication between involved systems is symmetric: if a system A can connect to a system B +then system B must also be able to connect to system A independently. + 2. The role of the communicating systems are symmetric in regards to connection patterns: there +is no system that only accepts connections, and there is no system that only initiates connections. + The consequence of these decisions is that it is not possible to safely create pure client-server setups with predefined roles (violates assumption 2). For client-server setups it is better to use HTTP or Akka I/O. @@ -75,11 +66,9 @@ containers violates assumption 1, unless additional steps are taken in the network configuration to allow symmetric communication between involved systems. In such situations Akka can be configured to bind to a different network address than the one used for establishing connections between Akka nodes. -See :ref:`remote-configuration-nat`. +See @ref:[Akka behind NAT or in a Docker container](../scala/remoting.md#remote-configuration-nat). - -Marking Points for Scaling Up with Routers ------------------------------------------- +## Marking Points for Scaling Up with Routers In addition to being able to run different parts of an actor system on different nodes of a cluster, it is also possible to scale up onto more cores @@ -92,4 +81,4 @@ up a configurable number of children of the desired type and route to them in the configured fashion. Once such a router has been declared, its configuration can be freely overridden from the configuration file, including mixing it with the remote deployment of (some of) the children. Read more about -this in :ref:`Routing (Scala) ` and :ref:`Routing (Java) `. +this in @ref:[Routing (Scala)](../scala/routing.md) and @ref:[Routing (Java)](../java/routing.md). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/stream/stream-configuration.md b/akka-docs/src/main/paradox/general/stream/stream-configuration.md index 5257351bf5..4c0ca931a0 100644 --- a/akka-docs/src/main/paradox/general/stream/stream-configuration.md +++ b/akka-docs/src/main/paradox/general/stream/stream-configuration.md @@ -1,7 +1,3 @@ -.. _stream-config: +# Configuration -############# -Configuration -############# - -.. literalinclude:: ../../../../akka-stream/src/main/resources/reference.conf \ No newline at end of file +@@snip [reference.conf]../../../../../../akka-stream/src/main/resources/reference.conf) \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/stream/stream-design.md b/akka-docs/src/main/paradox/general/stream/stream-design.md index 548838eacf..a6e173675a 100644 --- a/akka-docs/src/main/paradox/general/stream/stream-design.md +++ b/akka-docs/src/main/paradox/general/stream/stream-design.md @@ -1,104 +1,105 @@ -.. _stream-design: - -Design Principles behind Akka Streams -===================================== +# Design Principles behind Akka Streams It took quite a while until we were reasonably happy with the look and feel of the API and the architecture of the implementation, and while being guided by intuition the design phase was very much exploratory research. This section details the findings and codifies them into a set of principles that have emerged during the process. -.. note:: +@@@ note - As detailed in the introduction keep in mind that the Akka Streams API is completely decoupled from the Reactive Streams interfaces which are just an implementation detail for how to pass stream data between individual processing stages. +As detailed in the introduction keep in mind that the Akka Streams API is completely decoupled from the Reactive Streams interfaces which are just an implementation detail for how to pass stream data between individual processing stages. -What shall users of Akka Streams expect? ----------------------------------------- +@@@ + +## What shall users of Akka Streams expect? Akka is built upon a conscious decision to offer APIs that are minimal and consistent—as opposed to easy or intuitive. The credo is that we favor explicitness over magic, and if we provide a feature then it must work always, no exceptions. Another way to say this is that we minimize the number of rules a user has to learn instead of trying to keep the rules close to what we think users might expect. From this follows that the principles implemented by Akka Streams are: - * all features are explicit in the API, no magic - * supreme compositionality: combined pieces retain the function of each part - * exhaustive model of the domain of distributed bounded stream processing +> + * all features are explicit in the API, no magic + * supreme compositionality: combined pieces retain the function of each part + * exhaustive model of the domain of distributed bounded stream processing This means that we provide all the tools necessary to express any stream processing topology, that we model all the essential aspects of this domain (back-pressure, buffering, transformations, failure recovery, etc.) and that whatever the user builds is reusable in a larger context. -Akka Streams does not send dropped stream elements to the dead letter office -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Akka Streams does not send dropped stream elements to the dead letter office One important consequence of offering only features that can be relied upon is the restriction that Akka Streams cannot ensure that all objects sent through a processing topology will be processed. Elements can be dropped for a number of reasons: - * plain user code can consume one element in a `map(...)` stage and produce an entirely different one as its result - * common stream operators drop elements intentionally, e.g. take/drop/filter/conflate/buffer/… - * stream failure will tear down the stream without waiting for processing to finish, all elements that are in flight will be discarded - * stream cancellation will propagate upstream (e.g. from a `take` operator) leading to upstream processing steps being terminated without having processed all of their inputs +> + * plain user code can consume one element in a *map(...)* stage and produce an entirely different one as its result + * common stream operators drop elements intentionally, e.g. take/drop/filter/conflate/buffer/… + * stream failure will tear down the stream without waiting for processing to finish, all elements that are in flight will be discarded + * stream cancellation will propagate upstream (e.g. from a *take* operator) leading to upstream processing steps being terminated without having processed all of their inputs This means that sending JVM objects into a stream that need to be cleaned up will require the user to ensure that this happens outside of the Akka Streams facilities (e.g. by cleaning them up after a timeout or when their results are observed on the stream output, or by using other means like finalizers etc.). -Resulting Implementation Constraints -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Resulting Implementation Constraints Compositionality entails reusability of partial stream topologies, which led us to the lifted approach of describing data flows as (partial) graphs that can act as composite sources, flows (a.k.a. pipes) and sinks of data. These building blocks shall then be freely shareable, with the ability to combine them freely to form larger graphs. The representation of these pieces must therefore be an immutable blueprint that is materialized in an explicit step in order to start the stream processing. The resulting stream processing engine is then also immutable in the sense of having a fixed topology that is prescribed by the blueprint. Dynamic networks need to be modeled by explicitly using the Reactive Streams interfaces for plugging different engines together. The process of materialization will often create specific objects that are useful to interact with the processing engine once it is running, for example for shutting it down or for extracting metrics. This means that the materialization function produces a result termed the *materialized value of a graph*. -Interoperation with other Reactive Streams implementations ----------------------------------------------------------- +## Interoperation with other Reactive Streams implementations -Akka Streams fully implement the Reactive Streams specification and interoperate with all other conformant implementations. We chose to completely separate the Reactive Streams interfaces from the user-level API because we regard them to be an SPI that is not targeted at endusers. In order to obtain a :class:`Publisher` or :class:`Subscriber` from an Akka Stream topology, a corresponding ``Sink.asPublisher`` or ``Source.asSubscriber`` element must be used. +Akka Streams fully implement the Reactive Streams specification and interoperate with all other conformant implementations. We chose to completely separate the Reactive Streams interfaces from the user-level API because we regard them to be an SPI that is not targeted at endusers. In order to obtain a `Publisher` or `Subscriber` from an Akka Stream topology, a corresponding `Sink.asPublisher` or `Source.asSubscriber` element must be used. -All stream Processors produced by the default materialization of Akka Streams are restricted to having a single Subscriber, additional Subscribers will be rejected. The reason for this is that the stream topologies described using our DSL never require fan-out behavior from the Publisher sides of the elements, all fan-out is done using explicit elements like :class:`Broadcast[T]`. +All stream Processors produced by the default materialization of Akka Streams are restricted to having a single Subscriber, additional Subscribers will be rejected. The reason for this is that the stream topologies described using our DSL never require fan-out behavior from the Publisher sides of the elements, all fan-out is done using explicit elements like `Broadcast[T]`. -This means that ``Sink.asPublisher(true)`` (for enabling fan-out support) must be used where broadcast behavior is needed for interoperation with other Reactive Streams implementations. +This means that `Sink.asPublisher(true)` (for enabling fan-out support) must be used where broadcast behavior is needed for interoperation with other Reactive Streams implementations. -What shall users of streaming libraries expect? ------------------------------------------------ +## What shall users of streaming libraries expect? We expect libraries to be built on top of Akka Streams, in fact Akka HTTP is one such example that lives within the Akka project itself. In order to allow users to profit from the principles that are described for Akka Streams above, the following rules are established: - * libraries shall provide their users with reusable pieces, i.e. expose factories that return graphs, allowing full compositionality - * libraries may optionally and additionally provide facilities that consume and materialize graphs +> + * libraries shall provide their users with reusable pieces, i.e. expose factories that return graphs, allowing full compositionality + * libraries may optionally and additionally provide facilities that consume and materialize graphs The reasoning behind the first rule is that compositionality would be destroyed if different libraries only accepted graphs and expected to materialize them: using two of these together would be impossible because materialization can only happen once. As a consequence, the functionality of a library must be expressed such that materialization can be done by the user, outside of the library’s control. -The second rule allows a library to additionally provide nice sugar for the common case, an example of which is the Akka HTTP API that provides a ``handleWith`` method for convenient materialization. +The second rule allows a library to additionally provide nice sugar for the common case, an example of which is the Akka HTTP API that provides a `handleWith` method for convenient materialization. -.. note:: +@@@ note - One important consequence of this is that a reusable flow description cannot be bound to “live” resources, any connection to or allocation of such resources must be deferred until materialization time. Examples of “live” resources are already existing TCP connections, a multicast Publisher, etc.; a TickSource does not fall into this category if its timer is created only upon materialization (as is the case for our implementation). +One important consequence of this is that a reusable flow description cannot be bound to “live” resources, any connection to or allocation of such resources must be deferred until materialization time. Examples of “live” resources are already existing TCP connections, a multicast Publisher, etc.; a TickSource does not fall into this category if its timer is created only upon materialization (as is the case for our implementation). - Exceptions from this need to be well-justified and carefully documented. +Exceptions from this need to be well-justified and carefully documented. -Resulting Implementation Constraints -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@@ + +### Resulting Implementation Constraints Akka Streams must enable a library to express any stream processing utility in terms of immutable blueprints. The most common building blocks are - * Source: something with exactly one output stream - * Sink: something with exactly one input stream - * Flow: something with exactly one input and one output stream - * BidiFlow: something with exactly two input streams and two output streams that conceptually behave like two Flows of opposite direction - * Graph: a packaged stream processing topology that exposes a certain set of input and output ports, characterized by an object of type :class:`Shape`. +> + * Source: something with exactly one output stream + * Sink: something with exactly one input stream + * Flow: something with exactly one input and one output stream + * BidiFlow: something with exactly two input streams and two output streams that conceptually behave like two Flows of opposite direction + * Graph: a packaged stream processing topology that exposes a certain set of input and output ports, characterized by an object of type `Shape`. -.. note:: +@@@ note - A source that emits a stream of streams is still just a normal Source, the kind of elements that are produced does not play a role in the static stream topology that is being expressed. +A source that emits a stream of streams is still just a normal Source, the kind of elements that are produced does not play a role in the static stream topology that is being expressed. -The difference between Error and Failure ----------------------------------------- +@@@ -The starting point for this discussion is the `definition given by the Reactive Manifesto `_. Translated to streams this means that an error is accessible within the stream as a normal data element, while a failure means that the stream itself has failed and is collapsing. In concrete terms, on the Reactive Streams interface level data elements (including errors) are signaled via ``onNext`` while failures raise the ``onError`` signal. +## The difference between Error and Failure -.. note:: +The starting point for this discussion is the [definition given by the Reactive Manifesto](http://www.reactivemanifesto.org/glossary#Failure). Translated to streams this means that an error is accessible within the stream as a normal data element, while a failure means that the stream itself has failed and is collapsing. In concrete terms, on the Reactive Streams interface level data elements (including errors) are signaled via `onNext` while failures raise the `onError` signal. - Unfortunately the method name for signaling *failure* to a Subscriber is called ``onError`` for historical reasons. Always keep in mind that the Reactive Streams interfaces (Publisher/Subscription/Subscriber) are modeling the low-level infrastructure for passing streams between execution units, and errors on this level are precisely the failures that we are talking about on the higher level that is modeled by Akka Streams. +@@@ note -There is only limited support for treating ``onError`` in Akka Streams compared to the operators that are available for the transformation of data elements, which is intentional in the spirit of the previous paragraph. Since ``onError`` signals that the stream is collapsing, its ordering semantics are not the same as for stream completion: transformation stages of any kind will just collapse with the stream, possibly still holding elements in implicit or explicit buffers. This means that data elements emitted before a failure can still be lost if the ``onError`` overtakes them. +Unfortunately the method name for signaling *failure* to a Subscriber is called `onError` for historical reasons. Always keep in mind that the Reactive Streams interfaces (Publisher/Subscription/Subscriber) are modeling the low-level infrastructure for passing streams between execution units, and errors on this level are precisely the failures that we are talking about on the higher level that is modeled by Akka Streams. + +@@@ + +There is only limited support for treating `onError` in Akka Streams compared to the operators that are available for the transformation of data elements, which is intentional in the spirit of the previous paragraph. Since `onError` signals that the stream is collapsing, its ordering semantics are not the same as for stream completion: transformation stages of any kind will just collapse with the stream, possibly still holding elements in implicit or explicit buffers. This means that data elements emitted before a failure can still be lost if the `onError` overtakes them. The ability for failures to propagate faster than data elements is essential for tearing down streams that are back-pressured—especially since back-pressure can be the failure mode (e.g. by tripping upstream buffers which then abort because they cannot do anything else; or if a dead-lock occurred). -The semantics of stream recovery -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### The semantics of stream recovery -A recovery element (i.e. any transformation that absorbs an ``onError`` signal and turns that into possibly more data elements followed normal stream completion) acts as a bulkhead that confines a stream collapse to a given region of the stream topology. Within the collapsed region buffered elements may be lost, but the outside is not affected by the failure. +A recovery element (i.e. any transformation that absorbs an `onError` signal and turns that into possibly more data elements followed normal stream completion) acts as a bulkhead that confines a stream collapse to a given region of the stream topology. Within the collapsed region buffered elements may be lost, but the outside is not affected by the failure. -This works in the same fashion as a ``try``–``catch`` expression: it marks a region in which exceptions are caught, but the exact amount of code that was skipped within this region in case of a failure might not be known precisely—the placement of statements matters. +This works in the same fashion as a `try`–`catch` expression: it marks a region in which exceptions are caught, but the exact amount of code that was skipped within this region in case of a failure might not be known precisely—the placement of statements matters. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/supervision.md b/akka-docs/src/main/paradox/general/supervision.md index 5c4752d1fc..ea82351da8 100644 --- a/akka-docs/src/main/paradox/general/supervision.md +++ b/akka-docs/src/main/paradox/general/supervision.md @@ -1,18 +1,13 @@ -.. _supervision: - -Supervision and Monitoring -========================== +# Supervision and Monitoring This chapter outlines the concept behind supervision, the primitives offered and their semantics. For details on how that translates into real code, please refer to the corresponding chapters for Scala and Java APIs. -.. _supervision-directives: + +## What Supervision Means -What Supervision Means ----------------------- - -As described in :ref:`actor-systems` supervision describes a dependency +As described in @ref:[Actor Systems](actor-systems.md) supervision describes a dependency relationship between actors: the supervisor delegates tasks to subordinates and therefore must respond to their failures. When a subordinate detects a failure (i.e. throws an exception), it suspends itself and all its subordinates and @@ -20,10 +15,10 @@ sends a message to its supervisor, signaling failure. Depending on the nature of the work to be supervised and the nature of the failure, the supervisor has a choice of the following four options: -#. Resume the subordinate, keeping its accumulated internal state -#. Restart the subordinate, clearing out its accumulated internal state -#. Stop the subordinate permanently -#. Escalate the failure, thereby failing itself + 1. Resume the subordinate, keeping its accumulated internal state + 2. Restart the subordinate, clearing out its accumulated internal state + 3. Stop the subordinate permanently + 4. Escalate the failure, thereby failing itself It is important to always view an actor as part of a supervision hierarchy, which explains the existence of the fourth choice (as a supervisor also is @@ -31,8 +26,8 @@ subordinate to another supervisor higher up) and has implications on the first three: resuming an actor resumes all its subordinates, restarting an actor entails restarting all its subordinates (but see below for more details), similarly terminating an actor will also terminate all its subordinates. It -should be noted that the default behavior of the :meth:`preRestart` hook of the -:class:`Actor` class is to terminate all its children before restarting, but +should be noted that the default behavior of the `preRestart` hook of the +`Actor` class is to terminate all its children before restarting, but this hook can be overridden; the recursive restart applies to all children left after this hook has been executed. @@ -55,93 +50,85 @@ actors cannot be orphaned or attached to supervisors from the outside, which might otherwise catch them unawares. In addition, this yields a natural and clean shutdown procedure for (sub-trees of) actor applications. -.. warning:: +@@@ warning - Supervision related parent-child communication happens by special system - messages that have their own mailboxes separate from user messages. This - implies that supervision related events are not deterministically - ordered relative to ordinary messages. In general, the user cannot influence - the order of normal messages and failure notifications. For details and - example see the :ref:`message-ordering` section. +Supervision related parent-child communication happens by special system +messages that have their own mailboxes separate from user messages. This +implies that supervision related events are not deterministically +ordered relative to ordinary messages. In general, the user cannot influence +the order of normal messages and failure notifications. For details and +example see the @ref:[Discussion: Message Ordering](message-delivery-reliability.md#message-ordering) section. -.. _toplevel-supervisors: +@@@ -The Top-Level Supervisors -------------------------- + +## The Top-Level Supervisors -.. image:: guardians.png - :align: center - :width: 360 +![guardians.png](guardians.png) An actor system will during its creation start at least three actors, shown in the image above. For more information about the consequences for actor paths -see :ref:`toplevel-paths`. +see @ref:[Top-Level Scopes for Actor Paths](addressing.md#toplevel-paths). -.. _user-guardian: - -``/user``: The Guardian Actor -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +### `/user`: The Guardian Actor The actor which is probably most interacted with is the parent of all -user-created actors, the guardian named ``"/user"``. Actors created using -``system.actorOf()`` are children of this actor. This means that when this +user-created actors, the guardian named `"/user"`. Actors created using +`system.actorOf()` are children of this actor. This means that when this guardian terminates, all normal actors in the system will be shutdown, too. It also means that this guardian’s supervisor strategy determines how the top-level normal actors are supervised. Since Akka 2.1 it is possible to -configure this using the setting ``akka.actor.guardian-supervisor-strategy``, +configure this using the setting `akka.actor.guardian-supervisor-strategy`, which takes the fully-qualified class-name of a -:class:`SupervisorStrategyConfigurator`. When the guardian escalates a failure, +`SupervisorStrategyConfigurator`. When the guardian escalates a failure, the root guardian’s response will be to terminate the guardian, which in effect will shut down the whole actor system. -``/system``: The System Guardian -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### `/system`: The System Guardian This special guardian has been introduced in order to achieve an orderly shut-down sequence where logging remains active while all normal actors terminate, even though logging itself is implemented using actors. This is realized by having the system guardian watch the user guardian and initiate its own -shut-down upon reception of the :class:`Terminated` message. The top-level +shut-down upon reception of the `Terminated` message. The top-level system actors are supervised using a strategy which will restart indefinitely -upon all types of :class:`Exception` except for -:class:`ActorInitializationException` and :class:`ActorKilledException`, which +upon all types of `Exception` except for +`ActorInitializationException` and `ActorKilledException`, which will terminate the child in question. All other throwables are escalated, which will shut down the whole actor system. -``/``: The Root Guardian -^^^^^^^^^^^^^^^^^^^^^^^^ +### `/`: The Root Guardian The root guardian is the grand-parent of all so-called “top-level” actors and -supervises all the special actors mentioned in :ref:`toplevel-paths` using the -``SupervisorStrategy.stoppingStrategy``, whose purpose is to terminate the -child upon any type of :class:`Exception`. All other throwables will be +supervises all the special actors mentioned in @ref:[Top-Level Scopes for Actor Paths](addressing.md#toplevel-paths) using the +`SupervisorStrategy.stoppingStrategy`, whose purpose is to terminate the +child upon any type of `Exception`. All other throwables will be escalated … but to whom? Since every real actor has a supervisor, the supervisor of the root guardian cannot be a real actor. And because this means that it is “outside of the bubble”, it is called the “bubble-walker”. This is a -synthetic :class:`ActorRef` which in effect stops its child upon the first sign -of trouble and sets the actor system’s ``isTerminated`` status to ``true`` as +synthetic `ActorRef` which in effect stops its child upon the first sign +of trouble and sets the actor system’s `isTerminated` status to `true` as soon as the root guardian is fully terminated (all children recursively stopped). -.. _supervision-restart: - -What Restarting Means ---------------------- + +## What Restarting Means When presented with an actor which failed while processing a certain message, causes for the failure fall into three categories: -* Systematic (i.e. programming) error for the specific message received -* (Transient) failure of some external resource used during processing the message -* Corrupt internal state of the actor + * Systematic (i.e. programming) error for the specific message received + * (Transient) failure of some external resource used during processing the message + * Corrupt internal state of the actor Unless the failure is specifically recognizable, the third cause cannot be ruled out, which leads to the conclusion that the internal state needs to be cleared out. If the supervisor decides that its other children or itself is not affected by the corruption—e.g. because of conscious application of the error kernel pattern—it is therefore best to restart the child. This is carried out -by creating a new instance of the underlying :class:`Actor` class and replacing -the failed instance with the fresh one inside the child’s :class:`ActorRef`; +by creating a new instance of the underlying `Actor` class and replacing +the failed instance with the fresh one inside the child’s `ActorRef`; the ability to do this is one of the reasons for encapsulating actors within special references. The new actor then resumes processing its mailbox, meaning that the restart is not visible outside of the actor itself with the notable @@ -150,26 +137,27 @@ re-processed. The precise sequence of events during a restart is the following: -#. suspend the actor (which means that it will not process normal messages until - resumed), and recursively suspend all children -#. call the old instance’s :meth:`preRestart` hook (defaults to sending - termination requests to all children and calling :meth:`postStop`) -#. wait for all children which were requested to terminate (using - ``context.stop()``) during :meth:`preRestart` to actually terminate; - this—like all actor operations—is non-blocking, the termination notice from - the last killed child will effect the progression to the next step -#. create new actor instance by invoking the originally provided factory again -#. invoke :meth:`postRestart` on the new instance (which by default also calls :meth:`preStart`) -#. send restart request to all children which were not killed in step 3; - restarted children will follow the same process recursively, from step 2 -#. resume the actor + 1. suspend the actor (which means that it will not process normal messages until +resumed), and recursively suspend all children + 2. call the old instance’s `preRestart` hook (defaults to sending +termination requests to all children and calling `postStop`) + 3. wait for all children which were requested to terminate (using +`context.stop()`) during `preRestart` to actually terminate; +this—like all actor operations—is non-blocking, the termination notice from +the last killed child will effect the progression to the next step + 4. create new actor instance by invoking the originally provided factory again + 5. invoke `postRestart` on the new instance (which by default also calls `preStart`) + 6. send restart request to all children which were not killed in step 3; +restarted children will follow the same process recursively, from step 2 + 7. resume the actor -What Lifecycle Monitoring Means -------------------------------- +## What Lifecycle Monitoring Means -.. note:: +@@@ note - Lifecycle Monitoring in Akka is usually referred to as ``DeathWatch`` +Lifecycle Monitoring in Akka is usually referred to as `DeathWatch` + +@@@ In contrast to the special relationship between parent and child described above, each actor may monitor any other actor. Since actors emerge from @@ -179,12 +167,12 @@ from alive to dead. Monitoring is thus used to tie one actor to another so that it may react to the other actor’s termination, in contrast to supervision which reacts to failure. -Lifecycle monitoring is implemented using a :class:`Terminated` message to be +Lifecycle monitoring is implemented using a `Terminated` message to be received by the monitoring actor, where the default behavior is to throw a -special :class:`DeathPactException` if not otherwise handled. In order to start -listening for :class:`Terminated` messages, invoke -``ActorContext.watch(targetActorRef)``. To stop listening, invoke -``ActorContext.unwatch(targetActorRef)``. One important property is that the +special `DeathPactException` if not otherwise handled. In order to start +listening for `Terminated` messages, invoke +`ActorContext.watch(targetActorRef)`. To stop listening, invoke +`ActorContext.unwatch(targetActorRef)`. One important property is that the message will be delivered irrespective of the order in which the monitoring request and target’s termination occur, i.e. you still get the message even if at the time of registration the target is already dead. @@ -196,81 +184,81 @@ them or schedule itself to retry this at a later time. Another common use case is that an actor needs to fail in the absence of an external resource, which may also be one of its own children. If a third party -terminates a child by way of the ``system.stop(child)`` method or sending a -:class:`PoisonPill`, the supervisor might well be affected. +terminates a child by way of the `system.stop(child)` method or sending a +`PoisonPill`, the supervisor might well be affected. -.. _backoff-supervisor: + +### Delayed restarts with the BackoffSupervisor pattern -Delayed restarts with the BackoffSupervisor pattern -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Provided as a built-in pattern the ``akka.pattern.BackoffSupervisor`` implements the so-called +Provided as a built-in pattern the `akka.pattern.BackoffSupervisor` implements the so-called *exponential backoff supervision strategy*, starting a child actor again when it fails, each time with a growing time delay between restarts. -This pattern is useful when the started actor fails [#]_ because some external resource is not available, +This pattern is useful when the started actor fails [1] because some external resource is not available, and we need to give it some time to start-up again. One of the prime examples when this is useful is -when a :ref:`PersistentActor ` fails (by stopping) with a persistence failure - which indicates that +when a @ref:[PersistentActor](../scala/persistence.md) fails (by stopping) with a persistence failure - which indicates that the database may be down or overloaded, in such situations it makes most sense to give it a little bit of time to recover before the peristent actor is started. -.. [#] A failure can be indicated in two different ways; by an actor stopping or crashing. +> [1] A failure can be indicated in two different ways; by an actor stopping or crashing. The following Scala snippet shows how to create a backoff supervisor which will start the given echo actor after it has stopped because of a failure, in increasing intervals of 3, 6, 12, 24 and finally 30 seconds: -.. includecode:: ../scala/code/docs/pattern/BackoffSupervisorDocSpec.scala#backoff-stop +@@snip [BackoffSupervisorDocSpec.scala](../scala/code/docs/pattern/BackoffSupervisorDocSpec.scala) { #backoff-stop } The above is equivalent to this Java code: -.. includecode:: ../java/code/jdocs/pattern/BackoffSupervisorDocTest.java#backoff-imports -.. includecode:: ../java/code/jdocs/pattern/BackoffSupervisorDocTest.java#backoff-stop +@@snip [BackoffSupervisorDocTest.java](../java/code/jdocs/pattern/BackoffSupervisorDocTest.java) { #backoff-imports } -Using a ``randomFactor`` to add a little bit of additional variance to the backoff intervals +@@snip [BackoffSupervisorDocTest.java](../java/code/jdocs/pattern/BackoffSupervisorDocTest.java) { #backoff-stop } + +Using a `randomFactor` to add a little bit of additional variance to the backoff intervals is highly recommended, in order to avoid multiple actors re-start at the exact same point in time, for example because they were stopped due to a shared resource such as a database going down and re-starting after the same configured interval. By adding additional randomness to the re-start intervals the actors will start in slightly different points in time, thus avoiding large spikes of traffic hitting the recovering shared database or other resource that they all need to contact. -The ``akka.pattern.BackoffSupervisor`` actor can also be configured to restart the actor after a delay when the actor +The `akka.pattern.BackoffSupervisor` actor can also be configured to restart the actor after a delay when the actor crashes and the supervision strategy decides that it should restart. The following Scala snippet shows how to create a backoff supervisor which will start the given echo actor after it has crashed because of some exception, in increasing intervals of 3, 6, 12, 24 and finally 30 seconds: -.. includecode:: ../scala/code/docs/pattern/BackoffSupervisorDocSpec.scala#backoff-fail +@@snip [BackoffSupervisorDocSpec.scala](../scala/code/docs/pattern/BackoffSupervisorDocSpec.scala) { #backoff-fail } The above is equivalent to this Java code: -.. includecode:: ../java/code/jdocs/pattern/BackoffSupervisorDocTest.java#backoff-imports -.. includecode:: ../java/code/jdocs/pattern/BackoffSupervisorDocTest.java#backoff-fail +@@snip [BackoffSupervisorDocTest.java](../java/code/jdocs/pattern/BackoffSupervisorDocTest.java) { #backoff-imports } -The ``akka.pattern.BackoffOptions`` can be used to customize the behavior of the back-off supervisor actor, below are some examples: +@@snip [BackoffSupervisorDocTest.java](../java/code/jdocs/pattern/BackoffSupervisorDocTest.java) { #backoff-fail } -.. includecode:: ../scala/code/docs/pattern/BackoffSupervisorDocSpec.scala#backoff-custom-stop +The `akka.pattern.BackoffOptions` can be used to customize the behavior of the back-off supervisor actor, below are some examples: -The above code sets up a back-off supervisor that requires the child actor to send a ``akka.pattern.BackoffSupervisor.Reset`` message +@@snip [BackoffSupervisorDocSpec.scala](../scala/code/docs/pattern/BackoffSupervisorDocSpec.scala) { #backoff-custom-stop } + +The above code sets up a back-off supervisor that requires the child actor to send a `akka.pattern.BackoffSupervisor.Reset` message to its parent when a message is successfully processed, resetting the back-off. It also uses a default stopping strategy, any exception will cause the child to stop. -.. includecode:: ../scala/code/docs/pattern/BackoffSupervisorDocSpec.scala#backoff-custom-fail +@@snip [BackoffSupervisorDocSpec.scala](../scala/code/docs/pattern/BackoffSupervisorDocSpec.scala) { #backoff-custom-fail } The above code sets up a back-off supervisor that restarts the child after back-off if MyException is thrown, any other exception will be escalated. The back-off is automatically reset if the child does not throw any errors within 10 seconds. -One-For-One Strategy vs. All-For-One Strategy ---------------------------------------------- +## One-For-One Strategy vs. All-For-One Strategy There are two classes of supervision strategies which come with Akka: -:class:`OneForOneStrategy` and :class:`AllForOneStrategy`. Both are configured +`OneForOneStrategy` and `AllForOneStrategy`. Both are configured with a mapping from exception type to supervision directive (see -:ref:`above `) and limits on how often a child is allowed to fail +[above](#supervision-directives)) and limits on how often a child is allowed to fail before terminating it. The difference between them is that the former applies the obtained directive only to the failed child, whereas the latter applies it to all siblings as well. Normally, you should use the -:class:`OneForOneStrategy`, which also is the default if none is specified +`OneForOneStrategy`, which also is the default if none is specified explicitly. -The :class:`AllForOneStrategy` is applicable in cases where the ensemble of +The `AllForOneStrategy` is applicable in cases where the ensemble of children has such tight dependencies among them, that a failure of one child affects the function of the others, i.e. they are inextricably linked. Since a restart does not clear out the mailbox, it often is best to terminate the children @@ -281,15 +269,14 @@ but processed afterwards. Normally stopping a child (i.e. not in response to a failure) will not automatically terminate the other children in an all-for-one strategy; this can -easily be done by watching their lifecycle: if the :class:`Terminated` message -is not handled by the supervisor, it will throw a :class:`DeathPactException` +easily be done by watching their lifecycle: if the `Terminated` message +is not handled by the supervisor, it will throw a `DeathPactException` which (depending on its supervisor) will restart it, and the default -:meth:`preRestart` action will terminate all children. Of course this can be +`preRestart` action will terminate all children. Of course this can be handled explicitly as well. Please note that creating one-off actors from an all-for-one supervisor entails that failures escalated by the temporary actor will affect all the permanent ones. If this is not desired, install an intermediate supervisor; this can very easily be done by declaring a router of size 1 for the worker, see -:ref:`routing-scala` or :ref:`routing-java`. - +@ref:[Routing](../scala/routing.md) or @ref:[Routing](../java/routing.md). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/general/terminology.md b/akka-docs/src/main/paradox/general/terminology.md index 81593649a0..9a16c816bc 100644 --- a/akka-docs/src/main/paradox/general/terminology.md +++ b/akka-docs/src/main/paradox/general/terminology.md @@ -1,22 +1,17 @@ -.. _terminology: - -Terminology, Concepts -===================== +# Terminology, Concepts In this chapter we attempt to establish a common terminology to define a solid ground for communicating about concurrent, distributed systems which Akka targets. Please note that, for many of these terms, there is no single agreed definition. We simply seek to give working definitions that will be used in the scope of the Akka documentation. -Concurrency vs. Parallelism ---------------------------- +## Concurrency vs. Parallelism Concurrency and parallelism are related concepts, but there are small differences. *Concurrency* means that two or more tasks are making progress even though they might not be executing simultaneously. This can for example be realized with time slicing where parts of tasks are executed sequentially and mixed with parts of other tasks. *Parallelism* on the other hand arise when the execution can be truly simultaneous. -Asynchronous vs. Synchronous ----------------------------- +## Asynchronous vs. Synchronous A method call is considered *synchronous* if the caller cannot make progress until the method returns a value or throws an exception. On the other hand, an *asynchronous* call allows the caller to progress after a finite number of steps, and @@ -28,8 +23,7 @@ might give a similar behavior as blocking. In general, it is preferred to use as the system is able to progress. Actors are asynchronous by nature: an actor can progress after a message send without waiting for the actual delivery to happen. -Non-blocking vs. Blocking -------------------------- +## Non-blocking vs. Blocking We talk about *blocking* if the delay of one thread can indefinitely delay some of the other threads. A good example is a resource which can be used exclusively by one thread using mutual exclusion. If a thread holds on to the resource @@ -39,8 +33,7 @@ In contrast, *non-blocking* means that no thread is able to indefinitely delay o Non-blocking operations are preferred to blocking ones, as the overall progress of the system is not trivially guaranteed when it contains blocking operations. -Deadlock vs. Starvation vs. Live-lock -------------------------------------- +## Deadlock vs. Starvation vs. Live-lock *Deadlock* arises when several participants are waiting on each other to reach a specific state to be able to progress. As none of them can progress without some other participant to reach a certain state (a "Catch-22" problem) all affected @@ -59,31 +52,31 @@ check if the other needs the resource, too. If the resource is requested by the the other instance of the resource. In the unfortunate case it might happen that the two participants "bounce" between the two resources, never acquiring it, but always yielding to the other. -Race Condition --------------- +## Race Condition We call it a *Race condition* when an assumption about the ordering of a set of events might be violated by external non-deterministic effects. Race conditions often arise when multiple threads have a shared mutable state, and the operations of thread on the state might be interleaved causing unexpected behavior. While this is a common case, shared state is not necessary to have race conditions. One example could be a client sending unordered packets (e.g UDP -datagrams) ``P1``, ``P2`` to a server. As the packets might potentially travel via different network routes, it is possible that -the server receives ``P2`` first and ``P1`` afterwards. If the messages contain no information about their sending order it is +datagrams) `P1`, `P2` to a server. As the packets might potentially travel via different network routes, it is possible that +the server receives `P2` first and `P1` afterwards. If the messages contain no information about their sending order it is impossible to determine by the server that they were sent in a different order. Depending on the meaning of the packets this can cause race conditions. -.. note:: - The only guarantee that Akka provides about messages sent between a given pair of actors is that their order is - always preserved. see :ref:`message-delivery-reliability` +@@@ note -Non-blocking Guarantees (Progress Conditions) ---------------------------------------------- +The only guarantee that Akka provides about messages sent between a given pair of actors is that their order is +always preserved. see @ref:[Message Delivery Reliability](message-delivery-reliability.md) + +@@@ + +## Non-blocking Guarantees (Progress Conditions) As discussed in the previous sections blocking is undesirable for several reasons, including the dangers of deadlocks and reduced throughput in the system. In the following sections we discuss various non-blocking properties with different strength. -Wait-freedom -............ +### Wait-freedom A method is *wait-free* if every call is guaranteed to finish in a finite number of steps. If a method is *bounded wait-free* then the number of steps has a finite upper bound. @@ -92,16 +85,14 @@ From this definition it follows that wait-free methods are never blocking, there Additionally, as each participant can progress after a finite number of steps (when the call finishes), wait-free methods are free of starvation. -Lock-freedom -............ +### Lock-freedom *Lock-freedom* is a weaker property than *wait-freedom*. In the case of lock-free calls, infinitely often some method finishes in a finite number of steps. This definition implies that no deadlock is possible for lock-free calls. On the other hand, the guarantee that *some call finishes* in a finite number of steps is not enough to guarantee that *all of them eventually finish*. In other words, lock-freedom is not enough to guarantee the lack of starvation. -Obstruction-freedom -................... +### Obstruction-freedom *Obstruction-freedom* is the weakest non-blocking guarantee discussed here. A method is called *obstruction-free* if there is a point in time after which it executes in isolation (other threads make no steps, e.g.: become suspended), it @@ -113,9 +104,8 @@ tries to execute its operation on the shared object, but if a participant detect the modifications, and tries again according to some schedule. If there is a point in time, where one of the participants is the only one trying, the operation will succeed. -Recommended literature ----------------------- +## Recommended literature +> * The Art of Multiprocessor Programming, M. Herlihy and N Shavit, 2008. ISBN 978-0123705914 - * Java Concurrency in Practice, B. Goetz, T. Peierls, J. Bloch, J. Bowbeer, D. Holmes and D. Lea, 2006. ISBN 978-0321349606 - + * Java Concurrency in Practice, B. Goetz, T. Peierls, J. Bloch, J. Bowbeer, D. Holmes and D. Lea, 2006. ISBN 978-0321349606 \ No newline at end of file diff --git a/akka-docs/src/main/paradox/images/faulttolerancesample-failure-flow.png b/akka-docs/src/main/paradox/images/faulttolerancesample-failure-flow.png old mode 100755 new mode 100644 diff --git a/akka-docs/src/main/paradox/images/faulttolerancesample.graffle b/akka-docs/src/main/paradox/images/faulttolerancesample.graffle old mode 100755 new mode 100644 diff --git a/akka-docs/src/main/paradox/index.md b/akka-docs/src/main/paradox/index.md index 5b7dc2df2e..13fb3d140a 100644 --- a/akka-docs/src/main/paradox/index.md +++ b/akka-docs/src/main/paradox/index.md @@ -1,8 +1,11 @@ -# Akka Documentation +# Contents + +@@toc { depth=1 } @@@ index -* [Beginners Guide](guide/index.md) -* [Reference](reference/index.md) +* [guide](guide/index.md) +* [java](java.md) +* [scala](scala.md) @@@ diff --git a/akka-docs/src/main/paradox/index.rst b/akka-docs/src/main/paradox/index.rst deleted file mode 100644 index 300ed680fc..0000000000 --- a/akka-docs/src/main/paradox/index.rst +++ /dev/null @@ -1,8 +0,0 @@ -Contents -======== - -.. toctree:: - :maxdepth: 1 - - java - scala diff --git a/akka-docs/src/main/paradox/intro/deployment-scenarios.md b/akka-docs/src/main/paradox/intro/deployment-scenarios.md index 90b2007f56..2176d1f14c 100644 --- a/akka-docs/src/main/paradox/intro/deployment-scenarios.md +++ b/akka-docs/src/main/paradox/intro/deployment-scenarios.md @@ -1,53 +1,38 @@ -.. _deployment-scenarios: +# Use-case and Deployment Scenarios -################################### - Use-case and Deployment Scenarios -################################### - -How can I use and deploy Akka? -============================== +## How can I use and deploy Akka? Akka can be used in different ways: -- As a library: used as a regular JAR on the classpath and/or in a web app, to - be put into ``WEB-INF/lib`` + * As a library: used as a regular JAR on the classpath and/or in a web app, to +be put into `WEB-INF/lib` + * As an application packaged with [sbt-native-packager](https://github.com/sbt/sbt-native-packager) + * As an application packaged and deployed using [Lightbend ConductR](http://www.lightbend.com/products/conductr). -- As an application packaged with `sbt-native-packager `_ +## Native Packager -- As an application packaged and deployed using `Lightbend ConductR `_. - - -Native Packager -=============== - -`sbt-native-packager `_ is a tool for creating +[sbt-native-packager](https://github.com/sbt/sbt-native-packager) is a tool for creating distributions of any type of application, including Akka applications. -Define sbt version in ``project/build.properties`` file: +Define sbt version in `project/build.properties` file: -.. code-block:: none +```none +sbt.version=0.13.13 +``` - sbt.version=0.13.13 +Add [sbt-native-packager](https://github.com/sbt/sbt-native-packager) in `project/plugins.sbt` file: -Add `sbt-native-packager `_ in ``project/plugins.sbt`` file: +```none +addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.1.5") +``` -.. code-block:: none +Follow the instructions for the `JavaAppPackaging` in the [sbt-native-packager plugin documentation](http://sbt-native-packager.readthedocs.io/en/latest/archetypes/java_app/index.html). - addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.1.5") +## In a Docker container -Follow the instructions for the ``JavaAppPackaging`` in the `sbt-native-packager plugin documentation`_. - -.. _sbt-native-packager plugin documentation: http://sbt-native-packager.readthedocs.io/en/latest/archetypes/java_app/index.html - - -In a Docker container -===================== You can use both Akka remoting and Akka Cluster inside of Docker containers. But note that you will need to take special care with the network configuration when using Docker, -described here: :ref:`remote-configuration-nat` +described here: @ref:[Akka behind NAT or in a Docker container](../scala/remoting.md#remote-configuration-nat) For an example of how to set up a project using Akka Cluster and Docker take a look at the -`"akka-docker-cluster" sample`__. - -__ https://github.com/muuki88/activator-akka-docker - +["akka-docker-cluster" sample](https://github.com/muuki88/activator-akka-docker). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/intro/getting-started.md b/akka-docs/src/main/paradox/intro/getting-started.md index 03711a5592..c1951e7352 100644 --- a/akka-docs/src/main/paradox/intro/getting-started.md +++ b/akka-docs/src/main/paradox/intro/getting-started.md @@ -1,258 +1,244 @@ -Getting Started -=============== +# Getting Started -Prerequisites -------------- +## Prerequisites -Akka requires that you have `Java 8 `_ or +Akka requires that you have [Java 8](http://www.oracle.com/technetwork/java/javase/downloads/index.html) or later installed on your machine. -`Lightbend Inc. `_ provides a commercial build of Akka and related projects such as Scala or Play -as part of the `Lightbend Reactive Platform `_ which is made available +[Lightbend Inc.](http://www.lightbend.com) provides a commercial build of Akka and related projects such as Scala or Play +as part of the [Lightbend Reactive Platform](http://www.lightbend.com/platform) which is made available for Java 6 in case your project can not upgrade to Java 8 just yet. It also includes additional commercial features or libraries. -Download --------- +## Download There are several ways to download Akka. You can download it as part of the Lightbend Platform (as described above). You can download the full distribution, which includes all modules. Or you can use a build tool like Maven or SBT to download dependencies from the Akka Maven repository. -Modules -------- +## Modules Akka is very modular and consists of several JARs containing different features. -- ``akka-actor`` – Classic Actors, Typed Actors, IO Actor etc. - -- ``akka-agent`` – Agents, integrated with Scala STM - -- ``akka-camel`` – Apache Camel integration - -- ``akka-cluster`` – Cluster membership management, elastic routers. - -- ``akka-cluster-sharding`` – Cluster Sharding of actors. - -- ``akka-cluster-tools`` – Additoinal Cluster utilities, such as Singleton, Pub/Sub and Client. - -- ``akka-distributed-data`` – Cluster data with CRDTs. - -- ``akka-osgi`` – Utilities for using Akka in OSGi containers - -- ``akka-osgi-aries`` – Aries blueprint for provisioning actor systems - -- ``akka-remote`` – Remote Actors - -- ``akka-slf4j`` – SLF4J Logger (event bus listener) - -- ``akka-stream`` – Reactive stream processing - -- ``akka-testkit`` – Toolkit for testing Actor systems + * `akka-actor` – Classic Actors, Typed Actors, IO Actor etc. + * `akka-agent` – Agents, integrated with Scala STM + * `akka-camel` – Apache Camel integration + * `akka-cluster` – Cluster membership management, elastic routers. + * `akka-cluster-sharding` – Cluster Sharding of actors. + * `akka-cluster-tools` – Additoinal Cluster utilities, such as Singleton, Pub/Sub and Client. + * `akka-distributed-data` – Cluster data with CRDTs. + * `akka-osgi` – Utilities for using Akka in OSGi containers + * `akka-osgi-aries` – Aries blueprint for provisioning actor systems + * `akka-remote` – Remote Actors + * `akka-slf4j` – SLF4J Logger (event bus listener) + * `akka-stream` – Reactive stream processing + * `akka-testkit` – Toolkit for testing Actor systems In addition to these stable modules there are several which are on their way -into the stable core but are still marked :ref:`may change ` at this point. This +into the stable core but are still marked @ref:[may change](../common/may-change.md) at this point. This does not mean that they do not function as intended, it primarily means that their API has not yet solidified enough in order to be considered frozen. You can help accelerating this process by giving feedback on these modules on our mailing list. -- ``akka-contrib`` – an assortment of contributions which may or may not be - moved into core modules, see :ref:`akka-contrib` for more details. + * `akka-contrib` – an assortment of contributions which may or may not be +moved into core modules, see akka-contrib for more details. -The filename of the actual JAR is for example ``@jarName@`` (and analog for +The filename of the actual JAR is for example `@jarName@` (and analog for the other modules). How to see the JARs dependencies of each Akka module is described in the -:ref:`dependencies` section. + dependencies section. -Using a release distribution ----------------------------- +## Using a release distribution -Download the release you need from http://akka.io/downloads and unzip it. +Download the release you need from [http://akka.io/downloads](http://akka.io/downloads) and unzip it. -Using a snapshot version ------------------------- +## Using a snapshot version -The Akka nightly snapshots are published to http://repo.akka.io/snapshots/ and are -versioned with both ``SNAPSHOT`` and timestamps. You can choose a timestamped +The Akka nightly snapshots are published to [http://repo.akka.io/snapshots](http://repo.akka.io/snapshots)/ and are +versioned with both `SNAPSHOT` and timestamps. You can choose a timestamped version to work with and can decide when to update to a newer version. -.. warning:: +@@@ warning - The use of Akka SNAPSHOTs, nightlies and milestone releases is discouraged unless you know what you are doing. +The use of Akka SNAPSHOTs, nightlies and milestone releases is discouraged unless you know what you are doing. -.. _build-tool: +@@@ -Using a build tool ------------------- + +## Using a build tool Akka can be used with build tools that support Maven repositories. -Maven repositories ------------------- +## Maven repositories For Akka version 2.1-M2 and onwards: -`Maven Central `_ +[Maven Central](https://repo1.maven.org/maven2/) For previous Akka versions: -`Akka Repo `_ +[Akka Repo](http://repo.akka.io/releases/) -Using Akka with Maven ---------------------- +## Using Akka with Maven The simplest way to get started with Akka and Maven is to download the ready to run sample -named `Akka Main in Java <@exampleCodeService@/akka-samples-main-java>`_. +named [Akka Main in Java](@exampleCodeService@/akka-samples-main-java). Since Akka is published to Maven Central (for versions since 2.1-M2), it is enough to add the Akka dependencies to the POM. For example, here is the dependency for akka-actor: -.. code-block:: xml - - - com.typesafe.akka - akka-actor_@binVersion@ - @version@ - +```xml + + com.typesafe.akka + akka-actor_@binVersion@ + @version@ + +``` For snapshot versions, the snapshot repository needs to be added as well: -.. code-block:: xml +```xml + + + akka-snapshots + + true + + http://repo.akka.io/snapshots/ + + +``` - - - akka-snapshots - - true - - http://repo.akka.io/snapshots/ - - +**Note**: for snapshot versions both `SNAPSHOT` and timestamped versions are published. -**Note**: for snapshot versions both ``SNAPSHOT`` and timestamped versions are published. +## Using Akka with SBT +The simplest way to get started with Akka and SBT is to use a [Gitter8](http://www.foundweekends.org/giter8/) template +named [Hello Akka!](https://github.com/akka/hello-akka.g8). If you have *sbt* already installed, you can create a project +from this template by running: -Using Akka with SBT -------------------- - -The simplest way to get started with Akka and SBT is to use a `Gitter8 `_ template -named `Hello Akka! `_. If you have `sbt` already installed, you can create a project -from this template by running:: - - sbt new akka/hello-akka.g8 +``` +sbt new akka/hello-akka.g8 +``` Summary of the essential parts for using Akka with SBT: -SBT installation instructions on `http://www.scala-sbt.org/release/tutorial/Setup.html `_ +SBT installation instructions on [http://www.scala-sbt.org/release/tutorial/Setup.html](http://www.scala-sbt.org/release/tutorial/Setup.html) -``build.sbt`` file: +`build.sbt` file: -.. parsed-literal:: +``` +name := "My Project" - name := "My Project" +version := "1.0" - version := "1.0" +scalaVersion := "@scalaVersion@" - scalaVersion := "@scalaVersion@" - - libraryDependencies += - "com.typesafe.akka" %% "akka-actor" % "@version@" @crossString@ +libraryDependencies += + "com.typesafe.akka" %% "akka-actor" % "@version@" @crossString@``` **Note**: the libraryDependencies setting above is specific to SBT v0.12.x and higher. If you are using an older version of SBT, the libraryDependencies should look like this: -.. parsed-literal:: - - libraryDependencies += - "com.typesafe.akka" % "akka-actor_@binVersion@" % "@version@" +``` +libraryDependencies += + "com.typesafe.akka" % " +@ref:[akka-actor](../general/configuration.md#akka-actor) +@binVersion +@" % "@version@"``` For snapshot versions, the snapshot repository needs to be added as well: -.. parsed-literal:: +``` +resolvers += "Akka Snapshot Repository" at " +[http://repo.akka.io/snapshots](http://repo.akka.io/snapshots) +/"``` - resolvers += "Akka Snapshot Repository" at "http://repo.akka.io/snapshots/" +## Using Akka with Gradle +Requires at least [Gradle](https://gradle.org) 1.4 +Uses the [Scala plugin](http://www.gradle.org/docs/current/userguide/scala_plugin.html) -Using Akka with Gradle ----------------------- +``` +apply plugin: 'scala' -Requires at least `Gradle `_ 1.4 -Uses the `Scala plugin `_ +repositories { + mavenCentral() +} -.. parsed-literal:: +dependencies { + compile 'org.scala-lang:scala-library:@ +[scalaVersion@](mailto:scalaVersion@) +' +} - apply plugin: 'scala' +tasks.withType(ScalaCompile) { + scalaCompileOptions.useAnt = false +} - repositories { - mavenCentral() - } - - dependencies { - compile 'org.scala-lang:scala-library:@scalaVersion@' - } - - tasks.withType(ScalaCompile) { - scalaCompileOptions.useAnt = false - } - - dependencies { - compile group: 'com.typesafe.akka', name: 'akka-actor_@binVersion@', version: '@version@' - compile group: 'org.scala-lang', name: 'scala-library', version: '@scalaVersion@' - } +dependencies { + compile group: 'com.typesafe.akka', name: ' +@ref:[akka-actor](../general/configuration.md#akka-actor) +@ +[binVersion@](mailto:binVersion@) +', version: ' +[@version](mailto:@version) +@ +' + compile group: 'org.scala-lang', name: 'scala-library', version: ' +[@scalaVersion](mailto:@scalaVersion) +@ +' +}``` For snapshot versions, the snapshot repository needs to be added as well: -.. parsed-literal:: +``` +repositories { + mavenCentral() + maven { + url " +[http://repo.akka.io/snapshots](http://repo.akka.io/snapshots) +/" + } +}``` - repositories { - mavenCentral() - maven { - url "http://repo.akka.io/snapshots/" - } - } +## Using Akka with Eclipse +Setup SBT project and then use [sbteclipse](https://github.com/typesafehub/sbteclipse) to generate an Eclipse project. -Using Akka with Eclipse ------------------------ +## Using Akka with IntelliJ IDEA -Setup SBT project and then use `sbteclipse `_ to generate an Eclipse project. +Setup SBT project and then use [sbt-idea](https://github.com/mpeltonen/sbt-idea) to generate an IntelliJ IDEA project. -Using Akka with IntelliJ IDEA ------------------------------ +## Using Akka with NetBeans -Setup SBT project and then use `sbt-idea `_ to generate an IntelliJ IDEA project. +Setup SBT project and then use [nbsbt](https://github.com/dcaoyuan/nbsbt) to generate a NetBeans project. -Using Akka with NetBeans ------------------------- +You should also use [nbscala](https://github.com/dcaoyuan/nbscala) for general scala support in the IDE. -Setup SBT project and then use `nbsbt `_ to generate a NetBeans project. +## Do not use -optimize Scala compiler flag -You should also use `nbscala `_ for general scala support in the IDE. +@@@ warning -Do not use -optimize Scala compiler flag ----------------------------------------- +Akka has not been compiled or tested with -optimize Scala compiler flag. +Strange behavior has been reported by users that have tried it. -.. warning:: +@@@ - Akka has not been compiled or tested with -optimize Scala compiler flag. - Strange behavior has been reported by users that have tried it. +## Build from sources +Akka uses Git and is hosted at [Github](https://github.com). -Build from sources ------------------- + * Akka: clone the Akka repository from [https://github.com/akka/akka](https://github.com/akka/akka) -Akka uses Git and is hosted at `Github `_. +Continue reading the page on @ref:[Building Akka](../dev/building-akka.md) -* Akka: clone the Akka repository from ``_ +## Need help? -Continue reading the page on :ref:`building-akka` +If you have questions you can get help on the [Akka Mailing List](https://groups.google.com/group/akka-user). -Need help? ----------- +You can also ask for [commercial support](https://www.lightbend.com). -If you have questions you can get help on the `Akka Mailing List `_. - -You can also ask for `commercial support `_. - -Thanks for being a part of the Akka community. +Thanks for being a part of the Akka community. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/intro/index-java.md b/akka-docs/src/main/paradox/intro/index-java.md index a826e59bb2..52ddd5c3fe 100644 --- a/akka-docs/src/main/paradox/intro/index-java.md +++ b/akka-docs/src/main/paradox/intro/index-java.md @@ -1,13 +1,14 @@ -Introduction -============ +# Introduction -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - what-is-akka - why-akka - getting-started - ../java/hello-world - deployment-scenarios - use-cases +@@@ index +* [what-is-akka](what-is-akka.md) +* [why-akka](why-akka.md) +* [getting-started](getting-started.md) +* [../java/hello-world](../java/hello-world.md) +* [deployment-scenarios](deployment-scenarios.md) +* [use-cases](use-cases.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/intro/index-scala.md b/akka-docs/src/main/paradox/intro/index-scala.md index 7cf8450db1..7467a5dc7a 100644 --- a/akka-docs/src/main/paradox/intro/index-scala.md +++ b/akka-docs/src/main/paradox/intro/index-scala.md @@ -1,13 +1,14 @@ -Introduction -============ +# Introduction -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - what-is-akka - why-akka - getting-started - ../scala/hello-world - deployment-scenarios - use-cases +@@@ index +* [what-is-akka](what-is-akka.md) +* [why-akka](why-akka.md) +* [getting-started](getting-started.md) +* [../scala/hello-world](../scala/hello-world.md) +* [deployment-scenarios](deployment-scenarios.md) +* [use-cases](use-cases.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/intro/use-cases.md b/akka-docs/src/main/paradox/intro/use-cases.md index d5f493483b..c0f1d60d02 100644 --- a/akka-docs/src/main/paradox/intro/use-cases.md +++ b/akka-docs/src/main/paradox/intro/use-cases.md @@ -1,9 +1,4 @@ - -.. _use-cases: - -################################ - Examples of use-cases for Akka -################################ +# Examples of use-cases for Akka We see Akka being adopted by many large organizations in a big range of industries all from investment and merchant banking, retail and social media, simulation, @@ -12,48 +7,56 @@ and much more. Any system that have the need for high-throughput and low latency is a good candidate for using Akka. There is a great discussion on use-cases for Akka with some good write-ups by production -users `here `_ +users [here](http://stackoverflow.com/questions/4493001/good-use-case-for-akka/4494512#4494512) -Here are some of the areas where Akka is being deployed into production -======================================================================= +## Here are some of the areas where Akka is being deployed into production -Transaction processing (Online Gaming, Finance/Banking, Trading, Statistics, Betting, Social Media, Telecom) ------------------------------------------------------------------------------------------------------------- - Scale up, scale out, fault-tolerance / HA +### Transaction processing (Online Gaming, Finance/Banking, Trading, Statistics, Betting, Social Media, Telecom) -Service backend (any industry, any app) ---------------------------------------- - Service REST, SOAP, Cometd, WebSockets etc - Act as message hub / integration layer - Scale up, scale out, fault-tolerance / HA +> +Scale up, scale out, fault-tolerance / HA -Concurrency/parallelism (any app) ---------------------------------- - Correct - Simple to work with and understand - Just add the jars to your existing JVM project (use Scala, Java, Groovy or JRuby) +### Service backend (any industry, any app) -Simulation ----------- - Master/Worker, Compute Grid, MapReduce etc. +> +Service REST, SOAP, Cometd, WebSockets etc +Act as message hub / integration layer +Scale up, scale out, fault-tolerance / HA -Batch processing (any industry) -------------------------------- - Camel integration to hook up with batch data sources - Actors divide and conquer the batch workloads +### Concurrency/parallelism (any app) -Communications Hub (Telecom, Web media, Mobile media) ------------------------------------------------------ - Scale up, scale out, fault-tolerance / HA +> +Correct +Simple to work with and understand +Just add the jars to your existing JVM project (use Scala, Java, Groovy or JRuby) -Gaming and Betting (MOM, online gaming, betting) ------------------------------------------------- - Scale up, scale out, fault-tolerance / HA +### Simulation -Business Intelligence/Data Mining/general purpose crunching ------------------------------------------------------------ - Scale up, scale out, fault-tolerance / HA +> +Master/Worker, Compute Grid, MapReduce etc. -Complex Event Stream Processing -------------------------------- - Scale up, scale out, fault-tolerance / HA +### Batch processing (any industry) + +> +Camel integration to hook up with batch data sources +Actors divide and conquer the batch workloads + +### Communications Hub (Telecom, Web media, Mobile media) + +> +Scale up, scale out, fault-tolerance / HA + +### Gaming and Betting (MOM, online gaming, betting) + +> +Scale up, scale out, fault-tolerance / HA + +### Business Intelligence/Data Mining/general purpose crunching + +> +Scale up, scale out, fault-tolerance / HA + +### Complex Event Stream Processing + +> +Scale up, scale out, fault-tolerance / HA \ No newline at end of file diff --git a/akka-docs/src/main/paradox/intro/what-is-akka.md b/akka-docs/src/main/paradox/intro/what-is-akka.md index cf9d1c4ece..f5f5bfd796 100644 --- a/akka-docs/src/main/paradox/intro/what-is-akka.md +++ b/akka-docs/src/main/paradox/intro/what-is-akka.md @@ -1,8 +1,4 @@ -.. _what-is-akka: - -############### - What is Akka? -############### +# What is Akka? **«resilient elastic distributed real-time transaction processing»** @@ -10,8 +6,8 @@ We believe that writing correct distributed, concurrent, fault-tolerant and scal applications is too hard. Most of the time it's because we are using the wrong tools and the wrong level of abstraction. Akka is here to change that. Using the Actor Model we raise the abstraction level and provide a better platform to -build scalable, resilient and responsive applications—see the `Reactive -Manifesto `_ for more details. For +build scalable, resilient and responsive applications—see the [Reactive +Manifesto](http://reactivemanifesto.org/) for more details. For fault-tolerance we adopt the "let it crash" model which the telecom industry has used with great success to build applications that self-heal and systems that never stop. Actors also provide the abstraction for transparent @@ -19,61 +15,53 @@ distribution and the basis for truly scalable and fault-tolerant applications. Akka is Open Source and available under the Apache 2 License. -Download from http://akka.io/downloads. +Download from [http://akka.io/downloads](http://akka.io/downloads). Please note that all code samples compile, so if you want direct access to the sources, have a look -over at the Akka Docs subproject on github: for `Java <@github@/akka-docs/rst/java/code/docs>`_ -and `Scala <@github@/akka-docs/rst/scala/code/docs>`_. +over at the Akka Docs subproject on github: for [Java](@github@/akka-docs/rst/java/code/docs) +and [Scala](@github@/akka-docs/rst/scala/code/docs). +## Akka implements a unique hybrid -Akka implements a unique hybrid -=============================== - -Actors ------- +### Actors Actors give you: -- Simple and high-level abstractions for distribution, concurrency and parallelism. -- Asynchronous, non-blocking and highly performant message-driven programming model. -- Very lightweight event-driven processes (several million actors per GB of heap memory). + * Simple and high-level abstractions for distribution, concurrency and parallelism. + * Asynchronous, non-blocking and highly performant message-driven programming model. + * Very lightweight event-driven processes (several million actors per GB of heap memory). -See the chapter for :ref:`Scala ` or :ref:`Java `. +See the chapter for @ref:[Scala](../scala/actors.md) or @ref:[Java](../java/actors.md). -Fault Tolerance ---------------- +### Fault Tolerance -- Supervisor hierarchies with "let-it-crash" semantics. -- Actor systems can span over multiple JVMs to provide truly fault-tolerant systems. -- Excellent for writing highly fault-tolerant systems that self-heal and never stop. + * Supervisor hierarchies with "let-it-crash" semantics. + * Actor systems can span over multiple JVMs to provide truly fault-tolerant systems. + * Excellent for writing highly fault-tolerant systems that self-heal and never stop. -See :ref:`Fault Tolerance (Scala) ` and :ref:`Fault Tolerance (Java) `. +See @ref:[Fault Tolerance (Scala)](../scala/fault-tolerance.md) and @ref:[Fault Tolerance (Java)](../java/fault-tolerance.md). + +### Location Transparency -Location Transparency ---------------------- Everything in Akka is designed to work in a distributed environment: all interactions of actors use pure message passing and everything is asynchronous. -For an overview of the cluster support see the :ref:`Java ` -and :ref:`Scala ` documentation chapters. +For an overview of the cluster support see the @ref:[Java](../java/cluster-usage.md) +and @ref:[Scala](../scala/cluster-usage.md) documentation chapters. -Persistence ------------ +### Persistence State changes experienced by an actor can optionally be persisted and replayed when the actor is started or restarted. This allows actors to recover their state, even after JVM crashes or when being migrated to another node. -You can find more details in the respective chapter for :ref:`Java ` or :ref:`Scala `. +You can find more details in the respective chapter for @ref:[Java](../java/persistence.md) or @ref:[Scala](../scala/persistence.md). -Scala and Java APIs -=================== +## Scala and Java APIs -Akka has both a :ref:`scala-api` and a :ref:`java-api`. +Akka has both a @ref:[Scala Documentation](../scala.md) and a @ref:[Java Documentation](../java.md). - -Akka can be used in different ways -================================== +## Akka can be used in different ways Akka is a toolkit, not a framework: you integrate it into your build like any other library without having to follow a particular source code layout. When expressing your systems as collaborating @@ -82,16 +70,11 @@ there is a natural separation between business logic and inter-component communi Akka applications are typically deployed as follows: -- as a library: used as a regular JAR on the classpath or in a web app. + * as a library: used as a regular JAR on the classpath or in a web app. + * packaged with [sbt-native-packager](https://github.com/sbt/sbt-native-packager). + * packaged and deployed using [Lightbend ConductR](http://www.lightbend.com/products/conductr). -- packaged with `sbt-native-packager `_. - -- packaged and deployed using `Lightbend ConductR `_. - -Commercial Support -================== +## Commercial Support Akka is available from Lightbend Inc. under a commercial license which includes -development or production support, read more `here -`_. - +development or production support, read more [here](http://www.lightbend.com/how/subscription). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/intro/why-akka.md b/akka-docs/src/main/paradox/intro/why-akka.md index 66325df8ee..120bd84c2a 100644 --- a/akka-docs/src/main/paradox/intro/why-akka.md +++ b/akka-docs/src/main/paradox/intro/why-akka.md @@ -1,16 +1,14 @@ -Why Akka? -========= +# Why Akka? -What features can the Akka platform offer, over the competition? ----------------------------------------------------------------- +## What features can the Akka platform offer, over the competition? Akka provides scalable real-time transaction processing. Akka is a unified runtime and programming model for: -- Scale up (Concurrency) -- Scale out (Remoting) -- Fault tolerance + * Scale up (Concurrency) + * Scale out (Remoting) + * Fault tolerance One thing to learn and admin, with high cohesion and coherent semantics. @@ -25,20 +23,18 @@ provides outstanding performance even if you're only running it on one machine. Akka also supplies a wide array of concurrency-paradigms, allowing users to choose the right tool for the job. - -What's a good use-case for Akka? --------------------------------- +## What's a good use-case for Akka? We see Akka being adopted by many large organizations in a big range of industries: -- Investment and Merchant Banking -- Retail -- Social Media -- Simulation -- Gaming and Betting -- Automobile and Traffic Systems -- Health Care -- Data Analytics + * Investment and Merchant Banking + * Retail + * Social Media + * Simulation + * Gaming and Betting + * Automobile and Traffic Systems + * Health Care + * Data Analytics and much more. Any system with the need for high-throughput and low latency is a good candidate for using Akka. @@ -48,6 +44,6 @@ strategies, timeouts and processing-isolation), as well as both horizontal and vertical scalability (add more cores and/or add more machines). Here's what some of the Akka users have to say about how they are using Akka: -http://stackoverflow.com/questions/4493001/good-use-case-for-akka +[http://stackoverflow.com/questions/4493001/good-use-case-for-akka](http://stackoverflow.com/questions/4493001/good-use-case-for-akka) -All this in the ApacheV2-licensed open source project. +All this in the ApacheV2-licensed open source project. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java.md b/akka-docs/src/main/paradox/java.md index e7f8349d10..da40c62c37 100644 --- a/akka-docs/src/main/paradox/java.md +++ b/akka-docs/src/main/paradox/java.md @@ -1,23 +1,22 @@ -.. _java-api: +# Java Documentation -Java Documentation -================== +@@toc { depth=2 } -.. toctree:: - :maxdepth: 2 +@@@ index - security/index - intro/index-java - general/index - java/index-actors - java/index-futures - java/index-network - java/index-utilities - java/stream/index - java/http/index - java/howto - java/scala-compat - dev/index - project/index - additional/index +* [security/index](security/index.md) +* [intro/index-java](intro/index-java.md) +* [general/index](general/index.md) +* [java/index-actors](java/index-actors.md) +* [java/index-futures](java/index-futures.md) +* [java/index-network](java/index-network.md) +* [java/index-utilities](java/index-utilities.md) +* [java/stream/index](java/stream/index.md) +* [java/http/index](java/http/index.md) +* [java/howto](java/howto.md) +* [java/scala-compat](java/scala-compat.md) +* [dev/index](dev/index.md) +* [project/index](project/index.md) +* [additional/index](additional/index.md) +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/actors.md b/akka-docs/src/main/paradox/java/actors.md index 320143da98..5a0e2227c5 100644 --- a/akka-docs/src/main/paradox/java/actors.md +++ b/akka-docs/src/main/paradox/java/actors.md @@ -1,10 +1,6 @@ -.. _actors-java: +# Actors -######## - Actors -######## - -The `Actor Model`_ provides a higher level of abstraction for writing concurrent +The [Actor Model](http://en.wikipedia.org/wiki/Actor_model) provides a higher level of abstraction for writing concurrent and distributed systems. It alleviates the developer from having to deal with explicit locking and thread management, making it easier to write correct concurrent and parallel systems. Actors were defined in the 1973 paper by Carl @@ -15,316 +11,302 @@ systems. The API of Akka’s Actors is similar to Scala Actors which has borrowed some of its syntax from Erlang. -.. _Actor Model: http://en.wikipedia.org/wiki/Actor_model +## Creating Actors -Creating Actors -=============== +@@@ note -.. note:: +Since Akka enforces parental supervision every actor is supervised and +(potentially) the supervisor of its children, it is advisable that you +familiarize yourself with @ref:[Actor Systems](../general/actor-systems.md) and supervision and it +may also help to read @ref:[Actor References, Paths and Addresses](../general/addressing.md). - Since Akka enforces parental supervision every actor is supervised and - (potentially) the supervisor of its children, it is advisable that you - familiarize yourself with :ref:`actor-systems` and :ref:`supervision` and it - may also help to read :ref:`addressing`. +@@@ -Defining an Actor class ------------------------ +### Defining an Actor class -Actor classes are implemented by extending the :class:`AbstractActor` class and setting -the “initial behavior” in the constructor by calling the :meth:`receive` method in -the :class:`AbstractActor`. +Actor classes are implemented by extending the `AbstractActor` class and setting +the “initial behavior” in the constructor by calling the `receive` method in +the `AbstractActor`. -The argument to the :meth:`receive` method is a ``PartialFunction`` +The argument to the `receive` method is a `PartialFunction` that defines which messages your Actor can handle, along with the implementation of how the messages should be processed. Don't let the type signature scare you. To allow you to easily build up a partial -function there is a builder named ``ReceiveBuilder`` that you can use. +function there is a builder named `ReceiveBuilder` that you can use. Here is an example: -.. includecode:: code/jdocs/actor/MyActor.java - :include: imports,my-actor +@@snip [MyActor.java](code/jdocs/actor/MyActor.java) { #imports #my-actor } -Please note that the Akka Actor ``receive`` message loop is exhaustive, which +Please note that the Akka Actor `receive` message loop is exhaustive, which is different compared to Erlang and the late Scala Actors. This means that you need to provide a pattern match for all messages that it can accept and if you want to be able to handle unknown messages then you need to have a default case -as in the example above. Otherwise an ``akka.actor.UnhandledMessage(message, -sender, recipient)`` will be published to the ``ActorSystem``'s -``EventStream``. +as in the example above. Otherwise an `akka.actor.UnhandledMessage(message, +sender, recipient)` will be published to the `ActorSystem`'s +`EventStream`. -Note further that the return type of the behavior defined above is ``Unit``; if +Note further that the return type of the behavior defined above is `Unit`; if the actor shall reply to the received message then this must be done explicitly as explained below. -The argument to the :meth:`receive` method is a partial function object, which is -stored within the actor as its “initial behavior”, see `Become/Unbecome`_ for +The argument to the `receive` method is a partial function object, which is +stored within the actor as its “initial behavior”, see [Become/Unbecome](#become-unbecome) for further information on changing the behavior of an actor after its construction. -Props ------ +### Props -:class:`Props` is a configuration class to specify options for the creation +`Props` is a configuration class to specify options for the creation of actors, think of it as an immutable and thus freely shareable recipe for creating an actor including associated deployment information (e.g. which dispatcher to use, see more below). Here are some examples of how to create a -:class:`Props` instance. +`Props` instance. -.. includecode:: code/jdocs/actor/ActorDocTest.java#import-props -.. includecode:: code/jdocs/actor/ActorDocTest.java#creating-props +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #import-props } + +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #creating-props } The second variant shows how to pass constructor arguments to the -:class:`Actor` being created, but it should only be used outside of actors as +`Actor` being created, but it should only be used outside of actors as explained below. The last line shows a possibility to pass constructor arguments regardless of the context it is being used in. The presence of a matching constructor is -verified during construction of the :class:`Props` object, resulting in an -:class:`IllegalArgumentException` if no or multiple matching constructors are +verified during construction of the `Props` object, resulting in an +`IllegalArgumentException` if no or multiple matching constructors are found. -Dangerous Variants -^^^^^^^^^^^^^^^^^^ +#### Dangerous Variants -.. includecode:: code/jdocs/actor/ActorDocTest.java#creating-props-deprecated +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #creating-props-deprecated } This method is not recommended to be used within another actor because it encourages to close over the enclosing scope, resulting in non-serializable -:class:`Props` and possibly race conditions (breaking the actor encapsulation). -On the other hand using this variant in a :class:`Props` factory in the actor’s +`Props` and possibly race conditions (breaking the actor encapsulation). +On the other hand using this variant in a `Props` factory in the actor’s companion object as documented under “Recommended Practices” below is completely fine. There were two use-cases for these methods: passing constructor arguments to the actor—which is solved by the newly introduced -:meth:`Props.create(clazz, args)` method above or the recommended practice +`Props.create(clazz, args)` method above or the recommended practice below—and creating actors “on the spot” as anonymous classes. The latter should be solved by making these actors named classes instead (if they are not -declared within a top-level ``object`` then the enclosing instance’s ``this`` +declared within a top-level `object` then the enclosing instance’s `this` reference needs to be passed as the first argument). -.. warning:: +@@@ warning - Declaring one actor within another is very dangerous and breaks actor - encapsulation. Never pass an actor’s ``this`` reference into :class:`Props`! +Declaring one actor within another is very dangerous and breaks actor +encapsulation. Never pass an actor’s `this` reference into `Props`! -Recommended Practices -^^^^^^^^^^^^^^^^^^^^^ +@@@ + +#### Recommended Practices It is a good idea to provide factory methods on the companion object of each -:class:`Actor` which help keeping the creation of suitable :class:`Props` as +`Actor` which help keeping the creation of suitable `Props` as close to the actor definition as possible. This also avoids the pitfalls -associated with using the ``Props.create(...)`` method which takes a by-name +associated with using the `Props.create(...)` method which takes a by-name argument, since within a companion object the given code block will not retain a reference to its enclosing scope: -.. includecode:: code/jdocs/actor/ActorDocTest.java#props-factory +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #props-factory } Another good practice is to declare what messages an Actor can receive as close to the actor definition as possible (e.g. as static classes inside the Actor or using other suitable class), which makes it easier to know what it can receive. -.. includecode:: code/jdocs/actor/ActorDocTest.java#messages-in-companion +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #messages-in-companion } -Creating Actors with Props --------------------------- +### Creating Actors with Props -Actors are created by passing a :class:`Props` instance into the -:meth:`actorOf` factory method which is available on :class:`ActorSystem` and -:class:`ActorContext`. +Actors are created by passing a `Props` instance into the +`actorOf` factory method which is available on `ActorSystem` and +`ActorContext`. -.. includecode:: code/jdocs/actor/ActorDocTest.java#import-actorRef -.. includecode:: code/jdocs/actor/ActorDocTest.java#system-actorOf +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #import-actorRef } -Using the :class:`ActorSystem` will create top-level actors, supervised by the +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #system-actorOf } + +Using the `ActorSystem` will create top-level actors, supervised by the actor system’s provided guardian actor, while using an actor’s context will create a child actor. -.. includecode:: code/jdocs/actor/ActorDocTest.java#context-actorOf - :exclude: plus-some-behavior +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #context-actorOf } It is recommended to create a hierarchy of children, grand-children and so on such that it fits the logical failure-handling structure of the application, -see :ref:`actor-systems`. +see @ref:[Actor Systems](../general/actor-systems.md). -The call to :meth:`actorOf` returns an instance of :class:`ActorRef`. This is a +The call to `actorOf` returns an instance of `ActorRef`. This is a handle to the actor instance and the only way to interact with it. The -:class:`ActorRef` is immutable and has a one to one relationship with the Actor -it represents. The :class:`ActorRef` is also serializable and network-aware. +`ActorRef` is immutable and has a one to one relationship with the Actor +it represents. The `ActorRef` is also serializable and network-aware. This means that you can serialize it, send it over the wire and use it on a remote host and it will still be representing the same Actor on the original node, across the network. The name parameter is optional, but you should preferably name your actors, since that is used in log messages and for identifying actors. The name must -not be empty or start with ``$``, but it may contain URL encoded characters -(eg. ``%20`` for a blank space). If the given name is already in use by -another child to the same parent an `InvalidActorNameException` is thrown. +not be empty or start with `$`, but it may contain URL encoded characters +(eg. `%20` for a blank space). If the given name is already in use by +another child to the same parent an *InvalidActorNameException* is thrown. Actors are automatically started asynchronously when created. -.. _actor-create-factory-java: - -Dependency Injection --------------------- + +### Dependency Injection If your actor has a constructor that takes parameters then those need to -be part of the :class:`Props` as well, as described `above`__. But there +be part of the `Props` as well, as described [above](Props_). But there are cases when a factory method must be used, for example when the actual constructor arguments are determined by a dependency injection framework. -__ Props_ +@@snip [DependencyInjectionDocTest.java](code/jdocs/actor/DependencyInjectionDocTest.java) { #import } -.. includecode:: code/jdocs/actor/DependencyInjectionDocTest.java#import -.. includecode:: code/jdocs/actor/DependencyInjectionDocTest.java - :include: creating-indirectly - :exclude: obtain-fresh-Actor-instance-from-DI-framework +@@snip [DependencyInjectionDocTest.java](code/jdocs/actor/DependencyInjectionDocTest.java) { #creating-indirectly } -.. warning:: +@@@ warning - You might be tempted at times to offer an :class:`IndirectActorProducer` - which always returns the same instance, e.g. by using a static field. This is - not supported, as it goes against the meaning of an actor restart, which is - described here: :ref:`supervision-restart`. +You might be tempted at times to offer an `IndirectActorProducer` +which always returns the same instance, e.g. by using a static field. This is +not supported, as it goes against the meaning of an actor restart, which is +described here: @ref:[What Restarting Means](../general/supervision.md#supervision-restart). - When using a dependency injection framework, actor beans *MUST NOT* have - singleton scope. +When using a dependency injection framework, actor beans *MUST NOT* have +singleton scope. + +@@@ Techniques for dependency injection and integration with dependency injection frameworks are described in more depth in the -`Using Akka with Dependency Injection `_ -guideline and the `Akka Java Spring `_ tutorial. +[Using Akka with Dependency Injection](http://letitcrash.com/post/55958814293/akka-dependency-injection) +guideline and the [Akka Java Spring](https://github.com/typesafehub/activator-akka-java-spring) tutorial. -The Inbox ---------- +### The Inbox When writing code outside of actors which shall communicate with actors, the -``ask`` pattern can be a solution (see below), but there are two things it -cannot do: receiving multiple replies (e.g. by subscribing an :class:`ActorRef` +`ask` pattern can be a solution (see below), but there are two things it +cannot do: receiving multiple replies (e.g. by subscribing an `ActorRef` to a notification service) and watching other actors’ lifecycle. For these -purposes there is the :class:`Inbox` class: +purposes there is the `Inbox` class: -.. includecode:: code/jdocs/actor/InboxDocTest.java#inbox +@@snip [InboxDocTest.java](code/jdocs/actor/InboxDocTest.java) { #inbox } -The :meth:`send` method wraps a normal :meth:`tell` and supplies the internal +The `send` method wraps a normal `tell` and supplies the internal actor’s reference as the sender. This allows the reply to be received on the last line. Watching an actor is quite simple as well: -.. includecode:: code/jdocs/actor/InboxDocTest.java#watch +@@snip [InboxDocTest.java](code/jdocs/actor/InboxDocTest.java) { #watch } -Actor API -========= +## Actor API -The :class:`AbstractActor` class defines a method called :meth:`receive`, +The `AbstractActor` class defines a method called `receive`, that is used to set the “initial behavior” of the actor. If the current actor behavior does not match a received message, -:meth:`unhandled` is called, which by default publishes an -``akka.actor.UnhandledMessage(message, sender, recipient)`` on the actor +`unhandled` is called, which by default publishes an +`akka.actor.UnhandledMessage(message, sender, recipient)` on the actor system’s event stream (set configuration item -``akka.actor.debug.unhandled`` to ``on`` to have them converted into +`akka.actor.debug.unhandled` to `on` to have them converted into actual Debug messages). In addition, it offers: -* :meth:`getSelf()` reference to the :class:`ActorRef` of the actor - -* :meth:`getSender()` reference sender Actor of the last received message, typically used as described in :ref:`LambdaActor.Reply` - -* :meth:`supervisorStrategy()` user overridable definition the strategy to use for supervising child actors - - This strategy is typically declared inside the actor in order to have access - to the actor’s internal state within the decider function: since failure is - communicated as a message sent to the supervisor and processed like other - messages (albeit outside of the normal behavior), all values and variables - within the actor are available, as is the ``sender`` reference (which will - be the immediate child reporting the failure; if the original failure - occurred within a distant descendant it is still reported one level up at a - time). - -* :meth:`getContext()` exposes contextual information for the actor and the current message, such as: - - * factory methods to create child actors (:meth:`actorOf`) - * system that the actor belongs to - * parent supervisor - * supervised children - * lifecycle monitoring - * hotswap behavior stack as described in :ref:`actor-hotswap-java` + * `getSelf()` reference to the `ActorRef` of the actor + * `getSender()` reference sender Actor of the last received message, typically used as described in [LambdaActor.Reply](#lambdaactor-reply) + * + `supervisorStrategy()` user overridable definition the strategy to use for supervising child actors + This strategy is typically declared inside the actor in order to have access +to the actor’s internal state within the decider function: since failure is +communicated as a message sent to the supervisor and processed like other +messages (albeit outside of the normal behavior), all values and variables +within the actor are available, as is the `sender` reference (which will +be the immediate child reporting the failure; if the original failure +occurred within a distant descendant it is still reported one level up at a +time). + * + `getContext()` exposes contextual information for the actor and the current message, such as: + * factory methods to create child actors (`actorOf`) + * system that the actor belongs to + * parent supervisor + * supervised children + * lifecycle monitoring + * hotswap behavior stack as described in [Become/Unbecome](#actor-hotswap-java) The remaining visible methods are user-overridable life-cycle hooks which are described in the following: -.. includecode:: code/jdocs/actor/ActorDocTest.java#lifecycle-callbacks +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #lifecycle-callbacks } -The implementations shown above are the defaults provided by the :class:`AbstractActor` +The implementations shown above are the defaults provided by the `AbstractActor` class. -Actor Lifecycle ---------------- +### Actor Lifecycle -.. image:: ../images/actor_lifecycle.png - :align: center - :width: 680 +![actor_lifecycle.png](../images/actor_lifecycle.png) A path in an actor system represents a "place" which might be occupied by a living actor. Initially (apart from system initialized actors) a path is -empty. When ``actorOf()`` is called it assigns an *incarnation* of the actor -described by the passed ``Props`` to the given path. An actor incarnation is -identified by the path *and a UID*. A restart only swaps the ``Actor`` -instance defined by the ``Props`` but the incarnation and hence the UID remains +empty. When `actorOf()` is called it assigns an *incarnation* of the actor +described by the passed `Props` to the given path. An actor incarnation is +identified by the path *and a UID*. A restart only swaps the `Actor` +instance defined by the `Props` but the incarnation and hence the UID remains the same. The lifecycle of an incarnation ends when the actor is stopped. At that point the appropriate lifecycle events are called and watching actors are notified of the termination. After the incarnation is stopped, the path can -be reused again by creating an actor with ``actorOf()``. In this case the +be reused again by creating an actor with `actorOf()`. In this case the name of the new incarnation will be the same as the previous one but the UIDs will differ. -.. note:: +@@@ note - It is important to note that Actors do not stop automatically when no longer - referenced, every Actor that is created must also explicitly be destroyed. - The only simplification is that stopping a parent Actor will also recursively - stop all the child Actors that this parent has created. +It is important to note that Actors do not stop automatically when no longer +referenced, every Actor that is created must also explicitly be destroyed. +The only simplification is that stopping a parent Actor will also recursively +stop all the child Actors that this parent has created. -An ``ActorRef`` always represents an incarnation (path and UID) not just a +@@@ + +An `ActorRef` always represents an incarnation (path and UID) not just a given path. Therefore if an actor is stopped and a new one with the same -name is created an ``ActorRef`` of the old incarnation will not point +name is created an `ActorRef` of the old incarnation will not point to the new one. -``ActorSelection`` on the other hand points to the path (or multiple paths +`ActorSelection` on the other hand points to the path (or multiple paths if wildcards are used) and is completely oblivious to which incarnation is currently -occupying it. ``ActorSelection`` cannot be watched for this reason. It is -possible to resolve the current incarnation's ``ActorRef`` living under the -path by sending an ``Identify`` message to the ``ActorSelection`` which -will be replied to with an ``ActorIdentity`` containing the correct reference -(see :ref:`actorselection-java`). This can also be done with the ``resolveOne`` -method of the :class:`ActorSelection`, which returns a ``Future`` of the matching -:class:`ActorRef`. +occupying it. `ActorSelection` cannot be watched for this reason. It is +possible to resolve the current incarnation's `ActorRef` living under the +path by sending an `Identify` message to the `ActorSelection` which +will be replied to with an `ActorIdentity` containing the correct reference +(see [Identifying Actors via Actor Selection](#actorselection-java)). This can also be done with the `resolveOne` +method of the `ActorSelection`, which returns a `Future` of the matching +`ActorRef`. -.. _deathwatch-java: - -Lifecycle Monitoring aka DeathWatch ------------------------------------ + +### Lifecycle Monitoring aka DeathWatch In order to be notified when another actor terminates (i.e. stops permanently, not temporary failure and restart), an actor may register itself for reception -of the :class:`Terminated` message dispatched by the other actor upon -termination (see `Stopping Actors`_). This service is provided by the -:class:`DeathWatch` component of the actor system. +of the `Terminated` message dispatched by the other actor upon +termination (see [Stopping Actors](#stopping-actors)). This service is provided by the +`DeathWatch` component of the actor system. Registering a monitor is easy: -.. includecode:: code/jdocs/actor/ActorDocTest.java#import-terminated -.. includecode:: code/jdocs/actor/ActorDocTest.java#watch +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #import-terminated } -It should be noted that the :class:`Terminated` message is generated +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #watch } + +It should be noted that the `Terminated` message is generated independent of the order in which registration and termination occur. -In particular, the watching actor will receive a :class:`Terminated` message +In particular, the watching actor will receive a `Terminated` message even if the watched actor has already been terminated at the time of registration. Registering multiple times does not necessarily lead to multiple messages being @@ -333,159 +315,154 @@ received: if termination of the watched actor has generated and queued the message, and another registration is done before this message has been processed, then a second message will be queued, because registering for monitoring of an already terminated actor leads to the immediate generation of -the :class:`Terminated` message. +the `Terminated` message. It is also possible to deregister from watching another actor’s liveliness -using ``context.unwatch(target)``. This works even if the :class:`Terminated` -message has already been enqueued in the mailbox; after calling :meth:`unwatch` -no :class:`Terminated` message for that actor will be processed anymore. +using `context.unwatch(target)`. This works even if the `Terminated` +message has already been enqueued in the mailbox; after calling `unwatch` +no `Terminated` message for that actor will be processed anymore. -.. _start-hook-java: + +### Start Hook -Start Hook ----------- +Right after starting the actor, its `preStart` method is invoked. -Right after starting the actor, its :meth:`preStart` method is invoked. - -.. includecode:: code/jdocs/actor/ActorDocTest.java#preStart +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #preStart } This method is called when the actor is first created. During restarts it is -called by the default implementation of :meth:`postRestart`, which means that +called by the default implementation of `postRestart`, which means that by overriding that method you can choose whether the initialization code in this method is called only exactly once for this actor or for every restart. Initialization code which is part of the actor’s constructor will always be called when an instance of the actor class is created, which happens at every restart. -.. _restart-hook-java: - -Restart Hooks -------------- + +### Restart Hooks All actors are supervised, i.e. linked to another actor with a fault handling strategy. Actors may be restarted in case an exception is thrown while -processing a message (see :ref:`supervision`). This restart involves the hooks +processing a message (see supervision). This restart involves the hooks mentioned above: -1. The old actor is informed by calling :meth:`preRestart` with the exception - which caused the restart and the message which triggered that exception; the - latter may be ``None`` if the restart was not caused by processing a - message, e.g. when a supervisor does not trap the exception and is restarted - in turn by its supervisor, or if an actor is restarted due to a sibling’s - failure. If the message is available, then that message’s sender is also - accessible in the usual way (i.e. by calling ``sender``). - - This method is the best place for cleaning up, preparing hand-over to the - fresh actor instance, etc. By default it stops all children and calls - :meth:`postStop`. - -2. The initial factory from the ``actorOf`` call is used - to produce the fresh instance. - -3. The new actor’s :meth:`postRestart` method is invoked with the exception - which caused the restart. By default the :meth:`preStart` - is called, just as in the normal start-up case. + 1. + The old actor is informed by calling `preRestart` with the exception +which caused the restart and the message which triggered that exception; the +latter may be `None` if the restart was not caused by processing a +message, e.g. when a supervisor does not trap the exception and is restarted +in turn by its supervisor, or if an actor is restarted due to a sibling’s +failure. If the message is available, then that message’s sender is also +accessible in the usual way (i.e. by calling `sender`). + This method is the best place for cleaning up, preparing hand-over to the +fresh actor instance, etc. By default it stops all children and calls +`postStop`. + 2. The initial factory from the `actorOf` call is used +to produce the fresh instance. + 3. The new actor’s `postRestart` method is invoked with the exception +which caused the restart. By default the `preStart` +is called, just as in the normal start-up case. An actor restart replaces only the actual actor object; the contents of the mailbox is unaffected by the restart, so processing of messages will resume -after the :meth:`postRestart` hook returns. The message +after the `postRestart` hook returns. The message that triggered the exception will not be received again. Any message sent to an actor while it is being restarted will be queued to its mailbox as usual. -.. warning:: +@@@ warning - Be aware that the ordering of failure notifications relative to user messages - is not deterministic. In particular, a parent might restart its child before - it has processed the last messages sent by the child before the failure. - See :ref:`message-ordering` for details. +Be aware that the ordering of failure notifications relative to user messages +is not deterministic. In particular, a parent might restart its child before +it has processed the last messages sent by the child before the failure. +See @ref:[Discussion: Message Ordering](../general/message-delivery-reliability.md#message-ordering) for details. -.. _stop-hook-java: +@@@ -Stop Hook ---------- + +### Stop Hook -After stopping an actor, its :meth:`postStop` hook is called, which may be used +After stopping an actor, its `postStop` hook is called, which may be used e.g. for deregistering this actor from other services. This hook is guaranteed to run after message queuing has been disabled for this actor, i.e. messages -sent to a stopped actor will be redirected to the :obj:`deadLetters` of the -:obj:`ActorSystem`. +sent to a stopped actor will be redirected to the `deadLetters` of the +`ActorSystem`. -.. _actorselection-java: + +## Identifying Actors via Actor Selection -Identifying Actors via Actor Selection -====================================== - -As described in :ref:`addressing`, each actor has a unique logical path, which +As described in @ref:[Actor References, Paths and Addresses](../general/addressing.md), each actor has a unique logical path, which is obtained by following the chain of actors from child to parent until reaching the root of the actor system, and it has a physical path, which may differ if the supervision chain includes any remote supervisors. These paths are used by the system to look up actors, e.g. when a remote message is received and the recipient is searched, but they are also useful more directly: actors may look up other actors by specifying absolute or relative -paths—logical or physical—and receive back an :class:`ActorSelection` with the +paths—logical or physical—and receive back an `ActorSelection` with the result: -.. includecode:: code/jdocs/actor/ActorDocTest.java#selection-local +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #selection-local } -.. note:: +@@@ note - It is always preferable to communicate with other Actors using their ActorRef - instead of relying upon ActorSelection. Exceptions are +It is always preferable to communicate with other Actors using their ActorRef +instead of relying upon ActorSelection. Exceptions are - * sending messages using the :ref:`at-least-once-delivery-java` facility - * initiating first contact with a remote system +> + * sending messages using the @ref:[At-Least-Once Delivery](persistence.md#at-least-once-delivery-java) facility + * initiating first contact with a remote system - In all other cases ActorRefs can be provided during Actor creation or - initialization, passing them from parent to child or introducing Actors by - sending their ActorRefs to other Actors within messages. +In all other cases ActorRefs can be provided during Actor creation or +initialization, passing them from parent to child or introducing Actors by +sending their ActorRefs to other Actors within messages. -The supplied path is parsed as a :class:`java.net.URI`, which basically means -that it is split on ``/`` into path elements. If the path starts with ``/``, it +@@@ + +The supplied path is parsed as a `java.net.URI`, which basically means +that it is split on `/` into path elements. If the path starts with `/`, it is absolute and the look-up starts at the root guardian (which is the parent of -``"/user"``); otherwise it starts at the current actor. If a path element equals -``..``, the look-up will take a step “up” towards the supervisor of the +`"/user"`); otherwise it starts at the current actor. If a path element equals +`..`, the look-up will take a step “up” towards the supervisor of the currently traversed actor, otherwise it will step “down” to the named child. -It should be noted that the ``..`` in actor paths here always means the logical +It should be noted that the `..` in actor paths here always means the logical structure, i.e. the supervisor. The path elements of an actor selection may contain wildcard patterns allowing for broadcasting of messages to that section: -.. includecode:: code/jdocs/actor/ActorDocTest.java#selection-wildcard +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #selection-wildcard } -Messages can be sent via the :class:`ActorSelection` and the path of the -:class:`ActorSelection` is looked up when delivering each message. If the selection +Messages can be sent via the `ActorSelection` and the path of the +`ActorSelection` is looked up when delivering each message. If the selection does not match any actors the message will be dropped. -To acquire an :class:`ActorRef` for an :class:`ActorSelection` you need to send -a message to the selection and use the ``getSender()`` reference of the reply from -the actor. There is a built-in ``Identify`` message that all Actors will -understand and automatically reply to with a ``ActorIdentity`` message -containing the :class:`ActorRef`. This message is handled specially by the +To acquire an `ActorRef` for an `ActorSelection` you need to send +a message to the selection and use the `getSender()` reference of the reply from +the actor. There is a built-in `Identify` message that all Actors will +understand and automatically reply to with a `ActorIdentity` message +containing the `ActorRef`. This message is handled specially by the actors which are traversed in the sense that if a concrete name lookup fails (i.e. a non-wildcard path element does not correspond to a live actor) then a negative result is generated. Please note that this does not mean that delivery of that reply is guaranteed, it still is a normal message. -.. includecode:: code/jdocs/actor/ActorDocTest.java#import-identify -.. includecode:: code/jdocs/actor/ActorDocTest.java#identify +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #import-identify } -You can also acquire an :class:`ActorRef` for an :class:`ActorSelection` with -the ``resolveOne`` method of the :class:`ActorSelection`. It returns a -``Future`` of the matching :class:`ActorRef` if such an actor exists (see also -:ref:`scala-java-compat` for Java compatibility). It is completed with failure +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #identify } + +You can also acquire an `ActorRef` for an `ActorSelection` with +the `resolveOne` method of the `ActorSelection`. It returns a +`Future` of the matching `ActorRef` if such an actor exists (see also +@ref:[Java 8 and Scala Compatibility](scala-compat.md) for Java compatibility). It is completed with failure [[akka.actor.ActorNotFound]] if no such actor exists or the identification -didn't complete within the supplied `timeout`. +didn't complete within the supplied *timeout*. -Remote actor addresses may also be looked up, if :ref:`remoting ` is enabled: +Remote actor addresses may also be looked up, if @ref:[remoting](remoting.md) is enabled: -.. includecode:: code/jdocs/actor/ActorDocTest.java#selection-remote +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #selection-remote } -An example demonstrating actor look-up is given in :ref:`remote-sample-java`. +An example demonstrating actor look-up is given in @ref:[Remoting Sample](remoting.md#remote-sample-java). -Messages and immutability -========================= +## Messages and immutability **IMPORTANT**: Messages can be any kind of object but have to be immutable. Akka can’t enforce immutability (yet) so this has to be by @@ -493,231 +470,224 @@ convention. Here is an example of an immutable message: -.. includecode:: code/jdocs/actor/ImmutableMessage.java#immutable-message +@@snip [ImmutableMessage.java](code/jdocs/actor/ImmutableMessage.java) { #immutable-message } -Send messages -============= +## Send messages Messages are sent to an Actor through one of the following methods. -* ``tell`` means “fire-and-forget”, e.g. send a message asynchronously and return - immediately. -* ``ask`` sends a message asynchronously and returns a :class:`Future` - representing a possible reply. + * `tell` means “fire-and-forget”, e.g. send a message asynchronously and return +immediately. + * `ask` sends a message asynchronously and returns a `Future` +representing a possible reply. Message ordering is guaranteed on a per-sender basis. -.. note:: +@@@ note - There are performance implications of using ``ask`` since something needs to - keep track of when it times out, there needs to be something that bridges - a ``Promise`` into an ``ActorRef`` and it also needs to be reachable through - remoting. So always prefer ``tell`` for performance, and only ``ask`` if you must. +There are performance implications of using `ask` since something needs to +keep track of when it times out, there needs to be something that bridges +a `Promise` into an `ActorRef` and it also needs to be reachable through +remoting. So always prefer `tell` for performance, and only `ask` if you must. -In all these methods you have the option of passing along your own ``ActorRef``. +@@@ + +In all these methods you have the option of passing along your own `ActorRef`. Make it a practice of doing so because it will allow the receiver actors to be able to respond to your message, since the sender reference is sent along with the message. -.. _actors-tell-sender-java: - -Tell: Fire-forget ------------------ + +### Tell: Fire-forget This is the preferred way of sending messages. No blocking waiting for a message. This gives the best concurrency and scalability characteristics. -.. includecode:: code/jdocs/actor/ActorDocTest.java#tell +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #tell } The sender reference is passed along with the message and available within the -receiving actor via its :meth:`getSender()` method while processing this -message. Inside of an actor it is usually :meth:`getSelf()` who shall be the +receiving actor via its `getSender()` method while processing this +message. Inside of an actor it is usually `getSelf()` who shall be the sender, but there can be cases where replies shall be routed to some other -actor—e.g. the parent—in which the second argument to :meth:`tell` would be a +actor—e.g. the parent—in which the second argument to `tell` would be a different one. Outside of an actor and if no reply is needed the second -argument can be ``null``; if a reply is needed outside of an actor you can use +argument can be `null`; if a reply is needed outside of an actor you can use the ask-pattern described next.. -.. _actors-ask-java: + +### Ask: Send-And-Receive-Future -Ask: Send-And-Receive-Future ----------------------------- +The `ask` pattern involves actors as well as futures, hence it is offered as +a use pattern rather than a method on `ActorRef`: -The ``ask`` pattern involves actors as well as futures, hence it is offered as -a use pattern rather than a method on :class:`ActorRef`: +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #import-ask } -.. includecode:: code/jdocs/actor/ActorDocTest.java#import-ask -.. includecode:: code/jdocs/actor/ActorDocTest.java#ask-pipe +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #ask-pipe } -This example demonstrates ``ask`` together with the ``pipe`` pattern on +This example demonstrates `ask` together with the `pipe` pattern on futures, because this is likely to be a common combination. Please note that -all of the above is completely non-blocking and asynchronous: ``ask`` produces -a :class:`Future`, two of which are composed into a new future using the -:meth:`Futures.sequence` and :meth:`map` methods and then ``pipe`` installs -an ``onComplete``-handler on the future to effect the submission of the -aggregated :class:`Result` to another actor. +all of the above is completely non-blocking and asynchronous: `ask` produces +a `Future`, two of which are composed into a new future using the +`Futures.sequence` and `map` methods and then `pipe` installs +an `onComplete`-handler on the future to effect the submission of the +aggregated `Result` to another actor. -Using ``ask`` will send a message to the receiving Actor as with ``tell``, and -the receiving actor must reply with ``getSender().tell(reply, getSelf())`` in order to -complete the returned :class:`Future` with a value. The ``ask`` operation +Using `ask` will send a message to the receiving Actor as with `tell`, and +the receiving actor must reply with `getSender().tell(reply, getSelf())` in order to +complete the returned `Future` with a value. The `ask` operation involves creating an internal actor for handling this reply, which needs to have a timeout after which it is destroyed in order not to leak resources; see more below. -.. note:: - A variant of the ``ask`` pattern that returns a ``CompletionStage`` instead of a Scala ``Future`` - is available in the ``akka.pattern.PatternsCS`` object. +@@@ note -.. warning:: +A variant of the `ask` pattern that returns a `CompletionStage` instead of a Scala `Future` +is available in the `akka.pattern.PatternsCS` object. - To complete the future with an exception you need send a Failure message to the sender. - This is *not done automatically* when an actor throws an exception while processing a message. +@@@ -.. includecode:: code/jdocs/actor/ActorDocTest.java#reply-exception +@@@ warning + +To complete the future with an exception you need send a Failure message to the sender. +This is *not done automatically* when an actor throws an exception while processing a message. + +@@@ + +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #reply-exception } If the actor does not complete the future, it will expire after the timeout period, -specified as parameter to the ``ask`` method; this will complete the -:class:`Future` with an :class:`AskTimeoutException`. +specified as parameter to the `ask` method; this will complete the +`Future` with an `AskTimeoutException`. -See :ref:`futures-java` for more information on how to await or query a +See @ref:[Futures](futures.md) for more information on how to await or query a future. -The ``onComplete``, ``onSuccess``, or ``onFailure`` methods of the ``Future`` can be +The `onComplete`, `onSuccess`, or `onFailure` methods of the `Future` can be used to register a callback to get a notification when the Future completes. Gives you a way to avoid blocking. -.. warning:: +@@@ warning - When using future callbacks, inside actors you need to carefully avoid closing over - the containing actor’s reference, i.e. do not call methods or access mutable state - on the enclosing actor from within the callback. This would break the actor - encapsulation and may introduce synchronization bugs and race conditions because - the callback will be scheduled concurrently to the enclosing actor. Unfortunately - there is not yet a way to detect these illegal accesses at compile time. See also: - :ref:`jmm-shared-state` +When using future callbacks, inside actors you need to carefully avoid closing over +the containing actor’s reference, i.e. do not call methods or access mutable state +on the enclosing actor from within the callback. This would break the actor +encapsulation and may introduce synchronization bugs and race conditions because +the callback will be scheduled concurrently to the enclosing actor. Unfortunately +there is not yet a way to detect these illegal accesses at compile time. See also: +@ref:[Actors and shared mutable state](../general/jmm.md#jmm-shared-state) -Forward message ---------------- +@@@ + +### Forward message You can forward a message from one actor to another. This means that the original sender address/reference is maintained even though the message is going through a 'mediator'. This can be useful when writing actors that work as routers, load-balancers, replicators etc. -.. includecode:: code/jdocs/actor/ActorDocTest.java#forward +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #forward } -.. _actors-receive-java: - -Receive messages -================ + +## Receive messages An actor has to define its initial receive behavior by implementing -the :meth:`createReceive` method in the :class:`AbstractActor`: +the `createReceive` method in the `AbstractActor`: -.. includecode:: code/jdocs/actor/ActorDocTest.java#createReceive +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #createReceive } - -The return type is :class:`AbstractActor.Receive` that defines which messages your Actor can handle, +The return type is `AbstractActor.Receive` that defines which messages your Actor can handle, along with the implementation of how the messages should be processed. -You can build such behavior with a builder named ``ReceiveBuilder``. +You can build such behavior with a builder named `ReceiveBuilder`. Here is an example: -.. includecode:: code/jdocs/actor/MyActor.java - :include: imports,my-actor +@@snip [MyActor.java](code/jdocs/actor/MyActor.java) { #imports #my-actor } -In case you want to provide many :meth:`match` cases but want to avoid creating a long call +In case you want to provide many `match` cases but want to avoid creating a long call trail, you can split the creation of the builder into multiple statements as in the example: -.. includecode:: code/jdocs/actor/GraduallyBuiltActor.java - :include: imports,actor +@@snip [GraduallyBuiltActor.java](code/jdocs/actor/GraduallyBuiltActor.java) { #imports #actor } Using small methods is a good practice, also in actors. It's recommended to delegate the -actual work of the message processing to methods instead of defining a huge ``ReceiveBuilder`` +actual work of the message processing to methods instead of defining a huge `ReceiveBuilder` with lots of code in each lambda. A well structured actor can look like this: -.. includecode:: code/jdocs/actor/ActorDocTest.java#well-structured +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #well-structured } That has benefits such as: -* easier to see what kind of messages the actor can handle -* readable stack traces in case of exceptions -* works better with performance profiling tools -* Java HotSpot has a better opportunity for making optimizations + * easier to see what kind of messages the actor can handle + * readable stack traces in case of exceptions + * works better with performance profiling tools + * Java HotSpot has a better opportunity for making optimizations -The ``Receive`` can be implemented in other ways than using the ``ReceiveBuilder`` since it in the -end is just a wrapper around a Scala ``PartialFunction``. In Java, you can implement ``PartialFunction`` by -extending ``AbstractPartialFunction``. For example, one could implement an adapter -to `Javaslang Pattern Matching DSL `_. +The `Receive` can be implemented in other ways than using the `ReceiveBuilder` since it in the +end is just a wrapper around a Scala `PartialFunction`. In Java, you can implement `PartialFunction` by +extending `AbstractPartialFunction`. For example, one could implement an adapter +to [Javaslang Pattern Matching DSL](http://www.javaslang.io/javaslang-jdocs/#_pattern_matching). -If the validation of the ``ReceiveBuilder`` match logic turns out to be a bottleneck for some of your -actors you can consider to implement it at lower level by extending ``UntypedAbstractActor`` instead -of ``AbstractActor``. The partial functions created by the ``ReceiveBuilder`` consist of multiple lambda +If the validation of the `ReceiveBuilder` match logic turns out to be a bottleneck for some of your +actors you can consider to implement it at lower level by extending `UntypedAbstractActor` instead +of `AbstractActor`. The partial functions created by the `ReceiveBuilder` consist of multiple lambda expressions for every match statement, where each lambda is referencing the code to be run. This is something that the JVM can have problems optimizing and the resulting code might not be as performant as the -untyped version. When extending ``UntypedAbstractActor`` each message is received as an untyped -``Object`` and you have to inspect and cast it to the actual message type in other ways, like this: +untyped version. When extending `UntypedAbstractActor` each message is received as an untyped +`Object` and you have to inspect and cast it to the actual message type in other ways, like this: -.. includecode:: code/jdocs/actor/ActorDocTest.java#optimized +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #optimized } -.. _LambdaActor.Reply: - -Reply to messages -================= + +## Reply to messages If you want to have a handle for replying to a message, you can use -``getSender()``, which gives you an ActorRef. You can reply by sending to -that ActorRef with ``getSender().tell(replyMsg, getSelf())``. You can also store the ActorRef +`getSender()`, which gives you an ActorRef. You can reply by sending to +that ActorRef with `getSender().tell(replyMsg, getSelf())`. You can also store the ActorRef for replying later, or passing on to other actors. If there is no sender (a message was sent without an actor or future context) then the sender defaults to a 'dead-letter' actor ref. -.. includecode:: code/jdocs/actor/MyActor.java#reply +@@snip [MyActor.java](code/jdocs/actor/MyActor.java) { #reply } +## Receive timeout -Receive timeout -=============== - -The `ActorContext` :meth:`setReceiveTimeout` defines the inactivity timeout after which -the sending of a `ReceiveTimeout` message is triggered. -When specified, the receive function should be able to handle an `akka.actor.ReceiveTimeout` message. +The *ActorContext* `setReceiveTimeout` defines the inactivity timeout after which +the sending of a *ReceiveTimeout* message is triggered. +When specified, the receive function should be able to handle an *akka.actor.ReceiveTimeout* message. 1 millisecond is the minimum supported timeout. -Please note that the receive timeout might fire and enqueue the `ReceiveTimeout` message right after +Please note that the receive timeout might fire and enqueue the *ReceiveTimeout* message right after another message was enqueued; hence it is **not guaranteed** that upon reception of the receive timeout there must have been an idle period beforehand as configured via this method. Once set, the receive timeout stays in effect (i.e. continues firing repeatedly after inactivity -periods). Pass in `Duration.Undefined` to switch off this feature. +periods). Pass in *Duration.Undefined* to switch off this feature. -.. includecode:: code/jdocs/actor/ActorDocTest.java#receive-timeout +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #receive-timeout } -Messages marked with ``NotInfluenceReceiveTimeout`` will not reset the timer. This can be useful when -``ReceiveTimeout`` should be fired by external inactivity but not influenced by internal activity, +Messages marked with `NotInfluenceReceiveTimeout` will not reset the timer. This can be useful when +`ReceiveTimeout` should be fired by external inactivity but not influenced by internal activity, e.g. scheduled tick messages. -.. _stopping-actors-java: + +## Stopping actors -Stopping actors -=============== - -Actors are stopped by invoking the :meth:`stop` method of a ``ActorRefFactory``, -i.e. ``ActorContext`` or ``ActorSystem``. Typically the context is used for stopping +Actors are stopped by invoking the `stop` method of a `ActorRefFactory`, +i.e. `ActorContext` or `ActorSystem`. Typically the context is used for stopping child actors and the system for stopping top level actors. The actual termination of -the actor is performed asynchronously, i.e. :meth:`stop` may return before the actor is +the actor is performed asynchronously, i.e. `stop` may return before the actor is stopped. -.. includecode:: code/jdocs/actor/MyStoppingActor.java#my-stopping-actor +@@snip [MyStoppingActor.java](code/jdocs/actor/MyStoppingActor.java) { #my-stopping-actor } Processing of the current message, if any, will continue before the actor is stopped, but additional messages in the mailbox will not be processed. By default these -messages are sent to the :obj:`deadLetters` of the :obj:`ActorSystem`, but that +messages are sent to the `deadLetters` of the `ActorSystem`, but that depends on the mailbox implementation. Termination of an actor proceeds in two steps: first the actor suspends its mailbox processing and sends a stop command to all its children, then it keeps processing the internal termination notifications from its children until the last one is -gone, finally terminating itself (invoking :meth:`postStop`, dumping mailbox, -publishing :class:`Terminated` on the :ref:`DeathWatch `, telling +gone, finally terminating itself (invoking `postStop`, dumping mailbox, +publishing `Terminated` on the [DeathWatch](#deathwatch-java), telling its supervisor). This procedure ensures that actor system sub-trees terminate in an orderly fashion, propagating the stop command to the leaves and collecting their confirmation back to the stopped supervisor. If one of the @@ -725,262 +695,261 @@ actors does not respond (i.e. processing a message for extended periods of time and therefore not receiving the stop command), this whole process will be stuck. -Upon :meth:`ActorSystem.terminate()`, the system guardian actors will be +Upon `ActorSystem.terminate()`, the system guardian actors will be stopped, and the aforementioned process will ensure proper termination of the whole system. -The :meth:`postStop()` hook is invoked after an actor is fully stopped. This +The `postStop()` hook is invoked after an actor is fully stopped. This enables cleaning up of resources: -.. includecode:: code/jdocs/actor/ActorDocTest.java#postStop - :exclude: clean-up-some-resources +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #postStop } -.. note:: +@@@ note - Since stopping an actor is asynchronous, you cannot immediately reuse the - name of the child you just stopped; this will result in an - :class:`InvalidActorNameException`. Instead, :meth:`watch()` the terminating - actor and create its replacement in response to the :class:`Terminated` - message which will eventually arrive. +Since stopping an actor is asynchronous, you cannot immediately reuse the +name of the child you just stopped; this will result in an +`InvalidActorNameException`. Instead, `watch()` the terminating +actor and create its replacement in response to the `Terminated` +message which will eventually arrive. -.. _poison-pill-java: +@@@ -PoisonPill ----------- + +### PoisonPill -You can also send an actor the ``akka.actor.PoisonPill`` message, which will -stop the actor when the message is processed. ``PoisonPill`` is enqueued as +You can also send an actor the `akka.actor.PoisonPill` message, which will +stop the actor when the message is processed. `PoisonPill` is enqueued as ordinary messages and will be handled after messages that were already queued in the mailbox. -.. includecode:: code/jdocs/actor/ActorDocTest.java#poison-pill +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #poison-pill } -Graceful Stop -------------- +### Graceful Stop -:meth:`gracefulStop` is useful if you need to wait for termination or compose ordered +`gracefulStop` is useful if you need to wait for termination or compose ordered termination of several actors: -.. includecode:: code/jdocs/actor/ActorDocTest.java#import-gracefulStop +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #import-gracefulStop } -.. includecode:: code/jdocs/actor/ActorDocTest.java#gracefulStop +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #gracefulStop } -.. includecode:: code/jdocs/actor/ActorDocTest.java#gracefulStop-actor +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #gracefulStop-actor } -When ``gracefulStop()`` returns successfully, the actor’s ``postStop()`` hook +When `gracefulStop()` returns successfully, the actor’s `postStop()` hook will have been executed: there exists a happens-before edge between the end of -``postStop()`` and the return of ``gracefulStop()``. +`postStop()` and the return of `gracefulStop()`. -In the above example a custom ``Manager.Shutdown`` message is sent to the target -actor to initiate the process of stopping the actor. You can use ``PoisonPill`` for +In the above example a custom `Manager.Shutdown` message is sent to the target +actor to initiate the process of stopping the actor. You can use `PoisonPill` for this, but then you have limited possibilities to perform interactions with other actors -before stopping the target actor. Simple cleanup tasks can be handled in ``postStop``. +before stopping the target actor. Simple cleanup tasks can be handled in `postStop`. -.. warning:: +@@@ warning - Keep in mind that an actor stopping and its name being deregistered are - separate events which happen asynchronously from each other. Therefore it may - be that you will find the name still in use after ``gracefulStop()`` - returned. In order to guarantee proper deregistration, only reuse names from - within a supervisor you control and only in response to a :class:`Terminated` - message, i.e. not for top-level actors. +Keep in mind that an actor stopping and its name being deregistered are +separate events which happen asynchronously from each other. Therefore it may +be that you will find the name still in use after `gracefulStop()` +returned. In order to guarantee proper deregistration, only reuse names from +within a supervisor you control and only in response to a `Terminated` +message, i.e. not for top-level actors. -.. _coordinated-shutdown-java: +@@@ -Coordinated Shutdown --------------------- + +### Coordinated Shutdown -There is an extension named ``CoordinatedShutdown`` that will stop certain actors and +There is an extension named `CoordinatedShutdown` that will stop certain actors and services in a specific order and perform registered tasks during the shutdown process. -The order of the shutdown phases is defined in configuration ``akka.coordinated-shutdown.phases``. +The order of the shutdown phases is defined in configuration `akka.coordinated-shutdown.phases`. The default phases are defined as: -.. includecode:: ../../../akka-actor/src/main/resources/reference.conf#coordinated-shutdown-phases +@@snip [reference.conf]../../../../../akka-actor/src/main/resources/reference.conf) { #coordinated-shutdown-phases } More phases can be be added in the application's configuration if needed by overriding a phase with an -additional ``depends-on``. Especially the phases ``before-service-unbind``, ``before-cluster-shutdown`` and -``before-actor-system-terminate`` are intended for application specific phases or tasks. +additional `depends-on`. Especially the phases `before-service-unbind`, `before-cluster-shutdown` and +`before-actor-system-terminate` are intended for application specific phases or tasks. The default phases are defined in a single linear order, but the phases can be ordered as a directed acyclic graph (DAG) by defining the dependencies between the phases. -The phases are ordered with `topological `_ sort of the DAG. +The phases are ordered with [topological](https://en.wikipedia.org/wiki/Topological_sorting) sort of the DAG. Tasks can be added to a phase with: -.. includecode:: code/jdocs/actor/ActorDocTest.java#coordinated-shutdown-addTask +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #coordinated-shutdown-addTask } -The returned ``CompletionStage`` should be completed when the task is completed. The task name parameter +The returned `CompletionStage` should be completed when the task is completed. The task name parameter is only used for debugging/logging. Tasks added to the same phase are executed in parallel without any ordering assumptions. Next phase will not start until all tasks of previous phase have been completed. -If tasks are not completed within a configured timeout (see :ref:`reference.conf `) -the next phase will be started anyway. It is possible to configure ``recover=off`` for a phase +If tasks are not completed within a configured timeout (see @ref:[reference.conf](../general/configuration.md#config-akka-actor)) +the next phase will be started anyway. It is possible to configure `recover=off` for a phase to abort the rest of the shutdown process if a task fails or is not completed within the timeout. Tasks should typically be registered as early as possible after system startup. When running the coordinated shutdown tasks that have been registered will be performed but tasks that are added too late will not be run. -To start the coordinated shutdown process you can invoke ``runAll`` on the ``CoordinatedShutdown`` +To start the coordinated shutdown process you can invoke `runAll` on the `CoordinatedShutdown` extension: -.. includecode:: code/jdocs/actor/ActorDocTest.java#coordinated-shutdown-run +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #coordinated-shutdown-run } -It's safe to call the ``runAll`` method multiple times. It will only run once. +It's safe to call the `runAll` method multiple times. It will only run once. -That also means that the ``ActorSystem`` will be terminated in the last phase. By default, the +That also means that the `ActorSystem` will be terminated in the last phase. By default, the JVM is not forcefully stopped (it will be stopped if all non-daemon threads have been terminated). -To enable a hard ``System.exit`` as a final action you can configure:: +To enable a hard `System.exit` as a final action you can configure: - akka.coordinated-shutdown.exit-jvm = on +``` +akka.coordinated-shutdown.exit-jvm = on +``` -When using :ref:`Akka Cluster ` the ``CoordinatedShutdown`` will automatically run -when the cluster node sees itself as ``Exiting``, i.e. leaving from another node will trigger +When using @ref:[Akka Cluster](cluster-usage.md) the `CoordinatedShutdown` will automatically run +when the cluster node sees itself as `Exiting`, i.e. leaving from another node will trigger the shutdown process on the leaving node. Tasks for graceful leaving of cluster including graceful shutdown of Cluster Singletons and Cluster Sharding are added automatically when Akka Cluster is used, i.e. running the shutdown process will also trigger the graceful leaving if it's not already in progress. -By default, the ``CoordinatedShutdown`` will be run when the JVM process exits, e.g. -via ``kill SIGTERM`` signal (``SIGINT`` ctrl-c doesn't work). This behavior can be disabled with:: +By default, the `CoordinatedShutdown` will be run when the JVM process exits, e.g. +via `kill SIGTERM` signal (`SIGINT` ctrl-c doesn't work). This behavior can be disabled with: - akka.coordinated-shutdown.run-by-jvm-shutdown-hook=off +``` +akka.coordinated-shutdown.run-by-jvm-shutdown-hook=off +``` If you have application specific JVM shutdown hooks it's recommended that you register them via the -``CoordinatedShutdown`` so that they are running before Akka internal shutdown hooks, e.g. +`CoordinatedShutdown` so that they are running before Akka internal shutdown hooks, e.g. those shutting down Akka Remoting (Artery). -.. includecode:: code/jdocs/actor/ActorDocTest.java#coordinated-shutdown-jvm-hook +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #coordinated-shutdown-jvm-hook } -For some tests it might be undesired to terminate the ``ActorSystem`` via ``CoordinatedShutdown``. -You can disable that by adding the following to the configuration of the ``ActorSystem`` that is -used in the test:: +For some tests it might be undesired to terminate the `ActorSystem` via `CoordinatedShutdown`. +You can disable that by adding the following to the configuration of the `ActorSystem` that is +used in the test: - # Don't terminate ActorSystem via CoordinatedShutdown in tests - akka.coordinated-shutdown.terminate-actor-system = off - akka.coordinated-shutdown.run-by-jvm-shutdown-hook = off - akka.cluster.run-coordinated-shutdown-when-down = off +``` +# Don't terminate ActorSystem via CoordinatedShutdown in tests +akka.coordinated-shutdown.terminate-actor-system = off +akka.coordinated-shutdown.run-by-jvm-shutdown-hook = off +akka.cluster.run-coordinated-shutdown-when-down = off +``` -.. _actor-hotswap-java: - -Become/Unbecome -=============== + +## Become/Unbecome Akka supports hotswapping the Actor’s message loop (e.g. its implementation) at -runtime: invoke the ``context.become`` method from within the Actor. -:meth:`become` takes a ``PartialFunction`` that implements the new +runtime: invoke the `context.become` method from within the Actor. +`become` takes a `PartialFunction` that implements the new message handler. The hotswapped code is kept in a Stack which can be pushed and popped. -.. warning:: +@@@ warning - Please note that the actor will revert to its original behavior when restarted by its Supervisor. +Please note that the actor will revert to its original behavior when restarted by its Supervisor. -To hotswap the Actor behavior using ``become``: +@@@ -.. includecode:: code/jdocs/actor/ActorDocTest.java#hot-swap-actor +To hotswap the Actor behavior using `become`: -This variant of the :meth:`become` method is useful for many different things, -such as to implement a Finite State Machine (FSM, for an example see `Dining -Hakkers`_). It will replace the current behavior (i.e. the top of the behavior -stack), which means that you do not use :meth:`unbecome`, instead always the +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #hot-swap-actor } + +This variant of the `become` method is useful for many different things, +such as to implement a Finite State Machine (FSM, for an example see [Dining +Hakkers](http://www.lightbend.com/activator/template/akka-sample-fsm-java-lambda)). It will replace the current behavior (i.e. the top of the behavior +stack), which means that you do not use `unbecome`, instead always the next behavior is explicitly installed. -.. _Dining Hakkers: http://www.lightbend.com/activator/template/akka-sample-fsm-java-lambda - -The other way of using :meth:`become` does not replace but add to the top of +The other way of using `become` does not replace but add to the top of the behavior stack. In this case care must be taken to ensure that the number -of “pop” operations (i.e. :meth:`unbecome`) matches the number of “push” ones +of “pop” operations (i.e. `unbecome`) matches the number of “push” ones in the long run, otherwise this amounts to a memory leak (which is why this behavior is not the default). -.. includecode:: code/jdocs/actor/ActorDocTest.java#swapper +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #swapper } -.. _stash-java: + +## Stash -Stash -===== - -The ``AbstractActorWithStash`` class enables an actor to temporarily stash away messages +The `AbstractActorWithStash` class enables an actor to temporarily stash away messages that can not or should not be handled using the actor's current behavior. Upon changing the actor's message handler, i.e., right -before invoking ``getContext().become()`` or ``getContext().unbecome()``, all +before invoking `getContext().become()` or `getContext().unbecome()`, all stashed messages can be "unstashed", thereby prepending them to the actor's mailbox. This way, the stashed messages can be processed in the same order as they have been received originally. An actor that extends -``AbstractActorWithStash`` will automatically get a deque-based mailbox. +`AbstractActorWithStash` will automatically get a deque-based mailbox. -.. note:: +@@@ note - The abstract class ``AbstractActorWithStash`` implements the marker - interface ``RequiresMessageQueue`` - which requests the system to automatically choose a deque based - mailbox implementation for the actor. If you want more - control over the mailbox, see the documentation on mailboxes: :ref:`mailboxes-java`. +The abstract class `AbstractActorWithStash` implements the marker +interface `RequiresMessageQueue` +which requests the system to automatically choose a deque based +mailbox implementation for the actor. If you want more +control over the mailbox, see the documentation on mailboxes: @ref:[Mailboxes](mailboxes.md). -Here is an example of the ``AbstractActorWithStash`` class in action: +@@@ -.. includecode:: code/jdocs/actor/ActorDocTest.java#stash +Here is an example of the `AbstractActorWithStash` class in action: -Invoking ``stash()`` adds the current message (the message that the +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #stash } + +Invoking `stash()` adds the current message (the message that the actor received last) to the actor's stash. It is typically invoked when handling the default case in the actor's message handler to stash messages that aren't handled by the other cases. It is illegal to stash the same message twice; to do so results in an -``IllegalStateException`` being thrown. The stash may also be bounded -in which case invoking ``stash()`` may lead to a capacity violation, -which results in a ``StashOverflowException``. The capacity of the -stash can be configured using the ``stash-capacity`` setting (an ``Int``) of the +`IllegalStateException` being thrown. The stash may also be bounded +in which case invoking `stash()` may lead to a capacity violation, +which results in a `StashOverflowException`. The capacity of the +stash can be configured using the `stash-capacity` setting (an `Int`) of the mailbox's configuration. -Invoking ``unstashAll()`` enqueues messages from the stash to the +Invoking `unstashAll()` enqueues messages from the stash to the actor's mailbox until the capacity of the mailbox (if any) has been reached (note that messages from the stash are prepended to the mailbox). In case a bounded mailbox overflows, a -``MessageQueueAppendFailedException`` is thrown. -The stash is guaranteed to be empty after calling ``unstashAll()``. +`MessageQueueAppendFailedException` is thrown. +The stash is guaranteed to be empty after calling `unstashAll()`. -The stash is backed by a ``scala.collection.immutable.Vector``. As a +The stash is backed by a `scala.collection.immutable.Vector`. As a result, even a very large number of messages may be stashed without a major impact on performance. Note that the stash is part of the ephemeral actor state, unlike the mailbox. Therefore, it should be managed like other parts of the -actor's state which have the same property. The :class:`AbstractActorWithStash` -implementation of :meth:`preRestart` will call ``unstashAll()``, which is +actor's state which have the same property. The `AbstractActorWithStash` +implementation of `preRestart` will call `unstashAll()`, which is usually the desired behavior. -.. note:: +@@@ note - If you want to enforce that your actor can only work with an unbounded stash, - then you should use the ``AbstractActorWithUnboundedStash`` class instead. +If you want to enforce that your actor can only work with an unbounded stash, +then you should use the `AbstractActorWithUnboundedStash` class instead. +@@@ -.. _killing-actors-java: + +## Killing an Actor -Killing an Actor -================ - -You can kill an actor by sending a ``Kill`` message. This will cause the actor -to throw a :class:`ActorKilledException`, triggering a failure. The actor will +You can kill an actor by sending a `Kill` message. This will cause the actor +to throw a `ActorKilledException`, triggering a failure. The actor will suspend operation and its supervisor will be asked how to handle the failure, which may mean resuming the actor, restarting it or terminating it completely. -See :ref:`supervision-directives` for more information. +See @ref:[What Supervision Means](../general/supervision.md#supervision-directives) for more information. -Use ``Kill`` like this: +Use `Kill` like this: -.. includecode:: code/jdocs/actor/ActorDocTest.java#kill +@@snip [ActorDocTest.java](code/jdocs/actor/ActorDocTest.java) { #kill } -Actors and exceptions -===================== +## Actors and exceptions It can happen that while a message is being processed by an actor, that some kind of exception is thrown, e.g. a database exception. -What happens to the Message ---------------------------- +### What happens to the Message If an exception is thrown while a message is being processed (i.e. taken out of its mailbox and handed over to the current behavior), then this message will be @@ -989,81 +958,75 @@ if you want to retry processing of a message, you need to deal with it yourself by catching the exception and retry your flow. Make sure that you put a bound on the number of retries since you don't want a system to livelock (so consuming a lot of cpu cycles without making progress). Another possibility -would be to have a look at the :ref:`PeekMailbox pattern `. +would be to have a look at the mailbox-acking. -What happens to the mailbox ---------------------------- +### What happens to the mailbox If an exception is thrown while a message is being processed, nothing happens to the mailbox. If the actor is restarted, the same mailbox will be there. So all messages on that mailbox will be there as well. -What happens to the actor -------------------------- +### What happens to the actor If code within an actor throws an exception, that actor is suspended and the -supervision process is started (see :ref:`supervision`). Depending on the +supervision process is started (see supervision). Depending on the supervisor’s decision the actor is resumed (as if nothing happened), restarted (wiping out its internal state and starting from scratch) or terminated. -Initialization patterns -======================= +## Initialization patterns The rich lifecycle hooks of Actors provide a useful toolkit to implement various initialization patterns. During the -lifetime of an ``ActorRef``, an actor can potentially go through several restarts, where the old instance is replaced by -a fresh one, invisibly to the outside observer who only sees the ``ActorRef``. +lifetime of an `ActorRef`, an actor can potentially go through several restarts, where the old instance is replaced by +a fresh one, invisibly to the outside observer who only sees the `ActorRef`. One may think about the new instances as "incarnations". Initialization might be necessary for every incarnation of an actor, but sometimes one needs initialization to happen only at the birth of the first instance when the -``ActorRef`` is created. The following sections provide patterns for different initialization needs. +`ActorRef` is created. The following sections provide patterns for different initialization needs. -Initialization via constructor ------------------------------- +### Initialization via constructor -Using the constructor for initialization has various benefits. First of all, it makes it possible to use ``val`` fields to store +Using the constructor for initialization has various benefits. First of all, it makes it possible to use `val` fields to store any state that does not change during the life of the actor instance, making the implementation of the actor more robust. The constructor is invoked for every incarnation of the actor, therefore the internals of the actor can always assume that proper initialization happened. This is also the drawback of this approach, as there are cases when one would like to avoid reinitializing internals on restart. For example, it is often useful to preserve child actors across restarts. The following section provides a pattern for this case. -Initialization via preStart ---------------------------- +### Initialization via preStart -The method ``preStart()`` of an actor is only called once directly during the initialization of the first instance, that -is, at creation of its ``ActorRef``. In the case of restarts, ``preStart()`` is called from ``postRestart()``, therefore -if not overridden, ``preStart()`` is called on every incarnation. However, overriding ``postRestart()`` one can disable -this behavior, and ensure that there is only one call to ``preStart()``. +The method `preStart()` of an actor is only called once directly during the initialization of the first instance, that +is, at creation of its `ActorRef`. In the case of restarts, `preStart()` is called from `postRestart()`, therefore +if not overridden, `preStart()` is called on every incarnation. However, overriding `postRestart()` one can disable +this behavior, and ensure that there is only one call to `preStart()`. -One useful usage of this pattern is to disable creation of new ``ActorRefs`` for children during restarts. This can be -achieved by overriding ``preRestart()``: +One useful usage of this pattern is to disable creation of new `ActorRefs` for children during restarts. This can be +achieved by overriding `preRestart()`: -.. includecode:: code/jdocs/actor/InitializationDocTest.java#preStartInit +@@snip [InitializationDocTest.java](code/jdocs/actor/InitializationDocTest.java) { #preStartInit } -Please note, that the child actors are *still restarted*, but no new ``ActorRef`` is created. One can recursively apply -the same principles for the children, ensuring that their ``preStart()`` method is called only at the creation of their +Please note, that the child actors are *still restarted*, but no new `ActorRef` is created. One can recursively apply +the same principles for the children, ensuring that their `preStart()` method is called only at the creation of their refs. -For more information see :ref:`supervision-restart`. +For more information see @ref:[What Restarting Means](../general/supervision.md#supervision-restart). -Initialization via message passing ----------------------------------- +### Initialization via message passing There are cases when it is impossible to pass all the information needed for actor initialization in the constructor, for example in the presence of circular dependencies. In this case the actor should listen for an initialization message, -and use ``become()`` or a finite state-machine state transition to encode the initialized and uninitialized states +and use `become()` or a finite state-machine state transition to encode the initialized and uninitialized states of the actor. -.. includecode:: code/jdocs/actor/InitializationDocTest.java#messageInit +@@snip [InitializationDocTest.java](code/jdocs/actor/InitializationDocTest.java) { #messageInit } -If the actor may receive messages before it has been initialized, a useful tool can be the ``Stash`` to save messages +If the actor may receive messages before it has been initialized, a useful tool can be the `Stash` to save messages until the initialization finishes, and replaying them after the actor became initialized. -.. warning:: - - This pattern should be used with care, and applied only when none of the patterns above are applicable. One of - the potential issues is that messages might be lost when sent to remote actors. Also, publishing an ``ActorRef`` in - an uninitialized state might lead to the condition that it receives a user message before the initialization has been - done. +@@@ warning +This pattern should be used with care, and applied only when none of the patterns above are applicable. One of +the potential issues is that messages might be lost when sent to remote actors. Also, publishing an `ActorRef` in +an uninitialized state might lead to the condition that it receives a user message before the initialization has been +done. +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/agents.md b/akka-docs/src/main/paradox/java/agents.md index 99385e09b0..dd4e2ae6fb 100644 --- a/akka-docs/src/main/paradox/java/agents.md +++ b/akka-docs/src/main/paradox/java/agents.md @@ -1,17 +1,15 @@ -.. _agents-java: +# Agents -Agents -###### +Agents in Akka are inspired by [agents in Clojure](http://clojure.org/agents). -Agents in Akka are inspired by `agents in Clojure`_. +@@@ warning -.. warning:: - **Deprecation warning** - Agents have been deprecated and are scheduled for removal - in the next major version. We have found that their leaky abstraction (they do not - work over the network) make them inferior to pure Actors, and in face of the soon - inclusion of Akka Typed we see little value in maintaining the current Agents. +**Deprecation warning** - Agents have been deprecated and are scheduled for removal +in the next major version. We have found that their leaky abstraction (they do not +work over the network) make them inferior to pure Actors, and in face of the soon +inclusion of Akka Typed we see little value in maintaining the current Agents. -.. _agents in Clojure: http://clojure.org/agents +@@@ Agents provide asynchronous change of individual locations. Agents are bound to a single storage location for their lifetime, and only allow mutation of that @@ -20,96 +18,76 @@ functions that are asynchronously applied to the Agent's state and whose return value becomes the Agent's new state. The state of an Agent should be immutable. While updates to Agents are asynchronous, the state of an Agent is always -immediately available for reading by any thread (using ``get``) without any messages. +immediately available for reading by any thread (using `get`) without any messages. Agents are reactive. The update actions of all Agents get interleaved amongst -threads in an ``ExecutionContext``. At any point in time, at most one ``send`` action for +threads in an `ExecutionContext`. At any point in time, at most one `send` action for each Agent is being executed. Actions dispatched to an agent from another thread will occur in the order they were sent, potentially interleaved with actions dispatched to the same agent from other threads. +## Creating Agents +Agents are created by invoking `new Agent(value, executionContext)` – passing in the Agent's initial +value and providing an `ExecutionContext` to be used for it: -Creating Agents -=============== +@@snip [AgentDocTest.java](code/jdocs/agent/AgentDocTest.java) { #import-agent #create type=java } -Agents are created by invoking ``new Agent(value, executionContext)`` – passing in the Agent's initial -value and providing an ``ExecutionContext`` to be used for it: - -.. includecode:: code/jdocs/agent/AgentDocTest.java - :include: import-agent,create - :language: java - -Reading an Agent's value -======================== +## Reading an Agent's value Agents can be dereferenced (you can get an Agent's value) by invoking the Agent -with ``get()`` like this: +with `get()` like this: -.. includecode:: code/jdocs/agent/AgentDocTest.java#read-get - :language: java +@@snip [AgentDocTest.java](code/jdocs/agent/AgentDocTest.java) { #read-get type=java } Reading an Agent's current value does not involve any message passing and happens immediately. So while updates to an Agent are asynchronous, reading the state of an Agent is synchronous. -You can also get a ``Future`` to the Agents value, that will be completed after the +You can also get a `Future` to the Agents value, that will be completed after the currently queued updates have completed: -.. includecode:: code/jdocs/agent/AgentDocTest.java - :include: import-future,read-future - :language: java +@@snip [AgentDocTest.java](code/jdocs/agent/AgentDocTest.java) { #import-future #read-future type=java } -See :ref:`futures-java` for more information on ``Futures``. +See @ref:[Futures](futures.md) for more information on `Futures`. -Updating Agents (send & alter) -============================== +## Updating Agents (send & alter) -You update an Agent by sending a function (``akka.dispatch.Mapper``) that transforms the current value or +You update an Agent by sending a function (`akka.dispatch.Mapper`) that transforms the current value or by sending just a new value. The Agent will apply the new value or function atomically and asynchronously. The update is done in a fire-forget manner and you are only guaranteed that it will be applied. There is no guarantee of when the update will be applied but dispatches to an Agent from a single thread will -occur in order. You apply a value or a function by invoking the ``send`` +occur in order. You apply a value or a function by invoking the `send` function. -.. includecode:: code/jdocs/agent/AgentDocTest.java - :include: import-function,send - :language: java +@@snip [AgentDocTest.java](code/jdocs/agent/AgentDocTest.java) { #import-function #send type=java } You can also dispatch a function to update the internal state but on its own thread. This does not use the reactive thread pool and can be used for -long-running or blocking operations. You do this with the ``sendOff`` -method. Dispatches using either ``sendOff`` or ``send`` will still be executed +long-running or blocking operations. You do this with the `sendOff` +method. Dispatches using either `sendOff` or `send` will still be executed in order. -.. includecode:: code/jdocs/agent/AgentDocTest.java - :include: import-function,send-off - :language: java +@@snip [AgentDocTest.java](code/jdocs/agent/AgentDocTest.java) { #import-function #send-off type=java } -All ``send`` methods also have a corresponding ``alter`` method that returns a ``Future``. -See :ref:`futures-java` for more information on ``Futures``. +All `send` methods also have a corresponding `alter` method that returns a `Future`. +See @ref:[Futures](futures.md) for more information on `Futures`. -.. includecode:: code/jdocs/agent/AgentDocTest.java - :include: import-future,import-function,alter - :language: java +@@snip [AgentDocTest.java](code/jdocs/agent/AgentDocTest.java) { #import-future #import-function #alter type=java } -.. includecode:: code/jdocs/agent/AgentDocTest.java - :include: import-future,import-function,alter-off - :language: java +@@snip [AgentDocTest.java](code/jdocs/agent/AgentDocTest.java) { #import-future #import-function #alter-off type=java } -Configuration -============= +## Configuration There are several configuration properties for the agents module, please refer -to the :ref:`reference configuration `. +to the @ref:[reference configuration](../general/configuration.md#config-akka-agent). -Deprecated Transactional Agents -=============================== +## Deprecated Transactional Agents Agents participating in enclosing STM transaction is a deprecated feature in 2.3. -If an Agent is used within an enclosing ``Scala STM transaction``, then it will participate in +If an Agent is used within an enclosing `Scala STM transaction`, then it will participate in that transaction. If you send to an Agent within a transaction then the dispatch to the Agent will be held until that transaction commits, and discarded if the -transaction is aborted. +transaction is aborted. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/camel.md b/akka-docs/src/main/paradox/java/camel.md index 40b095cd69..21b5c5710f 100644 --- a/akka-docs/src/main/paradox/java/camel.md +++ b/akka-docs/src/main/paradox/java/camel.md @@ -1,17 +1,12 @@ +# Camel -.. _camel-java: +@@@ warning -Camel -##### +Akka Camel is deprecated in favour of [Alpakka](https://github.com/akka/alpakka) , the Akka Streams based collection of integrations to various endpoints (including Camel). -.. warning:: - Akka Camel is deprecated in favour of `Alpakka`_ , the Akka Streams based collection of integrations to various endpoints (including Camel). +@@@ -.. _Alpakka: https://github.com/akka/alpakka - - -Introduction -============ +## Introduction The akka-camel module allows Untyped Actors to receive and send messages over a great variety of protocols and APIs. @@ -19,59 +14,51 @@ In addition to the native Scala and Java actor API, actors can now exchange mess of protocols and APIs such as HTTP, SOAP, TCP, FTP, SMTP or JMS, to mention a few. At the moment, approximately 80 protocols and APIs are supported. -Apache Camel ------------- -The akka-camel module is based on `Apache Camel`_, a powerful and light-weight +### Apache Camel + +The akka-camel module is based on [Apache Camel](http://camel.apache.org/), a powerful and light-weight integration framework for the JVM. For an introduction to Apache Camel you may -want to read this `Apache Camel article`_. Camel comes with a -large number of `components`_ that provide bindings to different protocols and -APIs. The `camel-extra`_ project provides further components. +want to read this [Apache Camel article](http://architects.dzone.com/articles/apache-camel-integration). Camel comes with a +large number of [components](http://camel.apache.org/components.html) that provide bindings to different protocols and +APIs. The [camel-extra](http://code.google.com/p/camel-extra/) project provides further components. -.. _Apache Camel: http://camel.apache.org/ -.. _Apache Camel article: http://architects.dzone.com/articles/apache-camel-integration -.. _components: http://camel.apache.org/components.html -.. _camel-extra: http://code.google.com/p/camel-extra/ +### Consumer -Consumer --------- Here's an example of using Camel's integration components in Akka. -.. includecode:: code/jdocs/camel/MyEndpoint.java#Consumer-mina +@@snip [MyEndpoint.java](code/jdocs/camel/MyEndpoint.java) { #Consumer-mina } The above example exposes an actor over a TCP endpoint via Apache -Camel's `Mina component`_. The actor implements the `getEndpointUri` method to define +Camel's [Mina component](http://camel.apache.org/mina2.html). The actor implements the *getEndpointUri* method to define an endpoint from which it can receive messages. After starting the actor, TCP clients can immediately send messages to and receive responses from that -actor. If the message exchange should go over HTTP (via Camel's `Jetty -component`_), the actor's `getEndpointUri` method should return a different URI, for instance "jetty:http://localhost:8877/example". +actor. If the message exchange should go over HTTP (via Camel's `Jetty +component`_), the actor's *getEndpointUri* method should return a different URI, for instance "jetty:[http://localhost:8877/example](http://localhost:8877/example)". In the above case an extra constructor is added that can set the endpoint URI, which would result in -the `getEndpointUri` returning the URI that was set using this constructor. +the *getEndpointUri* returning the URI that was set using this constructor. -.. _Mina component: http://camel.apache.org/mina2.html -.. _Jetty component: http://camel.apache.org/jetty.html +### Producer -Producer --------- Actors can also trigger message exchanges with external systems i.e. produce to Camel endpoints. -.. includecode:: code/jdocs/camel/Orders.java#Producer +@@snip [Orders.java](code/jdocs/camel/Orders.java) { #Producer } In the above example, any message sent to this actor will be sent to -the JMS queue ``Orders``. Producer actors may choose from the same set of Camel +the JMS queue `Orders`. Producer actors may choose from the same set of Camel components as Consumer actors do. Below an example of how to send a message to the Orders producer. -.. includecode:: code/jdocs/camel/ProducerTestBase.java#TellProducer +@@snip [ProducerTestBase.java](code/jdocs/camel/ProducerTestBase.java) { #TellProducer } + +### CamelMessage -CamelMessage ------------- The number of Camel components is constantly increasing. The akka-camel module can support these in a plug-and-play manner. Just add them to your application's classpath, define a component-specific endpoint URI and use it to exchange messages over the component-specific protocols or APIs. This is possible because Camel components bind protocol-specific message formats to a Camel-specific -`normalized message format`__. The normalized message format hides +[normalized message format](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Message.java). The normalized message format hides protocol-specific details from Akka and makes it therefore very easy to support a large number of protocols through a uniform Camel component interface. The akka-camel module further converts mutable Camel messages into immutable @@ -80,103 +67,79 @@ matching, transformation, serialization or storage. In the above example of the the XML message is put in the body of a newly created Camel Message with an empty set of headers. You can also create a CamelMessage yourself with the appropriate body and headers as you see fit. -__ https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Message.java +### CamelExtension -CamelExtension --------------- -The akka-camel module is implemented as an Akka Extension, the ``CamelExtension`` object. -Extensions will only be loaded once per ``ActorSystem``, which will be managed by Akka. -The ``CamelExtension`` object provides access to the `Camel`_ interface. -The `Camel`_ interface in turn provides access to two important Apache Camel objects, the `CamelContext`_ and the `ProducerTemplate`_. +The akka-camel module is implemented as an Akka Extension, the `CamelExtension` object. +Extensions will only be loaded once per `ActorSystem`, which will be managed by Akka. +The `CamelExtension` object provides access to the [Camel](@github@/akka-camel/src/main/scala/akka/camel/Camel.scala) interface. +The [Camel](@github@/akka-camel/src/main/scala/akka/camel/Camel.scala) interface in turn provides access to two important Apache Camel objects, the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) and the `ProducerTemplate`_. Below you can see how you can get access to these Apache Camel objects. -.. includecode:: code/jdocs/camel/CamelExtensionTest.java#CamelExtension +@@snip [CamelExtensionTest.java](code/jdocs/camel/CamelExtensionTest.java) { #CamelExtension } -One ``CamelExtension`` is only loaded once for every one ``ActorSystem``, which makes it safe to call the ``CamelExtension`` at any point in your code to get to the -Apache Camel objects associated with it. There is one `CamelContext`_ and one `ProducerTemplate`_ for every one ``ActorSystem`` that uses a ``CamelExtension``. -By Default, a new `CamelContext`_ is created when the ``CamelExtension`` starts. If you want to inject your own context instead, -you can implement the `ContextProvider`_ interface and add the FQCN of your implementation in the config, as the value of the "akka.camel.context-provider". -This interface define a single method ``getContext()`` used to load the `CamelContext`_. +One `CamelExtension` is only loaded once for every one `ActorSystem`, which makes it safe to call the `CamelExtension` at any point in your code to get to the +Apache Camel objects associated with it. There is one [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) and one `ProducerTemplate`_ for every one `ActorSystem` that uses a `CamelExtension`. +By Default, a new [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) is created when the `CamelExtension` starts. If you want to inject your own context instead, +you can implement the [ContextProvider](@github@/akka-camel/src/main/scala/akka/camel/ContextProvider.scala) interface and add the FQCN of your implementation in the config, as the value of the "akka.camel.context-provider". +This interface define a single method `getContext()` used to load the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java). -.. _ContextProvider: @github@/akka-camel/src/main/scala/akka/camel/ContextProvider.scala +Below an example on how to add the ActiveMQ component to the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java), which is required when you would like to use the ActiveMQ component. -Below an example on how to add the ActiveMQ component to the `CamelContext`_, which is required when you would like to use the ActiveMQ component. +@@snip [CamelExtensionTest.java](code/jdocs/camel/CamelExtensionTest.java) { #CamelExtensionAddComponent } -.. includecode:: code/jdocs/camel/CamelExtensionTest.java#CamelExtensionAddComponent +The [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) joins the lifecycle of the `ActorSystem` and `CamelExtension` it is associated with; the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) is started when +the `CamelExtension` is created, and it is shut down when the associated `ActorSystem` is shut down. The same is true for the `ProducerTemplate`_. -The `CamelContext`_ joins the lifecycle of the ``ActorSystem`` and ``CamelExtension`` it is associated with; the `CamelContext`_ is started when -the ``CamelExtension`` is created, and it is shut down when the associated ``ActorSystem`` is shut down. The same is true for the `ProducerTemplate`_. - -The ``CamelExtension`` is used by both `Producer` and `Consumer` actors to interact with Apache Camel internally. -You can access the ``CamelExtension`` inside a `Producer` or a `Consumer` using the ``camel`` method, or get straight at the `CamelContext` -using the ``getCamelContext`` method or to the `ProducerTemplate` using the `getProducerTemplate` method. -Actors are created and started asynchronously. When a `Consumer` actor is created, the `Consumer` is published at its Camel endpoint -(more precisely, the route is added to the `CamelContext`_ from the `Endpoint`_ to the actor). -When a `Producer` actor is created, a `SendProcessor`_ and `Endpoint`_ are created so that the Producer can send messages to it. +The `CamelExtension` is used by both *Producer* and *Consumer* actors to interact with Apache Camel internally. +You can access the `CamelExtension` inside a *Producer* or a *Consumer* using the `camel` method, or get straight at the *CamelContext* +using the `getCamelContext` method or to the *ProducerTemplate* using the *getProducerTemplate* method. +Actors are created and started asynchronously. When a *Consumer* actor is created, the *Consumer* is published at its Camel endpoint +(more precisely, the route is added to the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) from the [Endpoint](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Endpoint.java) to the actor). +When a *Producer* actor is created, a [SendProcessor](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/processor/SendProcessor.java) and [Endpoint](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Endpoint.java) are created so that the Producer can send messages to it. Publication is done asynchronously; setting up an endpoint may still be in progress after you have requested the actor to be created. Some Camel components can take a while to startup, and in some cases you might want to know when the endpoints are activated and ready to be used. -The `Camel`_ interface allows you to find out when the endpoint is activated or deactivated. +The [Camel](@github@/akka-camel/src/main/scala/akka/camel/Camel.scala) interface allows you to find out when the endpoint is activated or deactivated. -.. includecode:: code/jdocs/camel/ActivationTestBase.java#CamelActivation +@@snip [ActivationTestBase.java](code/jdocs/camel/ActivationTestBase.java) { #CamelActivation } -The above code shows that you can get a ``Future`` to the activation of the route from the endpoint to the actor, or you can wait in a blocking fashion on the activation of the route. -An ``ActivationTimeoutException`` is thrown if the endpoint could not be activated within the specified timeout. Deactivation works in a similar fashion: +The above code shows that you can get a `Future` to the activation of the route from the endpoint to the actor, or you can wait in a blocking fashion on the activation of the route. +An `ActivationTimeoutException` is thrown if the endpoint could not be activated within the specified timeout. Deactivation works in a similar fashion: -.. includecode:: code/jdocs/camel/ActivationTestBase.java#CamelDeactivation +@@snip [ActivationTestBase.java](code/jdocs/camel/ActivationTestBase.java) { #CamelDeactivation } -Deactivation of a Consumer or a Producer actor happens when the actor is terminated. For a Consumer, the route to the actor is stopped. For a Producer, the `SendProcessor`_ is stopped. -A ``DeActivationTimeoutException`` is thrown if the associated camel objects could not be deactivated within the specified timeout. +Deactivation of a Consumer or a Producer actor happens when the actor is terminated. For a Consumer, the route to the actor is stopped. For a Producer, the [SendProcessor](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/processor/SendProcessor.java) is stopped. +A `DeActivationTimeoutException` is thrown if the associated camel objects could not be deactivated within the specified timeout. -.. _Camel: @github@/akka-camel/src/main/scala/akka/camel/Camel.scala -.. _CamelContext: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java -.. _ProducerTemplate: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/ProducerTemplate.java -.. _SendProcessor: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/processor/SendProcessor.java -.. _Endpoint: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Endpoint.java +## Consumer Actors -Consumer Actors -================ - -For objects to receive messages, they must inherit from the `UntypedConsumerActor`_ +For objects to receive messages, they must inherit from the [UntypedConsumerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedConsumer.scala) class. For example, the following actor class (Consumer1) implements the -`getEndpointUri` method, which is declared in the `UntypedConsumerActor`_ class, in order to receive -messages from the ``file:data/input/actor`` Camel endpoint. +*getEndpointUri* method, which is declared in the [UntypedConsumerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedConsumer.scala) class, in order to receive +messages from the `file:data/input/actor` Camel endpoint. -.. _UntypedConsumerActor: @github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedConsumer.scala - -.. includecode:: code/jdocs/camel/Consumer1.java#Consumer1 +@@snip [Consumer1.java](code/jdocs/camel/Consumer1.java) { #Consumer1 } Whenever a file is put into the data/input/actor directory, its content is -picked up by the Camel `file component`_ and sent as message to the +picked up by the Camel [file component](http://camel.apache.org/file2.html) and sent as message to the actor. Messages consumed by actors from Camel endpoints are of type -`CamelMessage`_. These are immutable representations of Camel messages. - -.. _file component: http://camel.apache.org/file2.html -.. _Message: @github@/akka-camel/src/main/scala/akka/camel/CamelMessage.scala - +[CamelMessage](#camelmessage). These are immutable representations of Camel messages. Here's another example that sets the endpointUri to -``jetty:http://localhost:8877/camel/default``. It causes Camel's `Jetty -component`_ to start an embedded `Jetty`_ server, accepting HTTP connections +`jetty:http://localhost:8877/camel/default`. It causes Camel's `Jetty +component`_ to start an embedded [Jetty](http://www.eclipse.org/jetty/) server, accepting HTTP connections from localhost on port 8877. -.. _Jetty component: http://camel.apache.org/jetty.html -.. _Jetty: http://www.eclipse.org/jetty/ - -.. includecode:: code/jdocs/camel/Consumer2.java#Consumer2 +@@snip [Consumer2.java](code/jdocs/camel/Consumer2.java) { #Consumer2 } After starting the actor, clients can send messages to that actor by POSTing to -``http://localhost:8877/camel/default``. The actor sends a response by using the -``getSender().tell`` method. For returning a message body and headers to the HTTP -client the response type should be `CamelMessage`_. For any other response type, a +`http://localhost:8877/camel/default`. The actor sends a response by using the +`getSender().tell` method. For returning a message body and headers to the HTTP +client the response type should be [CamelMessage](#camelmessage). For any other response type, a new CamelMessage object is created by akka-camel with the actor response as message body. -.. _Message: @github@/akka-camel/src/main/scala/akka/camel/CamelMessage.scala - -.. _camel-acknowledgements-java: - -Delivery acknowledgements -------------------------- + +### Delivery acknowledgements With in-out message exchanges, clients usually know that a message exchange is done when they receive a reply from a consumer actor. The reply message can be a @@ -188,163 +151,144 @@ is added to the consumer actor's mailbox. Any failure or exception that occurs during processing of that message by the consumer actor cannot be reported back to the endpoint in this case. To allow consumer actors to positively or negatively acknowledge the receipt of a message from an in-only message -exchange, they need to override the ``autoAck`` method to return false. +exchange, they need to override the `autoAck` method to return false. In this case, consumer actors must reply either with a special akka.camel.Ack message (positive acknowledgement) or a akka.actor.Status.Failure (negative acknowledgement). -.. includecode:: code/jdocs/camel/Consumer3.java#Consumer3 +@@snip [Consumer3.java](code/jdocs/camel/Consumer3.java) { #Consumer3 } -.. _camel-timeout-java: - -Consumer timeout ----------------- + +### Consumer timeout Camel Exchanges (and their corresponding endpoints) that support two-way communications need to wait for a response from an actor before returning it to the initiating client. For some endpoint types, timeout values can be defined in an endpoint-specific -way which is described in the documentation of the individual `Camel +way which is described in the documentation of the individual `Camel components`_. Another option is to configure timeouts on the level of consumer actors. -.. _Camel components: http://camel.apache.org/components.html - Two-way communications between a Camel endpoint and an actor are -initiated by sending the request message to the actor with the `ask`_ pattern +initiated by sending the request message to the actor with the [ask](@github@/akka-actor/src/main/scala/akka/pattern/Patterns.scala) pattern and the actor replies to the endpoint when the response is ready. The ask request to the actor can timeout, which will -result in the `Exchange`_ failing with a TimeoutException set on the failure of the `Exchange`_. -The timeout on the consumer actor can be overridden with the ``replyTimeout``, as shown below. +result in the [Exchange](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Exchange.java) failing with a TimeoutException set on the failure of the [Exchange](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Exchange.java). +The timeout on the consumer actor can be overridden with the `replyTimeout`, as shown below. -.. includecode:: code/jdocs/camel/Consumer4.java#Consumer4 -.. _Exchange: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Exchange.java -.. _ask: @github@/akka-actor/src/main/scala/akka/pattern/Patterns.scala +@@snip [Consumer4.java](code/jdocs/camel/Consumer4.java) { #Consumer4 } -Producer Actors -=============== +## Producer Actors -For sending messages to Camel endpoints, actors need to inherit from the `UntypedProducerActor`_ class and implement the getEndpointUri method. +For sending messages to Camel endpoints, actors need to inherit from the [UntypedProducerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedProducerActor.scala) class and implement the getEndpointUri method. -.. includecode:: code/jdocs/camel/Producer1.java#Producer1 +@@snip [Producer1.java](code/jdocs/camel/Producer1.java) { #Producer1 } Producer1 inherits a default implementation of the onReceive method from the -`UntypedProducerActor`_ class. To customize a producer actor's default behavior you must override the `UntypedProducerActor`_.onTransformResponse and -`UntypedProducerActor`_.onTransformOutgoingMessage methods. This is explained later in more detail. -Producer Actors cannot override the `UntypedProducerActor`_.onReceive method. +[UntypedProducerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedProducerActor.scala) class. To customize a producer actor's default behavior you must override the [UntypedProducerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedProducerActor.scala).onTransformResponse and +[UntypedProducerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedProducerActor.scala).onTransformOutgoingMessage methods. This is explained later in more detail. +Producer Actors cannot override the [UntypedProducerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedProducerActor.scala).onReceive method. Any message sent to a Producer actor will be sent to the associated Camel endpoint, in the above example to -``http://localhost:8080/news``. The `UntypedProducerActor`_ always sends messages asynchronously. Response messages (if supported by the +`http://localhost:8080/news`. The [UntypedProducerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedProducerActor.scala) always sends messages asynchronously. Response messages (if supported by the configured endpoint) will, by default, be returned to the original sender. The following example uses the ask pattern to send a message to a Producer actor and waits for a response. -.. includecode:: code/jdocs/camel/ProducerTestBase.java#AskProducer +@@snip [ProducerTestBase.java](code/jdocs/camel/ProducerTestBase.java) { #AskProducer } -The future contains the response CamelMessage, or an ``AkkaCamelException`` when an error occurred, which contains the headers of the response. +The future contains the response CamelMessage, or an `AkkaCamelException` when an error occurred, which contains the headers of the response. -.. _camel-custom-processing-java: - -Custom Processing ------------------ + +### Custom Processing Instead of replying to the initial sender, producer actors can implement custom response processing by overriding the onRouteResponse method. In the following example, the response message is forwarded to a target actor instead of being replied to the original sender. -.. includecode:: code/jdocs/camel/ResponseReceiver.java#RouteResponse -.. includecode:: code/jdocs/camel/Forwarder.java#RouteResponse -.. includecode:: code/jdocs/camel/OnRouteResponseTestBase.java#RouteResponse +@@snip [ResponseReceiver.java](code/jdocs/camel/ResponseReceiver.java) { #RouteResponse } + +@@snip [Forwarder.java](code/jdocs/camel/Forwarder.java) { #RouteResponse } + +@@snip [OnRouteResponseTestBase.java](code/jdocs/camel/OnRouteResponseTestBase.java) { #RouteResponse } Before producing messages to endpoints, producer actors can pre-process them by -overriding the `UntypedProducerActor`_.onTransformOutgoingMessage method. +overriding the [UntypedProducerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedProducerActor.scala).onTransformOutgoingMessage method. -.. includecode:: code/jdocs/camel/Transformer.java#TransformOutgoingMessage +@@snip [Transformer.java](code/jdocs/camel/Transformer.java) { #TransformOutgoingMessage } -Producer configuration options ------------------------------- +### Producer configuration options The interaction of producer actors with Camel endpoints can be configured to be one-way or two-way (by initiating in-only or in-out message exchanges, respectively). By default, the producer initiates an in-out message exchange with the endpoint. For initiating an in-only exchange, producer actors have to override the isOneway method to return true. -.. includecode:: code/jdocs/camel/OnewaySender.java#Oneway +@@snip [OnewaySender.java](code/jdocs/camel/OnewaySender.java) { #Oneway } -Message correlation -------------------- +### Message correlation To correlate request with response messages, applications can set the -`Message.MessageExchangeId` message header. +*Message.MessageExchangeId* message header. -.. includecode:: code/jdocs/camel/ProducerTestBase.java#Correlate +@@snip [ProducerTestBase.java](code/jdocs/camel/ProducerTestBase.java) { #Correlate } -ProducerTemplate ----------------- +### ProducerTemplate -The `UntypedProducerActor`_ class is a very convenient way for actors to produce messages to Camel endpoints. -Actors may also use a Camel `ProducerTemplate`_ for producing messages to endpoints. +The [UntypedProducerActor](@github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedProducerActor.scala) class is a very convenient way for actors to produce messages to Camel endpoints. +Actors may also use a Camel `ProducerTemplate`_ for producing messages to endpoints. -.. includecode:: code/jdocs/camel/MyActor.java#ProducerTemplate +@@snip [MyActor.java](code/jdocs/camel/MyActor.java) { #ProducerTemplate } For initiating a two-way message exchange, one of the -``ProducerTemplate.request*`` methods must be used. +`ProducerTemplate.request*` methods must be used. -.. includecode:: code/jdocs/camel/RequestBodyActor.java#RequestProducerTemplate +@@snip [RequestBodyActor.java](code/jdocs/camel/RequestBodyActor.java) { #RequestProducerTemplate } -.. _UntypedProducerActor: @github@/akka-camel/src/main/scala/akka/camel/javaapi/UntypedProducerActor.scala -.. _ProducerTemplate: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/ProducerTemplate.java - -.. _camel-asynchronous-routing-java: - -Asynchronous routing -==================== + +## Asynchronous routing In-out message exchanges between endpoints and actors are designed to be asynchronous. This is the case for both, consumer and producer actors. -* A consumer endpoint sends request messages to its consumer actor using the ``tell`` - method and the actor returns responses with ``getSender().tell`` once they are - ready. - -* A producer actor sends request messages to its endpoint using Camel's - asynchronous routing engine. Asynchronous responses are wrapped and added to the - producer actor's mailbox for later processing. By default, response messages are - returned to the initial sender but this can be overridden by Producer - implementations (see also description of the ``onRouteResponse`` method - in :ref:`camel-custom-processing-java`). + * A consumer endpoint sends request messages to its consumer actor using the `tell` +method and the actor returns responses with `getSender().tell` once they are +ready. + * A producer actor sends request messages to its endpoint using Camel's +asynchronous routing engine. Asynchronous responses are wrapped and added to the +producer actor's mailbox for later processing. By default, response messages are +returned to the initial sender but this can be overridden by Producer +implementations (see also description of the `onRouteResponse` method +in [Custom Processing](#camel-custom-processing-java)). However, asynchronous two-way message exchanges, without allocating a thread for the full duration of exchange, cannot be generically supported by Camel's asynchronous routing engine alone. This must be supported by the individual -`Camel components`_ (from which endpoints are created) as well. They must be + `Camel components`_ (from which endpoints are created) as well. They must be able to suspend any work started for request processing (thereby freeing threads to do other work) and resume processing when the response is ready. This is -currently the case for a `subset of components`_ such as the `Jetty component`_. +currently the case for a [subset of components](http://camel.apache.org/asynchronous-routing-engine.html) such as the `Jetty component`_. All other Camel components can still be used, of course, but they will cause allocation of a thread for the duration of an in-out message exchange. There's -also :ref:`camel-examples-java` that implements both, an asynchronous +also [Examples](#camel-examples-java) that implements both, an asynchronous consumer and an asynchronous producer, with the jetty component. If the used Camel component is blocking it might be necessary to use a separate -:ref:`dispatcher ` for the producer. The Camel processor is +@ref:[dispatcher](dispatchers.md) for the producer. The Camel processor is invoked by a child actor of the producer and the dispatcher can be defined in the deployment section of the configuration. For example, if your producer actor -has path ``/user/integration/output`` the dispatcher of the child actor can be -defined with:: +has path `/user/integration/output` the dispatcher of the child actor can be +defined with: - akka.actor.deployment { - /integration/output/* { - dispatcher = my-dispatcher - } +``` +akka.actor.deployment { + /integration/output/* { + dispatcher = my-dispatcher } +} +``` -.. _Camel components: http://camel.apache.org/components.html -.. _subset of components: http://camel.apache.org/asynchronous-routing-engine.html -.. _Jetty component: http://camel.apache.org/jetty.html - -Custom Camel routes -=================== +## Custom Camel routes In all the examples so far, routes to consumer actors have been automatically constructed by akka-camel, when the actor was started. Although the default @@ -353,107 +297,87 @@ most use cases, some applications may require more specialized routes to actors. The akka-camel module provides two mechanisms for customizing routes to actors, which will be explained in this section. These are: -* Usage of :ref:`camel-components-java` to access actors. - Any Camel route can use these components to access Akka actors. + * Usage of [Akka Camel components](#camel-components-java) to access actors. +Any Camel route can use these components to access Akka actors. + * [Intercepting route construction](#camel-intercepting-route-construction-java) to actors. +This option gives you the ability to change routes that have already been added to Camel. +Consumer actors have a hook into the route definition process which can be used to change the route. -* :ref:`camel-intercepting-route-construction-java` to actors. - This option gives you the ability to change routes that have already been added to Camel. - Consumer actors have a hook into the route definition process which can be used to change the route. + +### Akka Camel components - -.. _camel-components-java: - -Akka Camel components ---------------------- - -Akka actors can be accessed from Camel routes using the `actor`_ Camel component. This component can be used to +Akka actors can be accessed from Camel routes using the `actor`_ Camel component. This component can be used to access any Akka actor (not only consumer actors) from Camel routes, as described in the following sections. -.. _actor: @github@/akka-camel/src/main/scala/akka/camel/internal/component/ActorComponent.scala + +### Access to actors -.. _access-to-actors-java: - -Access to actors ----------------- - -To access actors from custom Camel routes, the `actor`_ Camel -component should be used. It fully supports Camel's `asynchronous routing -engine`_. - -.. _actor: @github@/akka-camel/src/main/scala/akka/camel/internal/component/ActorComponent.scala -.. _asynchronous routing engine: http://camel.apache.org/asynchronous-routing-engine.html +To access actors from custom Camel routes, the `actor`_ Camel +component should be used. It fully supports Camel's [asynchronous routing +engine](http://camel.apache.org/asynchronous-routing-engine.html). This component accepts the following endpoint URI format: -* ``[]?`` + * `[]?` -where ```` is the ``ActorPath`` to the actor. The ```` are -name-value pairs separated by ``&`` (i.e. ``name1=value1&name2=value2&...``). +where `` is the `ActorPath` to the actor. The `` are +name-value pairs separated by `&` (i.e. `name1=value1&name2=value2&...`). - -URI options -^^^^^^^^^^^ +#### URI options The following URI options are supported: -.. tabularcolumns:: |l|l|l|L| +|Name | Type | Default | Description | +|-------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|replyTimeout | Duration | false | +The reply timeout, specified in the same +way that you use the duration in akka, +for instance `10 seconds` except that +in the url it is handy to use a + +between the amount and the unit, like +for example `200+millis` +See also [Consumer timeout](#camel-timeout-java).| +|autoAck | Boolean | true | +If set to true, in-only message exchanges +are auto-acknowledged when the message is +added to the actor's mailbox. If set to +false, actors must acknowledge the +receipt of the message. +See also [Delivery acknowledgements](#camel-acknowledgements-java). | -+--------------+----------+---------+------------------------------------------------+ -| Name | Type | Default | Description | -+==============+==========+=========+================================================+ -| replyTimeout | Duration | false | The reply timeout, specified in the same | -| | | | way that you use the duration in akka, | -| | | | for instance ``10 seconds`` except that | -| | | | in the url it is handy to use a + | -| | | | between the amount and the unit, like | -| | | | for example ``200+millis`` | -| | | | | -| | | | See also :ref:`camel-timeout-java`. | -+--------------+----------+---------+------------------------------------------------+ -| autoAck | Boolean | true | If set to true, in-only message exchanges | -| | | | are auto-acknowledged when the message is | -| | | | added to the actor's mailbox. If set to | -| | | | false, actors must acknowledge the | -| | | | receipt of the message. | -| | | | | -| | | | See also :ref:`camel-acknowledgements-java`. | -+--------------+----------+---------+------------------------------------------------+ +Here's an actor endpoint URI example containing an actor path: -Here's an actor endpoint URI example containing an actor path:: - - akka://some-system/user/myconsumer?autoAck=false&replyTimeout=100+millis +``` +akka://some-system/user/myconsumer?autoAck=false&replyTimeout=100+millis +``` In the following example, a custom route to an actor is created, using the actor's path. -.. includecode:: code/jdocs/camel/Responder.java#CustomRoute -.. includecode:: code/jdocs/camel/CustomRouteBuilder.java#CustomRoute -.. includecode:: code/jdocs/camel/CustomRouteTestBase.java#CustomRoute +@@snip [Responder.java](code/jdocs/camel/Responder.java) { #CustomRoute } -The `CamelPath.toCamelUri` converts the `ActorRef` to the Camel actor component URI format which points to the actor endpoint as described above. +@@snip [CustomRouteBuilder.java](code/jdocs/camel/CustomRouteBuilder.java) { #CustomRoute } + +@@snip [CustomRouteTestBase.java](code/jdocs/camel/CustomRouteTestBase.java) { #CustomRoute } + +The *CamelPath.toCamelUri* converts the *ActorRef* to the Camel actor component URI format which points to the actor endpoint as described above. When a message is received on the jetty endpoint, it is routed to the Responder actor, which in return replies back to the client of the HTTP request. + +### Intercepting route construction -.. _camel-intercepting-route-construction-java: - -Intercepting route construction -------------------------------- - -The previous section, :ref:`camel-components-java`, explained how to setup a route to +The previous section, [Akka Camel components](#camel-components-java), explained how to setup a route to an actor manually. It was the application's responsibility to define the route and add it to the current CamelContext. This section explains a more convenient way to define custom routes: akka-camel is still setting up the routes to consumer actors (and adds these routes to the current CamelContext) but applications can define extensions to these routes. -Extensions can be defined with Camel's `Java DSL`_ or `Scala DSL`_. For example, an extension could be a custom error handler that redelivers messages from an endpoint to an actor's bounded mailbox when the mailbox was full. - -.. _Java DSL: http://camel.apache.org/dsl.html -.. _Scala DSL: http://camel.apache.org/scala-dsl.html +Extensions can be defined with Camel's [Java DSL](http://camel.apache.org/dsl.html) or [Scala DSL](http://camel.apache.org/scala-dsl.html). For example, an extension could be a custom error handler that redelivers messages from an endpoint to an actor's bounded mailbox when the mailbox was full. The following examples demonstrate how to extend a route to a consumer actor for handling exceptions thrown by that actor. -.. includecode:: code/jdocs/camel/ErrorThrowingConsumer.java#ErrorThrowingConsumer +@@snip [ErrorThrowingConsumer.java](code/jdocs/camel/ErrorThrowingConsumer.java) { #ErrorThrowingConsumer } The above ErrorThrowingConsumer sends the Failure back to the sender in preRestart because the Exception that is thrown in the actor would @@ -462,54 +386,44 @@ otherwise just crash the actor, by default the actor would be restarted, and the The akka-camel module creates a RouteDefinition instance by calling from(endpointUri) on a Camel RouteBuilder (where endpointUri is the endpoint URI of the consumer actor) and passes that instance as argument to the route -definition handler \*). The route definition handler then extends the route and +definition handler *). The route definition handler then extends the route and returns a ProcessorDefinition (in the above example, the ProcessorDefinition -returned by the end method. See the `org.apache.camel.model`__ package for +returned by the end method. See the [org.apache.camel.model](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/model/) package for details). After executing the route definition handler, akka-camel finally calls a to(targetActorUri) on the returned ProcessorDefinition to complete the -route to the consumer actor (where targetActorUri is the actor component URI as described in :ref:`access-to-actors-java`). -If the actor cannot be found, a `ActorNotRegisteredException` is thrown. +route to the consumer actor (where targetActorUri is the actor component URI as described in [Access to actors](#access-to-actors-java)). +If the actor cannot be found, a *ActorNotRegisteredException* is thrown. -\*) Before passing the RouteDefinition instance to the route definition handler, +*) Before passing the RouteDefinition instance to the route definition handler, akka-camel may make some further modifications to it. -__ https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/model/ + +## Examples -.. _camel-examples-java: - -Examples -======== - -The sample named `Akka Camel Samples with Java <@exampleCodeService@/akka-samples-camel-java>`_ (`source code <@samples@/akka-sample-camel-java>`_) +The sample named [Akka Camel Samples with Java](@exampleCodeService@/akka-samples-camel-java) ([source code](@samples@/akka-sample-camel-java)) contains 3 samples: +> * Asynchronous routing and transformation - This example demonstrates how to implement consumer and - producer actors that support :ref:`camel-asynchronous-routing-java` with their Camel endpoints. - - * Custom Camel route - Demonstrates the combined usage of a ``Producer`` and a - ``Consumer`` actor as well as the inclusion of a custom Camel route. - +producer actors that support [Asynchronous routing](#camel-asynchronous-routing-java) with their Camel endpoints. + * Custom Camel route - Demonstrates the combined usage of a `Producer` and a +`Consumer` actor as well as the inclusion of a custom Camel route. * Quartz Scheduler Example - Showing how simple is to implement a cron-style scheduler by - using the Camel Quartz component +using the Camel Quartz component -Configuration -============= +## Configuration There are several configuration properties for the Camel module, please refer -to the :ref:`reference configuration `. +to the @ref:[reference configuration](../general/configuration.md#config-akka-camel). -Additional Resources -==================== -For an introduction to akka-camel 2, see also the Peter Gabryanczyk's talk `Migrating akka-camel module to Akka 2.x`_. +## Additional Resources -For an introduction to akka-camel 1, see also the `Appendix E - Akka and Camel`_ -(pdf) of the book `Camel in Action`_. +For an introduction to akka-camel 2, see also the Peter Gabryanczyk's talk [Migrating akka-camel module to Akka 2.x](http://skillsmatter.com/podcast/scala/akka-2-x). -.. _Appendix E - Akka and Camel: http://www.manning.com/ibsen/appEsample.pdf -.. _Camel in Action: http://www.manning.com/ibsen/ -.. _Migrating akka-camel module to Akka 2.x: http://skillsmatter.com/podcast/scala/akka-2-x +For an introduction to akka-camel 1, see also the [Appendix E - Akka and Camel](http://www.manning.com/ibsen/appEsample.pdf) +(pdf) of the book [Camel in Action](http://www.manning.com/ibsen/). Other, more advanced external articles (for version 1) are: -* `Akka Consumer Actors: New Features and Best Practices `_ -* `Akka Producer Actors: New Features and Best Practices `_ + * [Akka Consumer Actors: New Features and Best Practices](http://krasserm.blogspot.com/2011/02/akka-consumer-actors-new-features-and.html) + * [Akka Producer Actors: New Features and Best Practices](http://krasserm.blogspot.com/2011/02/akka-producer-actor-new-features-and.html) \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/cluster-client.md b/akka-docs/src/main/paradox/java/cluster-client.md index 0c5a627a58..29b3e44f8c 100644 --- a/akka-docs/src/main/paradox/java/cluster-client.md +++ b/akka-docs/src/main/paradox/java/cluster-client.md @@ -1,53 +1,52 @@ -.. _cluster-client-java: - -Cluster Client -============== +# Cluster Client An actor system that is not part of the cluster can communicate with actors -somewhere in the cluster via this ``ClusterClient``. The client can of course be part of +somewhere in the cluster via this `ClusterClient`. The client can of course be part of another cluster. It only needs to know the location of one (or more) nodes to use as initial -contact points. It will establish a connection to a ``ClusterReceptionist`` somewhere in +contact points. It will establish a connection to a `ClusterReceptionist` somewhere in the cluster. It will monitor the connection to the receptionist and establish a new connection if the link goes down. When looking for a new receptionist it uses fresh contact points retrieved from previous establishment, or periodically refreshed contacts, i.e. not necessarily the initial contact points. -.. note:: +@@@ note - ``ClusterClient`` should not be used when sending messages to actors that run - within the same cluster. Similar functionality as the ``ClusterClient`` is - provided in a more efficient way by :ref:`distributed-pub-sub-java` for actors that - belong to the same cluster. +`ClusterClient` should not be used when sending messages to actors that run +within the same cluster. Similar functionality as the `ClusterClient` is +provided in a more efficient way by @ref:[Distributed Publish Subscribe in Cluster](distributed-pub-sub.md) for actors that +belong to the same cluster. -Also, note it's necessary to change ``akka.actor.provider`` from ``local`` -to ``remote`` or ``cluster`` when using +@@@ + +Also, note it's necessary to change `akka.actor.provider` from `local` +to `remote` or `cluster` when using the cluster client. The receptionist is supposed to be started on all nodes, or all nodes with specified role, -in the cluster. The receptionist can be started with the ``ClusterClientReceptionist`` extension +in the cluster. The receptionist can be started with the `ClusterClientReceptionist` extension or as an ordinary actor. -You can send messages via the ``ClusterClient`` to any actor in the cluster that is registered -in the ``DistributedPubSubMediator`` used by the ``ClusterReceptionist``. -The ``ClusterClientReceptionist`` provides methods for registration of actors that -should be reachable from the client. Messages are wrapped in ``ClusterClient.Send``, -``ClusterClient.SendToAll`` or ``ClusterClient.Publish``. +You can send messages via the `ClusterClient` to any actor in the cluster that is registered +in the `DistributedPubSubMediator` used by the `ClusterReceptionist`. +The `ClusterClientReceptionist` provides methods for registration of actors that +should be reachable from the client. Messages are wrapped in `ClusterClient.Send`, +`ClusterClient.SendToAll` or `ClusterClient.Publish`. -Both the ``ClusterClient`` and the ``ClusterClientReceptionist`` emit events that can be subscribed to. -The ``ClusterClient`` sends out notifications in relation to having received a list of contact points -from the ``ClusterClientReceptionist``. One use of this list might be for the client to record its +Both the `ClusterClient` and the `ClusterClientReceptionist` emit events that can be subscribed to. +The `ClusterClient` sends out notifications in relation to having received a list of contact points +from the `ClusterClientReceptionist`. One use of this list might be for the client to record its contact points. A client that is restarted could then use this information to supersede any previously configured contact points. -The ``ClusterClientReceptionist`` sends out notifications in relation to having received a contact -from a ``ClusterClient``. This notification enables the server containing the receptionist to become aware of +The `ClusterClientReceptionist` sends out notifications in relation to having received a contact +from a `ClusterClient`. This notification enables the server containing the receptionist to become aware of what clients are connected. **1. ClusterClient.Send** The message will be delivered to one recipient with a matching path, if any such exists. If several entries match the path the message will be delivered -to one random destination. The ``sender`` of the message can specify that local +to one random destination. The `sender` of the message can specify that local affinity is preferred, i.e. the message is sent to an actor in the same local actor system as the used receptionist actor, if any such exists, otherwise random to any other matching entry. @@ -63,130 +62,133 @@ to the named topic. Response messages from the destination actor are tunneled via the receptionist to avoid inbound connections from other cluster nodes to the client, i.e. -the ``sender``, as seen by the destination actor, is not the client itself. -The ``sender`` of the response messages, as seen by the client, is ``deadLetters`` -since the client should normally send subsequent messages via the ``ClusterClient``. +the `sender`, as seen by the destination actor, is not the client itself. +The `sender` of the response messages, as seen by the client, is `deadLetters` +since the client should normally send subsequent messages via the `ClusterClient`. It is possible to pass the original sender inside the reply messages if the client is supposed to communicate directly to the actor in the cluster. -While establishing a connection to a receptionist the ``ClusterClient`` will buffer +While establishing a connection to a receptionist the `ClusterClient` will buffer messages and send them when the connection is established. If the buffer is full -the ``ClusterClient`` will drop old messages when new messages are sent via the client. +the `ClusterClient` will drop old messages when new messages are sent via the client. The size of the buffer is configurable and it can be disabled by using a buffer size of 0. It's worth noting that messages can always be lost because of the distributed nature of these actors. As always, additional logic should be implemented in the destination (acknowledgement) and in the client (retry) actors to ensure at-least-once message delivery. -An Example ----------- +## An Example On the cluster nodes first start the receptionist. Note, it is recommended to load the extension -when the actor system is started by defining it in the ``akka.extensions`` configuration property:: +when the actor system is started by defining it in the `akka.extensions` configuration property: - akka.extensions = ["akka.cluster.client.ClusterClientReceptionist"] +``` +akka.extensions = ["akka.cluster.client.ClusterClientReceptionist"] +``` Next, register the actors that should be available for the client. -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java#server +@@snip [ClusterClientTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java) { #server } -On the client you create the ``ClusterClient`` actor and use it as a gateway for sending +On the client you create the `ClusterClient` actor and use it as a gateway for sending messages to the actors identified by their path (without address information) somewhere in the cluster. -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java#client +@@snip [ClusterClientTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java) { #client } -The ``initialContacts`` parameter is a ``Set``, which can be created like this: +The `initialContacts` parameter is a `Set`, which can be created like this: -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java#initialContacts +@@snip [ClusterClientTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java) { #initialContacts } You will probably define the address information of the initial contact points in configuration or system property. -See also :ref:`cluster-client-config-java`. +See also [Configuration](#cluster-client-config-java). -A more comprehensive sample is available in the tutorial named `Distributed workers with Akka and Java! `_. +A more comprehensive sample is available in the tutorial named [Distributed workers with Akka and Java!](https://github.com/typesafehub/activator-akka-distributed-workers-java). -ClusterClientReceptionist Extension ------------------------------------ +## ClusterClientReceptionist Extension -In the example above the receptionist is started and accessed with the ``akka.cluster.client.ClusterClientReceptionist`` extension. +In the example above the receptionist is started and accessed with the `akka.cluster.client.ClusterClientReceptionist` extension. That is convenient and perfectly fine in most cases, but it can be good to know that it is possible to -start the ``akka.cluster.client.ClusterReceptionist`` actor as an ordinary actor and you can have several +start the `akka.cluster.client.ClusterReceptionist` actor as an ordinary actor and you can have several different receptionists at the same time, serving different types of clients. -Note that the ``ClusterClientReceptionist`` uses the ``DistributedPubSub`` extension, which is described -in :ref:`distributed-pub-sub-java`. +Note that the `ClusterClientReceptionist` uses the `DistributedPubSub` extension, which is described +in @ref:[Distributed Publish Subscribe in Cluster](distributed-pub-sub.md). It is recommended to load the extension when the actor system is started by defining it in the -``akka.extensions`` configuration property:: +`akka.extensions` configuration property: - akka.extensions = ["akka.cluster.client.ClusterClientReceptionist"] +``` +akka.extensions = ["akka.cluster.client.ClusterClientReceptionist"] +``` -Events ------- -As mentioned earlier, both the ``ClusterClient`` and ``ClusterClientReceptionist`` emit events that can be subscribed to. +## Events + +As mentioned earlier, both the `ClusterClient` and `ClusterClientReceptionist` emit events that can be subscribed to. The following code snippet declares an actor that will receive notifications on contact points (addresses to the available -receptionists), as they become available. The code illustrates subscribing to the events and receiving the ``ClusterClient`` +receptionists), as they become available. The code illustrates subscribing to the events and receiving the `ClusterClient` initial state. -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java#clientEventsListener +@@snip [ClusterClientTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java) { #clientEventsListener } -Similarly we can have an actor that behaves in a similar fashion for learning what cluster clients contact a ``ClusterClientReceptionist``: +Similarly we can have an actor that behaves in a similar fashion for learning what cluster clients contact a `ClusterClientReceptionist`: -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java#receptionistEventsListener +@@snip [ClusterClientTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/client/ClusterClientTest.java) { #receptionistEventsListener } -Dependencies ------------- +## Dependencies To use the Cluster Client you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` -maven:: +maven: - - com.typesafe.akka - akka-cluster-tools_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-cluster-tools_@binVersion@ + @version@ + +``` -.. _cluster-client-config-java: + +## Configuration -Configuration -------------- - -The ``ClusterClientReceptionist`` extension (or ``ClusterReceptionistSettings``) can be configured +The `ClusterClientReceptionist` extension (or `ClusterReceptionistSettings`) can be configured with the following properties: -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#receptionist-ext-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #receptionist-ext-config } -The following configuration properties are read by the ``ClusterClientSettings`` -when created with a ``ActorSystem`` parameter. It is also possible to amend the ``ClusterClientSettings`` -or create it from another config section with the same layout as below. ``ClusterClientSettings`` is -a parameter to the ``ClusterClient.props`` factory method, i.e. each client can be configured +The following configuration properties are read by the `ClusterClientSettings` +when created with a `ActorSystem` parameter. It is also possible to amend the `ClusterClientSettings` +or create it from another config section with the same layout as below. `ClusterClientSettings` is +a parameter to the `ClusterClient.props` factory method, i.e. each client can be configured with different settings if needed. -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#cluster-client-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #cluster-client-config } + +## Failure handling -Failure handling ----------------- When the cluster client is started it must be provided with a list of initial contacts which are cluster nodes where receptionists are running. It will then repeatedly (with an interval configurable -by ``establishing-get-contacts-interval``) try to contact those until it gets in contact with one of them. +by `establishing-get-contacts-interval`) try to contact those until it gets in contact with one of them. While running, the list of contacts are continuously updated with data from the receptionists (again, with an -interval configurable with ``refresh-contacts-interval``), so that if there are more receptionists in the cluster +interval configurable with `refresh-contacts-interval`), so that if there are more receptionists in the cluster than the initial contacts provided to the client the client will learn about them. While the client is running it will detect failures in its connection to the receptionist by heartbeats if more than a configurable amount of heartbeats are missed the client will try to reconnect to its known set of contacts to find a receptionist it can access. -When the cluster cannot be reached at all ------------------------------------------ +## When the cluster cannot be reached at all + It is possible to make the cluster client stop entirely if it cannot find a receptionist it can talk to -within a configurable interval. This is configured with the ``reconnect-timeout``, which defaults to ``off``. +within a configurable interval. This is configured with the `reconnect-timeout`, which defaults to `off`. This can be useful when initial contacts are provided from some kind of service registry, cluster node addresses are entirely dynamic and the entire cluster might shut down or crash, be restarted on new addresses. Since the -client will be stopped in that case a monitoring actor can watch it and upon ``Terminate`` a new set of initial -contacts can be fetched and a new cluster client started. +client will be stopped in that case a monitoring actor can watch it and upon `Terminate` a new set of initial +contacts can be fetched and a new cluster client started. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/cluster-metrics.md b/akka-docs/src/main/paradox/java/cluster-metrics.md index 4f24281462..a7f87e8bb3 100644 --- a/akka-docs/src/main/paradox/java/cluster-metrics.md +++ b/akka-docs/src/main/paradox/java/cluster-metrics.md @@ -1,11 +1,6 @@ +# Cluster Metrics Extension -.. _cluster_metrics_java: - -Cluster Metrics Extension -========================= - -Introduction ------------- +## Introduction The member nodes of the cluster can collect system health metrics and publish that to other cluster nodes and to the registered subscribers on the system event bus with the help of Cluster Metrics Extension. @@ -14,66 +9,69 @@ Cluster metrics information is primarily used for load-balancing routers, and can also be used to implement advanced metrics-based node life cycles, such as "Node Let-it-crash" when CPU steal time becomes excessive. -Cluster Metrics Extension is a separate Akka module delivered in ``akka-cluster-metrics`` jar. +Cluster Metrics Extension is a separate Akka module delivered in `akka-cluster-metrics` jar. To enable usage of the extension you need to add the following dependency to your project: -:: +: - - com.typesafe.akka - akka-cluster-metrics_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-cluster-metrics_@binVersion@ + @version@ + +``` -and add the following configuration stanza to your ``application.conf`` -:: +and add the following configuration stanza to your `application.conf` +: - akka.extensions = [ "akka.cluster.metrics.ClusterMetricsExtension" ] +``` +akka.extensions = [ "akka.cluster.metrics.ClusterMetricsExtension" ] +``` -Cluster members with status :ref:`WeaklyUp `, +Cluster members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-java), will participate in Cluster Metrics collection and dissemination. -Metrics Collector ------------------ +## Metrics Collector -Metrics collection is delegated to an implementation of ``akka.cluster.metrics.MetricsCollector``. +Metrics collection is delegated to an implementation of `akka.cluster.metrics.MetricsCollector`. Different collector implementations provide different subsets of metrics published to the cluster. Certain message routing and let-it-crash functions may not work when Sigar is not provisioned. Cluster metrics extension comes with two built-in collector implementations: -#. ``akka.cluster.metrics.SigarMetricsCollector``, which requires Sigar provisioning, and is more rich/precise -#. ``akka.cluster.metrics.JmxMetricsCollector``, which is used as fall back, and is less rich/precise + 1. `akka.cluster.metrics.SigarMetricsCollector`, which requires Sigar provisioning, and is more rich/precise + 2. `akka.cluster.metrics.JmxMetricsCollector`, which is used as fall back, and is less rich/precise You can also plug-in your own metrics collector implementation. By default, metrics extension will use collector provider fall back and will try to load them in this order: -#. configured user-provided collector -#. built-in ``akka.cluster.metrics.SigarMetricsCollector`` -#. and finally ``akka.cluster.metrics.JmxMetricsCollector`` + 1. configured user-provided collector + 2. built-in `akka.cluster.metrics.SigarMetricsCollector` + 3. and finally `akka.cluster.metrics.JmxMetricsCollector` -Metrics Events --------------- +## Metrics Events Metrics extension periodically publishes current snapshot of the cluster metrics to the node system event bus. -The publication interval is controlled by the ``akka.cluster.metrics.collector.sample-interval`` setting. +The publication interval is controlled by the `akka.cluster.metrics.collector.sample-interval` setting. -The payload of the ``akka.cluster.metrics.ClusterMetricsChanged`` event will contain +The payload of the `akka.cluster.metrics.ClusterMetricsChanged` event will contain latest metrics of the node as well as other cluster member nodes metrics gossip which was received during the collector sample interval. You can subscribe your metrics listener actors to these events in order to implement custom node lifecycle -:: +: - ClusterMetricsExtension.get(system).subscribe(metricsListenerActor); +``` +ClusterMetricsExtension.get(system).subscribe(metricsListenerActor); +``` -Hyperic Sigar Provisioning --------------------------- +## Hyperic Sigar Provisioning -Both user-provided and built-in metrics collectors can optionally use `Hyperic Sigar `_ +Both user-provided and built-in metrics collectors can optionally use [Hyperic Sigar](http://www.hyperic.com/products/sigar) for a wider and more accurate range of metrics compared to what can be retrieved from ordinary JMX MBeans. Sigar is using a native o/s library, and requires library provisioning, i.e. @@ -81,117 +79,116 @@ deployment, extraction and loading of the o/s native library into JVM at runtime User can provision Sigar classes and native library in one of the following ways: -#. Use `Kamon sigar-loader `_ as a project dependency for the user project. - Metrics extension will extract and load sigar library on demand with help of Kamon sigar provisioner. -#. Use `Kamon sigar-loader `_ as java agent: ``java -javaagent:/path/to/sigar-loader.jar``. - Kamon sigar loader agent will extract and load sigar library during JVM start. -#. Place ``sigar.jar`` on the ``classpath`` and Sigar native library for the o/s on the ``java.library.path``. - User is required to manage both project dependency and library deployment manually. + 1. Use [Kamon sigar-loader](https://github.com/kamon-io/sigar-loader) as a project dependency for the user project. +Metrics extension will extract and load sigar library on demand with help of Kamon sigar provisioner. + 2. Use [Kamon sigar-loader](https://github.com/kamon-io/sigar-loader) as java agent: `java -javaagent:/path/to/sigar-loader.jar`. +Kamon sigar loader agent will extract and load sigar library during JVM start. + 3. Place `sigar.jar` on the `classpath` and Sigar native library for the o/s on the `java.library.path`. +User is required to manage both project dependency and library deployment manually. -.. warning:: +@@@ warning - When using `Kamon sigar-loader `_ and running multiple - instances of the same application on the same host, you have to make sure that sigar library is extracted to a - unique per instance directory. You can control the extract directory with the - ``akka.cluster.metrics.native-library-extract-folder`` configuration setting. +When using [Kamon sigar-loader](https://github.com/kamon-io/sigar-loader) and running multiple +instances of the same application on the same host, you have to make sure that sigar library is extracted to a +unique per instance directory. You can control the extract directory with the +`akka.cluster.metrics.native-library-extract-folder` configuration setting. + +@@@ To enable usage of Sigar you can add the following dependency to the user project -:: +: - - io.kamon - sigar-loader - @sigarLoaderVersion@ - +``` + + io.kamon + sigar-loader + @sigarLoaderVersion@ + +``` -You can download Kamon sigar-loader from `Maven Central `_ +You can download Kamon sigar-loader from [Maven Central](http://search.maven.org/#search%7Cga%7C1%7Csigar-loader) -Adaptive Load Balancing ------------------------ +## Adaptive Load Balancing -The ``AdaptiveLoadBalancingPool`` / ``AdaptiveLoadBalancingGroup`` performs load balancing of messages to cluster nodes based on the cluster metrics data. +The `AdaptiveLoadBalancingPool` / `AdaptiveLoadBalancingGroup` performs load balancing of messages to cluster nodes based on the cluster metrics data. It uses random selection of routees with probabilities derived from the remaining capacity of the corresponding node. It can be configured to use a specific MetricsSelector to produce the probabilities, a.k.a. weights: -* ``heap`` / ``HeapMetricsSelector`` - Used and max JVM heap memory. Weights based on remaining heap capacity; (max - used) / max -* ``load`` / ``SystemLoadAverageMetricsSelector`` - System load average for the past 1 minute, corresponding value can be found in ``top`` of Linux systems. The system is possibly nearing a bottleneck if the system load average is nearing number of cpus/cores. Weights based on remaining load capacity; 1 - (load / processors) -* ``cpu`` / ``CpuMetricsSelector`` - CPU utilization in percentage, sum of User + Sys + Nice + Wait. Weights based on remaining cpu capacity; 1 - utilization -* ``mix`` / ``MixMetricsSelector`` - Combines heap, cpu and load. Weights based on mean of remaining capacity of the combined selectors. -* Any custom implementation of ``akka.cluster.metrics.MetricsSelector`` + * `heap` / `HeapMetricsSelector` - Used and max JVM heap memory. Weights based on remaining heap capacity; (max - used) / max + * `load` / `SystemLoadAverageMetricsSelector` - System load average for the past 1 minute, corresponding value can be found in `top` of Linux systems. The system is possibly nearing a bottleneck if the system load average is nearing number of cpus/cores. Weights based on remaining load capacity; 1 - (load / processors) + * `cpu` / `CpuMetricsSelector` - CPU utilization in percentage, sum of User + Sys + Nice + Wait. Weights based on remaining cpu capacity; 1 - utilization + * `mix` / `MixMetricsSelector` - Combines heap, cpu and load. Weights based on mean of remaining capacity of the combined selectors. + * Any custom implementation of `akka.cluster.metrics.MetricsSelector` -The collected metrics values are smoothed with `exponential weighted moving average `_. In the :ref:`cluster_configuration_java` you can adjust how quickly past data is decayed compared to new data. +The collected metrics values are smoothed with [exponential weighted moving average](http://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average). In the @ref:[cluster_configuration_java](cluster-usage.md#cluster-configuration-java) you can adjust how quickly past data is decayed compared to new data. Let's take a look at this router in action. What can be more demanding than calculating factorials? The backend worker that performs the factorial calculation: -.. includecode:: code/jdocs/cluster/FactorialBackend.java#backend +@@snip [FactorialBackend.java](code/jdocs/cluster/FactorialBackend.java) { #backend } The frontend that receives user jobs and delegates to the backends via the router: -.. includecode:: code/jdocs/cluster/FactorialFrontend.java#frontend - +@@snip [FactorialFrontend.java](code/jdocs/cluster/FactorialFrontend.java) { #frontend } As you can see, the router is defined in the same way as other routers, and in this case it is configured as follows: -:: - - akka.actor.deployment { - /factorialFrontend/factorialBackendRouter = { - # Router type provided by metrics extension. - router = cluster-metrics-adaptive-group - # Router parameter specific for metrics extension. - # metrics-selector = heap - # metrics-selector = load - # metrics-selector = cpu - metrics-selector = mix - # - routees.paths = ["/user/factorialBackend"] - cluster { - enabled = on - use-role = backend - allow-local-routees = off - } +``` +akka.actor.deployment { + /factorialFrontend/factorialBackendRouter = { + # Router type provided by metrics extension. + router = cluster-metrics-adaptive-group + # Router parameter specific for metrics extension. + # metrics-selector = heap + # metrics-selector = load + # metrics-selector = cpu + metrics-selector = mix + # + routees.paths = ["/user/factorialBackend"] + cluster { + enabled = on + use-role = backend + allow-local-routees = off } } +} +``` -It is only ``router`` type and the ``metrics-selector`` parameter that is specific to this router, +It is only `router` type and the `metrics-selector` parameter that is specific to this router, other things work in the same way as other routers. The same type of router could also have been defined in code: -.. includecode:: code/jdocs/cluster/FactorialFrontend.java#router-lookup-in-code +@@snip [FactorialFrontend.java](code/jdocs/cluster/FactorialFrontend.java) { #router-lookup-in-code } -.. includecode:: code/jdocs/cluster/FactorialFrontend.java#router-deploy-in-code +@@snip [FactorialFrontend.java](code/jdocs/cluster/FactorialFrontend.java) { #router-deploy-in-code } The easiest way to run **Adaptive Load Balancing** example yourself is to download the ready to run -`Akka Cluster Sample with Scala <@exampleCodeService@/akka-samples-cluster-java>`_ +[Akka Cluster Sample with Scala](@exampleCodeService@/akka-samples-cluster-java) together with the tutorial. It contains instructions on how to run the **Adaptive Load Balancing** sample. -The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-cluster-java>`_. +The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-cluster-java). -Subscribe to Metrics Events ---------------------------- +## Subscribe to Metrics Events It is possible to subscribe to the metrics events directly to implement other functionality. -.. includecode:: code/jdocs/cluster/MetricsListener.java#metrics-listener +@@snip [MetricsListener.java](code/jdocs/cluster/MetricsListener.java) { #metrics-listener } -Custom Metrics Collector ------------------------- +## Custom Metrics Collector -Metrics collection is delegated to the implementation of ``akka.cluster.metrics.MetricsCollector`` +Metrics collection is delegated to the implementation of `akka.cluster.metrics.MetricsCollector` You can plug-in your own metrics collector instead of built-in -``akka.cluster.metrics.SigarMetricsCollector`` or ``akka.cluster.metrics.JmxMetricsCollector``. +`akka.cluster.metrics.SigarMetricsCollector` or `akka.cluster.metrics.JmxMetricsCollector`. Look at those two implementations for inspiration. Custom metrics collector implementation class must be specified in the -``akka.cluster.metrics.collector.provider`` configuration property. +`akka.cluster.metrics.collector.provider` configuration property. -Configuration -------------- +## Configuration The Cluster metrics extension can be configured with the following properties: -.. includecode:: ../../../akka-cluster-metrics/src/main/resources/reference.conf +@@snip [reference.conf]../../../../../akka-cluster-metrics/src/main/resources/reference.conf) { # } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/cluster-sharding.md b/akka-docs/src/main/paradox/java/cluster-sharding.md index 98d5bd4301..86eef8b719 100644 --- a/akka-docs/src/main/paradox/java/cluster-sharding.md +++ b/akka-docs/src/main/paradox/java/cluster-sharding.md @@ -1,7 +1,4 @@ -.. _cluster_sharding_java: - -Cluster Sharding -================ +# Cluster Sharding Cluster sharding is useful when you need to distribute actors across several nodes in the cluster and want to be able to interact with them using their logical identifier, but without having to care about @@ -13,61 +10,64 @@ but this feature is not limited to actors with persistent state. Cluster sharding is typically used when you have many stateful actors that together consume more resources (e.g. memory) than fit on one machine. If you only have a few stateful actors -it might be easier to run them on a :ref:`cluster-singleton-java` node. +it might be easier to run them on a @ref:[Cluster Singleton](cluster-singleton.md) node. In this context sharding means that actors with an identifier, so called entities, can be automatically distributed across multiple nodes in the cluster. Each entity actor runs only at one place, and messages can be sent to the entity without requiring the sender to know the location of the destination actor. This is achieved by sending -the messages via a ``ShardRegion`` actor provided by this extension, which knows how +the messages via a `ShardRegion` actor provided by this extension, which knows how to route the message with the entity id to the final destination. -Cluster sharding will not be active on members with status :ref:`WeaklyUp ` +Cluster sharding will not be active on members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-java) if that feature is enabled. -.. warning:: - **Don't use Cluster Sharding together with Automatic Downing**, - since it allows the cluster to split up into two separate clusters, which in turn will result - in *multiple shards and entities* being started, one in each separate cluster! - See :ref:`automatic-vs-manual-downing-java`. +@@@ warning -An Example ----------- +**Don't use Cluster Sharding together with Automatic Downing**, +since it allows the cluster to split up into two separate clusters, which in turn will result +in *multiple shards and entities* being started, one in each separate cluster! +See @ref:[Downing](cluster-usage.md#automatic-vs-manual-downing-java). + +@@@ + +## An Example This is how an entity actor may look like: -.. includecode:: ../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java#counter-actor +@@snip [ClusterShardingTest.java]../../../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java) { #counter-actor } -The above actor uses event sourcing and the support provided in ``AbstractPersistentActor`` to store its state. +The above actor uses event sourcing and the support provided in `AbstractPersistentActor` to store its state. It does not have to be a persistent actor, but in case of failure or migration of entities between nodes it must be able to recover its state if it is valuable. -Note how the ``persistenceId`` is defined. The name of the actor is the entity identifier (utf-8 URL-encoded). +Note how the `persistenceId` is defined. The name of the actor is the entity identifier (utf-8 URL-encoded). You may define it another way, but it must be unique. When using the sharding extension you are first, typically at system startup on each node -in the cluster, supposed to register the supported entity types with the ``ClusterSharding.start`` -method. ``ClusterSharding.start`` gives you the reference which you can pass along. +in the cluster, supposed to register the supported entity types with the `ClusterSharding.start` +method. `ClusterSharding.start` gives you the reference which you can pass along. -.. includecode:: ../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java#counter-start +@@snip [ClusterShardingTest.java]../../../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java) { #counter-start } -The ``messageExtractor`` defines application specific methods to extract the entity +The `messageExtractor` defines application specific methods to extract the entity identifier and the shard identifier from incoming messages. -.. includecode:: ../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java#counter-extractor +@@snip [ClusterShardingTest.java]../../../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java) { #counter-extractor } This example illustrates two different ways to define the entity identifier in the messages: - * The ``Get`` message includes the identifier itself. - * The ``EntityEnvelope`` holds the identifier, and the actual message that is - sent to the entity actor is wrapped in the envelope. +> + * The `Get` message includes the identifier itself. + * The `EntityEnvelope` holds the identifier, and the actual message that is +sent to the entity actor is wrapped in the envelope. -Note how these two messages types are handled in the ``entityId`` and ``entityMessage`` methods shown above. -The message sent to the entity actor is what ``entityMessage`` returns and that makes it possible to unwrap envelopes +Note how these two messages types are handled in the `entityId` and `entityMessage` methods shown above. +The message sent to the entity actor is what `entityMessage` returns and that makes it possible to unwrap envelopes if needed. A shard is a group of entities that will be managed together. The grouping is defined by the -``extractShardId`` function shown above. For a specific entity identifier the shard identifier must always +`extractShardId` function shown above. For a specific entity identifier the shard identifier must always be the same. Otherwise the entity actor might accidentally be started in several places at the same time. Creating a good sharding algorithm is an interesting challenge in itself. Try to produce a uniform distribution, @@ -78,119 +78,116 @@ overhead, and increased latency because the coordinator is involved in the routi shard. The sharding algorithm must be the same on all nodes in a running cluster. It can be changed after stopping all nodes in the cluster. -A simple sharding algorithm that works fine in most cases is to take the absolute value of the ``hashCode`` of +A simple sharding algorithm that works fine in most cases is to take the absolute value of the `hashCode` of the entity identifier modulo number of shards. As a convenience this is provided by the -``ShardRegion.HashCodeMessageExtractor``. +`ShardRegion.HashCodeMessageExtractor`. -Messages to the entities are always sent via the local ``ShardRegion``. The ``ShardRegion`` actor reference for a -named entity type is returned by ``ClusterSharding.start`` and it can also be retrieved with ``ClusterSharding.shardRegion``. -The ``ShardRegion`` will lookup the location of the shard for the entity if it does not already know its location. It will +Messages to the entities are always sent via the local `ShardRegion`. The `ShardRegion` actor reference for a +named entity type is returned by `ClusterSharding.start` and it can also be retrieved with `ClusterSharding.shardRegion`. +The `ShardRegion` will lookup the location of the shard for the entity if it does not already know its location. It will delegate the message to the right node and it will create the entity actor on demand, i.e. when the first message for a specific entity is delivered. -.. includecode:: ../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java#counter-usage +@@snip [ClusterShardingTest.java]../../../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java) { #counter-usage } -How it works ------------- +## How it works -The ``ShardRegion`` actor is started on each node in the cluster, or group of nodes -tagged with a specific role. The ``ShardRegion`` is created with two application specific +The `ShardRegion` actor is started on each node in the cluster, or group of nodes +tagged with a specific role. The `ShardRegion` is created with two application specific functions to extract the entity identifier and the shard identifier from incoming messages. A shard is a group of entities that will be managed together. For the first message in a -specific shard the ``ShardRegion`` request the location of the shard from a central coordinator, -the ``ShardCoordinator``. +specific shard the `ShardRegion` request the location of the shard from a central coordinator, +the `ShardCoordinator`. -The ``ShardCoordinator`` decides which ``ShardRegion`` shall own the ``Shard`` and informs -that ``ShardRegion``. The region will confirm this request and create the ``Shard`` supervisor -as a child actor. The individual ``Entities`` will then be created when needed by the ``Shard`` -actor. Incoming messages thus travel via the ``ShardRegion`` and the ``Shard`` to the target -``Entity``. +The `ShardCoordinator` decides which `ShardRegion` shall own the `Shard` and informs +that `ShardRegion`. The region will confirm this request and create the `Shard` supervisor +as a child actor. The individual `Entities` will then be created when needed by the `Shard` +actor. Incoming messages thus travel via the `ShardRegion` and the `Shard` to the target +`Entity`. -If the shard home is another ``ShardRegion`` instance messages will be forwarded -to that ``ShardRegion`` instance instead. While resolving the location of a +If the shard home is another `ShardRegion` instance messages will be forwarded +to that `ShardRegion` instance instead. While resolving the location of a shard incoming messages for that shard are buffered and later delivered when the shard home is known. Subsequent messages to the resolved shard can be delivered -to the target destination immediately without involving the ``ShardCoordinator``. +to the target destination immediately without involving the `ShardCoordinator`. Scenario 1: -#. Incoming message M1 to ``ShardRegion`` instance R1. -#. M1 is mapped to shard S1. R1 doesn't know about S1, so it asks the coordinator C for the location of S1. -#. C answers that the home of S1 is R1. -#. R1 creates child actor for the entity E1 and sends buffered messages for S1 to E1 child -#. All incoming messages for S1 which arrive at R1 can be handled by R1 without C. It creates entity children as needed, and forwards messages to them. + 1. Incoming message M1 to `ShardRegion` instance R1. + 2. M1 is mapped to shard S1. R1 doesn't know about S1, so it asks the coordinator C for the location of S1. + 3. C answers that the home of S1 is R1. + 4. R1 creates child actor for the entity E1 and sends buffered messages for S1 to E1 child + 5. All incoming messages for S1 which arrive at R1 can be handled by R1 without C. It creates entity children as needed, and forwards messages to them. Scenario 2: -#. Incoming message M2 to R1. -#. M2 is mapped to S2. R1 doesn't know about S2, so it asks C for the location of S2. -#. C answers that the home of S2 is R2. -#. R1 sends buffered messages for S2 to R2 -#. All incoming messages for S2 which arrive at R1 can be handled by R1 without C. It forwards messages to R2. -#. R2 receives message for S2, ask C, which answers that the home of S2 is R2, and we are in Scenario 1 (but for R2). + 1. Incoming message M2 to R1. + 2. M2 is mapped to S2. R1 doesn't know about S2, so it asks C for the location of S2. + 3. C answers that the home of S2 is R2. + 4. R1 sends buffered messages for S2 to R2 + 5. All incoming messages for S2 which arrive at R1 can be handled by R1 without C. It forwards messages to R2. + 6. R2 receives message for S2, ask C, which answers that the home of S2 is R2, and we are in Scenario 1 (but for R2). To make sure that at most one instance of a specific entity actor is running somewhere in the cluster it is important that all nodes have the same view of where the shards are located. Therefore the shard allocation decisions are taken by the central -``ShardCoordinator``, which is running as a cluster singleton, i.e. one instance on +`ShardCoordinator`, which is running as a cluster singleton, i.e. one instance on the oldest member among all cluster nodes or a group of nodes tagged with a specific role. The logic that decides where a shard is to be located is defined in a pluggable shard -allocation strategy. The default implementation ``ShardCoordinator.LeastShardAllocationStrategy`` -allocates new shards to the ``ShardRegion`` with least number of previously allocated shards. +allocation strategy. The default implementation `ShardCoordinator.LeastShardAllocationStrategy` +allocates new shards to the `ShardRegion` with least number of previously allocated shards. This strategy can be replaced by an application specific implementation. To be able to use newly added members in the cluster the coordinator facilitates rebalancing of shards, i.e. migrate entities from one node to another. In the rebalance process the -coordinator first notifies all ``ShardRegion`` actors that a handoff for a shard has started. +coordinator first notifies all `ShardRegion` actors that a handoff for a shard has started. That means they will start buffering incoming messages for that shard, in the same way as if the shard location is unknown. During the rebalance process the coordinator will not answer any requests for the location of shards that are being rebalanced, i.e. local buffering will -continue until the handoff is completed. The ``ShardRegion`` responsible for the rebalanced shard -will stop all entities in that shard by sending the specified ``handOffStopMessage`` -(default ``PoisonPill``) to them. When all entities have been terminated the ``ShardRegion`` +continue until the handoff is completed. The `ShardRegion` responsible for the rebalanced shard +will stop all entities in that shard by sending the specified `handOffStopMessage` +(default `PoisonPill`) to them. When all entities have been terminated the `ShardRegion` owning the entities will acknowledge the handoff as completed to the coordinator. Thereafter the coordinator will reply to requests for the location of the shard and thereby allocate a new home for the shard and then buffered messages in the -``ShardRegion`` actors are delivered to the new location. This means that the state of the entities +`ShardRegion` actors are delivered to the new location. This means that the state of the entities are not transferred or migrated. If the state of the entities are of importance it should be -persistent (durable), e.g. with :ref:`persistence-java`, so that it can be recovered at the new +persistent (durable), e.g. with @ref:[Persistence](persistence.md), so that it can be recovered at the new location. The logic that decides which shards to rebalance is defined in a pluggable shard -allocation strategy. The default implementation ``ShardCoordinator.LeastShardAllocationStrategy`` -picks shards for handoff from the ``ShardRegion`` with most number of previously allocated shards. -They will then be allocated to the ``ShardRegion`` with least number of previously allocated shards, +allocation strategy. The default implementation `ShardCoordinator.LeastShardAllocationStrategy` +picks shards for handoff from the `ShardRegion` with most number of previously allocated shards. +They will then be allocated to the `ShardRegion` with least number of previously allocated shards, i.e. new members in the cluster. There is a configurable threshold of how large the difference must be to begin the rebalancing. This strategy can be replaced by an application specific implementation. -The state of shard locations in the ``ShardCoordinator`` is persistent (durable) with -:ref:`distributed_data_java` or :ref:`persistence-java` to survive failures. When a crashed or -unreachable coordinator node has been removed (via down) from the cluster a new ``ShardCoordinator`` singleton +The state of shard locations in the `ShardCoordinator` is persistent (durable) with +@ref:[distributed_data_java](distributed-data.md) or @ref:[Persistence](persistence.md) to survive failures. When a crashed or +unreachable coordinator node has been removed (via down) from the cluster a new `ShardCoordinator` singleton actor will take over and the state is recovered. During such a failure period shards with known location are still available, while messages for new (unknown) shards -are buffered until the new ``ShardCoordinator`` becomes available. +are buffered until the new `ShardCoordinator` becomes available. -As long as a sender uses the same ``ShardRegion`` actor to deliver messages to an entity +As long as a sender uses the same `ShardRegion` actor to deliver messages to an entity actor the order of the messages is preserved. As long as the buffer limit is not reached messages are delivered on a best effort basis, with at-most once delivery semantics, in the same way as ordinary message sending. Reliable end-to-end messaging, with -at-least-once semantics can be added by using ``AtLeastOnceDelivery`` in :ref:`persistence-java`. +at-least-once semantics can be added by using `AtLeastOnceDelivery` in @ref:[Persistence](persistence.md). Some additional latency is introduced for messages targeted to new or previously unused shards due to the round-trip to the coordinator. Rebalancing of shards may also add latency. This should be considered when designing the application specific shard resolution, e.g. to avoid too fine grained shards. -.. _cluster_sharding_mode_java: + +## Distributed Data vs. Persistence Mode -Distributed Data vs. Persistence Mode -------------------------------------- - -The state of the coordinator and the state of :ref:`cluster_sharding_remembering_java` of the shards -are persistent (durable) to survive failures. :ref:`distributed_data_java` or :ref:`persistence-java` +The state of the coordinator and the state of [cluster_sharding_remembering_java](#cluster-sharding-remembering-java) of the shards +are persistent (durable) to survive failures. @ref:[distributed_data_java](distributed-data.md) or @ref:[Persistence](persistence.md) can be used for the storage. Distributed Data is used by default. The functionality when using the two modes is the same. If your sharded entities are not using Akka Persistence @@ -201,222 +198,222 @@ no major reasons for using one mode over the the other. It's important to use the same mode on all nodes in the cluster, i.e. it's not possible to perform a rolling upgrade to change this setting. -Distributed Data Mode -^^^^^^^^^^^^^^^^^^^^^ +### Distributed Data Mode -This mode is enabled with configuration (enabled by default):: +This mode is enabled with configuration (enabled by default): - akka.cluster.sharding.state-store-mode = ddata +``` +akka.cluster.sharding.state-store-mode = ddata +``` -The state of the ``ShardCoordinator`` will be replicated inside a cluster by the -:ref:`distributed_data_java` module with ``WriteMajority``/``ReadMajority`` consistency. +The state of the `ShardCoordinator` will be replicated inside a cluster by the +@ref:[distributed_data_java](distributed-data.md) module with `WriteMajority`/`ReadMajority` consistency. The state of the coordinator is not durable, it's not stored to disk. When all nodes in the cluster have been stopped the state is lost and not needed any more. -The state of :ref:`cluster_sharding_remembering_java` is also durable, i.e. it is stored to +The state of [cluster_sharding_remembering_java](#cluster-sharding-remembering-java) is also durable, i.e. it is stored to disk. The stored entities are started also after a complete cluster restart. -Cluster Sharding is using its own Distributed Data ``Replicator`` per node role. In this way you can use a subset of +Cluster Sharding is using its own Distributed Data `Replicator` per node role. In this way you can use a subset of all nodes for some entity types and another subset for other entity types. Each such replicator has a name that contains the node role and therefore the role configuration must be the same on all nodes in the cluster, i.e. you can't change the roles when performing a rolling upgrade. - + The settings for Distributed Data is configured in the the section -``akka.cluster.sharding.distributed-data``. It's not possible to have different -``distributed-data`` settings for different sharding entity types. +`akka.cluster.sharding.distributed-data`. It's not possible to have different +`distributed-data` settings for different sharding entity types. -Persistence Mode -^^^^^^^^^^^^^^^^ +### Persistence Mode -This mode is enabled with configuration:: +This mode is enabled with configuration: - akka.cluster.sharding.state-store-mode = persistence +``` +akka.cluster.sharding.state-store-mode = persistence +``` -Since it is running in a cluster :ref:`persistence-java` must be configured with a distributed journal. +Since it is running in a cluster @ref:[Persistence](persistence.md) must be configured with a distributed journal. -Startup after minimum number of members ---------------------------------------- +## Startup after minimum number of members -It's good to use Cluster Sharding with the Cluster setting ``akka.cluster.min-nr-of-members`` or -``akka.cluster.role..min-nr-of-members``. That will defer the allocation of the shards +It's good to use Cluster Sharding with the Cluster setting `akka.cluster.min-nr-of-members` or +`akka.cluster.role..min-nr-of-members`. That will defer the allocation of the shards until at least that number of regions have been started and registered to the coordinator. This avoids that many shards are allocated to the first region that registers and only later are rebalanced to other nodes. -See :ref:`min-members_java` for more information about ``min-nr-of-members``. +See @ref:[min-members_java](cluster-usage.md#min-members-java) for more information about `min-nr-of-members`. -Proxy Only Mode ---------------- +## Proxy Only Mode -The ``ShardRegion`` actor can also be started in proxy only mode, i.e. it will not +The `ShardRegion` actor can also be started in proxy only mode, i.e. it will not host any entities itself, but knows how to delegate messages to the right location. -A ``ShardRegion`` is started in proxy only mode with the method ``ClusterSharding.startProxy`` +A `ShardRegion` is started in proxy only mode with the method `ClusterSharding.startProxy` method. -Passivation ------------ +## Passivation If the state of the entities are persistent you may stop entities that are not used to reduce memory consumption. This is done by the application specific implementation of -the entity actors for example by defining receive timeout (``context.setReceiveTimeout``). +the entity actors for example by defining receive timeout (`context.setReceiveTimeout`). If a message is already enqueued to the entity when it stops itself the enqueued message in the mailbox will be dropped. To support graceful passivation without losing such -messages the entity actor can send ``ShardRegion.Passivate`` to its parent ``Shard``. -The specified wrapped message in ``Passivate`` will be sent back to the entity, which is -then supposed to stop itself. Incoming messages will be buffered by the ``Shard`` -between reception of ``Passivate`` and termination of the entity. Such buffered messages +messages the entity actor can send `ShardRegion.Passivate` to its parent `Shard`. +The specified wrapped message in `Passivate` will be sent back to the entity, which is +then supposed to stop itself. Incoming messages will be buffered by the `Shard` +between reception of `Passivate` and termination of the entity. Such buffered messages are thereafter delivered to a new incarnation of the entity. -.. _cluster_sharding_remembering_java: + +## Remembering Entities -Remembering Entities --------------------- - -The list of entities in each ``Shard`` can be made persistent (durable) by setting -the ``rememberEntities`` flag to true in ``ClusterShardingSettings`` when calling -``ClusterSharding.start``. When configured to remember entities, whenever a ``Shard`` +The list of entities in each `Shard` can be made persistent (durable) by setting +the `rememberEntities` flag to true in `ClusterShardingSettings` when calling +`ClusterSharding.start`. When configured to remember entities, whenever a `Shard` is rebalanced onto another node or recovers after a crash it will recreate all the -entities which were previously running in that ``Shard``. To permanently stop entities, -a ``Passivate`` message must be sent to the parent of the entity actor, otherwise the +entities which were previously running in that `Shard`. To permanently stop entities, +a `Passivate` message must be sent to the parent of the entity actor, otherwise the entity will be automatically restarted after the entity restart backoff specified in the configuration. -When :ref:`Distributed Data mode ` is used the identifiers of the entities are -stored in :ref:`ddata_durable_java` of Distributed Data. You may want to change the +When [Distributed Data mode](#cluster-sharding-mode-java) is used the identifiers of the entities are +stored in @ref:[ddata_durable_java](distributed-data.md#ddata-durable-java) of Distributed Data. You may want to change the configuration of the akka.cluster.sharding.distributed-data.durable.lmdb.dir`, since the default directory contains the remote port of the actor system. If using a dynamically assigned port (0) it will be different each time and the previously stored data will not be loaded. -When ``rememberEntities`` is set to false, a ``Shard`` will not automatically restart any entities +When `rememberEntities` is set to false, a `Shard` will not automatically restart any entities after a rebalance or recovering from a crash. Entities will only be started once the first message -for that entity has been received in the ``Shard``. Entities will not be restarted if they stop without -using a ``Passivate``. +for that entity has been received in the `Shard`. Entities will not be restarted if they stop without +using a `Passivate`. Note that the state of the entities themselves will not be restored unless they have been made persistent, -e.g. with :ref:`persistence-java`. +e.g. with @ref:[Persistence](persistence.md). -The performance cost of ``rememberEntities`` is rather high when starting/stopping entities and when +The performance cost of `rememberEntities` is rather high when starting/stopping entities and when shards are rebalanced. This cost increases with number of entities per shard and we currently don't recommend using it with more than 10000 entities per shard. -Supervision ------------ +## Supervision -If you need to use another ``supervisorStrategy`` for the entity actors than the default (restarting) strategy -you need to create an intermediate parent actor that defines the ``supervisorStrategy`` to the +If you need to use another `supervisorStrategy` for the entity actors than the default (restarting) strategy +you need to create an intermediate parent actor that defines the `supervisorStrategy` to the child entity actor. -.. includecode:: ../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java#supervisor +@@snip [ClusterShardingTest.java]../../../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java) { #supervisor } You start such a supervisor in the same way as if it was the entity actor. -.. includecode:: ../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java#counter-supervisor-start +@@snip [ClusterShardingTest.java]../../../../../akka-cluster-sharding/src/test/java/akka/cluster/sharding/ClusterShardingTest.java) { #counter-supervisor-start } Note that stopped entities will be started again when a new message is targeted to the entity. -Graceful Shutdown ------------------ +## Graceful Shutdown -You can send the ``ShardRegion.gracefulShutdownInstance`` message -to the ``ShardRegion`` actor to handoff all shards that are hosted by that ``ShardRegion`` and then the -``ShardRegion`` actor will be stopped. You can ``watch`` the ``ShardRegion`` actor to know when it is completed. +You can send the `ShardRegion.gracefulShutdownInstance` message +to the `ShardRegion` actor to handoff all shards that are hosted by that `ShardRegion` and then the +`ShardRegion` actor will be stopped. You can `watch` the `ShardRegion` actor to know when it is completed. During this period other regions will buffer messages for those shards in the same way as when a rebalance is triggered by the coordinator. When the shards have been stopped the coordinator will allocate these shards elsewhere. -This is performed automatically by the :ref:`coordinated-shutdown-java` and is therefore part of the +This is performed automatically by the @ref:[Coordinated Shutdown](actors.md#coordinated-shutdown-java) and is therefore part of the graceful leaving process of a cluster member. -.. _RemoveInternalClusterShardingData-java: - -Removal of Internal Cluster Sharding Data ------------------------------------------ + +## Removal of Internal Cluster Sharding Data The Cluster Sharding coordinator stores the locations of the shards using Akka Persistence. This data can safely be removed when restarting the whole Akka Cluster. Note that this is not application data. -There is a utility program ``akka.cluster.sharding.RemoveInternalClusterShardingData`` +There is a utility program `akka.cluster.sharding.RemoveInternalClusterShardingData` that removes this data. - -.. warning:: - Never use this program while there are running Akka Cluster nodes that are - using Cluster Sharding. Stop all Cluster nodes before using this program. +@@@ warning + +Never use this program while there are running Akka Cluster nodes that are +using Cluster Sharding. Stop all Cluster nodes before using this program. + +@@@ It can be needed to remove the data if the Cluster Sharding coordinator cannot startup because of corrupt data, which may happen if accidentally two clusters were running at the same time, e.g. caused by using auto-down and there was a network partition. -.. warning:: - **Don't use Cluster Sharding together with Automatic Downing**, - since it allows the cluster to split up into two separate clusters, which in turn will result - in *multiple shards and entities* being started, one in each separate cluster! - See :ref:`automatic-vs-manual-downing-java`. +@@@ warning -Use this program as a standalone Java main program:: - - java -classpath - akka.cluster.sharding.RemoveInternalClusterShardingData - -2.3 entityType1 entityType2 entityType3 +**Don't use Cluster Sharding together with Automatic Downing**, +since it allows the cluster to split up into two separate clusters, which in turn will result +in *multiple shards and entities* being started, one in each separate cluster! +See @ref:[Downing](cluster-usage.md#automatic-vs-manual-downing-java). -The program is included in the ``akka-cluster-sharding`` jar file. It +@@@ + +Use this program as a standalone Java main program: + +``` +java -classpath + akka.cluster.sharding.RemoveInternalClusterShardingData + -2.3 entityType1 entityType2 entityType3 +``` + +The program is included in the `akka-cluster-sharding` jar file. It is easiest to run it with same classpath and configuration as your ordinary application. It can be run from sbt or maven in similar way. -Specify the entity type names (same as you use in the ``start`` method -of ``ClusterSharding``) as program arguments. +Specify the entity type names (same as you use in the `start` method +of `ClusterSharding`) as program arguments. -If you specify ``-2.3`` as the first program argument it will also try +If you specify `-2.3` as the first program argument it will also try to remove data that was stored by Cluster Sharding in Akka 2.3.x using different persistenceId. - -Dependencies ------------- +## Dependencies To use the Cluster Sharding you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-cluster-sharding" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-cluster-sharding" % "@version@" @crossString@ +``` -maven:: +maven: - - com.typesafe.akka - akka-cluster-sharding_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-cluster-sharding_@binVersion@ + @version@ + +``` -Configuration -------------- +## Configuration -The ``ClusterSharding`` extension can be configured with the following properties. These configuration -properties are read by the ``ClusterShardingSettings`` when created with a ``ActorSystem`` parameter. -It is also possible to amend the ``ClusterShardingSettings`` or create it from another config section -with the same layout as below. ``ClusterShardingSettings`` is a parameter to the ``start`` method of -the ``ClusterSharding`` extension, i.e. each each entity type can be configured with different settings +The `ClusterSharding` extension can be configured with the following properties. These configuration +properties are read by the `ClusterShardingSettings` when created with a `ActorSystem` parameter. +It is also possible to amend the `ClusterShardingSettings` or create it from another config section +with the same layout as below. `ClusterShardingSettings` is a parameter to the `start` method of +the `ClusterSharding` extension, i.e. each each entity type can be configured with different settings if needed. -.. includecode:: ../../../akka-cluster-sharding/src/main/resources/reference.conf#sharding-ext-config +@@snip [reference.conf]../../../../../akka-cluster-sharding/src/main/resources/reference.conf) { #sharding-ext-config } Custom shard allocation strategy can be defined in an optional parameter to -``ClusterSharding.start``. See the API documentation of ``AbstractShardAllocationStrategy`` for details +`ClusterSharding.start`. See the API documentation of `AbstractShardAllocationStrategy` for details of how to implement a custom shard allocation strategy. +## Inspecting cluster sharding state -Inspecting cluster sharding state ---------------------------------- Two requests to inspect the cluster state are available: -``ShardRegion.getShardRegionStateInstance`` which will return a ``ShardRegion.ShardRegionState`` that contains +`ShardRegion.getShardRegionStateInstance` which will return a `ShardRegion.ShardRegionState` that contains the identifiers of the shards running in a Region and what entities are alive for each of them. -``ShardRegion.GetClusterShardingStats`` which will query all the regions in the cluster and return -a ``ShardRegion.ClusterShardingStats`` containing the identifiers of the shards running in each region and a count +`ShardRegion.GetClusterShardingStats` which will query all the regions in the cluster and return +a `ShardRegion.ClusterShardingStats` containing the identifiers of the shards running in each region and a count of entities that are alive in each shard. The purpose of these messages is testing and monitoring, they are not provided to give access to -directly sending messages to the individual entities. +directly sending messages to the individual entities. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/cluster-singleton.md b/akka-docs/src/main/paradox/java/cluster-singleton.md index 951056d70f..87426cf447 100644 --- a/akka-docs/src/main/paradox/java/cluster-singleton.md +++ b/akka-docs/src/main/paradox/java/cluster-singleton.md @@ -1,34 +1,31 @@ -.. _cluster-singleton-java: - -Cluster Singleton -================= +# Cluster Singleton For some use cases it is convenient and sometimes also mandatory to ensure that you have exactly one actor of a certain type running somewhere in the cluster. Some examples: -* single point of responsibility for certain cluster-wide consistent decisions, or - coordination of actions across the cluster system -* single entry point to an external system -* single master, many workers -* centralized naming service, or routing logic + * single point of responsibility for certain cluster-wide consistent decisions, or +coordination of actions across the cluster system + * single entry point to an external system + * single master, many workers + * centralized naming service, or routing logic Using a singleton should not be the first design choice. It has several drawbacks, such as single-point of bottleneck. Single-point of failure is also a relevant concern, but for some cases this feature takes care of that by making sure that another singleton instance will eventually be started. -The cluster singleton pattern is implemented by ``akka.cluster.singleton.ClusterSingletonManager``. +The cluster singleton pattern is implemented by `akka.cluster.singleton.ClusterSingletonManager`. It manages one singleton actor instance among all cluster nodes or a group of nodes tagged with -a specific role. ``ClusterSingletonManager`` is an actor that is supposed to be started on +a specific role. `ClusterSingletonManager` is an actor that is supposed to be started on all nodes, or all nodes with specified role, in the cluster. The actual singleton actor is -started by the ``ClusterSingletonManager`` on the oldest node by creating a child actor from -supplied ``Props``. ``ClusterSingletonManager`` makes sure that at most one singleton instance +started by the `ClusterSingletonManager` on the oldest node by creating a child actor from +supplied `Props`. `ClusterSingletonManager` makes sure that at most one singleton instance is running at any point in time. The singleton actor is always running on the oldest member with specified role. -The oldest member is determined by ``akka.cluster.Member#isOlderThan``. +The oldest member is determined by `akka.cluster.Member#isOlderThan`. This can change when removing that member from the cluster. Be aware that there is a short time period when there is no active singleton during the hand-over process. @@ -38,46 +35,47 @@ take over and a new singleton actor is created. For these failure scenarios ther not be a graceful hand-over, but more than one active singletons is prevented by all reasonable means. Some corner cases are eventually resolved by configurable timeouts. -You can access the singleton actor by using the provided ``akka.cluster.singleton.ClusterSingletonProxy``, +You can access the singleton actor by using the provided `akka.cluster.singleton.ClusterSingletonProxy`, which will route all messages to the current instance of the singleton. The proxy will keep track of -the oldest node in the cluster and resolve the singleton's ``ActorRef`` by explicitly sending the -singleton's ``actorSelection`` the ``akka.actor.Identify`` message and waiting for it to reply. +the oldest node in the cluster and resolve the singleton's `ActorRef` by explicitly sending the +singleton's `actorSelection` the `akka.actor.Identify` message and waiting for it to reply. This is performed periodically if the singleton doesn't reply within a certain (configurable) time. -Given the implementation, there might be periods of time during which the ``ActorRef`` is unavailable, +Given the implementation, there might be periods of time during which the `ActorRef` is unavailable, e.g., when a node leaves the cluster. In these cases, the proxy will buffer the messages sent to the singleton and then deliver them when the singleton is finally available. If the buffer is full -the ``ClusterSingletonProxy`` will drop old messages when new messages are sent via the proxy. +the `ClusterSingletonProxy` will drop old messages when new messages are sent via the proxy. The size of the buffer is configurable and it can be disabled by using a buffer size of 0. It's worth noting that messages can always be lost because of the distributed nature of these actors. As always, additional logic should be implemented in the singleton (acknowledgement) and in the client (retry) actors to ensure at-least-once message delivery. -The singleton instance will not run on members with status :ref:`WeaklyUp `. +The singleton instance will not run on members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-java). -Potential problems to be aware of ---------------------------------- +## Potential problems to be aware of This pattern may seem to be very tempting to use at first, but it has several drawbacks, some of them are listed below: -* the cluster singleton may quickly become a *performance bottleneck*, -* you can not rely on the cluster singleton to be *non-stop* available — e.g. when the node on which the singleton has - been running dies, it will take a few seconds for this to be noticed and the singleton be migrated to another node, -* in the case of a *network partition* appearing in a Cluster that is using Automatic Downing (see docs for - :ref:`automatic-vs-manual-downing-java`), - it may happen that the isolated clusters each decide to spin up their own singleton, meaning that there might be multiple - singletons running in the system, yet the Clusters have no way of finding out about them (because of the partition). + * the cluster singleton may quickly become a *performance bottleneck*, + * you can not rely on the cluster singleton to be *non-stop* available — e.g. when the node on which the singleton has +been running dies, it will take a few seconds for this to be noticed and the singleton be migrated to another node, + * in the case of a *network partition* appearing in a Cluster that is using Automatic Downing (see docs for +@ref:[Downing](cluster-usage.md#automatic-vs-manual-downing-java)), +it may happen that the isolated clusters each decide to spin up their own singleton, meaning that there might be multiple +singletons running in the system, yet the Clusters have no way of finding out about them (because of the partition). Especially the last point is something you should be aware of — in general when using the Cluster Singleton pattern you should take care of downing nodes yourself and not rely on the timing based auto-down feature. -.. warning:: - **Don't use Cluster Singleton together with Automatic Downing**, - since it allows the cluster to split up into two separate clusters, which in turn will result - in *multiple Singletons* being started, one in each separate cluster! +@@@ warning -An Example ----------- +**Don't use Cluster Singleton together with Automatic Downing**, +since it allows the cluster to split up into two separate clusters, which in turn will result +in *multiple Singletons* being started, one in each separate cluster! + +@@@ + +## An Example Assume that we need one single entry point to an external system. An actor that receives messages from a JMS queue with the strict requirement that only one @@ -85,58 +83,59 @@ JMS consumer must exist to be make sure that the messages are processed in order That is perhaps not how one would like to design things, but a typical real-world scenario when integrating with external systems. -On each node in the cluster you need to start the ``ClusterSingletonManager`` and -supply the ``Props`` of the singleton actor, in this case the JMS queue consumer. +On each node in the cluster you need to start the `ClusterSingletonManager` and +supply the `Props` of the singleton actor, in this case the JMS queue consumer. -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/singleton/ClusterSingletonManagerTest.java#create-singleton-manager +@@snip [ClusterSingletonManagerTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/singleton/ClusterSingletonManagerTest.java) { #create-singleton-manager } -Here we limit the singleton to nodes tagged with the ``"worker"`` role, but all nodes, independent of -role, can be used by not specifying ``withRole``. +Here we limit the singleton to nodes tagged with the `"worker"` role, but all nodes, independent of +role, can be used by not specifying `withRole`. -Here we use an application specific ``terminationMessage`` to be able to close the -resources before actually stopping the singleton actor. Note that ``PoisonPill`` is a -perfectly fine ``terminationMessage`` if you only need to stop the actor. +Here we use an application specific `terminationMessage` to be able to close the +resources before actually stopping the singleton actor. Note that `PoisonPill` is a +perfectly fine `terminationMessage` if you only need to stop the actor. With the names given above, access to the singleton can be obtained from any cluster node using a properly configured proxy. -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/singleton/ClusterSingletonManagerTest.java#create-singleton-proxy +@@snip [ClusterSingletonManagerTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/singleton/ClusterSingletonManagerTest.java) { #create-singleton-proxy } -A more comprehensive sample is available in the tutorial named `Distributed workers with Akka and Java! `_. +A more comprehensive sample is available in the tutorial named [Distributed workers with Akka and Java!](https://github.com/typesafehub/activator-akka-distributed-workers-java). -Dependencies ------------- +## Dependencies To use the Cluster Singleton you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` -maven:: +maven: - - com.typesafe.akka - akka-cluster-tools_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-cluster-tools_@binVersion@ + @version@ + +``` +## Configuration -Configuration -------------- - -The following configuration properties are read by the ``ClusterSingletonManagerSettings`` -when created with a ``ActorSystem`` parameter. It is also possible to amend the ``ClusterSingletonManagerSettings`` -or create it from another config section with the same layout as below. ``ClusterSingletonManagerSettings`` is -a parameter to the ``ClusterSingletonManager.props`` factory method, i.e. each singleton can be configured +The following configuration properties are read by the `ClusterSingletonManagerSettings` +when created with a `ActorSystem` parameter. It is also possible to amend the `ClusterSingletonManagerSettings` +or create it from another config section with the same layout as below. `ClusterSingletonManagerSettings` is +a parameter to the `ClusterSingletonManager.props` factory method, i.e. each singleton can be configured with different settings if needed. -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#singleton-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #singleton-config } -The following configuration properties are read by the ``ClusterSingletonProxySettings`` -when created with a ``ActorSystem`` parameter. It is also possible to amend the ``ClusterSingletonProxySettings`` -or create it from another config section with the same layout as below. ``ClusterSingletonProxySettings`` is -a parameter to the ``ClusterSingletonProxy.props`` factory method, i.e. each singleton proxy can be configured +The following configuration properties are read by the `ClusterSingletonProxySettings` +when created with a `ActorSystem` parameter. It is also possible to amend the `ClusterSingletonProxySettings` +or create it from another config section with the same layout as below. `ClusterSingletonProxySettings` is +a parameter to the `ClusterSingletonProxy.props` factory method, i.e. each singleton proxy can be configured with different settings if needed. -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#singleton-proxy-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #singleton-proxy-config } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/cluster-usage.md b/akka-docs/src/main/paradox/java/cluster-usage.md index 479529d8d8..10a14d71ab 100644 --- a/akka-docs/src/main/paradox/java/cluster-usage.md +++ b/akka-docs/src/main/paradox/java/cluster-usage.md @@ -1,97 +1,92 @@ +# Cluster Usage -.. _cluster_usage_java: +For introduction to the Akka Cluster concepts please see cluster. -############# -Cluster Usage -############# +## Preparing Your Project for Clustering -For introduction to the Akka Cluster concepts please see :ref:`cluster`. +The Akka cluster is a separate jar file. Make sure that you have the following dependency in your project: -Preparing Your Project for Clustering -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``` + + com.typesafe.akka + akka-cluster_@binVersion@ + @version@ + +``` -The Akka cluster is a separate jar file. Make sure that you have the following dependency in your project:: + +## A Simple Cluster Example - - com.typesafe.akka - akka-cluster_@binVersion@ - @version@ - - -.. _cluster_simple_example_java: - -A Simple Cluster Example -^^^^^^^^^^^^^^^^^^^^^^^^ - -The following configuration enables the ``Cluster`` extension to be used. +The following configuration enables the `Cluster` extension to be used. It joins the cluster and an actor subscribes to cluster membership events and logs them. -The ``application.conf`` configuration looks like this: +The `application.conf` configuration looks like this: -:: - - akka { - actor { - provider = "cluster" - } - remote { - log-remote-lifecycle-events = off - netty.tcp { - hostname = "127.0.0.1" - port = 0 - } - } - - cluster { - seed-nodes = [ - "akka.tcp://ClusterSystem@127.0.0.1:2551", - "akka.tcp://ClusterSystem@127.0.0.1:2552"] - - # auto downing is NOT safe for production deployments. - # you may want to use it during development, read more about it in the docs. - # - # auto-down-unreachable-after = 10s - } +``` +akka { + actor { + provider = "cluster" + } + remote { + log-remote-lifecycle-events = off + netty.tcp { + hostname = "127.0.0.1" + port = 0 } + } - # Disable legacy metrics in akka-cluster. - akka.cluster.metrics.enabled=off + cluster { + seed-nodes = [ + "akka.tcp://ClusterSystem@127.0.0.1:2551", + "akka.tcp://ClusterSystem@127.0.0.1:2552"] - # Enable metrics extension in akka-cluster-metrics. - akka.extensions=["akka.cluster.metrics.ClusterMetricsExtension"] + # auto downing is NOT safe for production deployments. + # you may want to use it during development, read more about it in the docs. + # + # auto-down-unreachable-after = 10s + } +} - # Sigar native library extract location during tests. - # Note: use per-jvm-instance folder when running multiple jvm on one host. - akka.cluster.metrics.native-library-extract-folder=${user.dir}/target/native +# Disable legacy metrics in akka-cluster. +akka.cluster.metrics.enabled=off -To enable cluster capabilities in your Akka project you should, at a minimum, add the :ref:`remoting-java` -settings, but with ``cluster``. -The ``akka.cluster.seed-nodes`` should normally also be added to your ``application.conf`` file. +# Enable metrics extension in akka-cluster-metrics. +akka.extensions=["akka.cluster.metrics.ClusterMetricsExtension"] -.. note:: - If you are running Akka in a Docker container or the nodes for some other reason have separate internal and - external ip addresses you must configure remoting according to :ref:`remote-configuration-nat-java` +# Sigar native library extract location during tests. +# Note: use per-jvm-instance folder when running multiple jvm on one host. +akka.cluster.metrics.native-library-extract-folder=${user.dir}/target/native +``` + +To enable cluster capabilities in your Akka project you should, at a minimum, add the @ref:[Remoting](remoting.md) +settings, but with `cluster`. +The `akka.cluster.seed-nodes` should normally also be added to your `application.conf` file. + +@@@ note + +If you are running Akka in a Docker container or the nodes for some other reason have separate internal and +external ip addresses you must configure remoting according to @ref:[Akka behind NAT or in a Docker container](remoting.md#remote-configuration-nat-java) + +@@@ The seed nodes are configured contact points for initial, automatic, join of the cluster. Note that if you are going to start the nodes on different machines you need to specify the -ip-addresses or host names of the machines in ``application.conf`` instead of ``127.0.0.1`` +ip-addresses or host names of the machines in `application.conf` instead of `127.0.0.1` An actor that uses the cluster extension may look like this: -.. literalinclude:: code/jdocs/cluster/SimpleClusterListener.java - :language: java +@@snip [SimpleClusterListener.java](code/jdocs/cluster/SimpleClusterListener.java) { type=java } The actor registers itself as subscriber of certain cluster events. It receives events corresponding to the current state of the cluster when the subscription starts and then it receives events for changes that happen in the cluster. The easiest way to run this example yourself is to download the ready to run -`Akka Cluster Sample with Scala <@exampleCodeService@/akka-samples-cluster-java>`_ -together with the tutorial. It contains instructions on how to run the ``SimpleClusterApp``. -The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-cluster-java>`_. +[Akka Cluster Sample with Scala](@exampleCodeService@/akka-samples-cluster-java) +together with the tutorial. It contains instructions on how to run the `SimpleClusterApp`. +The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-cluster-java). -Joining to Seed Nodes -^^^^^^^^^^^^^^^^^^^^^ +## Joining to Seed Nodes You may decide if joining to the cluster should be done manually or automatically to configured initial contact points, so-called seed nodes. When a new node is started @@ -99,25 +94,29 @@ it sends a message to all seed nodes and then sends join command to the one that answers first. If no one of the seed nodes replied (might not be started yet) it retries this procedure until successful or shutdown. -You define the seed nodes in the :ref:`cluster_configuration_java` file (application.conf):: +You define the seed nodes in the [cluster_configuration_java](#cluster-configuration-java) file (application.conf): - akka.cluster.seed-nodes = [ - "akka.tcp://ClusterSystem@host1:2552", - "akka.tcp://ClusterSystem@host2:2552"] +``` +akka.cluster.seed-nodes = [ + "akka.tcp://ClusterSystem@host1:2552", + "akka.tcp://ClusterSystem@host2:2552"] +``` -This can also be defined as Java system properties when starting the JVM using the following syntax:: +This can also be defined as Java system properties when starting the JVM using the following syntax: - -Dakka.cluster.seed-nodes.0=akka.tcp://ClusterSystem@host1:2552 - -Dakka.cluster.seed-nodes.1=akka.tcp://ClusterSystem@host2:2552 +``` +-Dakka.cluster.seed-nodes.0=akka.tcp://ClusterSystem@host1:2552 +-Dakka.cluster.seed-nodes.1=akka.tcp://ClusterSystem@host2:2552 +``` The seed nodes can be started in any order and it is not necessary to have all -seed nodes running, but the node configured as the first element in the ``seed-nodes`` +seed nodes running, but the node configured as the first element in the `seed-nodes` configuration list must be started when initially starting a cluster, otherwise the other seed-nodes will not become initialized and no other node can join the cluster. The reason for the special first seed node is to avoid forming separated islands when starting from an empty cluster. It is quickest to start all configured seed nodes at the same time (order doesn't matter), -otherwise it can take up to the configured ``seed-node-timeout`` until the nodes +otherwise it can take up to the configured `seed-node-timeout` until the nodes can join. Once more than two seed nodes have been started it is no problem to shut down the first @@ -126,26 +125,26 @@ seed nodes in the existing cluster. If you don't configure seed nodes you need to join the cluster programmatically or manually. -Manual joining can be performed by using :ref:`cluster_jmx_java` or :ref:`cluster_http_java`. -Joining programmatically can be performed with ``Cluster.get(system).join``. Unsuccessful join attempts are -automatically retried after the time period defined in configuration property ``retry-unsuccessful-join-after``. -Retries can be disabled by setting the property to ``off``. +Manual joining can be performed by using [cluster_jmx_java](#cluster-jmx-java) or [cluster_http_java](#cluster-http-java). +Joining programmatically can be performed with `Cluster.get(system).join`. Unsuccessful join attempts are +automatically retried after the time period defined in configuration property `retry-unsuccessful-join-after`. +Retries can be disabled by setting the property to `off`. You can join to any node in the cluster. It does not have to be configured as a seed node. Note that you can only join to an existing cluster member, which means that for bootstrapping some node must join itself,and then the following nodes could join them to make up a cluster. -You may also use ``Cluster.get(system).joinSeedNodes`` to join programmatically, +You may also use `Cluster.get(system).joinSeedNodes` to join programmatically, which is attractive when dynamically discovering other nodes at startup by using some external tool or API. -When using ``joinSeedNodes`` you should not include the node itself except for the node that is -supposed to be the first seed node, and that should be placed first in parameter to ``joinSeedNodes``. +When using `joinSeedNodes` you should not include the node itself except for the node that is +supposed to be the first seed node, and that should be placed first in parameter to `joinSeedNodes`. Unsuccessful attempts to contact seed nodes are automatically retried after the time period defined in -configuration property ``seed-node-timeout``. Unsuccessful attempt to join a specific seed node is -automatically retried after the configured ``retry-unsuccessful-join-after``. Retrying means that it +configuration property `seed-node-timeout`. Unsuccessful attempt to join a specific seed node is +automatically retried after the configured `retry-unsuccessful-join-after`. Retrying means that it tries to contact all seed nodes and then joins the node that answers first. The first node in the list of seed nodes will join itself if it cannot contact any of the other seed nodes within the -configured ``seed-node-timeout``. +configured `seed-node-timeout`. An actor system can only join a cluster once. Additional attempts will be ignored. When it has successfully joined it must be restarted to be able to join another @@ -154,41 +153,42 @@ after the restart, when it come up as new incarnation of existing member in the trying to join in, then the existing one will be removed from the cluster and then it will be allowed to join. -.. note:: +@@@ note - The name of the ``ActorSystem`` must be the same for all members of a cluster. The name is given - when you start the ``ActorSystem``. +The name of the `ActorSystem` must be the same for all members of a cluster. The name is given +when you start the `ActorSystem`. -.. _automatic-vs-manual-downing-java: +@@@ -Downing -^^^^^^^ + +## Downing When a member is considered by the failure detector to be unreachable the leader is not allowed to perform its duties, such as changing status of new joining members to 'Up'. The node must first become reachable again, or the status of the unreachable member must be changed to 'Down'. Changing status to 'Down' can be performed automatically or manually. By default it must be done manually, using -:ref:`cluster_jmx_java` or :ref:`cluster_http_java`. +[cluster_jmx_java](#cluster-jmx-java) or [cluster_http_java](#cluster-http-java). -It can also be performed programmatically with ``Cluster.get(system).down(address)``. +It can also be performed programmatically with `Cluster.get(system).down(address)`. A pre-packaged solution for the downing problem is provided by -`Split Brain Resolver `_, -which is part of the `Lightbend Reactive Platform `_. -If you don’t use RP, you should anyway carefully read the `documentation `_ +[Split Brain Resolver](http://developer.lightbend.com/docs/akka-commercial-addons/current/split-brain-resolver.html), +which is part of the [Lightbend Reactive Platform](http://www.lightbend.com/platform). +If you don’t use RP, you should anyway carefully read the [documentation](http://developer.lightbend.com/docs/akka-commercial-addons/current/split-brain-resolver.html) of the Split Brain Resolver and make sure that the solution you are using handles the concerns described there. -Auto-downing (DO NOT USE) -------------------------- +### Auto-downing (DO NOT USE) -There is an atomatic downing feature that you should not use in production. For testing purpose you can enable it with configuration:: +There is an atomatic downing feature that you should not use in production. For testing purpose you can enable it with configuration: - akka.cluster.auto-down-unreachable-after = 120s +``` +akka.cluster.auto-down-unreachable-after = 120s +``` -This means that the cluster leader member will change the ``unreachable`` node -status to ``down`` automatically after the configured time of unreachability. +This means that the cluster leader member will change the `unreachable` node +status to `down` automatically after the configured time of unreachability. This is a naïve approach to remove unreachable nodes from the cluster membership. It works great for crashes and short transient network partitions, but not for long network @@ -197,16 +197,17 @@ and after a while remove it from its cluster membership. Since this happens on b sides the result is that two separate disconnected clusters have been created. This can also happen because of long GC pauses or system overload. -.. warning:: +@@@ warning - We recommend against using the auto-down feature of Akka Cluster in production. - This is crucial for correct behavior if you use :ref:`cluster-singleton-java` or - :ref:`cluster_sharding_java`, especially together with Akka :ref:`persistence-java`. - For Akka Persistence with Cluster Sharding it can result in corrupt data in case - of network partitions. +We recommend against using the auto-down feature of Akka Cluster in production. +This is crucial for correct behavior if you use @ref:[Cluster Singleton](cluster-singleton.md) or +@ref:[cluster_sharding_java](cluster-sharding.md), especially together with Akka @ref:[Persistence](persistence.md). +For Akka Persistence with Cluster Sharding it can result in corrupt data in case +of network partitions. -Leaving -^^^^^^^ +@@@ + +## Leaving There are two ways to remove a member from the cluster. @@ -215,94 +216,91 @@ as unreachable and removed after the automatic or manual downing as described above. A more graceful exit can be performed if you tell the cluster that a node shall leave. -This can be performed using :ref:`cluster_jmx_java` or :ref:`cluster_http_java`. +This can be performed using [cluster_jmx_java](#cluster-jmx-java) or [cluster_http_java](#cluster-http-java). It can also be performed programmatically with: -.. includecode:: code/jdocs/cluster/ClusterDocTest.java#leave +@@snip [ClusterDocTest.java](code/jdocs/cluster/ClusterDocTest.java) { #leave } Note that this command can be issued to any member in the cluster, not necessarily the one that is leaving. -The :ref:`coordinated-shutdown-java` will automatically run when the cluster node sees itself as -``Exiting``, i.e. leaving from another node will trigger the shutdown process on the leaving node. +The @ref:[Coordinated Shutdown](actors.md#coordinated-shutdown-java) will automatically run when the cluster node sees itself as +`Exiting`, i.e. leaving from another node will trigger the shutdown process on the leaving node. Tasks for graceful leaving of cluster including graceful shutdown of Cluster Singletons and Cluster Sharding are added automatically when Akka Cluster is used, i.e. running the shutdown process will also trigger the graceful leaving if it's not already in progress. Normally this is handled automatically, but in case of network failures during this process it might still -be necessary to set the node’s status to ``Down`` in order to complete the removal. +be necessary to set the node’s status to `Down` in order to complete the removal. -.. _weakly_up_java: + +## WeaklyUp Members -WeaklyUp Members -^^^^^^^^^^^^^^^^ - -If a node is ``unreachable`` then gossip convergence is not possible and therefore any -``leader`` actions are also not possible. However, we still might want new nodes to join +If a node is `unreachable` then gossip convergence is not possible and therefore any +`leader` actions are also not possible. However, we still might want new nodes to join the cluster in this scenario. -``Joining`` members will be promoted to ``WeaklyUp`` and become part of the cluster if -convergence can't be reached. Once gossip convergence is reached, the leader will move ``WeaklyUp`` -members to ``Up``. +`Joining` members will be promoted to `WeaklyUp` and become part of the cluster if +convergence can't be reached. Once gossip convergence is reached, the leader will move `WeaklyUp` +members to `Up`. -This feature is enabled by default, but it can be disabled with configuration option:: +This feature is enabled by default, but it can be disabled with configuration option: - akka.cluster.allow-weakly-up-members = off +``` +akka.cluster.allow-weakly-up-members = off +``` -You can subscribe to the ``WeaklyUp`` membership event to make use of the members that are +You can subscribe to the `WeaklyUp` membership event to make use of the members that are in this state, but you should be aware of that members on the other side of a network partition have no knowledge about the existence of the new members. You should for example not count -``WeaklyUp`` members in quorum decisions. +`WeaklyUp` members in quorum decisions. -.. _cluster_subscriber_java: - -Subscribe to Cluster Events -^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## Subscribe to Cluster Events You can subscribe to change notifications of the cluster membership by using -``Cluster.get(system).subscribe``. +`Cluster.get(system).subscribe`. -.. includecode:: code/jdocs/cluster/SimpleClusterListener2.java#subscribe +@@snip [SimpleClusterListener2.java](code/jdocs/cluster/SimpleClusterListener2.java) { #subscribe } -A snapshot of the full state, ``akka.cluster.ClusterEvent.CurrentClusterState``, is sent to the subscriber +A snapshot of the full state, `akka.cluster.ClusterEvent.CurrentClusterState`, is sent to the subscriber as the first message, followed by events for incremental updates. -Note that you may receive an empty ``CurrentClusterState``, containing no members, +Note that you may receive an empty `CurrentClusterState`, containing no members, if you start the subscription before the initial join procedure has completed. This is expected behavior. When the node has been accepted in the cluster you will -receive ``MemberUp`` for that node, and other nodes. +receive `MemberUp` for that node, and other nodes. -If you find it inconvenient to handle the ``CurrentClusterState`` you can use -``ClusterEvent.initialStateAsEvents()`` as parameter to ``subscribe``. -That means that instead of receiving ``CurrentClusterState`` as the first message you will receive +If you find it inconvenient to handle the `CurrentClusterState` you can use +`ClusterEvent.initialStateAsEvents()` as parameter to `subscribe`. +That means that instead of receiving `CurrentClusterState` as the first message you will receive the events corresponding to the current state to mimic what you would have seen if you were listening to the events when they occurred in the past. Note that those initial events only correspond to the current state and it is not the full history of all changes that actually has occurred in the cluster. -.. includecode:: code/jdocs/cluster/SimpleClusterListener.java#subscribe +@@snip [SimpleClusterListener.java](code/jdocs/cluster/SimpleClusterListener.java) { #subscribe } The events to track the life-cycle of members are: -* ``ClusterEvent.MemberJoined`` - A new member has joined the cluster and its status has been changed to ``Joining``. -* ``ClusterEvent.MemberUp`` - A new member has joined the cluster and its status has been changed to ``Up``. -* ``ClusterEvent.MemberExited`` - A member is leaving the cluster and its status has been changed to ``Exiting`` - Note that the node might already have been shutdown when this event is published on another node. -* ``ClusterEvent.MemberRemoved`` - Member completely removed from the cluster. -* ``ClusterEvent.UnreachableMember`` - A member is considered as unreachable, detected by the failure detector - of at least one other node. -* ``ClusterEvent.ReachableMember`` - A member is considered as reachable again, after having been unreachable. - All nodes that previously detected it as unreachable has detected it as reachable again. + * `ClusterEvent.MemberJoined` - A new member has joined the cluster and its status has been changed to `Joining`. + * `ClusterEvent.MemberUp` - A new member has joined the cluster and its status has been changed to `Up`. + * `ClusterEvent.MemberExited` - A member is leaving the cluster and its status has been changed to `Exiting` +Note that the node might already have been shutdown when this event is published on another node. + * `ClusterEvent.MemberRemoved` - Member completely removed from the cluster. + * `ClusterEvent.UnreachableMember` - A member is considered as unreachable, detected by the failure detector +of at least one other node. + * `ClusterEvent.ReachableMember` - A member is considered as reachable again, after having been unreachable. +All nodes that previously detected it as unreachable has detected it as reachable again. There are more types of change events, consult the API documentation -of classes that extends ``akka.cluster.ClusterEvent.ClusterDomainEvent`` +of classes that extends `akka.cluster.ClusterEvent.ClusterDomainEvent` for details about the events. Instead of subscribing to cluster events it can sometimes be convenient to only get the full membership state with -``Cluster.get(system).state()``. Note that this state is not necessarily in sync with the events published to a +`Cluster.get(system).state()`. Note that this state is not necessarily in sync with the events published to a cluster subscription. -Worker Dial-in Example ----------------------- +### Worker Dial-in Example Let's take a look at an example that illustrates how workers, here named *backend*, can detect and register to new master nodes, here named *frontend*. @@ -315,180 +313,179 @@ added or removed to the cluster dynamically. Messages: -.. includecode:: code/jdocs/cluster/TransformationMessages.java#messages +@@snip [TransformationMessages.java](code/jdocs/cluster/TransformationMessages.java) { #messages } The backend worker that performs the transformation job: -.. includecode:: code/jdocs/cluster/TransformationBackend.java#backend +@@snip [TransformationBackend.java](code/jdocs/cluster/TransformationBackend.java) { #backend } -Note that the ``TransformationBackend`` actor subscribes to cluster events to detect new, +Note that the `TransformationBackend` actor subscribes to cluster events to detect new, potential, frontend nodes, and send them a registration message so that they know that they can use the backend worker. The frontend that receives user jobs and delegates to one of the registered backend workers: -.. includecode:: code/jdocs/cluster/TransformationFrontend.java#frontend +@@snip [TransformationFrontend.java](code/jdocs/cluster/TransformationFrontend.java) { #frontend } -Note that the ``TransformationFrontend`` actor watch the registered backend +Note that the `TransformationFrontend` actor watch the registered backend to be able to remove it from its list of available backend workers. Death watch uses the cluster failure detector for nodes in the cluster, i.e. it detects network failures and JVM crashes, in addition to graceful termination of watched -actor. Death watch generates the ``Terminated`` message to the watching actor when the +actor. Death watch generates the `Terminated` message to the watching actor when the unreachable cluster node has been downed and removed. The Akka sample named -`Akka Cluster Sample with Java `_. +[Akka Cluster Sample with Java](https://github.com/akka/akka-samples/tree/master/akka-sample-cluster-java). contains the full source code and instructions of how to run the **Worker Dial-in Example**. -Node Roles -^^^^^^^^^^ +## Node Roles Not all nodes of a cluster need to perform the same function: there might be one sub-set which runs the web front-end, one which runs the data access layer and one for the number-crunching. Deployment of actors—for example by cluster-aware routers—can take node roles into account to achieve this distribution of responsibilities. -The roles of a node is defined in the configuration property named ``akka.cluster.roles`` +The roles of a node is defined in the configuration property named `akka.cluster.roles` and it is typically defined in the start script as a system property or environment variable. -The roles of the nodes is part of the membership information in ``MemberEvent`` that you can subscribe to. +The roles of the nodes is part of the membership information in `MemberEvent` that you can subscribe to. -.. _min-members_java: - -How To Startup when Cluster Size Reached -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## How To Startup when Cluster Size Reached A common use case is to start actors after the cluster has been initialized, members have joined, and the cluster has reached a certain size. With a configuration option you can define required number of members -before the leader changes member status of 'Joining' members to 'Up'.:: +before the leader changes member status of 'Joining' members to 'Up'.: - akka.cluster.min-nr-of-members = 3 +``` +akka.cluster.min-nr-of-members = 3 +``` In a similar way you can define required number of members of a certain role -before the leader changes member status of 'Joining' members to 'Up'.:: +before the leader changes member status of 'Joining' members to 'Up'.: - akka.cluster.role { - frontend.min-nr-of-members = 1 - backend.min-nr-of-members = 2 - } +``` +akka.cluster.role { + frontend.min-nr-of-members = 1 + backend.min-nr-of-members = 2 +} +``` -You can start the actors in a ``registerOnMemberUp`` callback, which will +You can start the actors in a `registerOnMemberUp` callback, which will be invoked when the current member status is changed to 'Up', i.e. the cluster has at least the defined number of members. -.. includecode:: code/jdocs/cluster/FactorialFrontendMain.java#registerOnUp +@@snip [FactorialFrontendMain.java](code/jdocs/cluster/FactorialFrontendMain.java) { #registerOnUp } This callback can be used for other things than starting actors. -How To Cleanup when Member is Removed -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## How To Cleanup when Member is Removed -You can do some clean up in a ``registerOnMemberRemoved`` callback, which will +You can do some clean up in a `registerOnMemberRemoved` callback, which will be invoked when the current member status is changed to 'Removed' or the cluster have been shutdown. -An alternative is to register tasks to the :ref:`coordinated-shutdown-java`. +An alternative is to register tasks to the @ref:[Coordinated Shutdown](actors.md#coordinated-shutdown-java). -.. note:: - Register a OnMemberRemoved callback on a cluster that have been shutdown, the callback will be invoked immediately on - the caller thread, otherwise it will be invoked later when the current member status changed to 'Removed'. You may - want to install some cleanup handling after the cluster was started up, but the cluster might already be shutting - down when you installing, and depending on the race is not healthy. +@@@ note -Cluster Singleton -^^^^^^^^^^^^^^^^^ +Register a OnMemberRemoved callback on a cluster that have been shutdown, the callback will be invoked immediately on +the caller thread, otherwise it will be invoked later when the current member status changed to 'Removed'. You may +want to install some cleanup handling after the cluster was started up, but the cluster might already be shutting +down when you installing, and depending on the race is not healthy. + +@@@ + +## Cluster Singleton For some use cases it is convenient and sometimes also mandatory to ensure that you have exactly one actor of a certain type running somewhere in the cluster. This can be implemented by subscribing to member events, but there are several corner cases to consider. Therefore, this specific use case is made easily accessible by the -:ref:`cluster-singleton-java`. +@ref:[Cluster Singleton](cluster-singleton.md). -Cluster Sharding -^^^^^^^^^^^^^^^^ +## Cluster Sharding Distributes actors across several nodes in the cluster and supports interaction with the actors using their logical identifier, but without having to care about their physical location in the cluster. -See :ref:`cluster_sharding_java`. +See @ref:[cluster_sharding_java](cluster-sharding.md). -Distributed Publish Subscribe -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## Distributed Publish Subscribe Publish-subscribe messaging between actors in the cluster, and point-to-point messaging using the logical path of the actors, i.e. the sender does not have to know on which node the destination actor is running. -See :ref:`distributed-pub-sub-java`. +See @ref:[Distributed Publish Subscribe in Cluster](distributed-pub-sub.md). -Cluster Client -^^^^^^^^^^^^^^ +## Cluster Client Communication from an actor system that is not part of the cluster to actors running somewhere in the cluster. The client does not have to know on which node the destination actor is running. -See :ref:`cluster-client-java`. +See @ref:[Cluster Client](cluster-client.md). -Distributed Data -^^^^^^^^^^^^^^^^ +## Distributed Data *Akka Distributed Data* is useful when you need to share data between nodes in an Akka Cluster. The data is accessed with an actor providing a key-value store like API. -See :ref:`distributed_data_java`. +See @ref:[distributed_data_java](distributed-data.md). -Failure Detector -^^^^^^^^^^^^^^^^ +## Failure Detector In a cluster each node is monitored by a few (default maximum 5) other nodes, and when -any of these detects the node as ``unreachable`` that information will spread to +any of these detects the node as `unreachable` that information will spread to the rest of the cluster through the gossip. In other words, only one node needs to -mark a node ``unreachable`` to have the rest of the cluster mark that node ``unreachable``. +mark a node `unreachable` to have the rest of the cluster mark that node `unreachable`. -The failure detector will also detect if the node becomes ``reachable`` again. When -all nodes that monitored the ``unreachable`` node detects it as ``reachable`` again -the cluster, after gossip dissemination, will consider it as ``reachable``. +The failure detector will also detect if the node becomes `reachable` again. When +all nodes that monitored the `unreachable` node detects it as `reachable` again +the cluster, after gossip dissemination, will consider it as `reachable`. If system messages cannot be delivered to a node it will be quarantined and then it -cannot come back from ``unreachable``. This can happen if the there are too many +cannot come back from `unreachable`. This can happen if the there are too many unacknowledged system messages (e.g. watch, Terminated, remote actor deployment, failures of actors supervised by remote parent). Then the node needs to be moved -to the ``down`` or ``removed`` states and the actor system of the quarantined node +to the `down` or `removed` states and the actor system of the quarantined node must be restarted before it can join the cluster again. The nodes in the cluster monitor each other by sending heartbeats to detect if a node is unreachable from the rest of the cluster. The heartbeat arrival times is interpreted by an implementation of -`The Phi Accrual Failure Detector `_. +[The Phi Accrual Failure Detector](http://www.jaist.ac.jp/~defago/files/pdf/IS_RR_2004_010.pdf). The suspicion level of failure is given by a value called *phi*. The basic idea of the phi failure detector is to express the value of *phi* on a scale that is dynamically adjusted to reflect current network conditions. -The value of *phi* is calculated as:: +The value of *phi* is calculated as: - phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` +phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` where F is the cumulative distribution function of a normal distribution with mean and standard deviation estimated from historical heartbeat inter-arrival times. -In the :ref:`cluster_configuration_java` you can adjust the ``akka.cluster.failure-detector.threshold`` +In the [cluster_configuration_java](#cluster-configuration-java) you can adjust the `akka.cluster.failure-detector.threshold` to define when a *phi* value is considered to be a failure. -A low ``threshold`` is prone to generate many false positives but ensures -a quick detection in the event of a real crash. Conversely, a high ``threshold`` +A low `threshold` is prone to generate many false positives but ensures +a quick detection in the event of a real crash. Conversely, a high `threshold` generates fewer mistakes but needs more time to detect actual crashes. The -default ``threshold`` is 8 and is appropriate for most situations. However in +default `threshold` is 8 and is appropriate for most situations. However in cloud environments, such as Amazon EC2, the value could be increased to 12 in order to account for network issues that sometimes occur on such platforms. The following chart illustrates how *phi* increase with increasing time since the previous heartbeat. -.. image:: ../images/phi1.png +![phi1.png](../images/phi1.png) Phi is calculated from the mean and standard deviation of historical inter arrival times. The previous chart is an example for standard deviation @@ -496,91 +493,92 @@ of 200 ms. If the heartbeats arrive with less deviation the curve becomes steepe i.e. it is possible to determine failure more quickly. The curve looks like this for a standard deviation of 100 ms. -.. image:: ../images/phi2.png +![phi2.png](../images/phi2.png) To be able to survive sudden abnormalities, such as garbage collection pauses and transient network failures the failure detector is configured with a margin, -``akka.cluster.failure-detector.acceptable-heartbeat-pause``. You may want to -adjust the :ref:`cluster_configuration_java` of this depending on you environment. -This is how the curve looks like for ``acceptable-heartbeat-pause`` configured to +`akka.cluster.failure-detector.acceptable-heartbeat-pause`. You may want to +adjust the [cluster_configuration_java](#cluster-configuration-java) of this depending on you environment. +This is how the curve looks like for `acceptable-heartbeat-pause` configured to 3 seconds. -.. image:: ../images/phi3.png +![phi3.png](../images/phi3.png) Death watch uses the cluster failure detector for nodes in the cluster, i.e. it detects network failures and JVM crashes, in addition to graceful termination of watched -actor. Death watch generates the ``Terminated`` message to the watching actor when the +actor. Death watch generates the `Terminated` message to the watching actor when the unreachable cluster node has been downed and removed. If you encounter suspicious false positives when the system is under load you should -define a separate dispatcher for the cluster actors as described in :ref:`cluster_dispatcher_java`. +define a separate dispatcher for the cluster actors as described in [cluster_dispatcher_java](#cluster-dispatcher-java). -Cluster Aware Routers -^^^^^^^^^^^^^^^^^^^^^ +## Cluster Aware Routers -All :ref:`routers ` can be made aware of member nodes in the cluster, i.e. +All @ref:[routers](routing.md) can be made aware of member nodes in the cluster, i.e. deploying new routees or looking up routees on nodes in the cluster. When a node becomes unreachable or leaves the cluster the routees of that node are automatically unregistered from the router. When new nodes join the cluster additional routees are added to the router, according to the configuration. Routees are also added when a node becomes reachable again, after having been unreachable. -Cluster aware routers make use of members with status :ref:`WeaklyUp `. +Cluster aware routers make use of members with status [WeaklyUp](#weakly-up-java). There are two distinct types of routers. -* **Group - router that sends messages to the specified path using actor selection** - The routees can be shared between routers running on different nodes in the cluster. - One example of a use case for this type of router is a service running on some backend - nodes in the cluster and used by routers running on front-end nodes in the cluster. + * **Group - router that sends messages to the specified path using actor selection** +The routees can be shared between routers running on different nodes in the cluster. +One example of a use case for this type of router is a service running on some backend +nodes in the cluster and used by routers running on front-end nodes in the cluster. + * **Pool - router that creates routees as child actors and deploys them on remote nodes.** +Each router will have its own routee instances. For example, if you start a router +on 3 nodes in a 10 nodes cluster you will have 30 routee actors in total if the router is +configured to use one instance per node. The routees created by the different routers +will not be shared between the routers. One example of a use case for this type of router +is a single master that coordinate jobs and delegates the actual work to routees running +on other nodes in the cluster. -* **Pool - router that creates routees as child actors and deploys them on remote nodes.** - Each router will have its own routee instances. For example, if you start a router - on 3 nodes in a 10 nodes cluster you will have 30 routee actors in total if the router is - configured to use one instance per node. The routees created by the different routers - will not be shared between the routers. One example of a use case for this type of router - is a single master that coordinate jobs and delegates the actual work to routees running - on other nodes in the cluster. +### Router with Group of Routees -Router with Group of Routees ----------------------------- +When using a `Group` you must start the routee actors on the cluster member nodes. +That is not done by the router. The configuration for a group looks like this:: -When using a ``Group`` you must start the routee actors on the cluster member nodes. -That is not done by the router. The configuration for a group looks like this::: - - akka.actor.deployment { - /statsService/workerRouter { - router = consistent-hashing-group - routees.paths = ["/user/statsWorker"] - cluster { - enabled = on - allow-local-routees = on - use-role = compute - } - } +``` +akka.actor.deployment { + /statsService/workerRouter { + router = consistent-hashing-group + routees.paths = ["/user/statsWorker"] + cluster { + enabled = on + allow-local-routees = on + use-role = compute + } } +} +``` -.. note:: - The routee actors should be started as early as possible when starting the actor system, because - the router will try to use them as soon as the member status is changed to 'Up'. +@@@ note -The actor paths without address information that are defined in ``routees.paths`` are used for selecting the +The routee actors should be started as early as possible when starting the actor system, because +the router will try to use them as soon as the member status is changed to 'Up'. + +@@@ + +The actor paths without address information that are defined in `routees.paths` are used for selecting the actors to which the messages will be forwarded to by the router. -Messages will be forwarded to the routees using :ref:`ActorSelection `, so the same delivery semantics should be expected. -It is possible to limit the lookup of routees to member nodes tagged with a certain role by specifying ``use-role``. +Messages will be forwarded to the routees using @ref:[ActorSelection](actors.md#actorselection-java), so the same delivery semantics should be expected. +It is possible to limit the lookup of routees to member nodes tagged with a certain role by specifying `use-role`. -``max-total-nr-of-instances`` defines total number of routees in the cluster. By default ``max-total-nr-of-instances`` +`max-total-nr-of-instances` defines total number of routees in the cluster. By default `max-total-nr-of-instances` is set to a high value (10000) that will result in new routees added to the router when nodes join the cluster. Set it to a lower value if you want to limit total number of routees. The same type of router could also have been defined in code: -.. includecode:: code/jdocs/cluster/StatsService.java#router-lookup-in-code +@@snip [StatsService.java](code/jdocs/cluster/StatsService.java) { #router-lookup-in-code } -See :ref:`cluster_configuration_java` section for further descriptions of the settings. +See [cluster_configuration_java](#cluster-configuration-java) section for further descriptions of the settings. -Router Example with Group of Routees ------------------------------------- +### Router Example with Group of Routees Let's take a look at how to use a cluster aware router with a group of routees, i.e. router sending to the paths of the routees. @@ -593,231 +591,231 @@ the average number of characters per word when all results have been collected. Messages: -.. includecode:: code/jdocs/cluster/StatsMessages.java#messages +@@snip [StatsMessages.java](code/jdocs/cluster/StatsMessages.java) { #messages } The worker that counts number of characters in each word: -.. includecode:: code/jdocs/cluster/StatsWorker.java#worker +@@snip [StatsWorker.java](code/jdocs/cluster/StatsWorker.java) { #worker } The service that receives text from users and splits it up into words, delegates to workers and aggregates: -.. includecode:: code/jdocs/cluster/StatsService.java#service - -.. includecode:: code/jdocs/cluster/StatsAggregator.java#aggregator +@@snip [StatsService.java](code/jdocs/cluster/StatsService.java) { #service } +@@snip [StatsAggregator.java](code/jdocs/cluster/StatsAggregator.java) { #aggregator } Note, nothing cluster specific so far, just plain actors. -All nodes start ``StatsService`` and ``StatsWorker`` actors. Remember, routees are the workers in this case. -The router is configured with ``routees.paths``::: +All nodes start `StatsService` and `StatsWorker` actors. Remember, routees are the workers in this case. +The router is configured with `routees.paths`:: - akka.actor.deployment { - /statsService/workerRouter { - router = consistent-hashing-group - routees.paths = ["/user/statsWorker"] - cluster { - enabled = on - allow-local-routees = on - use-role = compute - } - } +``` +akka.actor.deployment { + /statsService/workerRouter { + router = consistent-hashing-group + routees.paths = ["/user/statsWorker"] + cluster { + enabled = on + allow-local-routees = on + use-role = compute } + } +} +``` -This means that user requests can be sent to ``StatsService`` on any node and it will use -``StatsWorker`` on all nodes. +This means that user requests can be sent to `StatsService` on any node and it will use +`StatsWorker` on all nodes. The Akka sample named -`Akka Cluster Sample with Java `_. +[Akka Cluster Sample with Java](https://github.com/akka/akka-samples/tree/master/akka-sample-cluster-java). contains the full source code and instructions of how to run the **Router Example with Group of Routees**. -Router with Pool of Remote Deployed Routees -------------------------------------------- +### Router with Pool of Remote Deployed Routees -When using a ``Pool`` with routees created and deployed on the cluster member nodes -the configuration for a router looks like this::: +When using a `Pool` with routees created and deployed on the cluster member nodes +the configuration for a router looks like this:: - akka.actor.deployment { - /statsService/singleton/workerRouter { - router = consistent-hashing-pool - cluster { - enabled = on - max-nr-of-instances-per-node = 3 - allow-local-routees = on - use-role = compute - } - } +``` +akka.actor.deployment { + /statsService/singleton/workerRouter { + router = consistent-hashing-pool + cluster { + enabled = on + max-nr-of-instances-per-node = 3 + allow-local-routees = on + use-role = compute + } } +} +``` It is possible to limit the deployment of routees to member nodes tagged with a certain role by -specifying ``use-role``. +specifying `use-role`. -``max-total-nr-of-instances`` defines total number of routees in the cluster, but the number of routees -per node, ``max-nr-of-instances-per-node``, will not be exceeded. By default ``max-total-nr-of-instances`` +`max-total-nr-of-instances` defines total number of routees in the cluster, but the number of routees +per node, `max-nr-of-instances-per-node`, will not be exceeded. By default `max-total-nr-of-instances` is set to a high value (10000) that will result in new routees added to the router when nodes join the cluster. Set it to a lower value if you want to limit total number of routees. The same type of router could also have been defined in code: -.. includecode:: code/jdocs/cluster/StatsService.java#router-deploy-in-code +@@snip [StatsService.java](code/jdocs/cluster/StatsService.java) { #router-deploy-in-code } -See :ref:`cluster_configuration_java` section for further descriptions of the settings. +See [cluster_configuration_java](#cluster-configuration-java) section for further descriptions of the settings. -Router Example with Pool of Remote Deployed Routees ---------------------------------------------------- +### Router Example with Pool of Remote Deployed Routees Let's take a look at how to use a cluster aware router on single master node that creates -and deploys workers. To keep track of a single master we use the :ref:`cluster-singleton-java` -in the cluster-tools module. The ``ClusterSingletonManager`` is started on each node. +and deploys workers. To keep track of a single master we use the @ref:[Cluster Singleton](cluster-singleton.md) +in the cluster-tools module. The `ClusterSingletonManager` is started on each node. -.. includecode:: code/jdocs/cluster/StatsSampleOneMasterMain.java#create-singleton-manager +@@snip [StatsSampleOneMasterMain.java](code/jdocs/cluster/StatsSampleOneMasterMain.java) { #create-singleton-manager } We also need an actor on each node that keeps track of where current single master exists and -delegates jobs to the ``StatsService``. That is provided by the ``ClusterSingletonProxy``. +delegates jobs to the `StatsService`. That is provided by the `ClusterSingletonProxy`. -.. includecode:: code/jdocs/cluster/StatsSampleOneMasterMain.java#singleton-proxy +@@snip [StatsSampleOneMasterMain.java](code/jdocs/cluster/StatsSampleOneMasterMain.java) { #singleton-proxy } -The ``ClusterSingletonProxy`` receives text from users and delegates to the current ``StatsService``, the single -master. It listens to cluster events to lookup the ``StatsService`` on the oldest node. +The `ClusterSingletonProxy` receives text from users and delegates to the current `StatsService`, the single +master. It listens to cluster events to lookup the `StatsService` on the oldest node. -All nodes start ``ClusterSingletonProxy`` and the ``ClusterSingletonManager``. The router is now configured like this::: +All nodes start `ClusterSingletonProxy` and the `ClusterSingletonManager`. The router is now configured like this:: - akka.actor.deployment { - /statsService/singleton/workerRouter { - router = consistent-hashing-pool - cluster { - enabled = on - max-nr-of-instances-per-node = 3 - allow-local-routees = on - use-role = compute - } - } +``` +akka.actor.deployment { + /statsService/singleton/workerRouter { + router = consistent-hashing-pool + cluster { + enabled = on + max-nr-of-instances-per-node = 3 + allow-local-routees = on + use-role = compute } + } +} +``` The Akka sample named -`Akka Cluster Sample with Java `_. +[Akka Cluster Sample with Java](https://github.com/akka/akka-samples/tree/master/akka-sample-cluster-java). contains the full source code and instructions of how to run the **Router Example with Pool of Remote Deployed Routees**. -Cluster Metrics -^^^^^^^^^^^^^^^ +## Cluster Metrics The member nodes of the cluster can collect system health metrics and publish that to other cluster nodes -and to the registered subscribers on the system event bus with the help of :doc:`cluster-metrics`. +and to the registered subscribers on the system event bus with the help of `cluster-metrics`. +## Management -Management -^^^^^^^^^^ - -.. _cluster_http_java: - -HTTP ----- + +### HTTP Information and management of the cluster is available with a HTTP API. -See documentation of `akka/akka-cluster-management `_. +See documentation of [akka/akka-cluster-management](https://github.com/akka/akka-cluster-management). -.. _cluster_jmx_java: + +### JMX -JMX ---- - -Information and management of the cluster is available as JMX MBeans with the root name ``akka.Cluster``. +Information and management of the cluster is available as JMX MBeans with the root name `akka.Cluster`. The JMX information can be displayed with an ordinary JMX console such as JConsole or JVisualVM. From JMX you can: -* see what members that are part of the cluster -* see status of this node -* see roles of each member -* join this node to another node in cluster -* mark any node in the cluster as down -* tell any node in the cluster to leave + * see what members that are part of the cluster + * see status of this node + * see roles of each member + * join this node to another node in cluster + * mark any node in the cluster as down + * tell any node in the cluster to leave -Member nodes are identified by their address, in format `akka.://@:`. +Member nodes are identified by their address, in format *akka.://@:*. -.. _cluster_command_line_java: + +### Command Line -Command Line ------------- +@@@ warning -.. warning:: - **Deprecation warning** - The command line script has been deprecated and is scheduled for removal - in the next major version. Use the :ref:`cluster_http_java` API with `curl `_ - or similar instead. +**Deprecation warning** - The command line script has been deprecated and is scheduled for removal +in the next major version. Use the [cluster_http_java](#cluster-http-java) API with [curl](https://curl.haxx.se/) +or similar instead. -The cluster can be managed with the script ``akka-cluster`` provided in the Akka github repository here: @github@/akka-cluster/jmx-client. Place the script and the ``jmxsh-R5.jar`` library in the same directory. +@@@ -Run it without parameters to see instructions about how to use the script:: +The cluster can be managed with the script `akka-cluster` provided in the Akka github repository here: @[github@/akka-cluster/jmx-client](mailto:github@/akka-cluster/jmx-client). Place the script and the `jmxsh-R5.jar` library in the same directory. - Usage: ./akka-cluster ... +Run it without parameters to see instructions about how to use the script: - Supported commands are: - join - Sends request a JOIN node with the specified URL - leave - Sends a request for node with URL to LEAVE the cluster - down - Sends a request for marking node with URL as DOWN - member-status - Asks the member node for its current status - members - Asks the cluster for addresses of current members - unreachable - Asks the cluster for addresses of unreachable members - cluster-status - Asks the cluster for its current status (member ring, - unavailable nodes, meta data etc.) - leader - Asks the cluster who the current leader is - is-singleton - Checks if the cluster is a singleton cluster (single - node cluster) - is-available - Checks if the member node is available - Where the should be on the format of - 'akka.://@:' +``` +Usage: ./akka-cluster ... - Examples: ./akka-cluster localhost 9999 is-available - ./akka-cluster localhost 9999 join akka.tcp://MySystem@darkstar:2552 - ./akka-cluster localhost 9999 cluster-status +Supported commands are: + join - Sends request a JOIN node with the specified URL + leave - Sends a request for node with URL to LEAVE the cluster + down - Sends a request for marking node with URL as DOWN + member-status - Asks the member node for its current status + members - Asks the cluster for addresses of current members + unreachable - Asks the cluster for addresses of unreachable members + cluster-status - Asks the cluster for its current status (member ring, + unavailable nodes, meta data etc.) + leader - Asks the cluster who the current leader is + is-singleton - Checks if the cluster is a singleton cluster (single + node cluster) + is-available - Checks if the member node is available +Where the should be on the format of + 'akka.://@:' +Examples: ./akka-cluster localhost 9999 is-available + ./akka-cluster localhost 9999 join akka.tcp://MySystem@darkstar:2552 + ./akka-cluster localhost 9999 cluster-status +``` To be able to use the script you must enable remote monitoring and management when starting the JVMs of the cluster nodes, -as described in `Monitoring and Management Using JMX Technology `_. +as described in [Monitoring and Management Using JMX Technology](http://docs.oracle.com/javase/8/jdocs/technotes/guides/management/agent.html). Make sure you understand the security implications of enabling remote monitoring and management. -.. _cluster_configuration_java: - -Configuration -^^^^^^^^^^^^^ + +## Configuration There are several configuration properties for the cluster. We refer to the -:ref:`reference configuration ` for more information. +@ref:[reference configuration](../general/configuration.md#config-akka-cluster) for more information. -Cluster Info Logging --------------------- +### Cluster Info Logging -You can silence the logging of cluster events at info level with configuration property:: +You can silence the logging of cluster events at info level with configuration property: - akka.cluster.log-info = off +``` +akka.cluster.log-info = off +``` -.. _cluster_dispatcher_java: - -Cluster Dispatcher ------------------- + +### Cluster Dispatcher Under the hood the cluster extension is implemented with actors and it can be necessary to create a bulkhead for those actors to avoid disturbance from other actors. Especially the heartbeating actors that is used for failure detection can generate false positives if they are not given a chance to run at regular intervals. -For this purpose you can define a separate dispatcher to be used for the cluster actors:: +For this purpose you can define a separate dispatcher to be used for the cluster actors: - akka.cluster.use-dispatcher = cluster-dispatcher +``` +akka.cluster.use-dispatcher = cluster-dispatcher - cluster-dispatcher { - type = "Dispatcher" - executor = "fork-join-executor" - fork-join-executor { - parallelism-min = 2 - parallelism-max = 4 - } +cluster-dispatcher { + type = "Dispatcher" + executor = "fork-join-executor" + fork-join-executor { + parallelism-min = 2 + parallelism-max = 4 } +} +``` -.. note:: - Normally it should not be necessary to configure a separate dispatcher for the Cluster. - The default-dispatcher should be sufficient for performing the Cluster tasks, i.e. ``akka.cluster.use-dispatcher`` - should not be changed. If you have Cluster related problems when using the default-dispatcher that is typically an - indication that you are running blocking or CPU intensive actors/tasks on the default-dispatcher. - Use dedicated dispatchers for such actors/tasks instead of running them on the default-dispatcher, - because that may starve system internal tasks. - Related config properties: ``akka.cluster.use-dispatcher = akka.cluster.cluster-dispatcher``. - Corresponding default values: ``akka.cluster.use-dispatcher =``. +@@@ note + +Normally it should not be necessary to configure a separate dispatcher for the Cluster. +The default-dispatcher should be sufficient for performing the Cluster tasks, i.e. `akka.cluster.use-dispatcher` +should not be changed. If you have Cluster related problems when using the default-dispatcher that is typically an +indication that you are running blocking or CPU intensive actors/tasks on the default-dispatcher. +Use dedicated dispatchers for such actors/tasks instead of running them on the default-dispatcher, +because that may starve system internal tasks. +Related config properties: `akka.cluster.use-dispatcher = akka.cluster.cluster-dispatcher`. +Corresponding default values: `akka.cluster.use-dispatcher =`. + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/dispatchers.md b/akka-docs/src/main/paradox/java/dispatchers.md index 1982bd687c..9cf11f280d 100644 --- a/akka-docs/src/main/paradox/java/dispatchers.md +++ b/akka-docs/src/main/paradox/java/dispatchers.md @@ -1,152 +1,138 @@ -.. _dispatchers-java: +# Dispatchers -Dispatchers -=========== +An Akka `MessageDispatcher` is what makes Akka Actors "tick", it is the engine of the machine so to speak. +All `MessageDispatcher` implementations are also an `ExecutionContext`, which means that they can be used +to execute arbitrary code, for instance @ref:[Futures](futures.md). -An Akka ``MessageDispatcher`` is what makes Akka Actors "tick", it is the engine of the machine so to speak. -All ``MessageDispatcher`` implementations are also an ``ExecutionContext``, which means that they can be used -to execute arbitrary code, for instance :ref:`futures-java`. +## Default dispatcher -Default dispatcher ------------------- - -Every ``ActorSystem`` will have a default dispatcher that will be used in case nothing else is configured for an ``Actor``. -The default dispatcher can be configured, and is by default a ``Dispatcher`` with the specified ``default-executor``. +Every `ActorSystem` will have a default dispatcher that will be used in case nothing else is configured for an `Actor`. +The default dispatcher can be configured, and is by default a `Dispatcher` with the specified `default-executor`. If an ActorSystem is created with an ExecutionContext passed in, this ExecutionContext will be used as the default executor for all dispatchers in this ActorSystem. If no ExecutionContext is given, it will fallback to the executor specified in -``akka.actor.default-dispatcher.default-executor.fallback``. By default this is a "fork-join-executor", which +`akka.actor.default-dispatcher.default-executor.fallback`. By default this is a "fork-join-executor", which gives excellent performance in most cases. -.. _dispatcher-lookup-java: + +## Looking up a Dispatcher -Looking up a Dispatcher ------------------------ +Dispatchers implement the `ExecutionContext` interface and can thus be used to run `Future` invocations etc. -Dispatchers implement the :class:`ExecutionContext` interface and can thus be used to run :class:`Future` invocations etc. +@@snip [DispatcherDocTest.java](code/jdocs/dispatcher/DispatcherDocTest.java) { #lookup } -.. includecode:: code/jdocs/dispatcher/DispatcherDocTest.java#lookup +## Setting the dispatcher for an Actor -Setting the dispatcher for an Actor ------------------------------------ - -So in case you want to give your ``Actor`` a different dispatcher than the default, you need to do two things, of which the first is +So in case you want to give your `Actor` a different dispatcher than the default, you need to do two things, of which the first is is to configure the dispatcher: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#my-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #my-dispatcher-config } -.. note:: - Note that the ``parallelism-max`` does not set the upper bound on the total number of threads - allocated by the ForkJoinPool. It is a setting specifically talking about the number of *hot* - threads the pool keep running in order to reduce the latency of handling a new incoming task. - You can read more about parallelism in the JDK's `ForkJoinPool documentation`_. +@@@ note + +Note that the `parallelism-max` does not set the upper bound on the total number of threads +allocated by the ForkJoinPool. It is a setting specifically talking about the number of *hot* +threads the pool keep running in order to reduce the latency of handling a new incoming task. +You can read more about parallelism in the JDK's [ForkJoinPool documentation](https://docs.oracle.com/javase/8/jdocs/api/java/util/concurrent/ForkJoinPool.html). + +@@@ Another example that uses the "thread-pool-executor": - -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#fixed-pool-size-dispatcher-config -.. note:: - The thread pool executor dispatcher is implemented using by a ``java.util.concurrent.ThreadPoolExecutor``. - You can read more about it in the JDK's `ThreadPoolExecutor documentation`_. +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #fixed-pool-size-dispatcher-config } -For more options, see the default-dispatcher section of the :ref:`configuration`. +@@@ note + +The thread pool executor dispatcher is implemented using by a `java.util.concurrent.ThreadPoolExecutor`. +You can read more about it in the JDK's [ThreadPoolExecutor documentation](https://docs.oracle.com/javase/8/jdocs/api/java/util/concurrent/ThreadPoolExecutor.html). + +@@@ + +For more options, see the default-dispatcher section of the configuration. Then you create the actor as usual and define the dispatcher in the deployment configuration. -.. includecode:: ../java/code/jdocs/dispatcher/DispatcherDocTest.java#defining-dispatcher-in-config +@@snip [DispatcherDocTest.java](../java/code/jdocs/dispatcher/DispatcherDocTest.java) { #defining-dispatcher-in-config } -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#dispatcher-deployment-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #dispatcher-deployment-config } An alternative to the deployment configuration is to define the dispatcher in code. -If you define the ``dispatcher`` in the deployment configuration then this value will be used instead +If you define the `dispatcher` in the deployment configuration then this value will be used instead of programmatically provided parameter. -.. includecode:: ../java/code/jdocs/dispatcher/DispatcherDocTest.java#defining-dispatcher-in-code +@@snip [DispatcherDocTest.java](../java/code/jdocs/dispatcher/DispatcherDocTest.java) { #defining-dispatcher-in-code } -.. note:: - The dispatcher you specify in ``withDispatcher`` and the ``dispatcher`` property in the deployment - configuration is in fact a path into your configuration. - So in this example it's a top-level section, but you could for instance put it as a sub-section, - where you'd use periods to denote sub-sections, like this: ``"foo.bar.my-dispatcher"`` +@@@ note -.. _ForkJoinPool documentation: https://docs.oracle.com/javase/8/jdocs/api/java/util/concurrent/ForkJoinPool.html -.. _ThreadPoolExecutor documentation: https://docs.oracle.com/javase/8/jdocs/api/java/util/concurrent/ThreadPoolExecutor.html +The dispatcher you specify in `withDispatcher` and the `dispatcher` property in the deployment +configuration is in fact a path into your configuration. +So in this example it's a top-level section, but you could for instance put it as a sub-section, +where you'd use periods to denote sub-sections, like this: `"foo.bar.my-dispatcher"` -Types of dispatchers --------------------- +@@@ + +## Types of dispatchers There are 3 different types of message dispatchers: -* Dispatcher + * Dispatcher + * This is an event-based dispatcher that binds a set of Actors to a thread pool. It is the default dispatcher +used if one is not specified. + * Sharability: Unlimited + * Mailboxes: Any, creates one per Actor + * Use cases: Default dispatcher, Bulkheading + * + Driven by: + `java.util.concurrent.ExecutorService` + : specify using "executor" using "fork-join-executor", +"thread-pool-executor" or the FQCN of +an `akka.dispatcher.ExecutorServiceConfigurator` + + * PinnedDispatcher + * This dispatcher dedicates a unique thread for each actor using it; i.e. each actor will have its own thread pool with only one thread in the pool. + * Sharability: None + * Mailboxes: Any, creates one per Actor + * Use cases: Bulkheading + * + Driven by: Any + `akka.dispatch.ThreadPoolExecutorConfigurator` + : by default a "thread-pool-executor" + + * CallingThreadDispatcher + * This dispatcher runs invocations on the current thread only. This dispatcher does not create any new threads, +but it can be used from different threads concurrently for the same actor. See @ref:[Java-CallingThreadDispatcher](testing.md#java-callingthreaddispatcher) +for details and restrictions. + * Sharability: Unlimited + * Mailboxes: Any, creates one per Actor per Thread (on demand) + * Use cases: Testing + * Driven by: The calling thread (duh) - - This is an event-based dispatcher that binds a set of Actors to a thread pool. It is the default dispatcher - used if one is not specified. - - - Sharability: Unlimited - - - Mailboxes: Any, creates one per Actor - - - Use cases: Default dispatcher, Bulkheading - - - Driven by: ``java.util.concurrent.ExecutorService`` - specify using "executor" using "fork-join-executor", - "thread-pool-executor" or the FQCN of - an ``akka.dispatcher.ExecutorServiceConfigurator`` - -* PinnedDispatcher - - - This dispatcher dedicates a unique thread for each actor using it; i.e. each actor will have its own thread pool with only one thread in the pool. - - - Sharability: None - - - Mailboxes: Any, creates one per Actor - - - Use cases: Bulkheading - - - Driven by: Any ``akka.dispatch.ThreadPoolExecutorConfigurator`` - by default a "thread-pool-executor" - -* CallingThreadDispatcher - - - This dispatcher runs invocations on the current thread only. This dispatcher does not create any new threads, - but it can be used from different threads concurrently for the same actor. See :ref:`Java-CallingThreadDispatcher` - for details and restrictions. - - - Sharability: Unlimited - - - Mailboxes: Any, creates one per Actor per Thread (on demand) - - - Use cases: Testing - - - Driven by: The calling thread (duh) - -More dispatcher configuration examples -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### More dispatcher configuration examples Configuring a dispatcher with fixed thread pool size, e.g. for actors that perform blocking IO: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#fixed-pool-size-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #fixed-pool-size-dispatcher-config } And then using it: -.. includecode:: ../java/code/jdocs/dispatcher/DispatcherDocTest.java#defining-fixed-pool-size-dispatcher +@@snip [DispatcherDocTest.java](../java/code/jdocs/dispatcher/DispatcherDocTest.java) { #defining-fixed-pool-size-dispatcher } Another example that uses the thread pool based on the number of cores (e.g. for CPU bound tasks) -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#my-thread-pool-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #my-thread-pool-dispatcher-config } -Configuring a ``PinnedDispatcher``: +Configuring a `PinnedDispatcher`: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#my-pinned-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #my-pinned-dispatcher-config } And then using it: -.. includecode:: ../java/code/jdocs/dispatcher/DispatcherDocTest.java#defining-pinned-dispatcher +@@snip [DispatcherDocTest.java](../java/code/jdocs/dispatcher/DispatcherDocTest.java) { #defining-pinned-dispatcher } -Note that ``thread-pool-executor`` configuration as per the above ``my-thread-pool-dispatcher`` example is -NOT applicable. This is because every actor will have its own thread pool when using ``PinnedDispatcher``, +Note that `thread-pool-executor` configuration as per the above `my-thread-pool-dispatcher` example is +NOT applicable. This is because every actor will have its own thread pool when using `PinnedDispatcher`, and that pool will have only one thread. Note that it's not guaranteed that the *same* thread is used over time, since the core pool timeout -is used for ``PinnedDispatcher`` to keep resource usage down in case of idle actors. To use the same -thread all the time you need to add ``thread-pool-executor.allow-core-timeout=off`` to the -configuration of the ``PinnedDispatcher``. +is used for `PinnedDispatcher` to keep resource usage down in case of idle actors. To use the same +thread all the time you need to add `thread-pool-executor.allow-core-timeout=off` to the +configuration of the `PinnedDispatcher`. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/distributed-data.md b/akka-docs/src/main/paradox/java/distributed-data.md index 154bf3b21c..e746a7f829 100644 --- a/akka-docs/src/main/paradox/java/distributed-data.md +++ b/akka-docs/src/main/paradox/java/distributed-data.md @@ -1,9 +1,4 @@ - -.. _distributed_data_java: - -################## - Distributed Data -################## +# Distributed Data *Akka Distributed Data* is useful when you need to share data between nodes in an Akka Cluster. The data is accessed with an actor providing a key-value store like API. @@ -24,479 +19,479 @@ It is eventually consistent and geared toward providing high read and write avai (partition tolerance), with low latency. Note that in an eventually consistent system a read may return an out-of-date value. -Using the Replicator -==================== +## Using the Replicator -The ``akka.cluster.ddata.Replicator`` actor provides the API for interacting with the data. -The ``Replicator`` actor must be started on each node in the cluster, or group of nodes tagged -with a specific role. It communicates with other ``Replicator`` instances with the same path +The `akka.cluster.ddata.Replicator` actor provides the API for interacting with the data. +The `Replicator` actor must be started on each node in the cluster, or group of nodes tagged +with a specific role. It communicates with other `Replicator` instances with the same path (without address) that are running on other nodes . For convenience it can be used with the -``akka.cluster.ddata.DistributedData`` extension but it can also be started as an ordinary -actor using the ``Replicator.props``. If it is started as an ordinary actor it is important +`akka.cluster.ddata.DistributedData` extension but it can also be started as an ordinary +actor using the `Replicator.props`. If it is started as an ordinary actor it is important that it is given the same name, started on same path, on all nodes. -Cluster members with status :ref:`WeaklyUp `, +Cluster members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-java), will participate in Distributed Data. This means that the data will be replicated to the -:ref:`WeaklyUp ` nodes with the background gossip protocol. Note that it +@ref:[WeaklyUp](cluster-usage.md#weakly-up-java) nodes with the background gossip protocol. Note that it will not participate in any actions where the consistency mode is to read/write from all -nodes or the majority of nodes. The :ref:`WeaklyUp ` node is not counted -as part of the cluster. So 3 nodes + 5 :ref:`WeaklyUp ` is essentially a +nodes or the majority of nodes. The @ref:[WeaklyUp](cluster-usage.md#weakly-up-java) node is not counted +as part of the cluster. So 3 nodes + 5 @ref:[WeaklyUp](cluster-usage.md#weakly-up-java) is essentially a 3 node cluster as far as consistent actions are concerned. Below is an example of an actor that schedules tick messages to itself and for each tick -adds or removes elements from a ``ORSet`` (observed-remove set). It also subscribes to +adds or removes elements from a `ORSet` (observed-remove set). It also subscribes to changes of this. -.. includecode:: code/jdocs/ddata/DataBot.java#data-bot +@@snip [DataBot.java](code/jdocs/ddata/DataBot.java) { #data-bot } -.. _replicator_update_java: + +### Update -Update ------- +To modify and replicate a data value you send a `Replicator.Update` message to the local +`Replicator`. -To modify and replicate a data value you send a ``Replicator.Update`` message to the local -``Replicator``. - -The current data value for the ``key`` of the ``Update`` is passed as parameter to the ``modify`` -function of the ``Update``. The function is supposed to return the new value of the data, which +The current data value for the `key` of the `Update` is passed as parameter to the `modify` +function of the `Update`. The function is supposed to return the new value of the data, which will then be replicated according to the given consistency level. -The ``modify`` function is called by the ``Replicator`` actor and must therefore be a pure +The `modify` function is called by the `Replicator` actor and must therefore be a pure function that only uses the data parameter and stable fields from enclosing scope. It must for example not access the sender reference of an enclosing actor. -``Update`` is intended to only be sent from an actor running in same local ``ActorSystem`` as - the ``Replicator``, because the ``modify`` function is typically not serializable. +`Update` + is intended to only be sent from an actor running in same local +`ActorSystem` + as +: the `Replicator`, because the `modify` function is typically not serializable. + You supply a write consistency level which has the following meaning: -* ``writeLocal`` the value will immediately only be written to the local replica, - and later disseminated with gossip -* ``WriteTo(n)`` the value will immediately be written to at least ``n`` replicas, - including the local replica -* ``WriteMajority`` the value will immediately be written to a majority of replicas, i.e. - at least **N/2 + 1** replicas, where N is the number of nodes in the cluster - (or cluster role group) -* ``WriteAll`` the value will immediately be written to all nodes in the cluster - (or all nodes in the cluster role group) + * `writeLocal` the value will immediately only be written to the local replica, +and later disseminated with gossip + * `WriteTo(n)` the value will immediately be written to at least `n` replicas, +including the local replica + * `WriteMajority` the value will immediately be written to a majority of replicas, i.e. +at least **N/2 + 1** replicas, where N is the number of nodes in the cluster +(or cluster role group) + * `WriteAll` the value will immediately be written to all nodes in the cluster +(or all nodes in the cluster role group) -When you specify to write to ``n`` out of ``x`` nodes, the update will first replicate to ``n`` nodes. If there are not - enough Acks after 1/5th of the timeout, the update will be replicated to ``n`` other nodes. If there are less than n nodes - left all of the remaining nodes are used. Reachable nodes are prefered over unreachable nodes. +When you specify to write to +`n` + out of +`x` + nodes, the update will first replicate to +`n` + nodes. If there are not +: enough Acks after 1/5th of the timeout, the update will be replicated to `n` other nodes. If there are less than n nodes +left all of the remaining nodes are used. Reachable nodes are prefered over unreachable nodes. -Note that ``WriteMajority`` has a ``minCap`` parameter that is useful to specify to achieve better safety for small clusters. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#update +Note that `WriteMajority` has a `minCap` parameter that is useful to specify to achieve better safety for small clusters. -As reply of the ``Update`` a ``Replicator.UpdateSuccess`` is sent to the sender of the -``Update`` if the value was successfully replicated according to the supplied consistency -level within the supplied timeout. Otherwise a ``Replicator.UpdateFailure`` subclass is -sent back. Note that a ``Replicator.UpdateTimeout`` reply does not mean that the update completely failed +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #update } + +As reply of the `Update` a `Replicator.UpdateSuccess` is sent to the sender of the +`Update` if the value was successfully replicated according to the supplied consistency +level within the supplied timeout. Otherwise a `Replicator.UpdateFailure` subclass is +sent back. Note that a `Replicator.UpdateTimeout` reply does not mean that the update completely failed or was rolled back. It may still have been replicated to some nodes, and will eventually be replicated to all nodes with the gossip protocol. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#update-response1 +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #update-response1 } -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#update-response2 +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #update-response2 } -You will always see your own writes. For example if you send two ``Update`` messages -changing the value of the same ``key``, the ``modify`` function of the second message will -see the change that was performed by the first ``Update`` message. +You will always see your own writes. For example if you send two `Update` messages +changing the value of the same `key`, the `modify` function of the second message will +see the change that was performed by the first `Update` message. -In the ``Update`` message you can pass an optional request context, which the ``Replicator`` +In the `Update` message you can pass an optional request context, which the `Replicator` does not care about, but is included in the reply messages. This is a convenient -way to pass contextual information (e.g. original sender) without having to use ``ask`` +way to pass contextual information (e.g. original sender) without having to use `ask` or maintain local correlation data structures. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#update-request-context +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #update-request-context } -.. _replicator_get_java: + +### Get -Get ---- +To retrieve the current value of a data you send `Replicator.Get` message to the +`Replicator`. You supply a consistency level which has the following meaning: -To retrieve the current value of a data you send ``Replicator.Get`` message to the -``Replicator``. You supply a consistency level which has the following meaning: + * `readLocal` the value will only be read from the local replica + * `ReadFrom(n)` the value will be read and merged from `n` replicas, +including the local replica + * `ReadMajority` the value will be read and merged from a majority of replicas, i.e. +at least **N/2 + 1** replicas, where N is the number of nodes in the cluster +(or cluster role group) + * `ReadAll` the value will be read and merged from all nodes in the cluster +(or all nodes in the cluster role group) -* ``readLocal`` the value will only be read from the local replica -* ``ReadFrom(n)`` the value will be read and merged from ``n`` replicas, - including the local replica -* ``ReadMajority`` the value will be read and merged from a majority of replicas, i.e. - at least **N/2 + 1** replicas, where N is the number of nodes in the cluster - (or cluster role group) -* ``ReadAll`` the value will be read and merged from all nodes in the cluster - (or all nodes in the cluster role group) +Note that `ReadMajority` has a `minCap` parameter that is useful to specify to achieve better safety for small clusters. -Note that ``ReadMajority`` has a ``minCap`` parameter that is useful to specify to achieve better safety for small clusters. +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #get } -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#get +As reply of the `Get` a `Replicator.GetSuccess` is sent to the sender of the +`Get` if the value was successfully retrieved according to the supplied consistency +level within the supplied timeout. Otherwise a `Replicator.GetFailure` is sent. +If the key does not exist the reply will be `Replicator.NotFound`. -As reply of the ``Get`` a ``Replicator.GetSuccess`` is sent to the sender of the -``Get`` if the value was successfully retrieved according to the supplied consistency -level within the supplied timeout. Otherwise a ``Replicator.GetFailure`` is sent. -If the key does not exist the reply will be ``Replicator.NotFound``. +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #get-response1 } -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#get-response1 +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #get-response2 } -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#get-response2 +You will always read your own writes. For example if you send a `Update` message +followed by a `Get` of the same `key` the `Get` will retrieve the change that was +performed by the preceding `Update` message. However, the order of the reply messages are +not defined, i.e. in the previous example you may receive the `GetSuccess` before +the `UpdateSuccess`. -You will always read your own writes. For example if you send a ``Update`` message -followed by a ``Get`` of the same ``key`` the ``Get`` will retrieve the change that was -performed by the preceding ``Update`` message. However, the order of the reply messages are -not defined, i.e. in the previous example you may receive the ``GetSuccess`` before -the ``UpdateSuccess``. +In the `Get` message you can pass an optional request context in the same way as for the +`Update` message, described above. For example the original sender can be passed and replied +to after receiving and transforming `GetSuccess`. -In the ``Get`` message you can pass an optional request context in the same way as for the -``Update`` message, described above. For example the original sender can be passed and replied -to after receiving and transforming ``GetSuccess``. +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #get-request-context } -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#get-request-context +### Consistency -Consistency ------------ - -The consistency level that is supplied in the :ref:`replicator_update_java` and :ref:`replicator_get_java` +The consistency level that is supplied in the [replicator_update_java](#replicator-update-java) and [replicator_get_java](#replicator-get-java) specifies per request how many replicas that must respond successfully to a write and read request. -For low latency reads you use ``ReadLocal`` with the risk of retrieving stale data, i.e. updates +For low latency reads you use `ReadLocal` with the risk of retrieving stale data, i.e. updates from other nodes might not be visible yet. -When using ``writeLocal`` the update is only written to the local replica and then disseminated +When using `writeLocal` the update is only written to the local replica and then disseminated in the background with the gossip protocol, which can take few seconds to spread to all nodes. -``WriteAll`` and ``ReadAll`` is the strongest consistency level, but also the slowest and with -lowest availability. For example, it is enough that one node is unavailable for a ``Get`` request +`WriteAll` and `ReadAll` is the strongest consistency level, but also the slowest and with +lowest availability. For example, it is enough that one node is unavailable for a `Get` request and you will not receive the value. If consistency is important, you can ensure that a read always reflects the most recent -write by using the following formula:: +write by using the following formula: - (nodes_written + nodes_read) > N +``` +(nodes_written + nodes_read) > N +``` where N is the total number of nodes in the cluster, or the number of nodes with the role that is -used for the ``Replicator``. +used for the `Replicator`. For example, in a 7 node cluster this these consistency properties are achieved by writing to 4 nodes and reading from 4 nodes, or writing to 5 nodes and reading from 3 nodes. -By combining ``WriteMajority`` and ``ReadMajority`` levels a read always reflects the most recent write. -The ``Replicator`` writes and reads to a majority of replicas, i.e. **N / 2 + 1**. For example, +By combining `WriteMajority` and `ReadMajority` levels a read always reflects the most recent write. +The `Replicator` writes and reads to a majority of replicas, i.e. **N / 2 + 1**. For example, in a 5 node cluster it writes to 3 nodes and reads from 3 nodes. In a 6 node cluster it writes to 4 nodes and reads from 4 nodes. For small clusters (<7) the risk of membership changes between a WriteMajority and ReadMajority is rather high and then the nice properties of combining majority write and reads are not -guaranteed. Therefore the ``ReadMajority`` and ``WriteMajority`` have a ``minCap`` parameter that +guaranteed. Therefore the `ReadMajority` and `WriteMajority` have a `minCap` parameter that is useful to specify to achieve better safety for small clusters. It means that if the cluster -size is smaller than the majority size it will use the ``minCap`` number of nodes but at most +size is smaller than the majority size it will use the `minCap` number of nodes but at most the total size of the cluster. -Here is an example of using ``writeMajority`` and ``readMajority``: +Here is an example of using `writeMajority` and `readMajority`: -.. includecode:: code/jdocs/ddata/ShoppingCart.java#read-write-majority +@@snip [ShoppingCart.java](code/jdocs/ddata/ShoppingCart.java) { #read-write-majority } -.. includecode:: code/jdocs/ddata/ShoppingCart.java#get-cart +@@snip [ShoppingCart.java](code/jdocs/ddata/ShoppingCart.java) { #get-cart } -.. includecode:: code/jdocs/ddata/ShoppingCart.java#add-item +@@snip [ShoppingCart.java](code/jdocs/ddata/ShoppingCart.java) { #add-item } -In some rare cases, when performing an ``Update`` it is needed to first try to fetch latest data from -other nodes. That can be done by first sending a ``Get`` with ``ReadMajority`` and then continue with -the ``Update`` when the ``GetSuccess``, ``GetFailure`` or ``NotFound`` reply is received. This might be -needed when you need to base a decision on latest information or when removing entries from ``ORSet`` -or ``ORMap``. If an entry is added to an ``ORSet`` or ``ORMap`` from one node and removed from another +In some rare cases, when performing an `Update` it is needed to first try to fetch latest data from +other nodes. That can be done by first sending a `Get` with `ReadMajority` and then continue with +the `Update` when the `GetSuccess`, `GetFailure` or `NotFound` reply is received. This might be +needed when you need to base a decision on latest information or when removing entries from `ORSet` +or `ORMap`. If an entry is added to an `ORSet` or `ORMap` from one node and removed from another node the entry will only be removed if the added entry is visible on the node where the removal is performed (hence the name observed-removed set). The following example illustrates how to do that: -.. includecode:: ../../../akka-docs/rst/java/code/jdocs/ddata/ShoppingCart.java#remove-item +@@snip [ShoppingCart.java]../../../../../akka-docs/rst/java/code/jdocs/ddata/ShoppingCart.java) { #remove-item } -.. warning:: +@@@ warning - *Caveat:* Even if you use ``writeMajority`` and ``readMajority`` there is small risk that you may - read stale data if the cluster membership has changed between the ``Update`` and the ``Get``. - For example, in cluster of 5 nodes when you ``Update`` and that change is written to 3 nodes: - n1, n2, n3. Then 2 more nodes are added and a ``Get`` request is reading from 4 nodes, which - happens to be n4, n5, n6, n7, i.e. the value on n1, n2, n3 is not seen in the response of the - ``Get`` request. +*Caveat:* Even if you use `writeMajority` and `readMajority` there is small risk that you may +read stale data if the cluster membership has changed between the `Update` and the `Get`. +For example, in cluster of 5 nodes when you `Update` and that change is written to 3 nodes: +n1, n2, n3. Then 2 more nodes are added and a `Get` request is reading from 4 nodes, which +happens to be n4, n5, n6, n7, i.e. the value on n1, n2, n3 is not seen in the response of the +`Get` request. -Subscribe ---------- +@@@ -You may also register interest in change notifications by sending ``Replicator.Subscribe`` -message to the ``Replicator``. It will send ``Replicator.Changed`` messages to the registered +### Subscribe + +You may also register interest in change notifications by sending `Replicator.Subscribe` +message to the `Replicator`. It will send `Replicator.Changed` messages to the registered subscriber when the data for the subscribed key is updated. Subscribers will be notified -periodically with the configured ``notify-subscribers-interval``, and it is also possible to -send an explicit ``Replicator.FlushChanges`` message to the ``Replicator`` to notify the subscribers +periodically with the configured `notify-subscribers-interval`, and it is also possible to +send an explicit `Replicator.FlushChanges` message to the `Replicator` to notify the subscribers immediately. The subscriber is automatically removed if the subscriber is terminated. A subscriber can -also be deregistered with the ``Replicator.Unsubscribe`` message. +also be deregistered with the `Replicator.Unsubscribe` message. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#subscribe +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #subscribe } -Delete ------- +### Delete -A data entry can be deleted by sending a ``Replicator.Delete`` message to the local -local ``Replicator``. As reply of the ``Delete`` a ``Replicator.DeleteSuccess`` is sent to -the sender of the ``Delete`` if the value was successfully deleted according to the supplied -consistency level within the supplied timeout. Otherwise a ``Replicator.ReplicationDeleteFailure`` -is sent. Note that ``ReplicationDeleteFailure`` does not mean that the delete completely failed or +A data entry can be deleted by sending a `Replicator.Delete` message to the local +local `Replicator`. As reply of the `Delete` a `Replicator.DeleteSuccess` is sent to +the sender of the `Delete` if the value was successfully deleted according to the supplied +consistency level within the supplied timeout. Otherwise a `Replicator.ReplicationDeleteFailure` +is sent. Note that `ReplicationDeleteFailure` does not mean that the delete completely failed or was rolled back. It may still have been replicated to some nodes, and may eventually be replicated to all nodes. A deleted key cannot be reused again, but it is still recommended to delete unused data entries because that reduces the replication overhead when new nodes join the cluster. -Subsequent ``Delete``, ``Update`` and ``Get`` requests will be replied with ``Replicator.DataDeleted``. -Subscribers will receive ``Replicator.DataDeleted``. +Subsequent `Delete`, `Update` and `Get` requests will be replied with `Replicator.DataDeleted`. +Subscribers will receive `Replicator.DataDeleted`. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#delete +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #delete } -.. warning:: +@@@ warning - As deleted keys continue to be included in the stored data on each node as well as in gossip - messages, a continuous series of updates and deletes of top-level entities will result in - growing memory usage until an ActorSystem runs out of memory. To use Akka Distributed Data - where frequent adds and removes are required, you should use a fixed number of top-level data - types that support both updates and removals, for example ``ORMap`` or ``ORSet``. +As deleted keys continue to be included in the stored data on each node as well as in gossip +messages, a continuous series of updates and deletes of top-level entities will result in +growing memory usage until an ActorSystem runs out of memory. To use Akka Distributed Data +where frequent adds and removes are required, you should use a fixed number of top-level data +types that support both updates and removals, for example `ORMap` or `ORSet`. -.. _delta_crdt_java: +@@@ -delta-CRDT ----------- + +### delta-CRDT -`Delta State Replicated Data Types `_ +[Delta State Replicated Data Types](http://arxiv.org/abs/1603.01529) are supported. delta-CRDT is a way to reduce the need for sending the full state -for updates. For example adding element ``'c'`` and ``'d'`` to set ``{'a', 'b'}`` would -result in sending the delta ``{'c', 'd'}`` and merge that with the state on the -receiving side, resulting in set ``{'a', 'b', 'c', 'd'}``. +for updates. For example adding element `'c'` and `'d'` to set `{'a', 'b'}` would +result in sending the delta `{'c', 'd'}` and merge that with the state on the +receiving side, resulting in set `{'a', 'b', 'c', 'd'}`. The protocol for replicating the deltas supports causal consistency if the data type -is marked with ``RequiresCausalDeliveryOfDeltas``. Otherwise it is only eventually -consistent. Without causal consistency it means that if elements ``'c'`` and ``'d'`` are -added in two separate `Update` operations these deltas may occasionally be propagated +is marked with `RequiresCausalDeliveryOfDeltas`. Otherwise it is only eventually +consistent. Without causal consistency it means that if elements `'c'` and `'d'` are +added in two separate *Update* operations these deltas may occasionally be propagated to nodes in different order than the causal order of the updates. For this example it -can result in that set ``{'a', 'b', 'd'}`` can be seen before element 'c' is seen. Eventually -it will be ``{'a', 'b', 'c', 'd'}``. +can result in that set `{'a', 'b', 'd'}` can be seen before element 'c' is seen. Eventually +it will be `{'a', 'b', 'c', 'd'}`. Note that the full state is occasionally also replicated for delta-CRDTs, for example when new nodes are added to the cluster or when deltas could not be propagated because of network partitions or similar problems. -The the delta propagation can be disabled with configuration property:: +The the delta propagation can be disabled with configuration property: - akka.cluster.distributed-data.delta-crdt.enabled=off +``` +akka.cluster.distributed-data.delta-crdt.enabled=off +``` -Data Types -========== +## Data Types -The data types must be convergent (stateful) CRDTs and implement the ``ReplicatedData`` trait, +The data types must be convergent (stateful) CRDTs and implement the `ReplicatedData` trait, i.e. they provide a monotonic merge function and the state changes always converge. -You can use your own custom ``AbstractReplicatedData`` or ``AbstractDeltaReplicatedData`` types, +You can use your own custom `AbstractReplicatedData` or `AbstractDeltaReplicatedData` types, and several types are provided by this package, such as: -* Counters: ``GCounter``, ``PNCounter`` -* Sets: ``GSet``, ``ORSet`` -* Maps: ``ORMap``, ``ORMultiMap``, ``LWWMap``, ``PNCounterMap`` -* Registers: ``LWWRegister``, ``Flag`` + * Counters: `GCounter`, `PNCounter` + * Sets: `GSet`, `ORSet` + * Maps: `ORMap`, `ORMultiMap`, `LWWMap`, `PNCounterMap` + * Registers: `LWWRegister`, `Flag` -Counters --------- +### Counters -``GCounter`` is a "grow only counter". It only supports increments, no decrements. +`GCounter` is a "grow only counter". It only supports increments, no decrements. It works in a similar way as a vector clock. It keeps track of one counter per node and the total -value is the sum of these counters. The ``merge`` is implemented by taking the maximum count for +value is the sum of these counters. The `merge` is implemented by taking the maximum count for each node. -If you need both increments and decrements you can use the ``PNCounter`` (positive/negative counter). +If you need both increments and decrements you can use the `PNCounter` (positive/negative counter). It is tracking the increments (P) separate from the decrements (N). Both P and N are represented -as two internal ``GCounter``. Merge is handled by merging the internal P and N counters. +as two internal `GCounter`. Merge is handled by merging the internal P and N counters. The value of the counter is the value of the P counter minus the value of the N counter. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#pncounter +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #pncounter } -``GCounter`` and ``PNCounter`` have support for :ref:`delta_crdt_java` and don't need causal +`GCounter` and `PNCounter` have support for [delta_crdt_java](#delta-crdt-java) and don't need causal delivery of deltas. -Several related counters can be managed in a map with the ``PNCounterMap`` data type. -When the counters are placed in a ``PNCounterMap`` as opposed to placing them as separate top level +Several related counters can be managed in a map with the `PNCounterMap` data type. +When the counters are placed in a `PNCounterMap` as opposed to placing them as separate top level values they are guaranteed to be replicated together as one unit, which is sometimes necessary for related data. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#pncountermap +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #pncountermap } -Sets ----- +### Sets -If you only need to add elements to a set and not remove elements the ``GSet`` (grow-only set) is +If you only need to add elements to a set and not remove elements the `GSet` (grow-only set) is the data type to use. The elements can be any type of values that can be serialized. Merge is simply the union of the two sets. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#gset +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #gset } -``GSet`` has support for :ref:`delta_crdt_java` and it doesn't require causal delivery of deltas. +`GSet` has support for [delta_crdt_java](#delta-crdt-java) and it doesn't require causal delivery of deltas. -If you need add and remove operations you should use the ``ORSet`` (observed-remove set). +If you need add and remove operations you should use the `ORSet` (observed-remove set). Elements can be added and removed any number of times. If an element is concurrently added and removed, the add will win. You cannot remove an element that you have not seen. -The ``ORSet`` has a version vector that is incremented when an element is added to the set. +The `ORSet` has a version vector that is incremented when an element is added to the set. The version for the node that added the element is also tracked for each element in a so -called "birth dot". The version vector and the dots are used by the ``merge`` function to +called "birth dot". The version vector and the dots are used by the `merge` function to track causality of the operations and resolve concurrent updates. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#orset +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #orset } -``ORSet`` has support for :ref:`delta_crdt_java` and it requires causal delivery of deltas. +`ORSet` has support for [delta_crdt_java](#delta-crdt-java) and it requires causal delivery of deltas. -Maps ----- +### Maps -``ORMap`` (observed-remove map) is a map with keys of ``Any`` type and the values are ``ReplicatedData`` +`ORMap` (observed-remove map) is a map with keys of `Any` type and the values are `ReplicatedData` types themselves. It supports add, remove and delete any number of times for a map entry. If an entry is concurrently added and removed, the add will win. You cannot remove an entry that -you have not seen. This is the same semantics as for the ``ORSet``. +you have not seen. This is the same semantics as for the `ORSet`. If an entry is concurrently updated to different values the values will be merged, hence the -requirement that the values must be ``ReplicatedData`` types. +requirement that the values must be `ReplicatedData` types. -It is rather inconvenient to use the ``ORMap`` directly since it does not expose specific types -of the values. The ``ORMap`` is intended as a low level tool for building more specific maps, +It is rather inconvenient to use the `ORMap` directly since it does not expose specific types +of the values. The `ORMap` is intended as a low level tool for building more specific maps, such as the following specialized maps. -``ORMultiMap`` (observed-remove multi-map) is a multi-map implementation that wraps an -``ORMap`` with an ``ORSet`` for the map's value. +`ORMultiMap` (observed-remove multi-map) is a multi-map implementation that wraps an +`ORMap` with an `ORSet` for the map's value. -``PNCounterMap`` (positive negative counter map) is a map of named counters. It is a specialized -``ORMap`` with ``PNCounter`` values. +`PNCounterMap` (positive negative counter map) is a map of named counters. It is a specialized +`ORMap` with `PNCounter` values. -``LWWMap`` (last writer wins map) is a specialized ``ORMap`` with ``LWWRegister`` (last writer wins register) +`LWWMap` (last writer wins map) is a specialized `ORMap` with `LWWRegister` (last writer wins register) values. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#ormultimap +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #ormultimap } When a data entry is changed the full state of that entry is replicated to other nodes, i.e. -when you update a map the whole map is replicated. Therefore, instead of using one ``ORMap`` -with 1000 elements it is more efficient to split that up in 10 top level ``ORMap`` entries +when you update a map the whole map is replicated. Therefore, instead of using one `ORMap` +with 1000 elements it is more efficient to split that up in 10 top level `ORMap` entries with 100 elements each. Top level entries are replicated individually, which has the trade-off that different entries may not be replicated at the same time and you may see inconsistencies between related entries. Separate top level entries cannot be updated atomically together. -Note that ``LWWRegister`` and therefore ``LWWMap`` relies on synchronized clocks and should only be used +Note that `LWWRegister` and therefore `LWWMap` relies on synchronized clocks and should only be used when the choice of value is not important for concurrent updates occurring within the clock skew. Read more -in the below section about ``LWWRegister``. +in the below section about `LWWRegister`. -Flags and Registers -------------------- +### Flags and Registers -``Flag`` is a data type for a boolean value that is initialized to ``false`` and can be switched -to ``true``. Thereafter it cannot be changed. ``true`` wins over ``false`` in merge. +`Flag` is a data type for a boolean value that is initialized to `false` and can be switched +to `true`. Thereafter it cannot be changed. `true` wins over `false` in merge. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#flag +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #flag } -``LWWRegister`` (last writer wins register) can hold any (serializable) value. +`LWWRegister` (last writer wins register) can hold any (serializable) value. -Merge of a ``LWWRegister`` takes the register with highest timestamp. Note that this -relies on synchronized clocks. `LWWRegister` should only be used when the choice of +Merge of a `LWWRegister` takes the register with highest timestamp. Note that this +relies on synchronized clocks. *LWWRegister* should only be used when the choice of value is not important for concurrent updates occurring within the clock skew. -Merge takes the register updated by the node with lowest address (``UniqueAddress`` is ordered) +Merge takes the register updated by the node with lowest address (`UniqueAddress` is ordered) if the timestamps are exactly the same. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#lwwregister +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #lwwregister } -Instead of using timestamps based on ``System.currentTimeMillis()`` time it is possible to +Instead of using timestamps based on `System.currentTimeMillis()` time it is possible to use a timestamp value based on something else, for example an increasing version number from a database record that is used for optimistic concurrency control. -.. includecode:: code/jdocs/ddata/DistributedDataDocTest.java#lwwregister-custom-clock +@@snip [DistributedDataDocTest.java](code/jdocs/ddata/DistributedDataDocTest.java) { #lwwregister-custom-clock } -For first-write-wins semantics you can use the ``LWWRegister#reverseClock`` instead of the -``LWWRegister#defaultClock``. +For first-write-wins semantics you can use the `LWWRegister#reverseClock` instead of the +`LWWRegister#defaultClock`. -The ``defaultClock`` is using max value of ``System.currentTimeMillis()`` and ``currentTimestamp + 1``. +The `defaultClock` is using max value of `System.currentTimeMillis()` and `currentTimestamp + 1`. This means that the timestamp is increased for changes on the same node that occurs within -the same millisecond. It also means that it is safe to use the ``LWWRegister`` without +the same millisecond. It also means that it is safe to use the `LWWRegister` without synchronized clocks when there is only one active writer, e.g. a Cluster Singleton. Such a -single writer should then first read current value with ``ReadMajority`` (or more) before -changing and writing the value with ``WriteMajority`` (or more). +single writer should then first read current value with `ReadMajority` (or more) before +changing and writing the value with `WriteMajority` (or more). -Custom Data Type ----------------- +### Custom Data Type You can rather easily implement your own data types. The only requirement is that it implements -the ``mergeData`` function of the ``AbstractReplicatedData`` class. +the `mergeData` function of the `AbstractReplicatedData` class. A nice property of stateful CRDTs is that they typically compose nicely, i.e. you can combine several -smaller data types to build richer data structures. For example, the ``PNCounter`` is composed of -two internal ``GCounter`` instances to keep track of increments and decrements separately. +smaller data types to build richer data structures. For example, the `PNCounter` is composed of +two internal `GCounter` instances to keep track of increments and decrements separately. -Here is s simple implementation of a custom ``TwoPhaseSet`` that is using two internal ``GSet`` types -to keep track of addition and removals. A ``TwoPhaseSet`` is a set where an element may be added and +Here is s simple implementation of a custom `TwoPhaseSet` that is using two internal `GSet` types +to keep track of addition and removals. A `TwoPhaseSet` is a set where an element may be added and removed, but never added again thereafter. -.. includecode:: code/jdocs/ddata/TwoPhaseSet.java#twophaseset +@@snip [TwoPhaseSet.java](code/jdocs/ddata/TwoPhaseSet.java) { #twophaseset } Data types should be immutable, i.e. "modifying" methods should return a new instance. -Implement the additional methods of ``AbstractDeltaReplicatedData`` if it has support for delta-CRDT replication. +Implement the additional methods of `AbstractDeltaReplicatedData` if it has support for delta-CRDT replication. -Serialization -^^^^^^^^^^^^^ +#### Serialization -The data types must be serializable with an :ref:`Akka Serializer `. +The data types must be serializable with an @ref:[Akka Serializer](serialization.md). It is highly recommended that you implement efficient serialization with Protobuf or similar -for your custom data types. The built in data types are marked with ``ReplicatedDataSerialization`` -and serialized with ``akka.cluster.ddata.protobuf.ReplicatedDataSerializer``. +for your custom data types. The built in data types are marked with `ReplicatedDataSerialization` +and serialized with `akka.cluster.ddata.protobuf.ReplicatedDataSerializer`. Serialization of the data types are used in remote messages and also for creating message digests (SHA-1) to detect changes. Therefore it is important that the serialization is efficient and produce the same bytes for the same content. For example sets and maps should be sorted deterministically in the serialization. -This is a protobuf representation of the above ``TwoPhaseSet``: +This is a protobuf representation of the above `TwoPhaseSet`: -.. includecode:: ../../src/main/protobuf/TwoPhaseSetMessages.proto#twophaseset +@@snip [TwoPhaseSetMessages.proto]../../protobuf/TwoPhaseSetMessages.proto) { #twophaseset } -The serializer for the ``TwoPhaseSet``: +The serializer for the `TwoPhaseSet`: -.. includecode:: code/jdocs/ddata/protobuf/TwoPhaseSetSerializer.java#serializer +@@snip [TwoPhaseSetSerializer.java](code/jdocs/ddata/protobuf/TwoPhaseSetSerializer.java) { #serializer } Note that the elements of the sets are sorted so the SHA-1 digests are the same for the same elements. You register the serializer in configuration: -.. includecode:: ../scala/code/docs/ddata/DistributedDataDocSpec.scala#japi-serializer-config +@@snip [DistributedDataDocSpec.scala](../scala/code/docs/ddata/DistributedDataDocSpec.scala) { #japi-serializer-config } Using compression can sometimes be a good idea to reduce the data size. Gzip compression is -provided by the ``akka.cluster.ddata.protobuf.SerializationSupport`` trait: +provided by the `akka.cluster.ddata.protobuf.SerializationSupport` trait: -.. includecode:: code/jdocs/ddata/protobuf/TwoPhaseSetSerializerWithCompression.java#compression +@@snip [TwoPhaseSetSerializerWithCompression.java](code/jdocs/ddata/protobuf/TwoPhaseSetSerializerWithCompression.java) { #compression } -The two embedded ``GSet`` can be serialized as illustrated above, but in general when composing +The two embedded `GSet` can be serialized as illustrated above, but in general when composing new data types from the existing built in types it is better to make use of the existing serializer for those types. This can be done by declaring those as bytes fields in protobuf: -.. includecode:: ../../src/main/protobuf/TwoPhaseSetMessages.proto#twophaseset2 +@@snip [TwoPhaseSetMessages.proto]../../protobuf/TwoPhaseSetMessages.proto) { #twophaseset2 } -and use the methods ``otherMessageToProto`` and ``otherMessageFromBinary`` that are provided -by the ``SerializationSupport`` trait to serialize and deserialize the ``GSet`` instances. This +and use the methods `otherMessageToProto` and `otherMessageFromBinary` that are provided +by the `SerializationSupport` trait to serialize and deserialize the `GSet` instances. This works with any type that has a registered Akka serializer. This is how such an serializer would -look like for the ``TwoPhaseSet``: +look like for the `TwoPhaseSet`: -.. includecode:: code/jdocs/ddata/protobuf/TwoPhaseSetSerializer2.java#serializer +@@snip [TwoPhaseSetSerializer2.java](code/jdocs/ddata/protobuf/TwoPhaseSetSerializer2.java) { #serializer } -.. _ddata_durable_java: - -Durable Storage ---------------- + +### Durable Storage By default the data is only kept in memory. It is redundant since it is replicated to other nodes in the cluster, but if you stop all nodes the data is lost, unless you have saved it @@ -505,30 +500,36 @@ elsewhere. Entries can be configured to be durable, i.e. stored on local disk on each node. The stored data will be loaded next time the replicator is started, i.e. when actor system is restarted. This means data will survive as long as at least one node from the old cluster takes part in a new cluster. The keys of the durable entries -are configured with:: +are configured with: - akka.cluster.distributed-data.durable.keys = ["a", "b", "durable*"] +``` +akka.cluster.distributed-data.durable.keys = ["a", "b", "durable*"] +``` -Prefix matching is supported by using ``*`` at the end of a key. +Prefix matching is supported by using `*` at the end of a key. -All entries can be made durable by specifying:: +All entries can be made durable by specifying: - akka.cluster.distributed-data.durable.keys = ["*"] +``` +akka.cluster.distributed-data.durable.keys = ["*"] +``` -`LMDB `_ is the default storage implementation. It is +[LMDB](https://github.com/lmdbjava/lmdbjava/) is the default storage implementation. It is possible to replace that with another implementation by implementing the actor protocol described in -``akka.cluster.ddata.DurableStore`` and defining the ``akka.cluster.distributed-data.durable.store-actor-class`` +`akka.cluster.ddata.DurableStore` and defining the `akka.cluster.distributed-data.durable.store-actor-class` property for the new implementation. -The location of the files for the data is configured with:: +The location of the files for the data is configured with: - # Directory of LMDB file. There are two options: - # 1. A relative or absolute path to a directory that ends with 'ddata' - # the full name of the directory will contain name of the ActorSystem - # and its remote port. - # 2. Otherwise the path is used as is, as a relative or absolute path to - # a directory. - akka.cluster.distributed-data.lmdb.dir = "ddata" +``` +# Directory of LMDB file. There are two options: +# 1. A relative or absolute path to a directory that ends with 'ddata' +# the full name of the directory will contain name of the ActorSystem +# and its remote port. +# 2. Otherwise the path is used as is, as a relative or absolute path to +# a directory. +akka.cluster.distributed-data.lmdb.dir = "ddata" +``` When running in production you may want to configure the directory to a specific path (alt 2), since the default directory contains the remote port of the @@ -537,58 +538,54 @@ port (0) it will be different each time and the previously stored data will not be loaded. Making the data durable has of course a performance cost. By default, each update is flushed -to disk before the ``UpdateSuccess`` reply is sent. For better performance, but with the risk of losing +to disk before the `UpdateSuccess` reply is sent. For better performance, but with the risk of losing the last writes if the JVM crashes, you can enable write behind mode. Changes are then accumulated during a time period before it is written to LMDB and flushed to disk. Enabling write behind is especially efficient when performing many writes to the same key, because it is only the last value for each key that will be serialized and stored. The risk of losing writes if the JVM crashes is small since the -data is typically replicated to other nodes immediately according to the given ``WriteConsistency``. +data is typically replicated to other nodes immediately according to the given `WriteConsistency`. -:: +``` +akka.cluster.distributed-data.lmdb.write-behind-interval = 200 ms +``` - akka.cluster.distributed-data.lmdb.write-behind-interval = 200 ms +Note that you should be prepared to receive `WriteFailure` as reply to an `Update` of a +durable entry if the data could not be stored for some reason. When enabling `write-behind-interval` +such errors will only be logged and `UpdateSuccess` will still be the reply to the `Update`. -Note that you should be prepared to receive ``WriteFailure`` as reply to an ``Update`` of a -durable entry if the data could not be stored for some reason. When enabling ``write-behind-interval`` -such errors will only be logged and ``UpdateSuccess`` will still be the reply to the ``Update``. - -There is one important caveat when it comes pruning of :ref:`crdt_garbage_java` for durable data. +There is one important caveat when it comes pruning of [crdt_garbage_java](#crdt-garbage-java) for durable data. If and old data entry that was never pruned is injected and merged with existing data after that the pruning markers have been removed the value will not be correct. The time-to-live of the markers is defined by configuration -``akka.cluster.distributed-data.durable.remove-pruning-marker-after`` and is in the magnitude of days. +`akka.cluster.distributed-data.durable.remove-pruning-marker-after` and is in the magnitude of days. This would be possible if a node with durable data didn't participate in the pruning (e.g. it was shutdown) and later started after this time. A node with durable data should not be stopped for longer time than this duration and if it is joining again after this duration its data should first be manually removed (from the lmdb directory). -.. _crdt_garbage_java: - -CRDT Garbage ------------- + +### CRDT Garbage One thing that can be problematic with CRDTs is that some data types accumulate history (garbage). -For example a ``GCounter`` keeps track of one counter per node. If a ``GCounter`` has been updated +For example a `GCounter` keeps track of one counter per node. If a `GCounter` has been updated from one node it will associate the identifier of that node forever. That can become a problem for long running systems with many cluster nodes being added and removed. To solve this problem -the ``Replicator`` performs pruning of data associated with nodes that have been removed from the -cluster. Data types that need pruning have to implement the ``RemovedNodePruning`` trait. See the -API documentation of the ``Replicator`` for details. +the `Replicator` performs pruning of data associated with nodes that have been removed from the +cluster. Data types that need pruning have to implement the `RemovedNodePruning` trait. See the +API documentation of the `Replicator` for details. -Samples -======= +## Samples Several interesting samples are included and described in the -tutorial named `Akka Distributed Data Samples with Java <@exampleCodeService@/akka-samples-distributed-data-java>`_ (`source code <@samples@/akka-sample-distributed-data-java>`_) +tutorial named [Akka Distributed Data Samples with Java](@exampleCodeService@/akka-samples-distributed-data-java) ([source code](@samples@/akka-sample-distributed-data-java)) -* Low Latency Voting Service -* Highly Available Shopping Cart -* Distributed Service Registry -* Replicated Cache -* Replicated Metrics + * Low Latency Voting Service + * Highly Available Shopping Cart + * Distributed Service Registry + * Replicated Cache + * Replicated Metrics -Limitations -=========== +## Limitations There are some limitations that you should be aware of. @@ -605,43 +602,44 @@ be able to improve this if needed, but the design is still not intended for bill All data is held in memory, which is another reason why it is not intended for *Big Data*. When a data entry is changed the full state of that entry may be replicated to other nodes -if it doesn't support :ref:`delta_crdt_java`. The full state is also replicated for delta-CRDTs, +if it doesn't support [delta_crdt_java](#delta-crdt-java). The full state is also replicated for delta-CRDTs, for example when new nodes are added to the cluster or when deltas could not be propagated because of network partitions or similar problems. This means that you cannot have too large data entries, because then the remote message size will be too large. -Learn More about CRDTs -====================== +## Learn More about CRDTs -* `The Final Causal Frontier `_ - talk by Sean Cribbs -* `Eventually Consistent Data Structures `_ - talk by Sean Cribbs -* `Strong Eventual Consistency and Conflict-free Replicated Data Types `_ - talk by Mark Shapiro -* `A comprehensive study of Convergent and Commutative Replicated Data Types `_ - paper by Mark Shapiro et. al. + * [The Final Causal Frontier](http://www.ustream.tv/recorded/61448875) +talk by Sean Cribbs + * [Eventually Consistent Data Structures](https://vimeo.com/43903960) +talk by Sean Cribbs + * [Strong Eventual Consistency and Conflict-free Replicated Data Types](http://research.microsoft.com/apps/video/default.aspx?id=153540&r=1) +talk by Mark Shapiro + * [A comprehensive study of Convergent and Commutative Replicated Data Types](http://hal.upmc.fr/file/index/docid/555588/filename/techreport.pdf) +paper by Mark Shapiro et. al. -Dependencies -============ +## Dependencies To use Distributed Data you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-distributed-data" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-distributed-data" % "@version@" @crossString@ +``` -maven:: +maven: - - com.typesafe.akka - akka-distributed-data_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-distributed-data_@binVersion@ + @version@ + +``` -Configuration -============= +## Configuration -The ``DistributedData`` extension can be configured with the following properties: +The `DistributedData` extension can be configured with the following properties: -.. includecode:: ../../../akka-distributed-data/src/main/resources/reference.conf#distributed-data +@@snip [reference.conf]../../../../../akka-distributed-data/src/main/resources/reference.conf) { #distributed-data } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/distributed-pub-sub.md b/akka-docs/src/main/paradox/java/distributed-pub-sub.md index f3ad71d5ba..0ceed24811 100644 --- a/akka-docs/src/main/paradox/java/distributed-pub-sub.md +++ b/akka-docs/src/main/paradox/java/distributed-pub-sub.md @@ -1,20 +1,17 @@ -.. _distributed-pub-sub-java: - -Distributed Publish Subscribe in Cluster -======================================== +# Distributed Publish Subscribe in Cluster How do I send a message to an actor without knowing which node it is running on? How do I send messages to all actors in the cluster that have registered interest in a named topic? -This pattern provides a mediator actor, ``akka.cluster.pubsub.DistributedPubSubMediator``, +This pattern provides a mediator actor, `akka.cluster.pubsub.DistributedPubSubMediator`, that manages a registry of actor references and replicates the entries to peer actors among all cluster nodes or a group of nodes tagged with a specific role. -The ``DistributedPubSubMediator`` actor is supposed to be started on all nodes, +The `DistributedPubSubMediator` actor is supposed to be started on all nodes, or all nodes with specified role, in the cluster. The mediator can be -started with the ``DistributedPubSub`` extension or as an ordinary actor. +started with the `DistributedPubSub` extension or as an ordinary actor. The registry is eventually consistent, i.e. changes are not immediately visible at other nodes, but typically they will be fully replicated to all other nodes after @@ -22,21 +19,19 @@ a few seconds. Changes are only performed in the own part of the registry and th changes are versioned. Deltas are disseminated in a scalable way to other nodes with a gossip protocol. -Cluster members with status :ref:`WeaklyUp `, +Cluster members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-java), will participate in Distributed Publish Subscribe, i.e. subscribers on nodes with -``WeaklyUp`` status will receive published messages if the publisher and subscriber are on +`WeaklyUp` status will receive published messages if the publisher and subscriber are on same side of a network partition. You can send messages via the mediator on any node to registered actors on any other node. There a two different modes of message delivery, explained in the sections -:ref:`distributed-pub-sub-publish-java` and :ref:`distributed-pub-sub-send-java` below. +[Publish](#distributed-pub-sub-publish-java) and [Send](#distributed-pub-sub-send-java) below. -.. _distributed-pub-sub-publish-java: - -Publish -------- + +## Publish This is the true pub/sub mode. A typical usage of this mode is a chat room in an instant messaging application. @@ -47,61 +42,60 @@ The message will be delivered to all subscribers of the topic. For efficiency the message is sent over the wire only once per node (that has a matching topic), and then delivered to all subscribers of the local topic representation. -You register actors to the local mediator with ``DistributedPubSubMediator.Subscribe``. -Successful ``Subscribe`` and ``Unsubscribe`` is acknowledged with -``DistributedPubSubMediator.SubscribeAck`` and ``DistributedPubSubMediator.UnsubscribeAck`` +You register actors to the local mediator with `DistributedPubSubMediator.Subscribe`. +Successful `Subscribe` and `Unsubscribe` is acknowledged with +`DistributedPubSubMediator.SubscribeAck` and `DistributedPubSubMediator.UnsubscribeAck` replies. The acknowledgment means that the subscription is registered, but it can still take some time until it is replicated to other nodes. -You publish messages by sending ``DistributedPubSubMediator.Publish`` message to the +You publish messages by sending `DistributedPubSubMediator.Publish` message to the local mediator. Actors are automatically removed from the registry when they are terminated, or you -can explicitly remove entries with ``DistributedPubSubMediator.Unsubscribe``. +can explicitly remove entries with `DistributedPubSubMediator.Unsubscribe`. An example of a subscriber actor: -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java#subscriber +@@snip [DistributedPubSubMediatorTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java) { #subscriber } Subscriber actors can be started on several nodes in the cluster, and all will receive messages published to the "content" topic. -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java#start-subscribers +@@snip [DistributedPubSubMediatorTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java) { #start-subscribers } A simple actor that publishes to this "content" topic: -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java#publisher +@@snip [DistributedPubSubMediatorTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java) { #publisher } It can publish messages to the topic from anywhere in the cluster: -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java#publish-message +@@snip [DistributedPubSubMediatorTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java) { #publish-message } -Topic Groups -^^^^^^^^^^^^ +### Topic Groups -Actors may also be subscribed to a named topic with a ``group`` id. +Actors may also be subscribed to a named topic with a `group` id. If subscribing with a group id, each message published to a topic with the -``sendOneMessageToEachGroup`` flag set to ``true`` is delivered via the supplied ``RoutingLogic`` +`sendOneMessageToEachGroup` flag set to `true` is delivered via the supplied `RoutingLogic` (default random) to one actor within each subscribing group. If all the subscribed actors have the same group id, then this works just like -``Send`` and each message is only delivered to one subscriber. +`Send` and each message is only delivered to one subscriber. If all the subscribed actors have different group names, then this works like -normal ``Publish`` and each message is broadcasted to all subscribers. +normal `Publish` and each message is broadcasted to all subscribers. -.. note:: +@@@ note - Note that if the group id is used it is part of the topic identifier. - Messages published with ``sendOneMessageToEachGroup=false`` will not be delivered - to subscribers that subscribed with a group id. - Messages published with ``sendOneMessageToEachGroup=true`` will not be delivered - to subscribers that subscribed without a group id. +Note that if the group id is used it is part of the topic identifier. +Messages published with `sendOneMessageToEachGroup=false` will not be delivered +to subscribers that subscribed with a group id. +Messages published with `sendOneMessageToEachGroup=true` will not be delivered +to subscribers that subscribed without a group id. -.. _distributed-pub-sub-send-java: +@@@ -Send ----- + +## Send This is a point-to-point mode where each message is delivered to one destination, but you still do not have to know where the destination is located. @@ -111,43 +105,43 @@ cluster aware router where the routees dynamically can register themselves. The message will be delivered to one recipient with a matching path, if any such exists in the registry. If several entries match the path because it has been registered -on several nodes the message will be sent via the supplied ``RoutingLogic`` (default random) +on several nodes the message will be sent via the supplied `RoutingLogic` (default random) to one destination. The sender of the message can specify that local affinity is preferred, i.e. the message is sent to an actor in the same local actor system as the used mediator actor, if any such exists, otherwise route to any other matching entry. -You register actors to the local mediator with ``DistributedPubSubMediator.Put``. -The ``ActorRef`` in ``Put`` must belong to the same local actor system as the mediator. +You register actors to the local mediator with `DistributedPubSubMediator.Put`. +The `ActorRef` in `Put` must belong to the same local actor system as the mediator. The path without address information is the key to which you send messages. On each node there can only be one actor for a given path, since the path is unique within one local actor system. -You send messages by sending ``DistributedPubSubMediator.Send`` message to the +You send messages by sending `DistributedPubSubMediator.Send` message to the local mediator with the path (without address information) of the destination actors. Actors are automatically removed from the registry when they are terminated, or you -can explicitly remove entries with ``DistributedPubSubMediator.Remove``. +can explicitly remove entries with `DistributedPubSubMediator.Remove`. An example of a destination actor: -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java#send-destination +@@snip [DistributedPubSubMediatorTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java) { #send-destination } Subscriber actors can be started on several nodes in the cluster, and all will receive messages published to the "content" topic. -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java#start-send-destinations +@@snip [DistributedPubSubMediatorTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java) { #start-send-destinations } A simple actor that publishes to this "content" topic: -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java#sender +@@snip [DistributedPubSubMediatorTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java) { #sender } It can publish messages to the topic from anywhere in the cluster: -.. includecode:: ../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java#send-message +@@snip [DistributedPubSubMediatorTest.java]../../../../../akka-cluster-tools/src/test/java/akka/cluster/pubsub/DistributedPubSubMediatorTest.java) { #send-message } It is also possible to broadcast messages to the actors that have been registered with -``Put``. Send ``DistributedPubSubMediator.SendToAll`` message to the local mediator and the wrapped message +`Put`. Send `DistributedPubSubMediator.SendToAll` message to the local mediator and the wrapped message will then be delivered to all recipients with a matching path. Actors with the same path, without address information, can be registered on different nodes. On each node there can only be one such actor, since the path is unique within one @@ -155,52 +149,52 @@ local actor system. Typical usage of this mode is to broadcast messages to all replicas with the same path, e.g. 3 actors on different nodes that all perform the same actions, -for redundancy. You can also optionally specify a property (``allButSelf``) deciding +for redundancy. You can also optionally specify a property (`allButSelf`) deciding if the message should be sent to a matching path on the self node or not. -DistributedPubSub Extension ---------------------------- +## DistributedPubSub Extension -In the example above the mediator is started and accessed with the ``akka.cluster.pubsub.DistributedPubSub`` extension. +In the example above the mediator is started and accessed with the `akka.cluster.pubsub.DistributedPubSub` extension. That is convenient and perfectly fine in most cases, but it can be good to know that it is possible to start the mediator actor as an ordinary actor and you can have several different mediators at the same time to be able to divide a large number of actors/topics to different mediators. For example you might want to use different cluster roles for different mediators. -The ``DistributedPubSub`` extension can be configured with the following properties: +The `DistributedPubSub` extension can be configured with the following properties: -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#pub-sub-ext-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #pub-sub-ext-config } It is recommended to load the extension when the actor system is started by defining it in -``akka.extensions`` configuration property. Otherwise it will be activated when first used +`akka.extensions` configuration property. Otherwise it will be activated when first used and then it takes a while for it to be populated. -:: +``` +akka.extensions = ["akka.cluster.pubsub.DistributedPubSub"] +``` - akka.extensions = ["akka.cluster.pubsub.DistributedPubSub"] +## Delivery Guarantee -Delivery Guarantee ------------------- - -As in :ref:`message-delivery-reliability` of Akka, message delivery guarantee in distributed pub sub modes is **at-most-once delivery**. +As in @ref:[Message Delivery Reliability](../general/message-delivery-reliability.md) of Akka, message delivery guarantee in distributed pub sub modes is **at-most-once delivery**. In other words, messages can be lost over the wire. -If you are looking for at-least-once delivery guarantee, we recommend `Kafka Akka Streams integration `_. +If you are looking for at-least-once delivery guarantee, we recommend [Kafka Akka Streams integration](https://github.com/akka/reactive-kafka). - -Dependencies ------------- +## Dependencies To use Distributed Publish Subscribe you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` -maven:: +maven: - - com.typesafe.akka - akka-cluster-tools_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-cluster-tools_@binVersion@ + @version@ + +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/event-bus.md b/akka-docs/src/main/paradox/java/event-bus.md index c7a8c104f9..3bb4d96912 100644 --- a/akka-docs/src/main/paradox/java/event-bus.md +++ b/akka-docs/src/main/paradox/java/event-bus.md @@ -1,66 +1,58 @@ -.. _event-bus-java: - -########### - Event Bus -########### - +# Event Bus Originally conceived as a way to send messages to groups of actors, the -:class:`EventBus` has been generalized into a set of abstract base classes +`EventBus` has been generalized into a set of abstract base classes implementing a simple interface: -.. includecode:: code/jdocs/event/EventBusDocTest.java#event-bus-api +@@snip [EventBusDocTest.java](code/jdocs/event/EventBusDocTest.java) { #event-bus-api } -.. note:: +@@@ note - Please note that the EventBus does not preserve the sender of the - published messages. If you need a reference to the original sender - you have to provide it inside the message. +Please note that the EventBus does not preserve the sender of the +published messages. If you need a reference to the original sender +you have to provide it inside the message. -This mechanism is used in different places within Akka, e.g. the `Event Stream`_. +@@@ + +This mechanism is used in different places within Akka, e.g. the [Event Stream](#event-stream). Implementations can make use of the specific building blocks presented below. An event bus must define the following three type parameters: -- :class:`Event` (E) is the type of all events published on that bus - -- :class:`Subscriber` (S) is the type of subscribers allowed to register on that - event bus - -- :class:`Classifier` (C) defines the classifier to be used in selecting - subscribers for dispatching events + * `Event` (E) is the type of all events published on that bus + * `Subscriber` (S) is the type of subscribers allowed to register on that +event bus + * `Classifier` (C) defines the classifier to be used in selecting +subscribers for dispatching events The traits below are still generic in these types, but they need to be defined for any concrete implementation. -Classifiers -=========== +## Classifiers The classifiers presented here are part of the Akka distribution, but rolling your own in case you do not find a perfect match is not difficult, check the -implementation of the existing ones on `github <@github@/akka-actor/src/main/scala/akka/event/EventBus.scala>`_ +implementation of the existing ones on [github](@github@/akka-actor/src/main/scala/akka/event/EventBus.scala) -Lookup Classification ---------------------- +### Lookup Classification The simplest classification is just to extract an arbitrary classifier from each event and maintaining a set of subscribers for each possible classifier. This can be compared to tuning in on a radio station. The trait -:class:`LookupClassification` is still generic in that it abstracts over how to +`LookupClassification` is still generic in that it abstracts over how to compare subscribers and how exactly to classify. The necessary methods to be implemented are illustrated with the following example: -.. includecode:: code/jdocs/event/EventBusDocTest.java#lookup-bus +@@snip [EventBusDocTest.java](code/jdocs/event/EventBusDocTest.java) { #lookup-bus } A test for this implementation may look like this: -.. includecode:: code/jdocs/event/EventBusDocTest.java#lookup-bus-test +@@snip [EventBusDocTest.java](code/jdocs/event/EventBusDocTest.java) { #lookup-bus-test } This classifier is efficient in case no subscribers exist for a particular event. -Subchannel Classification -------------------------- +### Subchannel Classification If classifiers form a hierarchy and it is desired that subscription be possible not only at the leaf nodes, this classification may be just the right one. It @@ -72,11 +64,11 @@ classifier hierarchy. The necessary methods to be implemented are illustrated with the following example: -.. includecode:: code/jdocs/event/EventBusDocTest.java#subchannel-bus +@@snip [EventBusDocTest.java](code/jdocs/event/EventBusDocTest.java) { #subchannel-bus } A test for this implementation may look like this: -.. includecode:: code/jdocs/event/EventBusDocTest.java#subchannel-bus-test +@@snip [EventBusDocTest.java](code/jdocs/event/EventBusDocTest.java) { #subchannel-bus-test } This classifier is also efficient in case no subscribers are found for an event, but it uses conventional locking to synchronize an internal classifier @@ -84,8 +76,7 @@ cache, hence it is not well-suited to use cases in which subscriptions change with very high frequency (keep in mind that “opening” a classifier by sending the first message will also have to re-check all previous subscriptions). -Scanning Classification ------------------------ +### Scanning Classification The previous classifier was built for multi-classifier subscriptions which are strictly hierarchical, this classifier is useful if there are overlapping @@ -95,121 +86,119 @@ stations by geographical reachability (for old-school radio-wave transmission). The necessary methods to be implemented are illustrated with the following example: -.. includecode:: code/jdocs/event/EventBusDocTest.java#scanning-bus +@@snip [EventBusDocTest.java](code/jdocs/event/EventBusDocTest.java) { #scanning-bus } A test for this implementation may look like this: -.. includecode:: code/jdocs/event/EventBusDocTest.java#scanning-bus-test +@@snip [EventBusDocTest.java](code/jdocs/event/EventBusDocTest.java) { #scanning-bus-test } This classifier takes always a time which is proportional to the number of subscriptions, independent of how many actually match. -.. _actor-classification-java: - -Actor Classification --------------------- + +### Actor Classification This classification was originally developed specifically for implementing -:ref:`DeathWatch `: subscribers as well as classifiers are of -type :class:`ActorRef`. +@ref:[DeathWatch](actors.md#deathwatch-java): subscribers as well as classifiers are of +type `ActorRef`. -This classification requires an :class:`ActorSystem` in order to perform book-keeping +This classification requires an `ActorSystem` in order to perform book-keeping operations related to the subscribers being Actors, which can terminate without first unsubscribing from the EventBus. ManagedActorClassification maintains a system Actor which takes care of unsubscribing terminated actors automatically. The necessary methods to be implemented are illustrated with the following example: -.. includecode:: code/jdocs/event/EventBusDocTest.java#actor-bus +@@snip [EventBusDocTest.java](code/jdocs/event/EventBusDocTest.java) { #actor-bus } A test for this implementation may look like this: -.. includecode:: code/jdocs/event/EventBusDocTest.java#actor-bus-test +@@snip [EventBusDocTest.java](code/jdocs/event/EventBusDocTest.java) { #actor-bus-test } This classifier is still is generic in the event type, and it is efficient for all use cases. -.. _event-stream-java: - -Event Stream -============ + +## Event Stream The event stream is the main event bus of each actor system: it is used for -carrying :ref:`log messages ` and `Dead Letters`_ and may be -used by the user code for other purposes as well. It uses `Subchannel -Classification`_ which enables registering to related sets of channels (as is -used for :class:`RemotingLifecycleEvent`). The following example demonstrates +carrying @ref:[log messages](logging.md) and [Dead Letters](#dead-letters) and may be +used by the user code for other purposes as well. It uses [Subchannel +Classification](#subchannel-classification) which enables registering to related sets of channels (as is +used for `RemotingLifecycleEvent`). The following example demonstrates how a simple subscription works. Given a simple actor: -.. includecode:: code/jdocs/event/LoggingDocTest.java#imports-deadletter -.. includecode:: code/jdocs/event/LoggingDocTest.java#deadletter-actor +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #imports-deadletter } + +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #deadletter-actor } it can be subscribed like this: -.. includecode:: code/jdocs/event/LoggingDocTest.java#deadletters +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #deadletters } It is also worth pointing out that thanks to the way the subchannel classification is implemented in the event stream, it is possible to subscribe to a group of events, by subscribing to their common superclass as demonstrated in the following example: -.. includecode:: code/jdocs/event/LoggingDocTest.java#superclass-subscription-eventstream +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #superclass-subscription-eventstream } -Similarly to `Actor Classification`_, :class:`EventStream` will automatically remove subscribers when they terminate. +Similarly to [Actor Classification](#actor-classification), `EventStream` will automatically remove subscribers when they terminate. -.. note:: - The event stream is a *local facility*, meaning that it will *not* distribute events to other nodes in a clustered environment (unless you subscribe a Remote Actor to the stream explicitly). - If you need to broadcast events in an Akka cluster, *without* knowing your recipients explicitly (i.e. obtaining their ActorRefs), you may want to look into: :ref:`distributed-pub-sub-java`. +@@@ note -Default Handlers ----------------- +The event stream is a *local facility*, meaning that it will *not* distribute events to other nodes in a clustered environment (unless you subscribe a Remote Actor to the stream explicitly). +If you need to broadcast events in an Akka cluster, *without* knowing your recipients explicitly (i.e. obtaining their ActorRefs), you may want to look into: @ref:[Distributed Publish Subscribe in Cluster](distributed-pub-sub.md). + +@@@ + +### Default Handlers Upon start-up the actor system creates and subscribes actors to the event stream for logging: these are the handlers which are configured for example in -``application.conf``: +`application.conf`: -.. code-block:: text - - akka { - loggers = ["akka.event.Logging$DefaultLogger"] - } +```text +akka { + loggers = ["akka.event.Logging$DefaultLogger"] +} +``` The handlers listed here by fully-qualified class name will be subscribed to all log event classes with priority higher than or equal to the configured log-level and their subscriptions are kept in sync when changing the log-level -at runtime:: +at runtime: - system.eventStream.setLogLevel(Logging.DebugLevel()); +``` +system.eventStream.setLogLevel(Logging.DebugLevel()); +``` This means that log events for a level which will not be logged are typically not dispatched at all (unless manual subscriptions to the respective event class have been done) -Dead Letters ------------- +### Dead Letters -As described at :ref:`stopping-actors-java`, messages queued when an actor +As described at @ref:[Stopping actors](actors.md#stopping-actors-java), messages queued when an actor terminates or sent after its death are re-routed to the dead letter mailbox, -which by default will publish the messages wrapped in :class:`DeadLetter`. This +which by default will publish the messages wrapped in `DeadLetter`. This wrapper holds the original sender, receiver and message of the envelope which was redirected. -Some internal messages (marked with the :class:`DeadLetterSuppression` trait) will not end up as +Some internal messages (marked with the `DeadLetterSuppression` trait) will not end up as dead letters like normal messages. These are by design safe and expected to sometimes arrive at a terminated actor and since they are nothing to worry about, they are suppressed from the default dead letters logging mechanism. However, in case you find yourself in need of debugging these kinds of low level suppressed dead letters, it's still possible to subscribe to them explicitly: -.. includecode:: code/jdocs/event/LoggingDocTest.java#suppressed-deadletters +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #suppressed-deadletters } or all dead letters (including the suppressed ones): -.. includecode:: code/jdocs/event/LoggingDocTest.java#all-deadletters +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #all-deadletters } -Other Uses ----------- +### Other Uses The event stream is always there and ready to be used, just publish your own -events (it accepts ``Object``) and subscribe listeners to the corresponding JVM -classes. - +events (it accepts `Object`) and subscribe listeners to the corresponding JVM +classes. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/extending-akka.md b/akka-docs/src/main/paradox/java/extending-akka.md index ca98d40b08..7fcd8fcc1a 100644 --- a/akka-docs/src/main/paradox/java/extending-akka.md +++ b/akka-docs/src/main/paradox/java/extending-akka.md @@ -1,113 +1,96 @@ -.. _extending-akka-java: - -######################## - Akka Extensions -######################## - +# Akka Extensions If you want to add features to Akka, there is a very elegant, but powerful mechanism for doing so. -It's called Akka Extensions and is comprised of 2 basic components: an ``Extension`` and an ``ExtensionId``. +It's called Akka Extensions and is comprised of 2 basic components: an `Extension` and an `ExtensionId`. -Extensions will only be loaded once per ``ActorSystem``, which will be managed by Akka. -You can choose to have your Extension loaded on-demand or at ``ActorSystem`` creation time through the Akka configuration. +Extensions will only be loaded once per `ActorSystem`, which will be managed by Akka. +You can choose to have your Extension loaded on-demand or at `ActorSystem` creation time through the Akka configuration. Details on how to make that happens are below, in the "Loading from Configuration" section. -.. warning:: +@@@ warning - Since an extension is a way to hook into Akka itself, the implementor of the extension needs to - ensure the thread safety of his/her extension. +Since an extension is a way to hook into Akka itself, the implementor of the extension needs to +ensure the thread safety of his/her extension. +@@@ -Building an Extension -===================== +## Building an Extension So let's create a sample extension that just lets us count the number of times something has happened. -First, we define what our ``Extension`` should do: +First, we define what our `Extension` should do: -.. includecode:: code/jdocs/extension/ExtensionDocTest.java - :include: imports +@@snip [ExtensionDocTest.java](code/jdocs/extension/ExtensionDocTest.java) { #imports } -.. includecode:: code/jdocs/extension/ExtensionDocTest.java - :include: extension +@@snip [ExtensionDocTest.java](code/jdocs/extension/ExtensionDocTest.java) { #extension } -Then we need to create an ``ExtensionId`` for our extension so we can grab a hold of it. +Then we need to create an `ExtensionId` for our extension so we can grab a hold of it. -.. includecode:: code/jdocs/extension/ExtensionDocTest.java - :include: imports +@@snip [ExtensionDocTest.java](code/jdocs/extension/ExtensionDocTest.java) { #imports } -.. includecode:: code/jdocs/extension/ExtensionDocTest.java - :include: extensionid +@@snip [ExtensionDocTest.java](code/jdocs/extension/ExtensionDocTest.java) { #extensionid } Wicked! Now all we need to do is to actually use it: -.. includecode:: code/jdocs/extension/ExtensionDocTest.java - :include: extension-usage +@@snip [ExtensionDocTest.java](code/jdocs/extension/ExtensionDocTest.java) { #extension-usage } Or from inside of an Akka Actor: -.. includecode:: code/jdocs/extension/ExtensionDocTest.java - :include: extension-usage-actor +@@snip [ExtensionDocTest.java](code/jdocs/extension/ExtensionDocTest.java) { #extension-usage-actor } That's all there is to it! -Loading from Configuration -========================== +## Loading from Configuration -To be able to load extensions from your Akka configuration you must add FQCNs of implementations of either ``ExtensionId`` or ``ExtensionIdProvider`` -in the "akka.extensions" section of the config you provide to your ``ActorSystem``. +To be able to load extensions from your Akka configuration you must add FQCNs of implementations of either `ExtensionId` or `ExtensionIdProvider` +in the "akka.extensions" section of the config you provide to your `ActorSystem`. -:: +``` +akka { + extensions = ["docs.extension.ExtensionDocTest.CountExtension"] +} +``` - akka { - extensions = ["docs.extension.ExtensionDocTest.CountExtension"] - } - -Applicability -============= +## Applicability The sky is the limit! -By the way, did you know that Akka's ``Typed Actors``, ``Serialization`` and other features are implemented as Akka Extensions? +By the way, did you know that Akka's `Typed Actors`, `Serialization` and other features are implemented as Akka Extensions? -.. _extending-akka-java.settings: + +### Application specific settings -Application specific settings ------------------------------ - -The :ref:`configuration` can be used for application specific settings. A good practice is to place those settings in an Extension. +The configuration can be used for application specific settings. A good practice is to place those settings in an Extension. Sample configuration: -.. includecode:: ../scala/code/docs/extension/SettingsExtensionDocSpec.scala - :include: config +@@snip [SettingsExtensionDocSpec.scala](../scala/code/docs/extension/SettingsExtensionDocSpec.scala) { #config } -The ``Extension``: +The `Extension`: -.. includecode:: code/jdocs/extension/SettingsExtensionDocTest.java - :include: imports +@@snip [SettingsExtensionDocTest.java](code/jdocs/extension/SettingsExtensionDocTest.java) { #imports } -.. includecode:: code/jdocs/extension/SettingsExtensionDocTest.java - :include: extension,extensionid +@@snip [SettingsExtensionDocTest.java](code/jdocs/extension/SettingsExtensionDocTest.java) { #extension #extensionid } Use it: -.. includecode:: code/jdocs/extension/SettingsExtensionDocTest.java - :include: extension-usage-actor +@@snip [SettingsExtensionDocTest.java](code/jdocs/extension/SettingsExtensionDocTest.java) { #extension-usage-actor } + +## Library extensions -Library extensions -================== A third part library may register it's extension for auto-loading on actor system startup by appending it to -``akka.library-extensions`` in its ``reference.conf``. - -:: - - akka.library-extensions += "docs.extension.ExampleExtension" +`akka.library-extensions` in its `reference.conf`. +``` +akka.library-extensions += "docs.extension.ExampleExtension" +``` As there is no way to selectively remove such extensions, it should be used with care and only when there is no case where the user would ever want it disabled or have specific support for disabling such sub-features. One example where this could be important is in tests. -.. warning:: - The``akka.library-extensions`` must never be assigned (``= ["Extension"]``) instead of appending as this will break - the library-extension mechanism and make behavior depend on class path ordering. +@@@ warning + +The``akka.library-extensions`` must never be assigned (`= ["Extension"]`) instead of appending as this will break +the library-extension mechanism and make behavior depend on class path ordering. + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/fault-tolerance-sample.md b/akka-docs/src/main/paradox/java/fault-tolerance-sample.md index ea195c87f4..4f2e4616ef 100644 --- a/akka-docs/src/main/paradox/java/fault-tolerance-sample.md +++ b/akka-docs/src/main/paradox/java/fault-tolerance-sample.md @@ -1,53 +1,38 @@ -.. _fault-tolerance-sample-java: -Diagrams of the Fault Tolerance Sample ----------------------------------------------- + +# Diagrams of the Fault Tolerance Sample -.. image:: ../images/faulttolerancesample-normal-flow.png +![faulttolerancesample-normal-flow.png](../images/faulttolerancesample-normal-flow.png) *The above diagram illustrates the normal message flow.* **Normal flow:** -======= ================================================================================== -Step Description -======= ================================================================================== -1 The progress ``Listener`` starts the work. -2 The ``Worker`` schedules work by sending ``Do`` messages periodically to itself -3, 4, 5 When receiving ``Do`` the ``Worker`` tells the ``CounterService`` - to increment the counter, three times. The ``Increment`` message is forwarded - to the ``Counter``, which updates its counter variable and sends current value - to the ``Storage``. -6, 7 The ``Worker`` asks the ``CounterService`` of current value of the counter and pipes - the result back to the ``Listener``. -======= ================================================================================== +|Step | Description | +|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|1 | The progress `Listener` starts the work. | +|2 | The `Worker` schedules work by sending `Do` messages periodically to itself | +|3, 4, 5 | When receiving `Do` the `Worker` tells the `CounterService` to increment the counter, three times. The `Increment` message is forwarded to the `Counter`, which updates its counter variable and sends current value to the `Storage`.| +|6, 7 | The `Worker` asks the `CounterService` of current value of the counter and pipes the result back to the `Listener`. | - -.. image:: ../images/faulttolerancesample-failure-flow.png +![faulttolerancesample-failure-flow.png](../images/faulttolerancesample-failure-flow.png) *The above diagram illustrates what happens in case of storage failure.* **Failure flow:** -=========== ================================================================================== -Step Description -=========== ================================================================================== -1 The ``Storage`` throws ``StorageException``. -2 The ``CounterService`` is supervisor of the ``Storage`` and restarts the - ``Storage`` when ``StorageException`` is thrown. -3, 4, 5, 6 The ``Storage`` continues to fail and is restarted. -7 After 3 failures and restarts within 5 seconds the ``Storage`` is stopped by its - supervisor, i.e. the ``CounterService``. -8 The ``CounterService`` is also watching the ``Storage`` for termination and - receives the ``Terminated`` message when the ``Storage`` has been stopped ... -9, 10, 11 and tells the ``Counter`` that there is no ``Storage``. -12 The ``CounterService`` schedules a ``Reconnect`` message to itself. -13, 14 When it receives the ``Reconnect`` message it creates a new ``Storage`` ... -15, 16 and tells the ``Counter`` to use the new ``Storage`` -=========== ================================================================================== +|Step | Description | +|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------| +|1 | The `Storage` throws `StorageException`. | +|2 | The `CounterService` is supervisor of the `Storage` and restarts the `Storage` when `StorageException` is thrown. | +|3, 4, 5, 6 | The `Storage` continues to fail and is restarted. | +|7 | After 3 failures and restarts within 5 seconds the `Storage` is stopped by its supervisor, i.e. the `CounterService`. | +|8 | The `CounterService` is also watching the `Storage` for termination and receives the `Terminated` message when the `Storage` has been stopped ...| +|9, 10, 11 | and tells the `Counter` that there is no `Storage`. | +|12 | The `CounterService` schedules a `Reconnect` message to itself. | +|13, 14 | When it receives the `Reconnect` message it creates a new `Storage` ... | +|15, 16 | and tells the `Counter` to use the new `Storage` | -Full Source Code of the Fault Tolerance Sample ------------------------------------------------------- - -.. includecode:: code/jdocs/actor/FaultHandlingDocSample.java#all +# Full Source Code of the Fault Tolerance Sample +@@snip [FaultHandlingDocSample.java](code/jdocs/actor/FaultHandlingDocSample.java) { #all } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/fault-tolerance.md b/akka-docs/src/main/paradox/java/fault-tolerance.md index 5d996be115..00fb97eec9 100644 --- a/akka-docs/src/main/paradox/java/fault-tolerance.md +++ b/akka-docs/src/main/paradox/java/fault-tolerance.md @@ -1,15 +1,11 @@ -.. _fault-tolerance-java: +# Fault Tolerance -Fault Tolerance -=============== - -As explained in :ref:`actor-systems` each actor is the supervisor of its +As explained in @ref:[Actor Systems](../general/actor-systems.md) each actor is the supervisor of its children, and as such each actor defines fault handling supervisor strategy. This strategy cannot be changed afterwards as it is an integral part of the actor system’s structure. -Fault Handling in Practice --------------------------- +## Fault Handling in Practice First, let us look at a sample that illustrates one way to handle data store errors, which is a typical source of failure in real world applications. Of course it depends @@ -20,157 +16,145 @@ Read the following source code. The inlined comments explain the different piece the fault handling and why they are added. It is also highly recommended to run this sample as it is easy to follow the log output to understand what is happening in runtime. -.. toctree:: +@@toc - fault-tolerance-sample +@@@ index -Creating a Supervisor Strategy ------------------------------- +* [fault-tolerance-sample](fault-tolerance-sample.md) + +@@@ + +## Creating a Supervisor Strategy The following sections explain the fault handling mechanism and alternatives in more depth. For the sake of demonstration let us consider the following strategy: -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: strategy +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #strategy } I have chosen a few well-known exception types in order to demonstrate the -application of the fault handling directives described in :ref:`supervision`. +application of the fault handling directives described in supervision. First off, it is a one-for-one strategy, meaning that each child is treated separately (an all-for-one strategy works very similarly, the only difference is that any decision is applied to all children of the supervisor, not only the failing one). There are limits set on the restart frequency, namely maximum 10 -restarts per minute. ``-1`` and ``Duration.Inf()`` means that the respective limit +restarts per minute. `-1` and `Duration.Inf()` means that the respective limit does not apply, leaving the possibility to specify an absolute upper limit on the restarts or to make the restarts work infinitely. The child actor is stopped if the limit is exceeded. -.. note:: +@@@ note - If the strategy is declared inside the supervising actor (as opposed to - a separate class) its decider has access to all internal state of - the actor in a thread-safe fashion, including obtaining a reference to the - currently failed child (available as the ``sender`` of the failure message). +If the strategy is declared inside the supervising actor (as opposed to +a separate class) its decider has access to all internal state of +the actor in a thread-safe fashion, including obtaining a reference to the +currently failed child (available as the `sender` of the failure message). -Default Supervisor Strategy -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@@ -``Escalate`` is used if the defined strategy doesn't cover the exception that was thrown. +### Default Supervisor Strategy + +`Escalate` is used if the defined strategy doesn't cover the exception that was thrown. When the supervisor strategy is not defined for an actor the following exceptions are handled by default: -* ``ActorInitializationException`` will stop the failing child actor -* ``ActorKilledException`` will stop the failing child actor -* ``DeathPactException`` will stop the failing child actor -* ``Exception`` will restart the failing child actor -* Other types of ``Throwable`` will be escalated to parent actor + * `ActorInitializationException` will stop the failing child actor + * `ActorKilledException` will stop the failing child actor + * `DeathPactException` will stop the failing child actor + * `Exception` will restart the failing child actor + * Other types of `Throwable` will be escalated to parent actor If the exception escalate all the way up to the root guardian it will handle it in the same way as the default strategy defined above. -Stopping Supervisor Strategy -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Stopping Supervisor Strategy Closer to the Erlang way is the strategy to just stop children when they fail and then take corrective action in the supervisor when DeathWatch signals the loss of the child. This strategy is also provided pre-packaged as -:obj:`SupervisorStrategy.stoppingStrategy` with an accompanying -:class:`StoppingSupervisorStrategy` configurator to be used when you want the -``"/user"`` guardian to apply it. +`SupervisorStrategy.stoppingStrategy` with an accompanying +`StoppingSupervisorStrategy` configurator to be used when you want the +`"/user"` guardian to apply it. -Logging of Actor Failures -^^^^^^^^^^^^^^^^^^^^^^^^^ +### Logging of Actor Failures -By default the ``SupervisorStrategy`` logs failures unless they are escalated. +By default the `SupervisorStrategy` logs failures unless they are escalated. Escalated failures are supposed to be handled, and potentially logged, at a level higher in the hierarchy. -You can mute the default logging of a ``SupervisorStrategy`` by setting -``loggingEnabled`` to ``false`` when instantiating it. Customized logging -can be done inside the ``Decider``. Note that the reference to the currently -failed child is available as the ``sender`` when the ``SupervisorStrategy`` is +You can mute the default logging of a `SupervisorStrategy` by setting +`loggingEnabled` to `false` when instantiating it. Customized logging +can be done inside the `Decider`. Note that the reference to the currently +failed child is available as the `sender` when the `SupervisorStrategy` is declared inside the supervising actor. -You may also customize the logging in your own ``SupervisorStrategy`` implementation -by overriding the ``logFailure`` method. +You may also customize the logging in your own `SupervisorStrategy` implementation +by overriding the `logFailure` method. -Supervision of Top-Level Actors -------------------------------- +## Supervision of Top-Level Actors -Toplevel actors means those which are created using ``system.actorOf()``, and -they are children of the :ref:`User Guardian `. There are no +Toplevel actors means those which are created using `system.actorOf()`, and +they are children of the @ref:[User Guardian](../general/supervision.md#user-guardian). There are no special rules applied in this case, the guardian simply applies the configured strategy. -Test Application ----------------- +## Test Application The following section shows the effects of the different directives in practice, where a test setup is needed. First off, we need a suitable supervisor: -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: supervisor +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #supervisor } This supervisor will be used to create a child, with which we can experiment: -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: child +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #child } -The test is easier by using the utilities described in :ref:`akka-testkit`, -where ``TestProbe`` provides an actor ref useful for receiving and inspecting replies. +The test is easier by using the utilities described in akka-testkit, +where `TestProbe` provides an actor ref useful for receiving and inspecting replies. -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: testkit +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #testkit } Let us create actors: -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: create +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #create } -The first test shall demonstrate the ``Resume`` directive, so we try it out by +The first test shall demonstrate the `Resume` directive, so we try it out by setting some non-initial state in the actor and have it fail: -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: resume +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #resume } As you can see the value 42 survives the fault handling directive. Now, if we -change the failure to a more serious ``NullPointerException``, that will no +change the failure to a more serious `NullPointerException`, that will no longer be the case: -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: restart +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #restart } -And finally in case of the fatal ``IllegalArgumentException`` the child will be +And finally in case of the fatal `IllegalArgumentException` the child will be terminated by the supervisor: -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: stop +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #stop } Up to now the supervisor was completely unaffected by the child’s failure, -because the directives set did handle it. In case of an ``Exception``, this is not +because the directives set did handle it. In case of an `Exception`, this is not true anymore and the supervisor escalates the failure. -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: escalate-kill +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #escalate-kill } The supervisor itself is supervised by the top-level actor provided by the -:class:`ActorSystem`, which has the default policy to restart in case of all -``Exception`` cases (with the notable exceptions of -``ActorInitializationException`` and ``ActorKilledException``). Since the +`ActorSystem`, which has the default policy to restart in case of all +`Exception` cases (with the notable exceptions of +`ActorInitializationException` and `ActorKilledException`). Since the default directive in case of a restart is to kill all children, we expected our poor child not to survive this failure. In case this is not desired (which depends on the use case), we need to use a different supervisor which overrides this behavior. -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: supervisor2 +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #supervisor2 } With this parent, the child survives the escalated restart, as demonstrated in the last test: -.. includecode:: code/jdocs/actor/FaultHandlingTest.java - :include: escalate-restart - +@@snip [FaultHandlingTest.java](code/jdocs/actor/FaultHandlingTest.java) { #escalate-restart } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/fsm.md b/akka-docs/src/main/paradox/java/fsm.md index 073559560e..99934aa570 100644 --- a/akka-docs/src/main/paradox/java/fsm.md +++ b/akka-docs/src/main/paradox/java/fsm.md @@ -1,54 +1,43 @@ -.. _fsm-java: +# FSM -##### - FSM -##### - - -Overview -======== +## Overview The FSM (Finite State Machine) is available as an abstract base class that implements -an Akka Actor and is best described in the `Erlang design principles -`_ +an Akka Actor and is best described in the [Erlang design principles](http://www.erlang.org/documentation/doc-4.8.2/doc/design_principles/fsm.html) A FSM can be described as a set of relations of the form: - **State(S) x Event(E) -> Actions (A), State(S')** +> +**State(S) x Event(E) -> Actions (A), State(S')** These relations are interpreted as meaning: - *If we are in state S and the event E occurs, we should perform the actions A - and make a transition to the state S'.* +> +*If we are in state S and the event E occurs, we should perform the actions A +and make a transition to the state S'.* +## A Simple Example -A Simple Example -================ - -To demonstrate most of the features of the :class:`AbstractFSM` class, consider an +To demonstrate most of the features of the `AbstractFSM` class, consider an actor which shall receive and queue messages while they arrive in a burst and send them on after the burst ended or a flush request is received. First, consider all of the below to use these import statements: -.. includecode:: code/jdocs/actor/fsm/Buncher.java#simple-imports +@@snip [Buncher.java](code/jdocs/actor/fsm/Buncher.java) { #simple-imports } The contract of our “Buncher” actor is that it accepts or produces the following messages: -.. includecode:: code/jdocs/actor/fsm/Events.java - :include: simple-events - :exclude: boilerplate +@@snip [Events.java](code/jdocs/actor/fsm/Events.java) { #simple-events } -``SetTarget`` is needed for starting it up, setting the destination for the -``Batches`` to be passed on; ``Queue`` will add to the internal queue while -``Flush`` will mark the end of a burst. +`SetTarget` is needed for starting it up, setting the destination for the +`Batches` to be passed on; `Queue` will add to the internal queue while +`Flush` will mark the end of a burst. -The actor can be in two states: no message queued (aka ``Idle``) or some -message queued (aka ``Active``). The states and the state data is defined like this: +The actor can be in two states: no message queued (aka `Idle`) or some +message queued (aka `Active`). The states and the state data is defined like this: -.. includecode:: code/jdocs/actor/fsm/Buncher.java - :include: simple-state - :exclude: boilerplate +@@snip [Buncher.java](code/jdocs/actor/fsm/Buncher.java) { #simple-state } The actor starts out in the idle state. Once a message arrives it will go to the active state and stay there as long as messages keep arriving and no flush is @@ -57,214 +46,206 @@ reference to send the batches to and the actual queue of messages. Now let’s take a look at the skeleton for our FSM actor: -.. includecode:: code/jdocs/actor/fsm/Buncher.java - :include: simple-fsm - :exclude: transition-elided,unhandled-elided +@@snip [Buncher.java](code/jdocs/actor/fsm/Buncher.java) { #simple-fsm } -The basic strategy is to declare the actor, by inheriting the :class:`AbstractFSM` class +The basic strategy is to declare the actor, by inheriting the `AbstractFSM` class and specifying the possible states and data values as type parameters. Within the body of the actor a DSL is used for declaring the state machine: - * :meth:`startWith` defines the initial state and initial data - * then there is one :meth:`when() { ... }` declaration per state to be - handled (could potentially be multiple ones, the passed - :class:`PartialFunction` will be concatenated using :meth:`orElse`) - * finally starting it up using :meth:`initialize`, which performs the - transition into the initial state and sets up timers (if required). +> + * `startWith` defines the initial state and initial data + * then there is one `when() { ... }` declaration per state to be +handled (could potentially be multiple ones, the passed +`PartialFunction` will be concatenated using `orElse`) + * finally starting it up using `initialize`, which performs the +transition into the initial state and sets up timers (if required). -In this case, we start out in the ``Idle`` and ``Uninitialized`` state, where -only the ``SetTarget()`` message is handled; ``stay`` prepares to end this -event’s processing for not leaving the current state, while the ``using`` -modifier makes the FSM replace the internal state (which is ``Uninitialized`` -at this point) with a fresh ``Todo()`` object containing the target actor -reference. The ``Active`` state has a state timeout declared, which means that -if no message is received for 1 second, a ``FSM.StateTimeout`` message will be -generated. This has the same effect as receiving the ``Flush`` command in this -case, namely to transition back into the ``Idle`` state and resetting the +In this case, we start out in the `Idle` and `Uninitialized` state, where +only the `SetTarget()` message is handled; `stay` prepares to end this +event’s processing for not leaving the current state, while the `using` +modifier makes the FSM replace the internal state (which is `Uninitialized` +at this point) with a fresh `Todo()` object containing the target actor +reference. The `Active` state has a state timeout declared, which means that +if no message is received for 1 second, a `FSM.StateTimeout` message will be +generated. This has the same effect as receiving the `Flush` command in this +case, namely to transition back into the `Idle` state and resetting the internal queue to the empty vector. But how do messages get queued? Since this shall work identically in both states, we make use of the fact that any event -which is not handled by the ``when()`` block is passed to the -``whenUnhandled()`` block: +which is not handled by the `when()` block is passed to the +`whenUnhandled()` block: -.. includecode:: code/jdocs/actor/fsm/Buncher.java#unhandled-elided +@@snip [Buncher.java](code/jdocs/actor/fsm/Buncher.java) { #unhandled-elided } -The first case handled here is adding ``Queue()`` requests to the internal -queue and going to the ``Active`` state (this does the obvious thing of staying -in the ``Active`` state if already there), but only if the FSM data are not -``Uninitialized`` when the ``Queue()`` event is received. Otherwise—and in all +The first case handled here is adding `Queue()` requests to the internal +queue and going to the `Active` state (this does the obvious thing of staying +in the `Active` state if already there), but only if the FSM data are not +`Uninitialized` when the `Queue()` event is received. Otherwise—and in all other non-handled cases—the second case just logs a warning and does not change the internal state. -The only missing piece is where the ``Batches`` are actually sent to the -target, for which we use the ``onTransition`` mechanism: you can declare +The only missing piece is where the `Batches` are actually sent to the +target, for which we use the `onTransition` mechanism: you can declare multiple such blocks and all of them will be tried for matching behavior in case a state transition occurs (i.e. only when the state actually changes). -.. includecode:: code/jdocs/actor/fsm/Buncher.java#transition-elided +@@snip [Buncher.java](code/jdocs/actor/fsm/Buncher.java) { #transition-elided } The transition callback is a partial function which takes as input a pair of states—the current and the next state. During the state change, the old state -data is available via ``stateData`` as shown, and the new state data would be -available as ``nextStateData``. +data is available via `stateData` as shown, and the new state data would be +available as `nextStateData`. To verify that this buncher actually works, it is quite easy to write a test -using the :ref:`akka-testkit`, here using JUnit as an example: +using the akka-testkit, here using JUnit as an example: -.. includecode:: code/jdocs/actor/fsm/BuncherTest.java - :include: test-code +@@snip [BuncherTest.java](code/jdocs/actor/fsm/BuncherTest.java) { #test-code } -Reference -========= +## Reference -The AbstractFSM Class ---------------------- +### The AbstractFSM Class -The :class:`AbstractFSM` abstract class is the base class used to implement an FSM. It implements +The `AbstractFSM` abstract class is the base class used to implement an FSM. It implements Actor since an Actor is created to drive the FSM. -.. includecode:: code/jdocs/actor/fsm/Buncher.java - :include: simple-fsm - :exclude: fsm-body +@@snip [Buncher.java](code/jdocs/actor/fsm/Buncher.java) { #simple-fsm } -.. note:: +@@@ note - The AbstractFSM class defines a ``receive`` method which handles internal messages - and passes everything else through to the FSM logic (according to the - current state). When overriding the ``receive`` method, keep in mind that - e.g. state timeout handling depends on actually passing the messages through - the FSM logic. +The AbstractFSM class defines a `receive` method which handles internal messages +and passes everything else through to the FSM logic (according to the +current state). When overriding the `receive` method, keep in mind that +e.g. state timeout handling depends on actually passing the messages through +the FSM logic. -The :class:`AbstractFSM` class takes two type parameters: +@@@ - #. the supertype of all state names, usually an enum, - #. the type of the state data which are tracked by the :class:`AbstractFSM` module - itself. +The `AbstractFSM` class takes two type parameters: -.. _fsm-philosophy: +> + 1. the supertype of all state names, usually an enum, + 2. the type of the state data which are tracked by the `AbstractFSM` module +itself. -.. note:: +@@@ note - The state data together with the state name describe the internal state of - the state machine; if you stick to this scheme and do not add mutable fields - to the FSM class you have the advantage of making all changes of the - internal state explicit in a few well-known places. +The state data together with the state name describe the internal state of +the state machine; if you stick to this scheme and do not add mutable fields +to the FSM class you have the advantage of making all changes of the +internal state explicit in a few well-known places. -Defining States ---------------- +@@@ + +### Defining States A state is defined by one or more invocations of the method - :func:`when([, stateTimeout = ])(stateFunction)`. +> +`when([, stateTimeout = ])(stateFunction)`. The given name must be an object which is type-compatible with the first type -parameter given to the :class:`AbstractFSM` class. This object is used as a hash key, -so you must ensure that it properly implements :meth:`equals` and -:meth:`hashCode`; in particular it must not be mutable. The easiest fit for +parameter given to the `AbstractFSM` class. This object is used as a hash key, +so you must ensure that it properly implements `equals` and +`hashCode`; in particular it must not be mutable. The easiest fit for these requirements are case objects. -If the :meth:`stateTimeout` parameter is given, then all transitions into this +If the `stateTimeout` parameter is given, then all transitions into this state, including staying, receive this timeout by default. Initiating the transition with an explicit timeout may be used to override this default, see -`Initiating Transitions`_ for more information. The state timeout of any state +[Initiating Transitions](#initiating-transitions) for more information. The state timeout of any state may be changed during action processing with -:func:`setStateTimeout(state, duration)`. This enables runtime configuration +`setStateTimeout(state, duration)`. This enables runtime configuration e.g. via external message. -The :meth:`stateFunction` argument is a :class:`PartialFunction[Event, State]`, +The `stateFunction` argument is a `PartialFunction[Event, State]`, which is conveniently given using the state function builder syntax as demonstrated below: -.. includecode:: code/jdocs/actor/fsm/Buncher.java - :include: when-syntax +@@snip [Buncher.java](code/jdocs/actor/fsm/Buncher.java) { #when-syntax } -.. warning:: +@@@ warning - It is required that you define handlers for each of the possible FSM states, - otherwise there will be failures when trying to switch to undeclared states. +It is required that you define handlers for each of the possible FSM states, +otherwise there will be failures when trying to switch to undeclared states. + +@@@ It is recommended practice to declare the states as an enum and then verify that there is a -``when`` clause for each of the states. If you want to leave the handling of a state +`when` clause for each of the states. If you want to leave the handling of a state “unhandled” (more below), it still needs to be declared like this: -.. includecode:: code/jdocs/actor/fsm/FSMDocTest.java#NullFunction +@@snip [FSMDocTest.java](code/jdocs/actor/fsm/FSMDocTest.java) { #NullFunction } -Defining the Initial State --------------------------- +### Defining the Initial State Each FSM needs a starting point, which is declared using - :func:`startWith(state, data[, timeout])` +> +`startWith(state, data[, timeout])` The optionally given timeout argument overrides any specification given for the desired initial state. If you want to cancel a default timeout, use -:obj:`Duration.Inf`. +`Duration.Inf`. -Unhandled Events ----------------- +### Unhandled Events If a state doesn't handle a received event a warning is logged. If you want to do something else in this case you can specify that with -:func:`whenUnhandled(stateFunction)`: +`whenUnhandled(stateFunction)`: -.. includecode:: code/jdocs/actor/fsm/FSMDocTest.java - :include: unhandled-syntax +@@snip [FSMDocTest.java](code/jdocs/actor/fsm/FSMDocTest.java) { #unhandled-syntax } Within this handler the state of the FSM may be queried using the -:meth:`stateName` method. +`stateName` method. **IMPORTANT**: This handler is not stacked, meaning that each invocation of -:func:`whenUnhandled` replaces the previously installed handler. +`whenUnhandled` replaces the previously installed handler. -Initiating Transitions ----------------------- +### Initiating Transitions -The result of any :obj:`stateFunction` must be a definition of the next state -unless terminating the FSM, which is described in `Termination from Inside`_. +The result of any `stateFunction` must be a definition of the next state +unless terminating the FSM, which is described in [Termination from Inside](#termination-from-inside). The state definition can either be the current state, as described by the -:func:`stay` directive, or it is a different state as given by -:func:`goto(state)`. The resulting object allows further qualification by way +`stay` directive, or it is a different state as given by +`goto(state)`. The resulting object allows further qualification by way of the modifiers described in the following: -* :meth:`forMax(duration)` - - This modifier sets a state timeout on the next state. This means that a timer - is started which upon expiry sends a :obj:`StateTimeout` message to the FSM. - This timer is canceled upon reception of any other message in the meantime; - you can rely on the fact that the :obj:`StateTimeout` message will not be - processed after an intervening message. - - This modifier can also be used to override any default timeout which is - specified for the target state. If you want to cancel the default timeout, - use :obj:`Duration.Inf`. - -* :meth:`using(data)` - - This modifier replaces the old state data with the new data given. If you - follow the advice :ref:`above `, this is the only place where - internal state data are ever modified. - -* :meth:`replying(msg)` - - This modifier sends a reply to the currently processed message and otherwise - does not modify the state transition. + * + `forMax(duration)` + This modifier sets a state timeout on the next state. This means that a timer +is started which upon expiry sends a `StateTimeout` message to the FSM. +This timer is canceled upon reception of any other message in the meantime; +you can rely on the fact that the `StateTimeout` message will not be +processed after an intervening message. + This modifier can also be used to override any default timeout which is +specified for the target state. If you want to cancel the default timeout, +use `Duration.Inf`. + * + `using(data)` + This modifier replaces the old state data with the new data given. If you +follow the advice [above](#fsm-philosophy), this is the only place where +internal state data are ever modified. + * + `replying(msg)` + This modifier sends a reply to the currently processed message and otherwise +does not modify the state transition. All modifiers can be chained to achieve a nice and concise description: -.. includecode:: code/jdocs/actor/fsm/FSMDocTest.java - :include: modifier-syntax +@@snip [FSMDocTest.java](code/jdocs/actor/fsm/FSMDocTest.java) { #modifier-syntax } The parentheses are not actually needed in all cases, but they visually distinguish between modifiers and their arguments and therefore make the code even more pleasant to read for foreigners. -.. note:: +@@@ note - Please note that the ``return`` statement may not be used in :meth:`when` - blocks or similar; this is a Scala restriction. Either refactor your code - using ``if () ... else ...`` or move it into a method definition. +Please note that the `return` statement may not be used in `when` +blocks or similar; this is a Scala restriction. Either refactor your code +using `if () ... else ...` or move it into a method definition. -Monitoring Transitions ----------------------- +@@@ + +### Monitoring Transitions Transitions occur "between states" conceptually, which means after any actions you have put into the event handling block; this is obvious since the next @@ -273,186 +254,184 @@ not need to worry about the exact order with respect to setting the internal state variable, as everything within the FSM actor is running single-threaded anyway. -Internal Monitoring -^^^^^^^^^^^^^^^^^^^ +#### Internal Monitoring Up to this point, the FSM DSL has been centered on states and events. The dual view is to describe it as a series of transitions. This is enabled by the method - :func:`onTransition(handler)` +> +`onTransition(handler)` which associates actions with a transition instead of with a state and event. The handler is a partial function which takes a pair of states as input; no resulting state is needed as it is not possible to modify the transition in progress. -.. includecode:: code/jdocs/actor/fsm/FSMDocTest.java - :include: transition-syntax +@@snip [FSMDocTest.java](code/jdocs/actor/fsm/FSMDocTest.java) { #transition-syntax } It is also possible to pass a function object accepting two states to -:func:`onTransition`, in case your transition handling logic is implemented as +`onTransition`, in case your transition handling logic is implemented as a method: -.. includecode:: code/jdocs/actor/fsm/FSMDocTest.java - :include: alt-transition-syntax +@@snip [FSMDocTest.java](code/jdocs/actor/fsm/FSMDocTest.java) { #alt-transition-syntax } The handlers registered with this method are stacked, so you can intersperse -:func:`onTransition` blocks with :func:`when` blocks as suits your design. It +`onTransition` blocks with `when` blocks as suits your design. It should be noted, however, that *all handlers will be invoked for each transition*, not only the first matching one. This is designed specifically so you can put all transition handling for a certain aspect into one place without having to worry about earlier declarations shadowing later ones; the actions are still executed in declaration order, though. -.. note:: +@@@ note - This kind of internal monitoring may be used to structure your FSM according - to transitions, so that for example the cancellation of a timer upon leaving - a certain state cannot be forgot when adding new target states. +This kind of internal monitoring may be used to structure your FSM according +to transitions, so that for example the cancellation of a timer upon leaving +a certain state cannot be forgot when adding new target states. -External Monitoring -^^^^^^^^^^^^^^^^^^^ +@@@ + +#### External Monitoring External actors may be registered to be notified of state transitions by -sending a message :class:`SubscribeTransitionCallBack(actorRef)`. The named -actor will be sent a :class:`CurrentState(self, stateName)` message immediately -and will receive :class:`Transition(actorRef, oldState, newState)` messages +sending a message `SubscribeTransitionCallBack(actorRef)`. The named +actor will be sent a `CurrentState(self, stateName)` message immediately +and will receive `Transition(actorRef, oldState, newState)` messages whenever a new state is reached. External monitors may be unregistered by -sending :class:`UnsubscribeTransitionCallBack(actorRef)` to the FSM actor. +sending `UnsubscribeTransitionCallBack(actorRef)` to the FSM actor. Stopping a listener without unregistering will not remove the listener from the -subscription list; use :class:`UnsubscribeTransitionCallback` before stopping +subscription list; use `UnsubscribeTransitionCallback` before stopping the listener. -Timers ------- +### Timers -Besides state timeouts, FSM manages timers identified by :class:`String` names. +Besides state timeouts, FSM manages timers identified by `String` names. You may set a timer using - :func:`setTimer(name, msg, interval, repeat)` +> +`setTimer(name, msg, interval, repeat)` -where :obj:`msg` is the message object which will be sent after the duration -:obj:`interval` has elapsed. If :obj:`repeat` is :obj:`true`, then the timer is -scheduled at fixed rate given by the :obj:`interval` parameter. +where `msg` is the message object which will be sent after the duration +`interval` has elapsed. If `repeat` is `true`, then the timer is +scheduled at fixed rate given by the `interval` parameter. Any existing timer with the same name will automatically be canceled before adding the new timer. Timers may be canceled using - :func:`cancelTimer(name)` +> +`cancelTimer(name)` which is guaranteed to work immediately, meaning that the scheduled message will not be processed after this call even if the timer already fired and queued it. The status of any timer may be inquired with - :func:`isTimerActive(name)` +> +`isTimerActive(name)` These named timers complement state timeouts because they are not affected by intervening reception of other messages. -Termination from Inside ------------------------ +### Termination from Inside The FSM is stopped by specifying the result state as - :func:`stop([reason[, data]])` +> +`stop([reason[, data]])` -The reason must be one of :obj:`Normal` (which is the default), :obj:`Shutdown` -or :obj:`Failure(reason)`, and the second argument may be given to change the +The reason must be one of `Normal` (which is the default), `Shutdown` +or `Failure(reason)`, and the second argument may be given to change the state data which is available during termination handling. -.. note:: +@@@ note - It should be noted that :func:`stop` does not abort the actions and stop the - FSM immediately. The stop action must be returned from the event handler in - the same way as a state transition (but note that the ``return`` statement - may not be used within a :meth:`when` block). +It should be noted that `stop` does not abort the actions and stop the +FSM immediately. The stop action must be returned from the event handler in +the same way as a state transition (but note that the `return` statement +may not be used within a `when` block). -.. includecode:: code/jdocs/actor/fsm/FSMDocTest.java - :include: stop-syntax +@@@ -You can use :func:`onTermination(handler)` to specify custom code that is +@@snip [FSMDocTest.java](code/jdocs/actor/fsm/FSMDocTest.java) { #stop-syntax } + +You can use `onTermination(handler)` to specify custom code that is executed when the FSM is stopped. The handler is a partial function which takes -a :class:`StopEvent(reason, stateName, stateData)` as argument: +a `StopEvent(reason, stateName, stateData)` as argument: -.. includecode:: code/jdocs/actor/fsm/FSMDocTest.java - :include: termination-syntax +@@snip [FSMDocTest.java](code/jdocs/actor/fsm/FSMDocTest.java) { #termination-syntax } -As for the :func:`whenUnhandled` case, this handler is not stacked, so each -invocation of :func:`onTermination` replaces the previously installed handler. +As for the `whenUnhandled` case, this handler is not stacked, so each +invocation of `onTermination` replaces the previously installed handler. -Termination from Outside ------------------------- +### Termination from Outside -When an :class:`ActorRef` associated to a FSM is stopped using the -:meth:`stop()` method, its :meth:`postStop` hook will be executed. The default -implementation by the :class:`AbstractFSM` class is to execute the -:meth:`onTermination` handler if that is prepared to handle a -:obj:`StopEvent(Shutdown, ...)`. +When an `ActorRef` associated to a FSM is stopped using the +`stop()` method, its `postStop` hook will be executed. The default +implementation by the `AbstractFSM` class is to execute the +`onTermination` handler if that is prepared to handle a +`StopEvent(Shutdown, ...)`. -.. warning:: +@@@ warning - In case you override :meth:`postStop` and want to have your - :meth:`onTermination` handler called, do not forget to call - ``super.postStop``. +In case you override `postStop` and want to have your +`onTermination` handler called, do not forget to call +`super.postStop`. -Testing and Debugging Finite State Machines -=========================================== +@@@ + +## Testing and Debugging Finite State Machines During development and for trouble shooting FSMs need care just as any other -actor. There are specialized tools available as described in :ref:`TestFSMRef` +actor. There are specialized tools available as described in @ref:[TestFSMRef](../scala/testing.md#testfsmref) and in the following. -Event Tracing -------------- +### Event Tracing -The setting ``akka.actor.debug.fsm`` in :ref:`configuration` enables logging of an -event trace by :class:`LoggingFSM` instances: +The setting `akka.actor.debug.fsm` in configuration enables logging of an +event trace by `LoggingFSM` instances: -.. includecode:: code/jdocs/actor/fsm/FSMDocTest.java - :include: logging-fsm - :exclude: body-elided +@@snip [FSMDocTest.java](code/jdocs/actor/fsm/FSMDocTest.java) { #logging-fsm } This FSM will log at DEBUG level: - * all processed events, including :obj:`StateTimeout` and scheduled timer - messages - * every setting and cancellation of named timers - * all state transitions +> + * all processed events, including `StateTimeout` and scheduled timer +messages + * every setting and cancellation of named timers + * all state transitions Life cycle changes and special messages can be logged as described for -:ref:`Actors `. +@ref:[Actors](../scala/testing.md#actor-logging-scala). -Rolling Event Log ------------------ +### Rolling Event Log -The :class:`AbstractLoggingFSM` class adds one more feature to the FSM: a rolling event +The `AbstractLoggingFSM` class adds one more feature to the FSM: a rolling event log which may be used during debugging (for tracing how the FSM entered a certain failure state) or for other creative uses: -.. includecode:: code/jdocs/actor/fsm/FSMDocTest.java - :include: logging-fsm +@@snip [FSMDocTest.java](code/jdocs/actor/fsm/FSMDocTest.java) { #logging-fsm } -The :meth:`logDepth` defaults to zero, which turns off the event log. +The `logDepth` defaults to zero, which turns off the event log. -.. warning:: +@@@ warning - The log buffer is allocated during actor creation, which is why the - configuration is done using a virtual method call. If you want to override - with a ``val``, make sure that its initialization happens before the - initializer of :class:`LoggingFSM` runs, and do not change the value returned - by ``logDepth`` after the buffer has been allocated. +The log buffer is allocated during actor creation, which is why the +configuration is done using a virtual method call. If you want to override +with a `val`, make sure that its initialization happens before the +initializer of `LoggingFSM` runs, and do not change the value returned +by `logDepth` after the buffer has been allocated. -The contents of the event log are available using method :meth:`getLog`, which -returns an :class:`IndexedSeq[LogEntry]` where the oldest entry is at index +@@@ + +The contents of the event log are available using method `getLog`, which +returns an `IndexedSeq[LogEntry]` where the oldest entry is at index zero. -Examples -======== +## Examples -A bigger FSM example contrasted with Actor's :meth:`become`/:meth:`unbecome` can be -downloaded as a ready to run `Akka FSM sample <@exampleCodeService@/akka-samples-fsm-java>`_ +A bigger FSM example contrasted with Actor's `become`/`unbecome` can be +downloaded as a ready to run [Akka FSM sample](@exampleCodeService@/akka-samples-fsm-java) together with a tutorial. The source code of this sample can be found in the -`Akka Samples Repository <@samples@/akka-sample-fsm-java>`_. +[Akka Samples Repository](@samples@/akka-sample-fsm-java). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/futures.md b/akka-docs/src/main/paradox/java/futures.md index 59c64a2780..ba51fbc118 100644 --- a/akka-docs/src/main/paradox/java/futures.md +++ b/akka-docs/src/main/paradox/java/futures.md @@ -1,347 +1,296 @@ -.. _futures-java: +# Futures -Futures -=============== +## Introduction -Introduction ------------- - -In the Scala Standard Library, a `Future `_ is a data structure +In the Scala Standard Library, a [Future](http://en.wikipedia.org/wiki/Futures_and_promises) is a data structure used to retrieve the result of some concurrent operation. This result can be accessed synchronously (blocking) or asynchronously (non-blocking). To be able to use this from Java, Akka provides a java friendly interface -in ``akka.dispatch.Futures``. +in `akka.dispatch.Futures`. -See also :ref:`scala-java-compat` for Java compatibility. +See also @ref:[Java 8 and Scala Compatibility](scala-compat.md) for Java compatibility. -Execution Contexts ------------------- +## Execution Contexts -In order to execute callbacks and operations, Futures need something called an ``ExecutionContext``, -which is very similar to a ``java.util.concurrent.Executor``. if you have an ``ActorSystem`` in scope, -it will use its default dispatcher as the ``ExecutionContext``, or you can use the factory methods provided -by the ``ExecutionContexts`` class to wrap ``Executors`` and ``ExecutorServices``, or even create your own. +In order to execute callbacks and operations, Futures need something called an `ExecutionContext`, +which is very similar to a `java.util.concurrent.Executor`. if you have an `ActorSystem` in scope, +it will use its default dispatcher as the `ExecutionContext`, or you can use the factory methods provided +by the `ExecutionContexts` class to wrap `Executors` and `ExecutorServices`, or even create your own. -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: imports1 +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #imports1 } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: diy-execution-context +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #diy-execution-context } -Use with Actors ---------------- +## Use with Actors -There are generally two ways of getting a reply from an ``AbstractActor``: the first is by a sent message (``actorRef.tell(msg, sender)``), -which only works if the original sender was an ``AbstractActor``) and the second is through a ``Future``. +There are generally two ways of getting a reply from an `AbstractActor`: the first is by a sent message (`actorRef.tell(msg, sender)`), +which only works if the original sender was an `AbstractActor`) and the second is through a `Future`. -Using the ``ActorRef``\'s ``ask`` method to send a message will return a ``Future``. +Using the `ActorRef`'s `ask` method to send a message will return a `Future`. To wait for and retrieve the actual result the simplest method is: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: imports1 +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #imports1 } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: ask-blocking +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #ask-blocking } -This will cause the current thread to block and wait for the ``AbstractActor`` to 'complete' the ``Future`` with it's reply. +This will cause the current thread to block and wait for the `AbstractActor` to 'complete' the `Future` with it's reply. Blocking is discouraged though as it can cause performance problem. -The blocking operations are located in ``Await.result`` and ``Await.ready`` to make it easy to spot where blocking occurs. +The blocking operations are located in `Await.result` and `Await.ready` to make it easy to spot where blocking occurs. Alternatives to blocking are discussed further within this documentation. -Also note that the ``Future`` returned by an ``AbstractActor`` is a ``Future`` since an ``AbstractActor`` is dynamic. -That is why the cast to ``String`` is used in the above sample. +Also note that the `Future` returned by an `AbstractActor` is a `Future` since an `AbstractActor` is dynamic. +That is why the cast to `String` is used in the above sample. -.. warning:: +@@@ warning - ``Await.result`` and ``Await.ready`` are provided for exceptional situations where you **must** block, - a good rule of thumb is to only use them if you know why you **must** block. For all other cases, use - asynchronous composition as described below. +`Await.result` and `Await.ready` are provided for exceptional situations where you **must** block, +a good rule of thumb is to only use them if you know why you **must** block. For all other cases, use +asynchronous composition as described below. -To send the result of a ``Future`` to an ``Actor``, you can use the ``pipe`` construct: +@@@ -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: pipe-to +To send the result of a `Future` to an `Actor`, you can use the `pipe` construct: -Use Directly ------------- +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #pipe-to } + +## Use Directly A common use case within Akka is to have some computation performed concurrently without needing -the extra utility of an ``AbstractActor``. If you find yourself creating a pool of ``AbstractActor``\s for the sole reason +the extra utility of an `AbstractActor`. If you find yourself creating a pool of `AbstractActor`s for the sole reason of performing a calculation in parallel, there is an easier (and faster) way: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: imports2 +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #imports2 } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: future-eval +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #future-eval } -In the above code the block passed to ``future`` will be executed by the default ``Dispatcher``, -with the return value of the block used to complete the ``Future`` (in this case, the result would be the string: "HelloWorld"). -Unlike a ``Future`` that is returned from an ``AbstractActor``, this ``Future`` is properly typed, -and we also avoid the overhead of managing an ``AbstractActor``. +In the above code the block passed to `future` will be executed by the default `Dispatcher`, +with the return value of the block used to complete the `Future` (in this case, the result would be the string: "HelloWorld"). +Unlike a `Future` that is returned from an `AbstractActor`, this `Future` is properly typed, +and we also avoid the overhead of managing an `AbstractActor`. -You can also create already completed Futures using the ``Futures`` class, which can be either successes: +You can also create already completed Futures using the `Futures` class, which can be either successes: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: successful +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #successful } Or failures: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: failed +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #failed } -It is also possible to create an empty ``Promise``, to be filled later, and obtain the corresponding ``Future``: +It is also possible to create an empty `Promise`, to be filled later, and obtain the corresponding `Future`: -.. includecode:: code/jdocs/future/FutureDocTest.java#promise +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #promise } -For these examples ``PrintResult`` is defined as follows: +For these examples `PrintResult` is defined as follows: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: print-result +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #print-result } -Functional Futures ------------------- +## Functional Futures -Scala's ``Future`` has several monadic methods that are very similar to the ones used by ``Scala``'s collections. +Scala's `Future` has several monadic methods that are very similar to the ones used by `Scala`'s collections. These allow you to create 'pipelines' or 'streams' that the result will travel through. -Future is a Monad -^^^^^^^^^^^^^^^^^ +### Future is a Monad -The first method for working with ``Future`` functionally is ``map``. This method takes a ``Mapper`` which performs -some operation on the result of the ``Future``, and returning a new result. -The return value of the ``map`` method is another ``Future`` that will contain the new result: +The first method for working with `Future` functionally is `map`. This method takes a `Mapper` which performs +some operation on the result of the `Future`, and returning a new result. +The return value of the `map` method is another `Future` that will contain the new result: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: imports2 +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #imports2 } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: map +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #map } -In this example we are joining two strings together within a ``Future``. Instead of waiting for f1 to complete, -we apply our function that calculates the length of the string using the ``map`` method. -Now we have a second ``Future``, f2, that will eventually contain an ``Integer``. -When our original ``Future``, f1, completes, it will also apply our function and complete the second ``Future`` -with its result. When we finally ``get`` the result, it will contain the number 10. -Our original ``Future`` still contains the string "HelloWorld" and is unaffected by the ``map``. +In this example we are joining two strings together within a `Future`. Instead of waiting for f1 to complete, +we apply our function that calculates the length of the string using the `map` method. +Now we have a second `Future`, f2, that will eventually contain an `Integer`. +When our original `Future`, f1, completes, it will also apply our function and complete the second `Future` +with its result. When we finally `get` the result, it will contain the number 10. +Our original `Future` still contains the string "HelloWorld" and is unaffected by the `map`. -Something to note when using these methods: passed work is always dispatched on the provided ``ExecutionContext``. Even if -the ``Future`` has already been completed, when one of these methods is called. +Something to note when using these methods: passed work is always dispatched on the provided `ExecutionContext`. Even if +the `Future` has already been completed, when one of these methods is called. -Composing Futures -^^^^^^^^^^^^^^^^^ +### Composing Futures It is very often desirable to be able to combine different Futures with each other, below are some examples on how that can be done in a non-blocking fashion. -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: imports3 +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #imports3 } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: sequence +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #sequence } -To better explain what happened in the example, ``Future.sequence`` is taking the ``Iterable>`` -and turning it into a ``Future>``. We can then use ``map`` to work with the ``Iterable`` directly, -and we aggregate the sum of the ``Iterable``. +To better explain what happened in the example, `Future.sequence` is taking the `Iterable>` +and turning it into a `Future>`. We can then use `map` to work with the `Iterable` directly, +and we aggregate the sum of the `Iterable`. -The ``traverse`` method is similar to ``sequence``, but it takes a sequence of ``A`` and applies a function from ``A`` to ``Future`` -and returns a ``Future>``, enabling parallel ``map`` over the sequence, if you use ``Futures.future`` to create the ``Future``. +The `traverse` method is similar to `sequence`, but it takes a sequence of `A` and applies a function from `A` to `Future` +and returns a `Future>`, enabling parallel `map` over the sequence, if you use `Futures.future` to create the `Future`. -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: imports4 +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #imports4 } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: traverse +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #traverse } It's as simple as that! -Then there's a method that's called ``fold`` that takes a start-value, -a sequence of ``Future``:s and a function from the type of the start-value, a timeout, +Then there's a method that's called `fold` that takes a start-value, +a sequence of `Future`:s and a function from the type of the start-value, a timeout, and the type of the futures and returns something with the same type as the start-value, and then applies the function to all elements in the sequence of futures, non-blockingly, the execution will be started when the last of the Futures is completed. -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: imports5 +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #imports5 } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: fold +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #fold } That's all it takes! +If the sequence passed to `fold` is empty, it will return the start-value, in the case above, that will be empty String. +In some cases you don't have a start-value and you're able to use the value of the first completing `Future` +in the sequence as the start-value, you can use `reduce`, it works like this: -If the sequence passed to ``fold`` is empty, it will return the start-value, in the case above, that will be empty String. -In some cases you don't have a start-value and you're able to use the value of the first completing ``Future`` -in the sequence as the start-value, you can use ``reduce``, it works like this: +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #imports6 } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: imports6 +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #reduce } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: reduce - -Same as with ``fold``, the execution will be started when the last of the Futures is completed, you can also parallelize +Same as with `fold`, the execution will be started when the last of the Futures is completed, you can also parallelize it by chunking your futures into sub-sequences and reduce them, and then reduce the reduced results again. This is just a sample of what can be done. -Callbacks ---------- +## Callbacks -Sometimes you just want to listen to a ``Future`` being completed, and react to that not by creating a new Future, but by side-effecting. -For this Scala supports ``onComplete``, ``onSuccess`` and ``onFailure``, of which the last two are specializations of the first. +Sometimes you just want to listen to a `Future` being completed, and react to that not by creating a new Future, but by side-effecting. +For this Scala supports `onComplete`, `onSuccess` and `onFailure`, of which the last two are specializations of the first. -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: onSuccess +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #onSuccess } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: onFailure +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #onFailure } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: onComplete +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #onComplete } -Ordering --------- +## Ordering Since callbacks are executed in any order and potentially in parallel, it can be tricky at the times when you need sequential ordering of operations. -But there's a solution! And it's name is ``andThen``, and it creates a new ``Future`` with -the specified callback, a ``Future`` that will have the same result as the ``Future`` it's called on, +But there's a solution! And it's name is `andThen`, and it creates a new `Future` with +the specified callback, a `Future` that will have the same result as the `Future` it's called on, which allows for ordering like in the following sample: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: and-then +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #and-then } -Auxiliary methods ------------------ +## Auxiliary methods -``Future`` ``fallbackTo`` combines 2 Futures into a new ``Future``, and will hold the successful value of the second ``Future`` -if the first ``Future`` fails. +`Future` `fallbackTo` combines 2 Futures into a new `Future`, and will hold the successful value of the second `Future` +if the first `Future` fails. -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: fallback-to +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #fallback-to } -You can also combine two Futures into a new ``Future`` that will hold a tuple of the two Futures successful results, -using the ``zip`` operation. +You can also combine two Futures into a new `Future` that will hold a tuple of the two Futures successful results, +using the `zip` operation. -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: zip +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #zip } -Exceptions ----------- +## Exceptions -Since the result of a ``Future`` is created concurrently to the rest of the program, exceptions must be handled differently. -It doesn't matter if an ``AbstractActor`` or the dispatcher is completing the ``Future``, if an ``Exception`` is caught -the ``Future`` will contain it instead of a valid result. If a ``Future`` does contain an ``Exception``, -calling ``Await.result`` will cause it to be thrown again so it can be handled properly. +Since the result of a `Future` is created concurrently to the rest of the program, exceptions must be handled differently. +It doesn't matter if an `AbstractActor` or the dispatcher is completing the `Future`, if an `Exception` is caught +the `Future` will contain it instead of a valid result. If a `Future` does contain an `Exception`, +calling `Await.result` will cause it to be thrown again so it can be handled properly. -It is also possible to handle an ``Exception`` by returning a different result. -This is done with the ``recover`` method. For example: +It is also possible to handle an `Exception` by returning a different result. +This is done with the `recover` method. For example: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: recover +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #recover } -In this example, if the actor replied with a ``akka.actor.Status.Failure`` containing the ``ArithmeticException``, -our ``Future`` would have a result of 0. The ``recover`` method works very similarly to the standard try/catch blocks, -so multiple ``Exception``\s can be handled in this manner, and if an ``Exception`` is not handled this way -it will behave as if we hadn't used the ``recover`` method. +In this example, if the actor replied with a `akka.actor.Status.Failure` containing the `ArithmeticException`, +our `Future` would have a result of 0. The `recover` method works very similarly to the standard try/catch blocks, +so multiple `Exception`s can be handled in this manner, and if an `Exception` is not handled this way +it will behave as if we hadn't used the `recover` method. -You can also use the ``recoverWith`` method, which has the same relationship to ``recover`` as ``flatMap`` has to ``map``, +You can also use the `recoverWith` method, which has the same relationship to `recover` as `flatMap` has to `map`, and is use like this: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: try-recover +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #try-recover } -After ------ +## After -``akka.pattern.Patterns.after`` makes it easy to complete a ``Future`` with a value or exception after a timeout. +`akka.pattern.Patterns.after` makes it easy to complete a `Future` with a value or exception after a timeout. -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: imports7 +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #imports7 } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: after +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #after } -Java 8, CompletionStage and CompletableFuture ---------------------------------------------- +## Java 8, CompletionStage and CompletableFuture -Starting with Akka 2.4.2 we have begun to introduce Java 8 ``java.util.concurrent.CompletionStage`` in Java APIs. -It's a ``scala.concurrent.Future`` counterpart in Java; conversion from ``scala.concurrent.Future`` is done using -``scala-java8-compat`` library. +Starting with Akka 2.4.2 we have begun to introduce Java 8 `java.util.concurrent.CompletionStage` in Java APIs. +It's a `scala.concurrent.Future` counterpart in Java; conversion from `scala.concurrent.Future` is done using +`scala-java8-compat` library. -Unlike ``scala.concurrent.Future`` which has async methods only, ``CompletionStage`` has *async* and *non-async* methods. +Unlike `scala.concurrent.Future` which has async methods only, `CompletionStage` has *async* and *non-async* methods. -The ``scala-java8-compat`` library returns its own implementation of ``CompletionStage`` which delegates all *non-async* -methods to their *async* counterparts. The implementation extends standard Java ``CompletableFuture``. -Java 8 ``CompletableFuture`` creates a new instance of ``CompletableFuture`` for any new stage, -which means ``scala-java8-compat`` implementation is not used after the first mapping method. +The `scala-java8-compat` library returns its own implementation of `CompletionStage` which delegates all *non-async* +methods to their *async* counterparts. The implementation extends standard Java `CompletableFuture`. +Java 8 `CompletableFuture` creates a new instance of `CompletableFuture` for any new stage, +which means `scala-java8-compat` implementation is not used after the first mapping method. -.. note:: - After adding any additional computation stage to ``CompletionStage`` returned by ``scala-java8-compat`` - (e.g. ``CompletionStage`` instances returned by Akka) it falls back to standard behaviour of Java ``CompletableFuture``. +@@@ note + +After adding any additional computation stage to `CompletionStage` returned by `scala-java8-compat` +(e.g. `CompletionStage` instances returned by Akka) it falls back to standard behaviour of Java `CompletableFuture`. + +@@@ Actions supplied for dependent completions of *non-async* methods may be performed by the thread -that completes the current ``CompletableFuture``, or by any other caller of a completion method. +that completes the current `CompletableFuture`, or by any other caller of a completion method. -All *async* methods without an explicit Executor are performed using the ``ForkJoinPool.commonPool()`` executor. +All *async* methods without an explicit Executor are performed using the `ForkJoinPool.commonPool()` executor. -Non-async methods -^^^^^^^^^^^^^^^^^ +### Non-async methods -When non-async methods are applied on a not yet completed ``CompletionStage``, they are completed by -the thread which completes initial ``CompletionStage``: +When non-async methods are applied on a not yet completed `CompletionStage`, they are completed by +the thread which completes initial `CompletionStage`: -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: apply-completion-thread +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #apply-completion-thread } -In this example Scala ``Future`` is converted to ``CompletionStage`` just like Akka does. -The completion is delayed: we are calling ``thenApply`` multiple times on a not yet complete ``CompletionStage``, then -complete the ``Future``. +In this example Scala `Future` is converted to `CompletionStage` just like Akka does. +The completion is delayed: we are calling `thenApply` multiple times on a not yet complete `CompletionStage`, then +complete the `Future`. -First ``thenApply`` is actually performed on ``scala-java8-compat`` instance and computational stage (lambda) execution -is delegated to default Java ``thenApplyAsync`` which is executed on ``ForkJoinPool.commonPool()``. +First `thenApply` is actually performed on `scala-java8-compat` instance and computational stage (lambda) execution +is delegated to default Java `thenApplyAsync` which is executed on `ForkJoinPool.commonPool()`. -Second and third ``thenApply`` methods are executed on Java 8 ``CompletableFuture`` instance which executes computational -stages on the thread which completed the first stage. It is never executed on a thread of Scala ``Future`` because -default ``thenApply`` breaks the chain and executes on ``ForkJoinPool.commonPool()``. +Second and third `thenApply` methods are executed on Java 8 `CompletableFuture` instance which executes computational +stages on the thread which completed the first stage. It is never executed on a thread of Scala `Future` because +default `thenApply` breaks the chain and executes on `ForkJoinPool.commonPool()`. +In the next example `thenApply` methods are executed on an already completed `Future`/`CompletionStage`: -In the next example ``thenApply`` methods are executed on an already completed ``Future``/``CompletionStage``: +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #apply-main-thread } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: apply-main-thread - -First ``thenApply`` is still executed on ``ForkJoinPool.commonPool()`` (because it is actually ``thenApplyAsync`` +First `thenApply` is still executed on `ForkJoinPool.commonPool()` (because it is actually `thenApplyAsync` which is always executed on global Java pool). -Then we wait for stages to complete so second and third ``thenApply`` are executed on completed ``CompletionStage``, -and stages are executed on the current thread - the thread which called second and third ``thenApply``. +Then we wait for stages to complete so second and third `thenApply` are executed on completed `CompletionStage`, +and stages are executed on the current thread - the thread which called second and third `thenApply`. +### Async methods -Async methods -^^^^^^^^^^^^^ +As mentioned above, default *async* methods are always executed on `ForkJoinPool.commonPool()`: -As mentioned above, default *async* methods are always executed on ``ForkJoinPool.commonPool()``: +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #apply-async-default } -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: apply-async-default +`CompletionStage` also has *async* methods which take `Executor` as a second parameter, just like `Future`: +@@snip [FutureDocTest.java](code/jdocs/future/FutureDocTest.java) { #apply-async-executor } -``CompletionStage`` also has *async* methods which take ``Executor`` as a second parameter, just like ``Future``: +This example is behaving like `Future`: every stage is executed on an explicitly specified `Executor`. -.. includecode:: code/jdocs/future/FutureDocTest.java - :include: apply-async-executor +@@@ note -This example is behaving like ``Future``: every stage is executed on an explicitly specified ``Executor``. +When in doubt, async methods with explicit executor should be used. Always async methods with a dedicated +executor/dispatcher for long-running or blocking computations, such as IO operations. -.. note:: - When in doubt, async methods with explicit executor should be used. Always async methods with a dedicated - executor/dispatcher for long-running or blocking computations, such as IO operations. +@@@ See also: -- `CompletionStage `_ - -- `CompletableFuture `_ - -- `scala-java8-compat `_ - + * [CompletionStage](https://docs.oracle.com/javase/8/jdocs/api/java/util/concurrent/CompletionStage.html) + * [CompletableFuture](https://docs.oracle.com/javase/8/jdocs/api/java/util/concurrent/CompletableFuture.html) + * [scala-java8-compat](https://github.com/scala/scala-java8-compat) \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/hello-world.md b/akka-docs/src/main/paradox/java/hello-world.md index 0e00188127..cf2d40d7b9 100644 --- a/akka-docs/src/main/paradox/java/hello-world.md +++ b/akka-docs/src/main/paradox/java/hello-world.md @@ -1,22 +1,21 @@ -########################## -The Obligatory Hello World -########################## +# The Obligatory Hello World The actor based version of the tough problem of printing a -well-known greeting to the console is introduced in a ready to run `Akka Main sample <@exampleCodeService@/akka-samples-main-java>`_ +well-known greeting to the console is introduced in a ready to run [Akka Main sample](@exampleCodeService@/akka-samples-main-java) together with a tutorial. The source code of this sample can be found in the -`Akka Samples Repository <@samples@/akka-sample-main-java>`_. +[Akka Samples Repository](@samples@/akka-sample-main-java). -The tutorial illustrates the generic launcher class :class:`akka.Main` which expects only +The tutorial illustrates the generic launcher class `akka.Main` which expects only one command line argument: the class name of the application’s main actor. This main method will then create the infrastructure needed for running the actors, start the given main actor and arrange for the whole application to shut down once the main actor terminates. -There is also a `Gitter8 `_ template in the same problem domain -that is named `Hello Akka! `_. -It describes the basics of Akka in more depth. If you have `sbt` already installed, you can create a project -from this template by running:: - - sbt new akka/hello-akka.g8 +There is also a [Gitter8](http://www.foundweekends.org/giter8/) template in the same problem domain +that is named [Hello Akka!](https://github.com/akka/hello-akka.g8). +It describes the basics of Akka in more depth. If you have *sbt* already installed, you can create a project +from this template by running: +``` +sbt new akka/hello-akka.g8 +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/howto.md b/akka-docs/src/main/paradox/java/howto.md index 1a0eb93870..d1105b43e4 100644 --- a/akka-docs/src/main/paradox/java/howto.md +++ b/akka-docs/src/main/paradox/java/howto.md @@ -1,8 +1,4 @@ -.. _howto-java: - -###################### -HowTo: Common Patterns -###################### +# HowTo: Common Patterns This section lists common actor patterns which have been found to be useful, elegant or instructive. Anything is welcome, example topics being message @@ -10,45 +6,46 @@ routing strategies, supervision patterns, restart handling, etc. As a special bonus, additions to this section are marked with the contributor’s name, and it would be nice if every Akka user who finds a recurring pattern in his or her code could share it for the profit of all. Where applicable it might also make -sense to add to the ``akka.pattern`` package for creating an `OTP-like library -`_. +sense to add to the `akka.pattern` package for creating an [OTP-like library](http://www.erlang.org/doc/man_index.html). You might find some of the patterns described in the Scala chapter of -:ref:`howto-scala` useful even though the example code is written in Scala. +@ref:[HowTo: Common Patterns](../scala/howto.md) useful even though the example code is written in Scala. -Scheduling Periodic Messages -============================ +## Scheduling Periodic Messages This pattern describes how to schedule periodic messages to yourself in two different ways. The first way is to set up periodic message scheduling in the constructor of the actor, -and cancel that scheduled sending in ``postStop`` or else we might have multiple registered +and cancel that scheduled sending in `postStop` or else we might have multiple registered message sends to the same actor. -.. note:: +@@@ note - With this approach the scheduled periodic message send will be restarted with the actor on restarts. - This also means that the time period that elapses between two tick messages during a restart may drift - off based on when you restart the scheduled message sends relative to the time that the last message was - sent, and how long the initial delay is. Worst case scenario is ``interval`` plus ``initialDelay``. +With this approach the scheduled periodic message send will be restarted with the actor on restarts. +This also means that the time period that elapses between two tick messages during a restart may drift +off based on when you restart the scheduled message sends relative to the time that the last message was +sent, and how long the initial delay is. Worst case scenario is `interval` plus `initialDelay`. -.. includecode:: code/jdocs/pattern/SchedulerPatternTest.java#schedule-constructor +@@@ -The second variant sets up an initial one shot message send in the ``preStart`` method +@@snip [SchedulerPatternTest.java](code/jdocs/pattern/SchedulerPatternTest.java) { #schedule-constructor } + +The second variant sets up an initial one shot message send in the `preStart` method of the actor, and the then the actor when it receives this message sets up a new one shot -message send. You also have to override ``postRestart`` so we don't call ``preStart`` +message send. You also have to override `postRestart` so we don't call `preStart` and schedule the initial message send again. -.. note:: +@@@ note - With this approach we won't fill up the mailbox with tick messages if the actor is - under pressure, but only schedule a new tick message when we have seen the previous one. +With this approach we won't fill up the mailbox with tick messages if the actor is +under pressure, but only schedule a new tick message when we have seen the previous one. -.. includecode:: code/jdocs/pattern/SchedulerPatternTest.java#schedule-receive +@@@ -Single-Use Actor Trees with High-Level Error Reporting -====================================================== +@@snip [SchedulerPatternTest.java](code/jdocs/pattern/SchedulerPatternTest.java) { #schedule-receive } + +## Single-Use Actor Trees with High-Level Error Reporting *Contributed by: Rick Latrine* @@ -66,11 +63,11 @@ Such an actor is unlikely to be reused in a different actor hierarchy and contai This pattern provides a way to encapsulate supervision and error propagation to the temporary actor. Finally the promise returned by Patterns.ask() is fulfilled as a failure, including the exception -(see also :ref:`scala-java-compat` for Java compatibility). +(see also @ref:[Java 8 and Scala Compatibility](scala-compat.md) for Java compatibility). Let's have a look at the example code: -.. includecode:: code/jdocs/pattern/SupervisedAsk.java +@@snip [SupervisedAsk.java](code/jdocs/pattern/SupervisedAsk.java) { # } In the askOf method the SupervisorCreator is sent the user message. The SupervisorCreator creates a SupervisorActor and forwards the message. @@ -83,5 +80,4 @@ Afterwards the actor hierarchy is stopped. Finally we are able to execute an actor and receive the results or exceptions. -.. includecode:: code/jdocs/pattern/SupervisedAskSpec.java - +@@snip [SupervisedAskSpec.java](code/jdocs/pattern/SupervisedAskSpec.java) { # } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/http/index.md b/akka-docs/src/main/paradox/java/http/index.md index b8934cf2a9..2de51cb7a7 100644 --- a/akka-docs/src/main/paradox/java/http/index.md +++ b/akka-docs/src/main/paradox/java/http/index.md @@ -1,5 +1,4 @@ -Akka HTTP Documentation (Java) moved! -===================================== +# Akka HTTP Documentation (Java) moved! Akka HTTP has been released as independent stable module (from Akka HTTP 10.x onwards). -The documentation is available under `doc.akka.io/akka-http/current/ `_. +The documentation is available under [doc.akka.io/akka-http/current/](http://doc.akka.io/docs/akka-http/current/java.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/index-actors.md b/akka-docs/src/main/paradox/java/index-actors.md index 6f1f0a6895..2cfc6bb1ae 100644 --- a/akka-docs/src/main/paradox/java/index-actors.md +++ b/akka-docs/src/main/paradox/java/index-actors.md @@ -1,18 +1,20 @@ -Actors -====== +# Actors -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - actors - fault-tolerance - dispatchers - mailboxes - routing - fsm - persistence - persistence-schema-evolution - persistence-query - persistence-query-leveldb - testing - typed-actors +@@@ index + +* [actors](actors.md) +* [fault-tolerance](fault-tolerance.md) +* [dispatchers](dispatchers.md) +* [mailboxes](mailboxes.md) +* [routing](routing.md) +* [fsm](fsm.md) +* [persistence](persistence.md) +* [persistence-schema-evolution](persistence-schema-evolution.md) +* [persistence-query](persistence-query.md) +* [persistence-query-leveldb](persistence-query-leveldb.md) +* [testing](testing.md) +* [typed-actors](typed-actors.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/index-futures.md b/akka-docs/src/main/paradox/java/index-futures.md index 59b846d0d6..c0158c312f 100644 --- a/akka-docs/src/main/paradox/java/index-futures.md +++ b/akka-docs/src/main/paradox/java/index-futures.md @@ -1,8 +1,10 @@ -Futures and Agents -================== +# Futures and Agents -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - futures - agents +@@@ index + +* [futures](futures.md) +* [agents](agents.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/index-network.md b/akka-docs/src/main/paradox/java/index-network.md index 33ed035cc8..0c7ef30707 100644 --- a/akka-docs/src/main/paradox/java/index-network.md +++ b/akka-docs/src/main/paradox/java/index-network.md @@ -1,21 +1,23 @@ -Networking -========== +# Networking -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - ../common/cluster - cluster-usage - cluster-singleton - distributed-pub-sub - cluster-client - cluster-sharding - cluster-metrics - distributed-data - remoting - remoting-artery - serialization - io - io-tcp - io-udp - camel +@@@ index + +* [../common/cluster](../common/cluster.md) +* [cluster-usage](cluster-usage.md) +* [cluster-singleton](cluster-singleton.md) +* [distributed-pub-sub](distributed-pub-sub.md) +* [cluster-client](cluster-client.md) +* [cluster-sharding](cluster-sharding.md) +* [cluster-metrics](cluster-metrics.md) +* [distributed-data](distributed-data.md) +* [remoting](remoting.md) +* [remoting-artery](remoting-artery.md) +* [serialization](serialization.md) +* [io](io.md) +* [io-tcp](io-tcp.md) +* [io-udp](io-udp.md) +* [camel](camel.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/index-utilities.md b/akka-docs/src/main/paradox/java/index-utilities.md index 02e4ce5114..45fd046727 100644 --- a/akka-docs/src/main/paradox/java/index-utilities.md +++ b/akka-docs/src/main/paradox/java/index-utilities.md @@ -1,14 +1,15 @@ -Utilities -========= +# Utilities -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - event-bus - logging - scheduler - ../common/duration - ../common/circuitbreaker - extending-akka - ../intro/deployment-scenarios +@@@ index +* [event-bus](event-bus.md) +* [logging](logging.md) +* [scheduler](scheduler.md) +* [../common/duration](../common/duration.md) +* [../common/circuitbreaker](../common/circuitbreaker.md) +* [extending-akka](extending-akka.md) +* [../intro/deployment-scenarios](../intro/deployment-scenarios.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/io-tcp.md b/akka-docs/src/main/paradox/java/io-tcp.md index 17e3c856ec..87047e3e34 100644 --- a/akka-docs/src/main/paradox/java/io-tcp.md +++ b/akka-docs/src/main/paradox/java/io-tcp.md @@ -1,161 +1,154 @@ -.. _io-java-tcp: - -Using TCP -========= +# Using TCP The code snippets through-out this section assume the following imports: -.. includecode:: code/jdocs/io/japi/IODocTest.java#imports +@@snip [IODocTest.java](code/jdocs/io/japi/IODocTest.java) { #imports } All of the Akka I/O APIs are accessed through manager objects. When using an I/O API, the first step is to acquire a -reference to the appropriate manager. The code below shows how to acquire a reference to the ``Tcp`` manager. +reference to the appropriate manager. The code below shows how to acquire a reference to the `Tcp` manager. -.. includecode:: code/jdocs/io/japi/EchoManager.java#manager +@@snip [EchoManager.java](code/jdocs/io/japi/EchoManager.java) { #manager } The manager is an actor that handles the underlying low level I/O resources (selectors, channels) and instantiates workers for specific tasks, such as listening to incoming connections. -Connecting ----------- +## Connecting -.. includecode:: code/jdocs/io/japi/IODocTest.java#client +@@snip [IODocTest.java](code/jdocs/io/japi/IODocTest.java) { #client } -The first step of connecting to a remote address is sending a :class:`Connect` +The first step of connecting to a remote address is sending a `Connect` message to the TCP manager; in addition to the simplest form shown above there -is also the possibility to specify a local :class:`InetSocketAddress` to bind +is also the possibility to specify a local `InetSocketAddress` to bind to and a list of socket options to apply. -.. note:: +@@@ note - The SO_NODELAY (TCP_NODELAY on Windows) socket option defaults to true in - Akka, independently of the OS default settings. This setting disables Nagle's - algorithm, considerably improving latency for most applications. This setting - could be overridden by passing ``SO.TcpNoDelay(false)`` in the list of socket - options of the ``Connect`` message. +The SO_NODELAY (TCP_NODELAY on Windows) socket option defaults to true in +Akka, independently of the OS default settings. This setting disables Nagle's +algorithm, considerably improving latency for most applications. This setting +could be overridden by passing `SO.TcpNoDelay(false)` in the list of socket +options of the `Connect` message. -The TCP manager will then reply either with a :class:`CommandFailed` or it will +@@@ + +The TCP manager will then reply either with a `CommandFailed` or it will spawn an internal actor representing the new connection. This new actor will -then send a :class:`Connected` message to the original sender of the -:class:`Connect` message. +then send a `Connected` message to the original sender of the +`Connect` message. -In order to activate the new connection a :class:`Register` message must be +In order to activate the new connection a `Register` message must be sent to the connection actor, informing that one about who shall receive data from the socket. Before this step is done the connection cannot be used, and there is an internal timeout after which the connection actor will shut itself -down if no :class:`Register` message is received. +down if no `Register` message is received. The connection actor watches the registered handler and closes the connection when that one terminates, thereby cleaning up all internal resources associated with that connection. -The actor in the example above uses :meth:`become` to switch from unconnected +The actor in the example above uses `become` to switch from unconnected to connected operation, demonstrating the commands and events which are -observed in that state. For a discussion on :class:`CommandFailed` see -`Throttling Reads and Writes`_ below. :class:`ConnectionClosed` is a trait, +observed in that state. For a discussion on `CommandFailed` see +[Throttling Reads and Writes](#throttling-reads-and-writes) below. `ConnectionClosed` is a trait, which marks the different connection close events. The last line handles all connection close events in the same way. It is possible to listen for more -fine-grained connection close events, see `Closing Connections`_ below. +fine-grained connection close events, see [Closing Connections](#closing-connections) below. -Accepting connections ---------------------- +## Accepting connections -.. includecode:: code/jdocs/io/japi/IODocTest.java#server +@@snip [IODocTest.java](code/jdocs/io/japi/IODocTest.java) { #server } -To create a TCP server and listen for inbound connections, a :class:`Bind` +To create a TCP server and listen for inbound connections, a `Bind` command has to be sent to the TCP manager. This will instruct the TCP manager -to listen for TCP connections on a particular :class:`InetSocketAddress`; the -port may be specified as ``0`` in order to bind to a random port. +to listen for TCP connections on a particular `InetSocketAddress`; the +port may be specified as `0` in order to bind to a random port. -The actor sending the :class:`Bind` message will receive a :class:`Bound` +The actor sending the `Bind` message will receive a `Bound` message signaling that the server is ready to accept incoming connections; -this message also contains the :class:`InetSocketAddress` to which the socket +this message also contains the `InetSocketAddress` to which the socket was actually bound (i.e. resolved IP address and correct port number). From this point forward the process of handling connections is the same as for outgoing connections. The example demonstrates that handling the reads from a certain connection can be delegated to another actor by naming it as the -handler when sending the :class:`Register` message. Writes can be sent from any +handler when sending the `Register` message. Writes can be sent from any actor in the system to the connection actor (i.e. the actor which sent the -:class:`Connected` message). The simplistic handler is defined as: +`Connected` message). The simplistic handler is defined as: -.. includecode:: code/jdocs/io/japi/IODocTest.java#simplistic-handler +@@snip [IODocTest.java](code/jdocs/io/japi/IODocTest.java) { #simplistic-handler } For a more complete sample which also takes into account the possibility of -failures when sending please see `Throttling Reads and Writes`_ below. +failures when sending please see [Throttling Reads and Writes](#throttling-reads-and-writes) below. The only difference to outgoing connections is that the internal actor managing -the listen port—the sender of the :class:`Bound` message—watches the actor -which was named as the recipient for :class:`Connected` messages in the -:class:`Bind` message. When that actor terminates the listen port will be +the listen port—the sender of the `Bound` message—watches the actor +which was named as the recipient for `Connected` messages in the +`Bind` message. When that actor terminates the listen port will be closed and all resources associated with it will be released; existing connections will not be terminated at this point. -Closing connections -------------------- +## Closing connections -A connection can be closed by sending one of the commands ``Close``, ``ConfirmedClose`` or ``Abort`` to the connection +A connection can be closed by sending one of the commands `Close`, `ConfirmedClose` or `Abort` to the connection actor. -``Close`` will close the connection by sending a ``FIN`` message, but without waiting for confirmation from +`Close` will close the connection by sending a `FIN` message, but without waiting for confirmation from the remote endpoint. Pending writes will be flushed. If the close is successful, the listener will be notified with -``Closed``. +`Closed`. -``ConfirmedClose`` will close the sending direction of the connection by sending a ``FIN`` message, but data +`ConfirmedClose` will close the sending direction of the connection by sending a `FIN` message, but data will continue to be received until the remote endpoint closes the connection, too. Pending writes will be flushed. If the close is -successful, the listener will be notified with ``ConfirmedClosed``. +successful, the listener will be notified with `ConfirmedClosed`. -``Abort`` will immediately terminate the connection by sending a ``RST`` message to the remote endpoint. Pending -writes will be not flushed. If the close is successful, the listener will be notified with ``Aborted``. +`Abort` will immediately terminate the connection by sending a `RST` message to the remote endpoint. Pending +writes will be not flushed. If the close is successful, the listener will be notified with `Aborted`. -``PeerClosed`` will be sent to the listener if the connection has been closed by the remote endpoint. Per default, the +`PeerClosed` will be sent to the listener if the connection has been closed by the remote endpoint. Per default, the connection will then automatically be closed from this endpoint as well. To support half-closed connections set the -``keepOpenOnPeerClosed`` member of the ``Register`` message to ``true`` in which case the connection stays open until +`keepOpenOnPeerClosed` member of the `Register` message to `true` in which case the connection stays open until it receives one of the above close commands. -``ErrorClosed`` will be sent to the listener whenever an error happened that forced the connection to be closed. +`ErrorClosed` will be sent to the listener whenever an error happened that forced the connection to be closed. -All close notifications are sub-types of ``ConnectionClosed`` so listeners who do not need fine-grained close events +All close notifications are sub-types of `ConnectionClosed` so listeners who do not need fine-grained close events may handle all close events in the same way. -Writing to a connection ------------------------ +## Writing to a connection -Once a connection has been established data can be sent to it from any actor in the form of a ``Tcp.WriteCommand``. -``Tcp.WriteCommand`` is an abstract class with three concrete implementations: +Once a connection has been established data can be sent to it from any actor in the form of a `Tcp.WriteCommand`. +`Tcp.WriteCommand` is an abstract class with three concrete implementations: Tcp.Write - The simplest ``WriteCommand`` implementation which wraps a ``ByteString`` instance and an "ack" event. - A ``ByteString`` (as explained in :ref:`this section `) models one or more chunks of immutable - in-memory data with a maximum (total) size of 2 GB (2^31 bytes). +: The simplest `WriteCommand` implementation which wraps a `ByteString` instance and an "ack" event. +A `ByteString` (as explained in @ref:[this section](io.md#bytestring-java)) models one or more chunks of immutable +in-memory data with a maximum (total) size of 2 GB (2^31 bytes). Tcp.WriteFile - If you want to send "raw" data from a file you can do so efficiently with the ``Tcp.WriteFile`` command. - This allows you do designate a (contiguous) chunk of on-disk bytes for sending across the connection without - the need to first load them into the JVM memory. As such ``Tcp.WriteFile`` can "hold" more than 2GB of data and - an "ack" event if required. +: If you want to send "raw" data from a file you can do so efficiently with the `Tcp.WriteFile` command. +This allows you do designate a (contiguous) chunk of on-disk bytes for sending across the connection without +the need to first load them into the JVM memory. As such `Tcp.WriteFile` can "hold" more than 2GB of data and +an "ack" event if required. Tcp.CompoundWrite - Sometimes you might want to group (or interleave) several ``Tcp.Write`` and/or ``Tcp.WriteFile`` commands into - one atomic write command which gets written to the connection in one go. The ``Tcp.CompoundWrite`` allows you - to do just that and offers three benefits: +: +Sometimes you might want to group (or interleave) several `Tcp.Write` and/or `Tcp.WriteFile` commands into +one atomic write command which gets written to the connection in one go. The `Tcp.CompoundWrite` allows you +to do just that and offers three benefits: + 1. As explained in the following section the TCP connection actor can only handle one single write command at a time. +By combining several writes into one `CompoundWrite` you can have them be sent across the connection with +minimum overhead and without the need to spoon feed them to the connection actor via an *ACK-based* message +protocol. + 2. Because a `WriteCommand` is atomic you can be sure that no other actor can "inject" other writes into your +series of writes if you combine them into one single `CompoundWrite`. In scenarios where several actors write +to the same connection this can be an important feature which can be somewhat hard to achieve otherwise. + 3. The "sub writes" of a `CompoundWrite` are regular `Write` or `WriteFile` commands that themselves can request +"ack" events. These ACKs are sent out as soon as the respective "sub write" has been completed. This allows you to +attach more than one ACK to a `Write` or `WriteFile` (by combining it with an empty write that itself requests +an ACK) or to have the connection actor acknowledge the progress of transmitting the `CompoundWrite` by sending +out intermediate ACKs at arbitrary points. - 1. As explained in the following section the TCP connection actor can only handle one single write command at a time. - By combining several writes into one ``CompoundWrite`` you can have them be sent across the connection with - minimum overhead and without the need to spoon feed them to the connection actor via an *ACK-based* message - protocol. - 2. Because a ``WriteCommand`` is atomic you can be sure that no other actor can "inject" other writes into your - series of writes if you combine them into one single ``CompoundWrite``. In scenarios where several actors write - to the same connection this can be an important feature which can be somewhat hard to achieve otherwise. - - 3. The "sub writes" of a ``CompoundWrite`` are regular ``Write`` or ``WriteFile`` commands that themselves can request - "ack" events. These ACKs are sent out as soon as the respective "sub write" has been completed. This allows you to - attach more than one ACK to a ``Write`` or ``WriteFile`` (by combining it with an empty write that itself requests - an ACK) or to have the connection actor acknowledge the progress of transmitting the ``CompoundWrite`` by sending - out intermediate ACKs at arbitrary points. - -Throttling Reads and Writes ---------------------------- +## Throttling Reads and Writes The basic model of the TCP connection actor is that it has no internal buffering (i.e. it can only process one write at a time, meaning it can buffer @@ -164,84 +157,81 @@ needs to be handled at the user level, for both writes and reads. For back-pressuring writes there are three modes of operation -* *ACK-based:* every :class:`Write` command carries an arbitrary object, and if - this object is not ``Tcp.NoAck`` then it will be returned to the sender of - the :class:`Write` upon successfully writing all contained data to the - socket. If no other write is initiated before having received this - acknowledgement then no failures can happen due to buffer overrun. - -* *NACK-based:* every write which arrives while a previous write is not yet - completed will be replied to with a :class:`CommandFailed` message containing - the failed write. Just relying on this mechanism requires the implemented - protocol to tolerate skipping writes (e.g. if each write is a valid message - on its own and it is not required that all are delivered). This mode is - enabled by setting the ``useResumeWriting`` flag to ``false`` within the - :class:`Register` message during connection activation. - -* *NACK-based with write suspending:* this mode is very similar to the - NACK-based one, but once a single write has failed no further writes will - succeed until a :class:`ResumeWriting` message is received. This message will - be answered with a :class:`WritingResumed` message once the last accepted - write has completed. If the actor driving the connection implements buffering - and resends the NACK’ed messages after having awaited the - :class:`WritingResumed` signal then every message is delivered exactly once - to the network socket. + * *ACK-based:* every `Write` command carries an arbitrary object, and if +this object is not `Tcp.NoAck` then it will be returned to the sender of +the `Write` upon successfully writing all contained data to the +socket. If no other write is initiated before having received this +acknowledgement then no failures can happen due to buffer overrun. + * *NACK-based:* every write which arrives while a previous write is not yet +completed will be replied to with a `CommandFailed` message containing +the failed write. Just relying on this mechanism requires the implemented +protocol to tolerate skipping writes (e.g. if each write is a valid message +on its own and it is not required that all are delivered). This mode is +enabled by setting the `useResumeWriting` flag to `false` within the +`Register` message during connection activation. + * *NACK-based with write suspending:* this mode is very similar to the +NACK-based one, but once a single write has failed no further writes will +succeed until a `ResumeWriting` message is received. This message will +be answered with a `WritingResumed` message once the last accepted +write has completed. If the actor driving the connection implements buffering +and resends the NACK’ed messages after having awaited the +`WritingResumed` signal then every message is delivered exactly once +to the network socket. These write models (with the exception of the second which is rather specialised) are demonstrated in complete examples below. The full and contiguous source is -available `on GitHub <@github@/akka-docs/rst/java/code/jdocs/io/japi>`_. +available [on GitHub](@github@/akka-docs/rst/java/code/jdocs/io/japi). For back-pressuring reads there are two modes of operation -* *Push-reading:* in this mode the connection actor sends the registered reader actor - incoming data as soon as available as :class:`Received` events. Whenever the reader actor - wants to signal back-pressure to the remote TCP endpoint it can send a :class:`SuspendReading` - message to the connection actor to indicate that it wants to suspend the - reception of new data. No :class:`Received` events will arrive until a corresponding - :class:`ResumeReading` is sent indicating that the receiver actor is ready again. + * *Push-reading:* in this mode the connection actor sends the registered reader actor +incoming data as soon as available as `Received` events. Whenever the reader actor +wants to signal back-pressure to the remote TCP endpoint it can send a `SuspendReading` +message to the connection actor to indicate that it wants to suspend the +reception of new data. No `Received` events will arrive until a corresponding +`ResumeReading` is sent indicating that the receiver actor is ready again. + * *Pull-reading:* after sending a `Received` event the connection +actor automatically suspends accepting data from the socket until the reader actor signals +with a `ResumeReading` message that it is ready to process more input data. Hence +new data is "pulled" from the connection by sending `ResumeReading` messages. -* *Pull-reading:* after sending a :class:`Received` event the connection - actor automatically suspends accepting data from the socket until the reader actor signals - with a :class:`ResumeReading` message that it is ready to process more input data. Hence - new data is "pulled" from the connection by sending :class:`ResumeReading` messages. +@@@ note -.. note:: +It should be obvious that all these flow control schemes only work between +one writer/reader and one connection actor; as soon as multiple actors send write +commands to a single connection no consistent result can be achieved. - It should be obvious that all these flow control schemes only work between - one writer/reader and one connection actor; as soon as multiple actors send write - commands to a single connection no consistent result can be achieved. +@@@ -ACK-Based Write Back-Pressure ------------------------------ +## ACK-Based Write Back-Pressure For proper function of the following example it is important to configure the connection to remain half-open when the remote side closed its writing end: -this allows the example :class:`EchoHandler` to write all outstanding data back +this allows the example `EchoHandler` to write all outstanding data back to the client before fully closing the connection. This is enabled using a flag -upon connection activation (observe the :class:`Register` message): +upon connection activation (observe the `Register` message): -.. includecode:: code/jdocs/io/japi/EchoManager.java#echo-manager +@@snip [EchoManager.java](code/jdocs/io/japi/EchoManager.java) { #echo-manager } With this preparation let us dive into the handler itself: -.. includecode:: code/jdocs/io/japi/SimpleEchoHandler.java#simple-echo-handler - :exclude: storage-omitted +@@snip [SimpleEchoHandler.java](code/jdocs/io/japi/SimpleEchoHandler.java) { #simple-echo-handler } The principle is simple: when having written a chunk always wait for the -``Ack`` to come back before sending the next chunk. While waiting we switch +`Ack` to come back before sending the next chunk. While waiting we switch behavior such that new incoming data are buffered. The helper functions used are a bit lengthy but not complicated: -.. includecode:: code/jdocs/io/japi/SimpleEchoHandler.java#simple-helpers +@@snip [SimpleEchoHandler.java](code/jdocs/io/japi/SimpleEchoHandler.java) { #simple-helpers } -The most interesting part is probably the last: an ``Ack`` removes the oldest +The most interesting part is probably the last: an `Ack` removes the oldest data chunk from the buffer, and if that was the last chunk then we either close the connection (if the peer closed its half already) or return to the idle behavior; otherwise we just send the next buffered chunk and stay waiting for -the next ``Ack``. +the next `Ack`. Back-pressure can be propagated also across the reading side back to the writer -on the other end of the connection by sending the :class:`SuspendReading` +on the other end of the connection by sending the `SuspendReading` command to the connection actor. This will lead to no data being read from the socket anymore (although this does happen after a delay because it takes some time until the connection actor processes this command, hence appropriate @@ -251,43 +241,40 @@ the remote side from writing, filling up its write buffer, until finally the writer on the other side cannot push any data into the socket anymore. This is how end-to-end back-pressure is realized across a TCP connection. -NACK-Based Write Back-Pressure with Suspending ----------------------------------------------- +## NACK-Based Write Back-Pressure with Suspending -.. includecode:: code/jdocs/io/japi/EchoHandler.java#echo-handler - :exclude: buffering,closing,storage-omitted +@@snip [EchoHandler.java](code/jdocs/io/japi/EchoHandler.java) { #echo-handler } -The principle here is to keep writing until a :class:`CommandFailed` is +The principle here is to keep writing until a `CommandFailed` is received, using acknowledgements only to prune the resend buffer. When a such a failure was received, transition into a different state for handling and handle resending of all queued data: -.. includecode:: code/jdocs/io/japi/EchoHandler.java#buffering +@@snip [EchoHandler.java](code/jdocs/io/japi/EchoHandler.java) { #buffering } It should be noted that all writes which are currently buffered have also been sent to the connection actor upon entering this state, which means that the -:class:`ResumeWriting` message is enqueued after those writes, leading to the -reception of all outstanding :class:`CommandFailed` messages (which are ignored -in this state) before receiving the :class:`WritingResumed` signal. That latter +`ResumeWriting` message is enqueued after those writes, leading to the +reception of all outstanding `CommandFailed` messages (which are ignored +in this state) before receiving the `WritingResumed` signal. That latter message is sent by the connection actor only once the internally queued write has been fully completed, meaning that a subsequent write will not fail. This -is exploited by the :class:`EchoHandler` to switch to an ACK-based approach for +is exploited by the `EchoHandler` to switch to an ACK-based approach for the first ten writes after a failure before resuming the optimistic write-through behavior. -.. includecode:: code/jdocs/io/japi/EchoHandler.java#closing +@@snip [EchoHandler.java](code/jdocs/io/japi/EchoHandler.java) { #closing } Closing the connection while still sending all data is a bit more involved than in the ACK-based approach: the idea is to always send all outstanding messages and acknowledge all successful writes, and if a failure happens then switch -behavior to await the :class:`WritingResumed` event and start over. +behavior to await the `WritingResumed` event and start over. The helper functions are very similar to the ACK-based case: -.. includecode:: code/jdocs/io/japi/EchoHandler.java#helpers +@@snip [EchoHandler.java](code/jdocs/io/japi/EchoHandler.java) { #helpers } -Read Back-Pressure with Pull Mode ---------------------------------- +## Read Back-Pressure with Pull Mode When using push based reading, data coming from the socket is sent to the actor as soon as it is available. In the case of the previous Echo server example @@ -297,44 +284,43 @@ since the rate of writing might be slower than the rate of the arrival of new da With the Pull mode this buffer can be completely eliminated as the following snippet demonstrates: -.. includecode:: code/jdocs/io/JavaReadBackPressure.java#pull-reading-echo +@@snip [JavaReadBackPressure.java](code/jdocs/io/JavaReadBackPressure.java) { #pull-reading-echo } The idea here is that reading is not resumed until the previous write has been completely acknowledged by the connection actor. Every pull mode connection actor starts from suspended state. To start the flow of data we send a -``ResumeReading`` in the ``preStart`` method to tell the connection actor that +`ResumeReading` in the `preStart` method to tell the connection actor that we are ready to receive the first chunk of data. Since we only resume reading when the previous data chunk has been completely written there is no need for maintaining a buffer. -To enable pull reading on an outbound connection the ``pullMode`` parameter of -the :class:`Connect` should be set to ``true``: +To enable pull reading on an outbound connection the `pullMode` parameter of +the `Connect` should be set to `true`: -.. includecode:: code/jdocs/io/JavaReadBackPressure.java#pull-mode-connect +@@snip [JavaReadBackPressure.java](code/jdocs/io/JavaReadBackPressure.java) { #pull-mode-connect } -Pull Mode Reading for Inbound Connections -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Pull Mode Reading for Inbound Connections The previous section demonstrated how to enable pull reading mode for outbound connections but it is possible to create a listener actor with this mode of reading -by setting the ``pullMode`` parameter of the :class:`Bind` command to ``true``: +by setting the `pullMode` parameter of the `Bind` command to `true`: -.. includecode:: code/jdocs/io/JavaReadBackPressure.java#pull-mode-bind +@@snip [JavaReadBackPressure.java](code/jdocs/io/JavaReadBackPressure.java) { #pull-mode-bind } One of the effects of this setting is that all connections accepted by this listener actor will use pull mode reading. Another effect of this setting is that in addition of setting all inbound connections to pull mode, accepting connections becomes pull based, too. This means that after handling -one (or more) :class:`Connected` events the listener actor has to be resumed by sending -it a :class:`ResumeAccepting` message. +one (or more) `Connected` events the listener actor has to be resumed by sending +it a `ResumeAccepting` message. Listener actors with pull mode start suspended so to start accepting connections -a :class:`ResumeAccepting` command has to be sent to the listener actor after binding was successful: +a `ResumeAccepting` command has to be sent to the listener actor after binding was successful: -.. includecode:: code/jdocs/io/JavaReadBackPressure.java#pull-accepting +@@snip [JavaReadBackPressure.java](code/jdocs/io/JavaReadBackPressure.java) { #pull-accepting } As shown in the example after handling an incoming connection we need to resume accepting again. -The :class:`ResumeAccepting` message accepts a ``batchSize`` parameter that specifies how -many new connections are accepted before a next :class:`ResumeAccepting` message -is needed to resume handling of new connections. +The `ResumeAccepting` message accepts a `batchSize` parameter that specifies how +many new connections are accepted before a next `ResumeAccepting` message +is needed to resume handling of new connections. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/io-udp.md b/akka-docs/src/main/paradox/java/io-udp.md index b8844eccbc..781252607f 100644 --- a/akka-docs/src/main/paradox/java/io-udp.md +++ b/akka-docs/src/main/paradox/java/io-udp.md @@ -1,105 +1,101 @@ -.. _io-java-udp: - -Using UDP -========= +# Using UDP UDP is a connectionless datagram protocol which offers two different ways of communication on the JDK level: +> * sockets which are free to send datagrams to any destination and receive - datagrams from any origin - +datagrams from any origin * sockets which are restricted to communication with one specific remote - socket address +socket address In the low-level API the distinction is made—confusingly—by whether or not -:meth:`connect` has been called on the socket (even when connect has been +`connect` has been called on the socket (even when connect has been called the protocol is still connectionless). These two forms of UDP usage are offered using distinct IO extensions described below. -Unconnected UDP ---------------- +## Unconnected UDP -Simple Send -^^^^^^^^^^^ +### Simple Send -.. includecode:: code/jdocs/io/UdpDocTest.java#sender +@@snip [UdpDocTest.java](code/jdocs/io/UdpDocTest.java) { #sender } The simplest form of UDP usage is to just send datagrams without the need of getting a reply. To this end a “simple sender” facility is provided as demonstrated above. The UDP extension is queried using the -:meth:`simpleSender` message, which is answered by a :class:`SimpleSenderReady` +`simpleSender` message, which is answered by a `SimpleSenderReady` notification. The sender of this message is the newly created sender actor which from this point onward can be used to send datagrams to arbitrary destinations; in this example it will just send any UTF-8 encoded -:class:`String` it receives to a predefined remote address. +`String` it receives to a predefined remote address. -.. note:: +@@@ note - The simple sender will not shut itself down because it cannot know when you - are done with it. You will need to send it a :class:`PoisonPill` when you - want to close the ephemeral port the sender is bound to. +The simple sender will not shut itself down because it cannot know when you +are done with it. You will need to send it a `PoisonPill` when you +want to close the ephemeral port the sender is bound to. -Bind (and Send) -^^^^^^^^^^^^^^^ +@@@ -.. includecode:: code/jdocs/io/UdpDocTest.java#listener +### Bind (and Send) + +@@snip [UdpDocTest.java](code/jdocs/io/UdpDocTest.java) { #listener } If you want to implement a UDP server which listens on a socket for incoming -datagrams then you need to use the :meth:`bind` command as shown above. The +datagrams then you need to use the `bind` command as shown above. The local address specified may have a zero port in which case the operating system will automatically choose a free port and assign it to the new socket. Which -port was actually bound can be found out by inspecting the :class:`Bound` +port was actually bound can be found out by inspecting the `Bound` message. -The sender of the :class:`Bound` message is the actor which manages the new -socket. Sending datagrams is achieved by using the :meth:`send` message type -and the socket can be closed by sending a :meth:`unbind` command, in which -case the socket actor will reply with a :class:`Unbound` notification. +The sender of the `Bound` message is the actor which manages the new +socket. Sending datagrams is achieved by using the `send` message type +and the socket can be closed by sending a `unbind` command, in which +case the socket actor will reply with a `Unbound` notification. -Received datagrams are sent to the actor designated in the :meth:`bind` -message, whereas the :class:`Bound` message will be sent to the sender of the -:meth:`bind`. +Received datagrams are sent to the actor designated in the `bind` +message, whereas the `Bound` message will be sent to the sender of the +`bind`. -Connected UDP -------------- +## Connected UDP The service provided by the connection based UDP API is similar to the bind-and-send service we saw earlier, but the main difference is that a -connection is only able to send to the ``remoteAddress`` it was connected to, +connection is only able to send to the `remoteAddress` it was connected to, and will receive datagrams only from that address. -.. includecode:: code/jdocs/io/UdpDocTest.java#connected +@@snip [UdpDocTest.java](code/jdocs/io/UdpDocTest.java) { #connected } Consequently the example shown here looks quite similar to the previous one, the biggest difference is the absence of remote address information in -:meth:`send` and :class:`Received` messages. +`send` and `Received` messages. -.. note:: - - There is a small performance benefit in using connection based UDP API over - the connectionless one. If there is a SecurityManager enabled on the system, - every connectionless message send has to go through a security check, while - in the case of connection-based UDP the security check is cached after - connect, thus writes do not suffer an additional performance penalty. +@@@ note -UDP Multicast ------------------------------------------- +There is a small performance benefit in using connection based UDP API over +the connectionless one. If there is a SecurityManager enabled on the system, +every connectionless message send has to go through a security check, while +in the case of connection-based UDP the security check is cached after +connect, thus writes do not suffer an additional performance penalty. -Akka provides a way to control various options of ``DatagramChannel`` through the -``akka.io.Inet.SocketOption`` interface. The example below shows +@@@ + +## UDP Multicast + +Akka provides a way to control various options of `DatagramChannel` through the +`akka.io.Inet.SocketOption` interface. The example below shows how to setup a receiver of multicast messages using IPv6 protocol. -To select a Protocol Family you must extend ``akka.io.Inet.DatagramChannelCreator`` -class which implements ``akka.io.Inet.SocketOption``. Provide custom logic -for opening a datagram channel by overriding :meth:`create` method. +To select a Protocol Family you must extend `akka.io.Inet.DatagramChannelCreator` +class which implements `akka.io.Inet.SocketOption`. Provide custom logic +for opening a datagram channel by overriding `create` method. -.. includecode:: code/jdocs/io/JavaUdpMulticast.java#inet6-protocol-family +@@snip [JavaUdpMulticast.java](code/jdocs/io/JavaUdpMulticast.java) { #inet6-protocol-family } Another socket option will be needed to join a multicast group. -.. includecode:: code/jdocs/io/JavaUdpMulticast.java#multicast-group +@@snip [JavaUdpMulticast.java](code/jdocs/io/JavaUdpMulticast.java) { #multicast-group } -Socket options must be provided to :meth:`UdpMessage.bind` command. +Socket options must be provided to `UdpMessage.bind` command. -.. includecode:: code/jdocs/io/JavaUdpMulticast.java#bind +@@snip [JavaUdpMulticast.java](code/jdocs/io/JavaUdpMulticast.java) { #bind } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/io.md b/akka-docs/src/main/paradox/java/io.md index 1c75fc0f2a..7d1cd331d1 100644 --- a/akka-docs/src/main/paradox/java/io.md +++ b/akka-docs/src/main/paradox/java/io.md @@ -1,14 +1,10 @@ -.. _io-java: +# I/O -I/O -========== +## Introduction -Introduction ------------- - -The ``akka.io`` package has been developed in collaboration between the Akka -and `spray.io`_ teams. Its design combines experiences from the -``spray-io`` module with improvements that were jointly developed for +The `akka.io` package has been developed in collaboration between the Akka +and [spray.io](http://spray.io) teams. Its design combines experiences from the +`spray-io` module with improvements that were jointly developed for more general consumption as an actor-based service. The guiding design goal for this I/O implementation was to reach extreme @@ -18,35 +14,33 @@ asynchronous. The API is meant to be a solid foundation for the implementation of network protocols and building higher abstractions; it is not meant to be a full-service high-level NIO wrapper for end users. -Terminology, Concepts ---------------------- +## Terminology, Concepts + The I/O API is completely actor based, meaning that all operations are implemented with message passing instead of direct method calls. Every I/O driver (TCP, UDP) has a special actor, called a *manager* that serves as an entry point for the API. I/O is broken into several drivers. The manager for a particular driver -is accessible by querying an ``ActorSystem``. For example the following code -looks up the TCP manager and returns its ``ActorRef``: +is accessible by querying an `ActorSystem`. For example the following code +looks up the TCP manager and returns its `ActorRef`: -.. includecode:: code/jdocs/io/japi/EchoManager.java#manager +@@snip [EchoManager.java](code/jdocs/io/japi/EchoManager.java) { #manager } The manager receives I/O command messages and instantiates worker actors in response. The worker actors present -themselves to the API user in the reply to the command that was sent. For example after a ``Connect`` command sent to +themselves to the API user in the reply to the command that was sent. For example after a `Connect` command sent to the TCP manager the manager creates an actor representing the TCP connection. All operations related to the given TCP -connections can be invoked by sending messages to the connection actor which announces itself by sending a ``Connected`` +connections can be invoked by sending messages to the connection actor which announces itself by sending a `Connected` message. -DeathWatch and Resource Management -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### DeathWatch and Resource Management I/O worker actors receive commands and also send out events. They usually need a user-side counterpart actor listening for these events (such events could be inbound connections, incoming bytes or acknowledgements for writes). These worker actors *watch* their listener counterparts. If the listener stops then the worker will automatically release any resources that it holds. This design makes the API more robust against resource leaks. - + Thanks to the completely actor based approach of the I/O API the opposite direction works as well: a user actor responsible for handling a connection can watch the connection actor to be notified if it unexpectedly terminates. -Write models (Ack, Nack) -^^^^^^^^^^^^^^^^^^^^^^^^ +### Write models (Ack, Nack) I/O devices have a maximum throughput which limits the frequency and size of writes. When an application tries to push more data than a device can handle, the driver has to buffer bytes until the device @@ -55,60 +49,56 @@ is able to write them. With buffering it is possible to handle short bursts of i Akka supports two types of flow control: -* *Ack-based*, where the driver notifies the writer when writes have succeeded. - -* *Nack-based*, where the driver notifies the writer when writes have failed. + * *Ack-based*, where the driver notifies the writer when writes have succeeded. + * *Nack-based*, where the driver notifies the writer when writes have failed. Each of these models is available in both the TCP and the UDP implementations of Akka I/O. -Individual writes can be acknowledged by providing an ack object in the write message (``Write`` in the case of TCP and -``Send`` for UDP). When the write is complete the worker will send the ack object to the writing actor. This can be +Individual writes can be acknowledged by providing an ack object in the write message (`Write` in the case of TCP and +`Send` for UDP). When the write is complete the worker will send the ack object to the writing actor. This can be used to implement *ack-based* flow control; sending new data only when old data has been acknowledged. If a write (or any other command) fails, the driver notifies the actor that sent the command with a special message -(``CommandFailed`` in the case of UDP and TCP). This message will also notify the writer of a failed write, serving as a +(`CommandFailed` in the case of UDP and TCP). This message will also notify the writer of a failed write, serving as a nack for that write. Please note, that in a nack-based flow-control setting the writer has to be prepared for the fact that the failed write might not be the most recent write it sent. For example, the failure notification for a write -``W1`` might arrive after additional write commands ``W2`` and ``W3`` have been sent. If the writer wants to resend any +`W1` might arrive after additional write commands `W2` and `W3` have been sent. If the writer wants to resend any nacked messages it may need to keep a buffer of pending messages. -.. warning:: - An acknowledged write does not mean acknowledged delivery or storage; receiving an ack for a write simply signals that - the I/O driver has successfully processed the write. The Ack/Nack protocol described here is a means of flow control - not error handling. In other words, data may still be lost, even if every write is acknowledged. +@@@ warning -.. _bytestring_java: +An acknowledged write does not mean acknowledged delivery or storage; receiving an ack for a write simply signals that +the I/O driver has successfully processed the write. The Ack/Nack protocol described here is a means of flow control +not error handling. In other words, data may still be lost, even if every write is acknowledged. -ByteString -^^^^^^^^^^ +@@@ -To maintain isolation, actors should communicate with immutable objects only. ``ByteString`` is an + +### ByteString + +To maintain isolation, actors should communicate with immutable objects only. `ByteString` is an immutable container for bytes. It is used by Akka's I/O system as an efficient, immutable alternative -the traditional byte containers used for I/O on the JVM, such as ``byte[]`` and ``ByteBuffer``. +the traditional byte containers used for I/O on the JVM, such as `byte[]` and `ByteBuffer`. -``ByteString`` is a `rope-like `_ data structure that is immutable -and provides fast concatenation and slicing operations (perfect for I/O). When two ``ByteString``\s are concatenated -together they are both stored within the resulting ``ByteString`` instead of copying both to a new array. Operations -such as ``drop`` and ``take`` return ``ByteString``\s that still reference the original array, but just change the +`ByteString` is a [rope-like](http://en.wikipedia.org/wiki/Rope_\(computer_science\)) data structure that is immutable +and provides fast concatenation and slicing operations (perfect for I/O). When two `ByteString`s are concatenated +together they are both stored within the resulting `ByteString` instead of copying both to a new array. Operations +such as `drop` and `take` return `ByteString`s that still reference the original array, but just change the offset and length that is visible. Great care has also been taken to make sure that the internal array cannot be -modified. Whenever a potentially unsafe array is used to create a new ``ByteString`` a defensive copy is created. If -you require a ``ByteString`` that only blocks a much memory as necessary for it's content, use the ``compact`` method to -get a ``CompactByteString`` instance. If the ``ByteString`` represented only a slice of the original array, this will +modified. Whenever a potentially unsafe array is used to create a new `ByteString` a defensive copy is created. If +you require a `ByteString` that only blocks a much memory as necessary for it's content, use the `compact` method to +get a `CompactByteString` instance. If the `ByteString` represented only a slice of the original array, this will result in copying all bytes in that slice. -``ByteString`` inherits all methods from ``IndexedSeq``, and it also has some new ones. For more information, look up the ``akka.util.ByteString`` class and it's companion object in the ScalaDoc. +`ByteString` inherits all methods from `IndexedSeq`, and it also has some new ones. For more information, look up the `akka.util.ByteString` class and it's companion object in the ScalaDoc. -``ByteString`` also comes with its own optimized builder and iterator classes ``ByteStringBuilder`` and -``ByteIterator`` which provide extra features in addition to those of normal builders and iterators. +`ByteString` also comes with its own optimized builder and iterator classes `ByteStringBuilder` and +`ByteIterator` which provide extra features in addition to those of normal builders and iterators. -Compatibility with java.io -.......................... +#### Compatibility with java.io -A ``ByteStringBuilder`` can be wrapped in a ``java.io.OutputStream`` via the ``asOutputStream`` method. Likewise, ``ByteIterator`` can we wrapped in a ``java.io.InputStream`` via ``asInputStream``. Using these, ``akka.io`` applications can integrate legacy code based on ``java.io`` streams. +A `ByteStringBuilder` can be wrapped in a `java.io.OutputStream` via the `asOutputStream` method. Likewise, `ByteIterator` can we wrapped in a `java.io.InputStream` via `asInputStream`. Using these, `akka.io` applications can integrate legacy code based on `java.io` streams. -Architecture in-depth ---------------------- +## Architecture in-depth -For further details on the design and internal architecture see :ref:`io-layer`. - -.. _spray.io: http://spray.io +For further details on the design and internal architecture see @ref:[I/O Layer Design](../dev/io-layer.md). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/logging.md b/akka-docs/src/main/paradox/java/logging.md index a153db30bc..03c23b13b9 100644 --- a/akka-docs/src/main/paradox/java/logging.md +++ b/akka-docs/src/main/paradox/java/logging.md @@ -1,8 +1,4 @@ -.. _logging-java: - -################ - Logging -################ +# Logging Logging in Akka is not tied to a specific logging backend. By default log messages are printed to STDOUT, but you can plug-in a SLF4J logger or @@ -11,48 +7,45 @@ has minimal performance impact. Logging generally means IO and locks, which can slow down the operations of your code if it was performed synchronously. -How to Log -========== +## How to Log -Create a ``LoggingAdapter`` and use the ``error``, ``warning``, ``info``, or ``debug`` methods, +Create a `LoggingAdapter` and use the `error`, `warning`, `info`, or `debug` methods, as illustrated in this example: -.. includecode:: code/jdocs/event/LoggingDocTest.java - :include: imports +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #imports } -.. includecode:: code/jdocs/event/LoggingDocTest.java - :include: my-actor +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #my-actor } -The first parameter to ``Logging.getLogger`` could also be any -:class:`LoggingBus`, specifically ``system.eventStream()``; in the demonstrated -case, the actor system’s address is included in the ``akkaSource`` -representation of the log source (see `Logging Thread, Akka Source and Actor System in MDC`_) +The first parameter to `Logging.getLogger` could also be any +`LoggingBus`, specifically `system.eventStream()`; in the demonstrated +case, the actor system’s address is included in the `akkaSource` +representation of the log source (see [Logging Thread, Akka Source and Actor System in MDC](#logging-thread-akka-source-and-actor-system-in-mdc)) while in the second case this is not automatically done. The second parameter -to ``Logging.getLogger`` is the source of this logging channel. The source +to `Logging.getLogger` is the source of this logging channel. The source object is translated to a String according to the following rules: - * if it is an Actor or ActorRef, its path is used - * in case of a String it is used as is - * in case of a class an approximation of its simpleName - * and in all other cases the simpleName of its class +> + * if it is an Actor or ActorRef, its path is used + * in case of a String it is used as is + * in case of a class an approximation of its simpleName + * and in all other cases the simpleName of its class -The log message may contain argument placeholders ``{}``, which will be +The log message may contain argument placeholders `{}`, which will be substituted if the log level is enabled. Giving more arguments than placeholders results in a warning being appended to the log statement (i.e. on the same line with the same severity). You may pass a Java array as the only substitution argument to have its elements be treated individually: -.. includecode:: code/jdocs/event/LoggingDocTest.java#array +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #array } -The Java :class:`Class` of the log source is also included in the generated -:class:`LogEvent`. In case of a simple string this is replaced with a “marker” -class :class:`akka.event.DummyClassForStringSources` in order to allow special +The Java `Class` of the log source is also included in the generated +`LogEvent`. In case of a simple string this is replaced with a “marker” +class `akka.event.DummyClassForStringSources` in order to allow special treatment of this case, e.g. in the SLF4J event listener which will then use the string instead of the class’ name for looking up the logger instance to use. -Logging of Dead Letters ------------------------ +### Logging of Dead Letters By default messages sent to dead letters are logged at info level. Existence of dead letters does not necessarily indicate a problem, but they are logged by default for the sake of caution. @@ -62,437 +55,448 @@ logged. During system shutdown it is likely that you see dead letters, since pen messages in the actor mailboxes are sent to dead letters. You can also disable logging of dead letters during shutdown. -.. code-block:: ruby - - akka { - log-dead-letters = 10 - log-dead-letters-during-shutdown = on - } +```ruby +akka { + log-dead-letters = 10 + log-dead-letters-during-shutdown = on +} +``` To customize the logging further or take other actions for dead letters you can subscribe -to the :ref:`event-stream-java`. +to the @ref:[Event Stream](event-bus.md#event-stream-java). -Auxiliary logging options -------------------------- +### Auxiliary logging options Akka has a few configuration options for very low level debugging. These make more sense in development than in production. You almost definitely need to have logging set to DEBUG to use any of the options below: -.. code-block:: ruby - - akka { - loglevel = "DEBUG" - } +```ruby +akka { + loglevel = "DEBUG" +} +``` This config option is very good if you want to know what config settings are loaded by Akka: -.. code-block:: ruby - - akka { - # Log the complete configuration at INFO level when the actor system is started. - # This is useful when you are uncertain of what configuration is used. - log-config-on-start = on - } +```ruby +akka { + # Log the complete configuration at INFO level when the actor system is started. + # This is useful when you are uncertain of what configuration is used. + log-config-on-start = on +} +``` If you want very detailed logging of all automatically received messages that are processed by Actors: -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of all AutoReceiveMessages (Kill, PoisonPill et.c.) - autoreceive = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of all AutoReceiveMessages (Kill, PoisonPill et.c.) + autoreceive = on } + } +} +``` If you want very detailed logging of all lifecycle changes of Actors (restarts, deaths etc): -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of actor lifecycle changes - lifecycle = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of actor lifecycle changes + lifecycle = on } + } +} +``` If you want unhandled messages logged at DEBUG: -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of unhandled messages - unhandled = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of unhandled messages + unhandled = on } + } +} +``` If you want very detailed logging of all events, transitions and timers of FSM Actors that extend LoggingFSM: -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of all LoggingFSMs for events, transitions and timers - fsm = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of all LoggingFSMs for events, transitions and timers + fsm = on } + } +} +``` If you want to monitor subscriptions (subscribe/unsubscribe) on the ActorSystem.eventStream: -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of subscription changes on the eventStream - event-stream = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of subscription changes on the eventStream + event-stream = on } + } +} +``` -.. _logging-remote-java: - -Auxiliary remote logging options --------------------------------- + +### Auxiliary remote logging options If you want to see all messages that are sent through remoting at DEBUG log level, use the following config option. Note that this logs the messages as they are sent by the transport layer, not by an actor. -.. code-block:: ruby - - akka { - remote { - # If this is "on", Akka will log all outbound messages at DEBUG level, - # if off then they are not logged - log-sent-messages = on - } - } +```ruby +akka { + remote { + # If this is "on", Akka will log all outbound messages at DEBUG level, + # if off then they are not logged + log-sent-messages = on + } +} +``` If you want to see all messages that are received through remoting at DEBUG log level, use the following config option. Note that this logs the messages as they are sent by the transport layer, not by an actor. -.. code-block:: ruby - - akka { - remote { - # If this is "on", Akka will log all inbound messages at DEBUG level, - # if off then they are not logged - log-received-messages = on - } - } +```ruby +akka { + remote { + # If this is "on", Akka will log all inbound messages at DEBUG level, + # if off then they are not logged + log-received-messages = on + } +} +``` If you want to see message types with payload size in bytes larger than a specified limit at INFO log level: -.. code-block:: ruby - - akka { - remote { - # Logging of message types with payload size in bytes larger than - # this value. Maximum detected size per message type is logged once, - # with an increase threshold of 10%. - # By default this feature is turned off. Activate it by setting the property to - # a value in bytes, such as 1000b. Note that for all messages larger than this - # limit there will be extra performance and scalability cost. - log-frame-size-exceeding = 1000b - } - } - -Also see the logging options for TestKit: :ref:`actor.logging-java`. - -Turn Off Logging ----------------- - -To turn off logging you can configure the log levels to be ``OFF`` like this. - -.. code-block:: ruby - - akka { - stdout-loglevel = "OFF" - loglevel = "OFF" +```ruby +akka { + remote { + # Logging of message types with payload size in bytes larger than + # this value. Maximum detected size per message type is logged once, + # with an increase threshold of 10%. + # By default this feature is turned off. Activate it by setting the property to + # a value in bytes, such as 1000b. Note that for all messages larger than this + # limit there will be extra performance and scalability cost. + log-frame-size-exceeding = 1000b } +} +``` -The ``stdout-loglevel`` is only in effect during system startup and shutdown, and setting -it to ``OFF`` as well, ensures that nothing gets logged during system startup or shutdown. +Also see the logging options for TestKit: @ref:[actor.logging-java](testing.md#actor-logging-java). -Loggers -======= +### Turn Off Logging + +To turn off logging you can configure the log levels to be `OFF` like this. + +```ruby +akka { + stdout-loglevel = "OFF" + loglevel = "OFF" +} +``` + +The `stdout-loglevel` is only in effect during system startup and shutdown, and setting +it to `OFF` as well, ensures that nothing gets logged during system startup or shutdown. + +## Loggers Logging is performed asynchronously through an event bus. Log events are processed by an event handler actor that receives the log events in the same order they were emitted. -.. note:: - The event handler actor does not have a bounded inbox and is run on the default dispatcher. This means - that logging extreme amounts of data may affect your application badly. This can be somewhat mitigated by using an async logging backend though. (See :ref:`slf4j-directly-java`) +@@@ note + +The event handler actor does not have a bounded inbox and is run on the default dispatcher. This means +that logging extreme amounts of data may affect your application badly. This can be somewhat mitigated by using an async logging backend though. (See [Using the SLF4J API directly](#slf4j-directly-java)) + +@@@ You can configure which event handlers are created at system start-up and listen to logging events. That is done using the -``loggers`` element in the :ref:`configuration`. +`loggers` element in the configuration. Here you can also define the log level. More fine grained filtering based on the log source -can be implemented in a custom ``LoggingFilter``, which can be defined in the ``logging-filter`` +can be implemented in a custom `LoggingFilter`, which can be defined in the `logging-filter` configuration property. -.. code-block:: ruby +```ruby +akka { + # Loggers to register at boot time (akka.event.Logging$DefaultLogger logs + # to STDOUT) + loggers = ["akka.event.Logging$DefaultLogger"] + # Options: OFF, ERROR, WARNING, INFO, DEBUG + loglevel = "DEBUG" +} +``` - akka { - # Loggers to register at boot time (akka.event.Logging$DefaultLogger logs - # to STDOUT) - loggers = ["akka.event.Logging$DefaultLogger"] - # Options: OFF, ERROR, WARNING, INFO, DEBUG - loglevel = "DEBUG" - } - -The default one logs to STDOUT and is registered by default. It is not intended to be used for production. There is also an :ref:`slf4j-java` +The default one logs to STDOUT and is registered by default. It is not intended to be used for production. There is also an [SLF4J](#slf4j-java) logger available in the 'akka-slf4j' module. Example of creating a listener: -.. includecode:: code/jdocs/event/LoggingDocTest.java - :include: imports,imports-listener +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #imports #imports-listener } -.. includecode:: code/jdocs/event/LoggingDocTest.java - :include: my-event-listener +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #my-event-listener } -Logging to stdout during startup and shutdown -============================================= +## Logging to stdout during startup and shutdown -While the actor system is starting up and shutting down the configured ``loggers`` are not used. +While the actor system is starting up and shutting down the configured `loggers` are not used. Instead log messages are printed to stdout (System.out). The default log level for this -stdout logger is ``WARNING`` and it can be silenced completely by setting -``akka.stdout-loglevel=OFF``. +stdout logger is `WARNING` and it can be silenced completely by setting +`akka.stdout-loglevel=OFF`. -.. _slf4j-java: + +## SLF4J -SLF4J -===== +Akka provides a logger for [SL4FJ](http://www.slf4j.org/). This module is available in the 'akka-slf4j.jar'. +It has a single dependency: the slf4j-api jar. In your runtime, you also need a SLF4J backend. We recommend [Logback](http://logback.qos.ch/): -Akka provides a logger for `SL4FJ `_. This module is available in the 'akka-slf4j.jar'. -It has a single dependency: the slf4j-api jar. In your runtime, you also need a SLF4J backend. We recommend `Logback `_: +> +```xml + + ch.qos.logback + logback-classic + 1.1.3 + +``` - .. code-block:: xml - - - ch.qos.logback - logback-classic - 1.1.3 - - -You need to enable the Slf4jLogger in the ``loggers`` element in -the :ref:`configuration`. Here you can also define the log level of the event bus. +You need to enable the Slf4jLogger in the `loggers` element in +the configuration. Here you can also define the log level of the event bus. More fine grained log levels can be defined in the configuration of the SLF4J backend -(e.g. logback.xml). You should also define ``akka.event.slf4j.Slf4jLoggingFilter`` in -the ``logging-filter`` configuration property. It will filter the log events using the backend +(e.g. logback.xml). You should also define `akka.event.slf4j.Slf4jLoggingFilter` in +the `logging-filter` configuration property. It will filter the log events using the backend configuration (e.g. logback.xml) before they are published to the event bus. -.. warning:: - If you set the ``loglevel`` to a higher level than "DEBUG", any DEBUG events will be filtered - out already at the source and will never reach the logging backend, regardless of how the backend - is configured. +@@@ warning -.. code-block:: ruby +If you set the `loglevel` to a higher level than "DEBUG", any DEBUG events will be filtered +out already at the source and will never reach the logging backend, regardless of how the backend +is configured. + +@@@ + +```ruby +akka { + loggers = ["akka.event.slf4j.Slf4jLogger"] + loglevel = "DEBUG" + logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" +} +``` - akka { - loggers = ["akka.event.slf4j.Slf4jLogger"] - loglevel = "DEBUG" - logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" - } - One gotcha is that the timestamp is attributed in the event handler, not when actually doing the logging. The SLF4J logger selected for each log event is chosen based on the -:class:`Class` of the log source specified when creating the -:class:`LoggingAdapter`, unless that was given directly as a string in which -case that string is used (i.e. ``LoggerFactory.getLogger(Class c)`` is used in -the first case and ``LoggerFactory.getLogger(String s)`` in the second). +`Class` of the log source specified when creating the +`LoggingAdapter`, unless that was given directly as a string in which +case that string is used (i.e. `LoggerFactory.getLogger(Class c)` is used in +the first case and `LoggerFactory.getLogger(String s)` in the second). -.. note:: +@@@ note - Beware that the actor system’s name is appended to a :class:`String` log - source if the LoggingAdapter was created giving an :class:`ActorSystem` to - the factory. If this is not intended, give a :class:`LoggingBus` instead as - shown below: +Beware that the actor system’s name is appended to a `String` log +source if the LoggingAdapter was created giving an `ActorSystem` to +the factory. If this is not intended, give a `LoggingBus` instead as +shown below: -.. code-block:: scala +@@@ - final LoggingAdapter log = Logging.getLogger(system.eventStream(), "my.string"); +```scala +final LoggingAdapter log = Logging.getLogger(system.eventStream(), "my.string"); +``` -.. _slf4j-directly-java: + +### Using the SLF4J API directly -Using the SLF4J API directly ----------------------------- If you use the SLF4J API directly in your application, remember that the logging operations will block while the underlying infrastructure writes the log statements. This can be avoided by configuring the logging implementation to use -a non-blocking appender. Logback provides `AsyncAppender `_ -that does this. It also contains a feature which will drop ``INFO`` and ``DEBUG`` messages if the logging +a non-blocking appender. Logback provides [AsyncAppender](http://logback.qos.ch/manual/appenders.html#AsyncAppender) +that does this. It also contains a feature which will drop `INFO` and `DEBUG` messages if the logging load is high. -Logging Thread, Akka Source and Actor System in MDC ---------------------------------------------------- +### Logging Thread, Akka Source and Actor System in MDC Since the logging is done asynchronously the thread in which the logging was performed is captured in -Mapped Diagnostic Context (MDC) with attribute name ``sourceThread``. -With Logback the thread name is available with ``%X{sourceThread}`` specifier within the pattern layout configuration:: +Mapped Diagnostic Context (MDC) with attribute name `sourceThread`. +With Logback the thread name is available with `%X{sourceThread}` specifier within the pattern layout configuration: - - - %date{ISO8601} %-5level %logger{36} %X{sourceThread} - %msg%n - - +``` + + + %date{ISO8601} %-5level %logger{36} %X{sourceThread} - %msg%n + + +``` -.. note:: +@@@ note - It will probably be a good idea to use the ``sourceThread`` MDC value also in - non-Akka parts of the application in order to have this property consistently - available in the logs. +It will probably be a good idea to use the `sourceThread` MDC value also in +non-Akka parts of the application in order to have this property consistently +available in the logs. + +@@@ Another helpful facility is that Akka captures the actor’s address when instantiating a logger within it, meaning that the full instance identification is available for associating log messages e.g. with members of a router. This -information is available in the MDC with attribute name ``akkaSource``:: +information is available in the MDC with attribute name `akkaSource`: - - - %date{ISO8601} %-5level %logger{36} %X{akkaSource} - %msg%n - - +``` + + + %date{ISO8601} %-5level %logger{36} %X{akkaSource} - %msg%n + + +``` Finally, the actor system in which the logging was performed -is available in the MDC with attribute name ``sourceActorSystem``:: +is available in the MDC with attribute name `sourceActorSystem`: - - - %date{ISO8601} %-5level %logger{36} %X{sourceActorSystem} - %msg%n - - +``` + + + %date{ISO8601} %-5level %logger{36} %X{sourceActorSystem} - %msg%n + + +``` For more details on what this attribute contains—also for non-actors—please see -`How to Log`_. +[How to Log](#how-to-log). -More accurate timestamps for log output in MDC ----------------------------------------------- +### More accurate timestamps for log output in MDC Akka's logging is asynchronous which means that the timestamp of a log entry is taken from when the underlying logger implementation is called, which can be surprising at first. -If you want to more accurately output the timestamp, use the MDC attribute ``akkaTimestamp``:: +If you want to more accurately output the timestamp, use the MDC attribute `akkaTimestamp`: - - - %X{akkaTimestamp} %-5level %logger{36} %X{akkaSource} - %msg%n - - +``` + + + %X{akkaTimestamp} %-5level %logger{36} %X{akkaSource} - %msg%n + + +``` +### MDC values defined by the application -MDC values defined by the application -------------------------------------- - -One useful feature available in Slf4j is `MDC `_, +One useful feature available in Slf4j is [MDC](http://logback.qos.ch/manual/mdc.html), Akka has a way to let the application specify custom values, you just need to get a -specialized :class:`LoggingAdapter`, the :class:`DiagnosticLoggingAdapter`. In order to +specialized `LoggingAdapter`, the `DiagnosticLoggingAdapter`. In order to get it you can use the factory, providing an AbstractActor as logSource: -.. code-block:: scala - - // Within your AbstractActor - final DiagnosticLoggingAdapter log = Logging.getLogger(this); +```scala +// Within your AbstractActor +final DiagnosticLoggingAdapter log = Logging.getLogger(this); +``` Once you have the logger, you just need to add the custom values before you log something. This way, the values will be put in the SLF4J MDC right before appending the log and removed after. -.. note:: +@@@ note - The cleanup (removal) should be done in the actor at the end, - otherwise, the next message will log with same MDC values, - if it is not set to a new map. Use ``log.clearMDC()``. +The cleanup (removal) should be done in the actor at the end, +otherwise, the next message will log with same MDC values, +if it is not set to a new map. Use `log.clearMDC()`. -.. includecode:: code/jdocs/event/LoggingDocTest.java - :include: imports-mdc +@@@ -.. includecode:: code/jdocs/event/LoggingDocTest.java - :include: mdc-actor +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #imports-mdc } -Now, the values will be available in the MDC, so you can use them in the layout pattern:: +@@snip [LoggingDocTest.java](code/jdocs/event/LoggingDocTest.java) { #mdc-actor } - - - - %-5level %logger{36} [req: %X{requestId}, visitor: %X{visitorId}] - %msg%n - - - +Now, the values will be available in the MDC, so you can use them in the layout pattern: +``` + + + + %-5level %logger{36} [req: %X{requestId}, visitor: %X{visitorId}] - %msg%n + + + +``` -Using Markers -------------- +### Using Markers Some logging libraries allow, in addition to MDC data, attaching so called "markers" to log statements. These are used to filter out rare and special events, for example you might want to mark logs that detect -some malicious activity and mark them with a ``SECURITY`` tag, and in your appender configuration make these +some malicious activity and mark them with a `SECURITY` tag, and in your appender configuration make these trigger emails and other notifications immediately. -Markers are available through the LoggingAdapters, when obtained via ``Logging.withMarker``. -The first argument passed into all log calls then should be a ``akka.event.LogMarker``. +Markers are available through the LoggingAdapters, when obtained via `Logging.withMarker`. +The first argument passed into all log calls then should be a `akka.event.LogMarker`. -The slf4j bridge provided by akka in ``akka-slf4j`` will automatically pick up this marker value and make it available to SLF4J. -For example you could use it like this:: +The slf4j bridge provided by akka in `akka-slf4j` will automatically pick up this marker value and make it available to SLF4J. +For example you could use it like this: - %date{ISO8601} [%marker][%level] [%msg]%n +``` +%date{ISO8601} [%marker][%level] [%msg]%n +``` -A more advanced (including most Akka added information) example pattern would be:: +A more advanced (including most Akka added information) example pattern would be: - %date{ISO8601} level=[%level] marker=[%marker] logger=[%logger] akkaSource=[%X{akkaSource}] sourceActorSystem=[%X{sourceActorSystem}] sourceThread=[%X{sourceThread}] mdc=[ticket-#%X{ticketNumber}: %X{ticketDesc}] - msg=[%msg]%n----%n +``` +%date{ISO8601} level=[%level] marker=[%marker] logger=[%logger] akkaSource=[%X{akkaSource}] sourceActorSystem=[%X{sourceActorSystem}] sourceThread=[%X{sourceThread}] mdc=[ticket-#%X{ticketNumber}: %X{ticketDesc}] - msg=[%msg]%n----%n +``` -.. _jul-java: + +## java.util.logging -java.util.logging -================= +Akka includes a logger for [java.util.logging](https://docs.oracle.com/javase/8/jdocs/api/java/util/logging/package-summary.html#package.description). -Akka includes a logger for `java.util.logging `_. - -You need to enable the ``akka.event.jul.JavaLogger`` in the ``loggers`` element in -the :ref:`configuration`. Here you can also define the log level of the event bus. +You need to enable the `akka.event.jul.JavaLogger` in the `loggers` element in +the configuration. Here you can also define the log level of the event bus. More fine grained log levels can be defined in the configuration of the logging backend. -You should also define ``akka.event.jul.JavaLoggingFilter`` in -the ``logging-filter`` configuration property. It will filter the log events using the backend +You should also define `akka.event.jul.JavaLoggingFilter` in +the `logging-filter` configuration property. It will filter the log events using the backend configuration before they are published to the event bus. -.. warning:: - If you set the ``loglevel`` to a higher level than "DEBUG", any DEBUG events will be filtered - out already at the source and will never reach the logging backend, regardless of how the backend - is configured. +@@@ warning -.. code-block:: ruby +If you set the `loglevel` to a higher level than "DEBUG", any DEBUG events will be filtered +out already at the source and will never reach the logging backend, regardless of how the backend +is configured. - akka { - loglevel = DEBUG - loggers = ["akka.event.jul.JavaLogger"] - logging-filter = "akka.event.jul.JavaLoggingFilter" - } +@@@ + +```ruby +akka { + loglevel = DEBUG + loggers = ["akka.event.jul.JavaLogger"] + logging-filter = "akka.event.jul.JavaLoggingFilter" +} +``` One gotcha is that the timestamp is attributed in the event handler, not when actually doing the logging. -The ``java.util.logging.Logger`` selected for each log event is chosen based on the -:class:`Class` of the log source specified when creating the -:class:`LoggingAdapter`, unless that was given directly as a string in which -case that string is used (i.e. ``LoggerFactory.getLogger(Class c)`` is used in -the first case and ``LoggerFactory.getLogger(String s)`` in the second). +The `java.util.logging.Logger` selected for each log event is chosen based on the +`Class` of the log source specified when creating the +`LoggingAdapter`, unless that was given directly as a string in which +case that string is used (i.e. `LoggerFactory.getLogger(Class c)` is used in +the first case and `LoggerFactory.getLogger(String s)` in the second). -.. note:: +@@@ note - Beware that the actor system’s name is appended to a :class:`String` log - source if the LoggingAdapter was created giving an :class:`ActorSystem` to - the factory. If this is not intended, give a :class:`LoggingBus` instead as - shown below: +Beware that the actor system’s name is appended to a `String` log +source if the LoggingAdapter was created giving an `ActorSystem` to +the factory. If this is not intended, give a `LoggingBus` instead as +shown below: -.. code-block:: scala +@@@ - final LoggingAdapter log = Logging.getLogger(system.eventStream(), "my.string"); +```scala +final LoggingAdapter log = Logging.getLogger(system.eventStream(), "my.string"); +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/mailboxes.md b/akka-docs/src/main/paradox/java/mailboxes.md index 7f22963fe3..431d93bb19 100644 --- a/akka-docs/src/main/paradox/java/mailboxes.md +++ b/akka-docs/src/main/paradox/java/mailboxes.md @@ -1,336 +1,273 @@ -.. _mailboxes-java: +# Mailboxes -Mailboxes -######### - -An Akka ``Mailbox`` holds the messages that are destined for an ``Actor``. -Normally each ``Actor`` has its own mailbox, but with for example a ``BalancingPool`` +An Akka `Mailbox` holds the messages that are destined for an `Actor`. +Normally each `Actor` has its own mailbox, but with for example a `BalancingPool` all routees will share a single mailbox instance. -Mailbox Selection -================= +## Mailbox Selection -Requiring a Message Queue Type for an Actor -------------------------------------------- +### Requiring a Message Queue Type for an Actor It is possible to require a certain type of message queue for a certain type of actor -by having that actor implement the parameterized interface :class:`RequiresMessageQueue`. Here is +by having that actor implement the parameterized interface `RequiresMessageQueue`. Here is an example: -.. includecode:: code/jdocs/actor/MyBoundedActor.java#my-bounded-untyped-actor +@@snip [MyBoundedActor.java](code/jdocs/actor/MyBoundedActor.java) { #my-bounded-untyped-actor } -The type parameter to the :class:`RequiresMessageQueue` interface needs to be mapped to a mailbox in +The type parameter to the `RequiresMessageQueue` interface needs to be mapped to a mailbox in configuration like this: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala - :include: bounded-mailbox-config,required-mailbox-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #bounded-mailbox-config #required-mailbox-config } -Now every time you create an actor of type :class:`MyBoundedActor` it will try to get a bounded +Now every time you create an actor of type `MyBoundedActor` it will try to get a bounded mailbox. If the actor has a different mailbox configured in deployment, either directly or via a dispatcher with a specified mailbox type, then that will override this mapping. -.. note:: +@@@ note - The type of the queue in the mailbox created for an actor will be checked against the required type in the - interface and if the queue doesn't implement the required type then actor creation will fail. +The type of the queue in the mailbox created for an actor will be checked against the required type in the +interface and if the queue doesn't implement the required type then actor creation will fail. -Requiring a Message Queue Type for a Dispatcher ------------------------------------------------ +@@@ + +### Requiring a Message Queue Type for a Dispatcher A dispatcher may also have a requirement for the mailbox type used by the actors running on it. An example is the BalancingDispatcher which requires a message queue that is thread-safe for multiple concurrent consumers. Such a requirement is formulated within the dispatcher configuration section like -this:: +this: - my-dispatcher { - mailbox-requirement = org.example.MyInterface - } +``` +my-dispatcher { + mailbox-requirement = org.example.MyInterface +} +``` The given requirement names a class or interface which will then be ensured to be a supertype of the message queue’s implementation. In case of a conflict—e.g. if the actor requires a mailbox type which does not satisfy this requirement—then actor creation will fail. -How the Mailbox Type is Selected --------------------------------- +### How the Mailbox Type is Selected -When an actor is created, the :class:`ActorRefProvider` first determines the +When an actor is created, the `ActorRefProvider` first determines the dispatcher which will execute it. Then the mailbox is determined as follows: -1. If the actor’s deployment configuration section contains a ``mailbox`` key - then that names a configuration section describing the mailbox type to be - used. + 1. If the actor’s deployment configuration section contains a `mailbox` key +then that names a configuration section describing the mailbox type to be +used. + 2. If the actor’s `Props` contains a mailbox selection—i.e. `withMailbox` +was called on it—then that names a configuration section describing the +mailbox type to be used. + 3. If the dispatcher’s configuration section contains a `mailbox-type` key +the same section will be used to configure the mailbox type. + 4. If the actor requires a mailbox type as described above then the mapping for +that requirement will be used to determine the mailbox type to be used; if +that fails then the dispatcher’s requirement—if any—will be tried instead. + 5. If the dispatcher requires a mailbox type as described above then the +mapping for that requirement will be used to determine the mailbox type to +be used. + 6. The default mailbox `akka.actor.default-mailbox` will be used. -2. If the actor’s ``Props`` contains a mailbox selection—i.e. ``withMailbox`` - was called on it—then that names a configuration section describing the - mailbox type to be used. - -3. If the dispatcher’s configuration section contains a ``mailbox-type`` key - the same section will be used to configure the mailbox type. - -4. If the actor requires a mailbox type as described above then the mapping for - that requirement will be used to determine the mailbox type to be used; if - that fails then the dispatcher’s requirement—if any—will be tried instead. - -5. If the dispatcher requires a mailbox type as described above then the - mapping for that requirement will be used to determine the mailbox type to - be used. - -6. The default mailbox ``akka.actor.default-mailbox`` will be used. - -Default Mailbox ---------------- +### Default Mailbox When the mailbox is not specified as described above the default mailbox is used. By default it is an unbounded mailbox, which is backed by a -``java.util.concurrent.ConcurrentLinkedQueue``. +`java.util.concurrent.ConcurrentLinkedQueue`. -``SingleConsumerOnlyUnboundedMailbox`` is an even more efficient mailbox, and +`SingleConsumerOnlyUnboundedMailbox` is an even more efficient mailbox, and it can be used as the default mailbox, but it cannot be used with a BalancingDispatcher. -Configuration of ``SingleConsumerOnlyUnboundedMailbox`` as default mailbox:: +Configuration of `SingleConsumerOnlyUnboundedMailbox` as default mailbox: - akka.actor.default-mailbox { - mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" - } +``` +akka.actor.default-mailbox { + mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" +} +``` -Which Configuration is passed to the Mailbox Type -------------------------------------------------- +### Which Configuration is passed to the Mailbox Type -Each mailbox type is implemented by a class which extends :class:`MailboxType` -and takes two constructor arguments: a :class:`ActorSystem.Settings` object and -a :class:`Config` section. The latter is computed by obtaining the named +Each mailbox type is implemented by a class which extends `MailboxType` +and takes two constructor arguments: a `ActorSystem.Settings` object and +a `Config` section. The latter is computed by obtaining the named configuration section from the actor system’s configuration, overriding its -``id`` key with the configuration path of the mailbox type and adding a +`id` key with the configuration path of the mailbox type and adding a fall-back to the default mailbox configuration section. -Builtin Mailbox Implementations -=============================== +## Builtin Mailbox Implementations Akka comes shipped with a number of mailbox implementations: -* **UnboundedMailbox** (default) - - - The default mailbox - - - Backed by a ``java.util.concurrent.ConcurrentLinkedQueue`` - - - Blocking: No - - - Bounded: No - - - Configuration name: ``"unbounded"`` or ``"akka.dispatch.UnboundedMailbox"`` - -* **SingleConsumerOnlyUnboundedMailbox** - - This queue may or may not be faster than the default one depending on your use-case—be sure to benchmark properly! - - - Backed by a Multiple-Producer Single-Consumer queue, cannot be used with ``BalancingDispatcher`` - - - Blocking: No - - - Bounded: No - - - Configuration name: ``"akka.dispatch.SingleConsumerOnlyUnboundedMailbox"`` - -* **NonBlockingBoundedMailbox** - - - Backed by a very efficient Multiple-Producer Single-Consumer queue - - - Blocking: No (discards overflowing messages into deadLetters) - - - Bounded: Yes - - - Configuration name: ``"akka.dispatch.NonBlockingBoundedMailbox"`` - -* **UnboundedControlAwareMailbox** - - - Delivers messages that extend ``akka.dispatch.ControlMessage`` with higher priority - - - Backed by two ``java.util.concurrent.ConcurrentLinkedQueue`` - - - Blocking: No - - - Bounded: No - - - Configuration name: "akka.dispatch.UnboundedControlAwareMailbox" - -* **UnboundedPriorityMailbox** - - - Backed by a ``java.util.concurrent.PriorityBlockingQueue`` - - - Delivery order for messages of equal priority is undefined - contrast with the UnboundedStablePriorityMailbox - - - Blocking: No - - - Bounded: No - - - Configuration name: "akka.dispatch.UnboundedPriorityMailbox" - -* **UnboundedStablePriorityMailbox** - - - Backed by a ``java.util.concurrent.PriorityBlockingQueue`` wrapped in an ``akka.util.PriorityQueueStabilizer`` - - - FIFO order is preserved for messages of equal priority - contrast with the UnboundedPriorityMailbox - - - Blocking: No - - - Bounded: No - - - Configuration name: "akka.dispatch.UnboundedStablePriorityMailbox" + * + **UnboundedMailbox** (default) + * The default mailbox + * Backed by a `java.util.concurrent.ConcurrentLinkedQueue` + * Blocking: No + * Bounded: No + * Configuration name: `"unbounded"` or `"akka.dispatch.UnboundedMailbox"` + * + **SingleConsumerOnlyUnboundedMailbox** + This queue may or may not be faster than the default one depending on your use-case—be sure to benchmark properly! + * Backed by a Multiple-Producer Single-Consumer queue, cannot be used with `BalancingDispatcher` + * Blocking: No + * Bounded: No + * Configuration name: `"akka.dispatch.SingleConsumerOnlyUnboundedMailbox"` + * + **NonBlockingBoundedMailbox** + * Backed by a very efficient Multiple-Producer Single-Consumer queue + * Blocking: No (discards overflowing messages into deadLetters) + * Bounded: Yes + * Configuration name: `"akka.dispatch.NonBlockingBoundedMailbox"` + * + **UnboundedControlAwareMailbox** + * Delivers messages that extend `akka.dispatch.ControlMessage` with higher priority + * Backed by two `java.util.concurrent.ConcurrentLinkedQueue` + * Blocking: No + * Bounded: No + * Configuration name: "akka.dispatch.UnboundedControlAwareMailbox" + * + **UnboundedPriorityMailbox** + * Backed by a `java.util.concurrent.PriorityBlockingQueue` + * Delivery order for messages of equal priority is undefined - contrast with the UnboundedStablePriorityMailbox + * Blocking: No + * Bounded: No + * Configuration name: "akka.dispatch.UnboundedPriorityMailbox" + * + **UnboundedStablePriorityMailbox** + * Backed by a `java.util.concurrent.PriorityBlockingQueue` wrapped in an `akka.util.PriorityQueueStabilizer` + * FIFO order is preserved for messages of equal priority - contrast with the UnboundedPriorityMailbox + * Blocking: No + * Bounded: No + * Configuration name: "akka.dispatch.UnboundedStablePriorityMailbox" Other bounded mailbox implementations which will block the sender if the capacity is reached and -configured with non-zero ``mailbox-push-timeout-time``. +configured with non-zero `mailbox-push-timeout-time`. -.. note:: The following mailboxes should only be used with zero ``mailbox-push-timeout-time``. +@@@ note -* **BoundedMailbox** +The following mailboxes should only be used with zero `mailbox-push-timeout-time`. - - Backed by a ``java.util.concurrent.LinkedBlockingQueue`` +@@@ - - Blocking: Yes if used with non-zero ``mailbox-push-timeout-time``, otherwise No + * **BoundedMailbox** + * Backed by a `java.util.concurrent.LinkedBlockingQueue` + * Blocking: Yes if used with non-zero `mailbox-push-timeout-time`, otherwise No + * Bounded: Yes + * Configuration name: "bounded" or "akka.dispatch.BoundedMailbox" + * **BoundedPriorityMailbox** + * Backed by a `java.util.PriorityQueue` wrapped in an `akka.util.BoundedBlockingQueue` + * Delivery order for messages of equal priority is undefined - contrast with the `BoundedStablePriorityMailbox` + * Blocking: Yes if used with non-zero `mailbox-push-timeout-time`, otherwise No + * Bounded: Yes + * Configuration name: `"akka.dispatch.BoundedPriorityMailbox"` + * **BoundedStablePriorityMailbox** + * Backed by a `java.util.PriorityQueue` wrapped in an `akka.util.PriorityQueueStabilizer` and an `akka.util.BoundedBlockingQueue` + * FIFO order is preserved for messages of equal priority - contrast with the BoundedPriorityMailbox + * Blocking: Yes if used with non-zero `mailbox-push-timeout-time`, otherwise No + * Bounded: Yes + * Configuration name: "akka.dispatch.BoundedStablePriorityMailbox" + * **BoundedControlAwareMailbox** + * Delivers messages that extend `akka.dispatch.ControlMessage` with higher priority + * Backed by two `java.util.concurrent.ConcurrentLinkedQueue` and blocking on enqueue if capacity has been reached + * Blocking: Yes if used with non-zero `mailbox-push-timeout-time`, otherwise No + * Bounded: Yes + * Configuration name: "akka.dispatch.BoundedControlAwareMailbox" - - Bounded: Yes +## Mailbox configuration examples - - Configuration name: "bounded" or "akka.dispatch.BoundedMailbox" - -* **BoundedPriorityMailbox** - - - Backed by a ``java.util.PriorityQueue`` wrapped in an ``akka.util.BoundedBlockingQueue`` - - - Delivery order for messages of equal priority is undefined - contrast with the ``BoundedStablePriorityMailbox`` - - - Blocking: Yes if used with non-zero ``mailbox-push-timeout-time``, otherwise No - - - Bounded: Yes - - - Configuration name: ``"akka.dispatch.BoundedPriorityMailbox"`` - -* **BoundedStablePriorityMailbox** - - - Backed by a ``java.util.PriorityQueue`` wrapped in an ``akka.util.PriorityQueueStabilizer`` and an ``akka.util.BoundedBlockingQueue`` - - - FIFO order is preserved for messages of equal priority - contrast with the BoundedPriorityMailbox - - - Blocking: Yes if used with non-zero ``mailbox-push-timeout-time``, otherwise No - - - Bounded: Yes - - - Configuration name: "akka.dispatch.BoundedStablePriorityMailbox" - -* **BoundedControlAwareMailbox** - - - Delivers messages that extend ``akka.dispatch.ControlMessage`` with higher priority - - - Backed by two ``java.util.concurrent.ConcurrentLinkedQueue`` and blocking on enqueue if capacity has been reached - - - Blocking: Yes if used with non-zero ``mailbox-push-timeout-time``, otherwise No - - - Bounded: Yes - - - Configuration name: "akka.dispatch.BoundedControlAwareMailbox" - -Mailbox configuration examples -============================== - -PriorityMailbox ---------------- +### PriorityMailbox How to create a PriorityMailbox: -.. includecode:: ../java/code/jdocs/dispatcher/DispatcherDocTest.java#prio-mailbox +@@snip [DispatcherDocTest.java](../java/code/jdocs/dispatcher/DispatcherDocTest.java) { #prio-mailbox } And then add it to the configuration: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#prio-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #prio-dispatcher-config } And then an example on how you would use it: -.. includecode:: ../java/code/jdocs/dispatcher/DispatcherDocTest.java#prio-dispatcher +@@snip [DispatcherDocTest.java](../java/code/jdocs/dispatcher/DispatcherDocTest.java) { #prio-dispatcher } It is also possible to configure a mailbox type directly like this: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala - :include: prio-mailbox-config-java,mailbox-deployment-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #prio-mailbox-config-java #mailbox-deployment-config } And then use it either from deployment like this: -.. includecode:: code/jdocs/dispatcher/DispatcherDocTest.java#defining-mailbox-in-config +@@snip [DispatcherDocTest.java](code/jdocs/dispatcher/DispatcherDocTest.java) { #defining-mailbox-in-config } Or code like this: -.. includecode:: code/jdocs/dispatcher/DispatcherDocTest.java#defining-mailbox-in-code +@@snip [DispatcherDocTest.java](code/jdocs/dispatcher/DispatcherDocTest.java) { #defining-mailbox-in-code } -ControlAwareMailbox -------------------- +### ControlAwareMailbox -A ``ControlAwareMailbox`` can be very useful if an actor needs to be able to receive control messages +A `ControlAwareMailbox` can be very useful if an actor needs to be able to receive control messages immediately no matter how many other messages are already in its mailbox. It can be configured like this: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#control-aware-mailbox-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #control-aware-mailbox-config } -Control messages need to extend the ``ControlMessage`` trait: +Control messages need to extend the `ControlMessage` trait: -.. includecode:: ../java/code/jdocs/dispatcher/DispatcherDocTest.java#control-aware-mailbox-messages +@@snip [DispatcherDocTest.java](../java/code/jdocs/dispatcher/DispatcherDocTest.java) { #control-aware-mailbox-messages } And then an example on how you would use it: -.. includecode:: ../java/code/jdocs/dispatcher/DispatcherDocTest.java#control-aware-dispatcher +@@snip [DispatcherDocTest.java](../java/code/jdocs/dispatcher/DispatcherDocTest.java) { #control-aware-dispatcher } -Creating your own Mailbox type -============================== +## Creating your own Mailbox type An example is worth a thousand quacks: -.. includecode:: code/jdocs/dispatcher/MyUnboundedMailbox.java#mailbox-implementation-example +@@snip [MyUnboundedMailbox.java](code/jdocs/dispatcher/MyUnboundedMailbox.java) { #mailbox-implementation-example } -.. includecode:: code/jdocs/dispatcher/MyUnboundedMessageQueueSemantics.java#mailbox-implementation-example +@@snip [MyUnboundedMessageQueueSemantics.java](code/jdocs/dispatcher/MyUnboundedMessageQueueSemantics.java) { #mailbox-implementation-example } And then you just specify the FQCN of your MailboxType as the value of the "mailbox-type" in the dispatcher configuration, or the mailbox configuration. -.. note:: +@@@ note - Make sure to include a constructor which takes - ``akka.actor.ActorSystem.Settings`` and ``com.typesafe.config.Config`` - arguments, as this constructor is invoked reflectively to construct your - mailbox type. The config passed in as second argument is that section from - the configuration which describes the dispatcher or mailbox setting using - this mailbox type; the mailbox type will be instantiated once for each - dispatcher or mailbox setting using it. +Make sure to include a constructor which takes +`akka.actor.ActorSystem.Settings` and `com.typesafe.config.Config` +arguments, as this constructor is invoked reflectively to construct your +mailbox type. The config passed in as second argument is that section from +the configuration which describes the dispatcher or mailbox setting using +this mailbox type; the mailbox type will be instantiated once for each +dispatcher or mailbox setting using it. + +@@@ You can also use the mailbox as a requirement on the dispatcher like this: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#custom-mailbox-config-java - +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #custom-mailbox-config-java } Or by defining the requirement on your actor class like this: -.. includecode:: code/jdocs/dispatcher/DispatcherDocTest.java#require-mailbox-on-actor +@@snip [DispatcherDocTest.java](code/jdocs/dispatcher/DispatcherDocTest.java) { #require-mailbox-on-actor } +## Special Semantics of `system.actorOf` -Special Semantics of ``system.actorOf`` -======================================= - -In order to make ``system.actorOf`` both synchronous and non-blocking while -keeping the return type :class:`ActorRef` (and the semantics that the returned +In order to make `system.actorOf` both synchronous and non-blocking while +keeping the return type `ActorRef` (and the semantics that the returned ref is fully functional), special handling takes place for this case. Behind the scenes, a hollow kind of actor reference is constructed, which is sent to the system’s guardian actor who actually creates the actor and its context and puts those inside the reference. Until that has happened, messages sent to the -:class:`ActorRef` will be queued locally, and only upon swapping the real +`ActorRef` will be queued locally, and only upon swapping the real filling in will they be transferred into the real mailbox. Thus, -.. code-block:: scala - - final Props props = ... - // this actor uses MyCustomMailbox, which is assumed to be a singleton - system.actorOf(props.withDispatcher("myCustomMailbox").tell("bang", sender); - assert(MyCustomMailbox.getInstance().getLastEnqueued().equals("bang")); +```scala +final Props props = ... +// this actor uses MyCustomMailbox, which is assumed to be a singleton +system.actorOf(props.withDispatcher("myCustomMailbox").tell("bang", sender); +assert(MyCustomMailbox.getInstance().getLastEnqueued().equals("bang")); +``` will probably fail; you will have to allow for some time to pass and retry the -check à la :meth:`TestKit.awaitCond`. - +check à la `TestKit.awaitCond`. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/persistence-query-leveldb.md b/akka-docs/src/main/paradox/java/persistence-query-leveldb.md index 5d5cb1b213..6b23930f4a 100644 --- a/akka-docs/src/main/paradox/java/persistence-query-leveldb.md +++ b/akka-docs/src/main/paradox/java/persistence-query-leveldb.md @@ -1,112 +1,103 @@ -.. _persistence-query-leveldb-java: +# Persistence Query for LevelDB -############################# -Persistence Query for LevelDB -############################# - -This is documentation for the LevelDB implementation of the :ref:`persistence-query-java` API. +This is documentation for the LevelDB implementation of the @ref:[Persistence Query](persistence-query.md) API. Note that implementations for other journals may have different semantics. - -Dependencies -============ -Akka persistence LevelDB query implementation is bundled in the ``akka-persistence-query`` artifact. -Make sure that you have the following dependency in your project:: +## Dependencies - - com.typesafe.akka - akka-persistence-query_@binVersion@ - @version@ - +Akka persistence LevelDB query implementation is bundled in the `akka-persistence-query` artifact. +Make sure that you have the following dependency in your project: +``` + + com.typesafe.akka + akka-persistence-query_@binVersion@ + @version@ + +``` -How to get the ReadJournal -========================== +## How to get the ReadJournal -The ``ReadJournal`` is retrieved via the ``akka.persistence.query.PersistenceQuery`` +The `ReadJournal` is retrieved via the `akka.persistence.query.PersistenceQuery` extension: -.. includecode:: code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java#get-read-journal +@@snip [LeveldbPersistenceQueryDocTest.java](code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java) { #get-read-journal } -Supported Queries -================= +## Supported Queries -EventsByPersistenceIdQuery and CurrentEventsByPersistenceIdQuery ----------------------------------------------------------------- +### EventsByPersistenceIdQuery and CurrentEventsByPersistenceIdQuery -``eventsByPersistenceId`` is used for retrieving events for a specific ``PersistentActor`` -identified by ``persistenceId``. +`eventsByPersistenceId` is used for retrieving events for a specific `PersistentActor` +identified by `persistenceId`. -.. includecode:: code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java#EventsByPersistenceId +@@snip [LeveldbPersistenceQueryDocTest.java](code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java) { #EventsByPersistenceId } -You can retrieve a subset of all events by specifying ``fromSequenceNr`` and ``toSequenceNr`` -or use ``0L`` and ``Long.MAX_VALUE`` respectively to retrieve all events. Note that -the corresponding sequence number of each event is provided in the ``EventEnvelope``, +You can retrieve a subset of all events by specifying `fromSequenceNr` and `toSequenceNr` +or use `0L` and `Long.MAX_VALUE` respectively to retrieve all events. Note that +the corresponding sequence number of each event is provided in the `EventEnvelope`, which makes it possible to resume the stream at a later point from a given sequence number. The returned event stream is ordered by sequence number, i.e. the same order as the -``PersistentActor`` persisted the events. The same prefix of stream elements (in same order) +`PersistentActor` persisted the events. The same prefix of stream elements (in same order) are returned for multiple executions of the query, except for when events have been deleted. The stream is not completed when it reaches the end of the currently stored events, but it continues to push new events when new events are persisted. Corresponding query that is completed when it reaches the end of the currently -stored events is provided by ``currentEventsByPersistenceId``. +stored events is provided by `currentEventsByPersistenceId`. The LevelDB write journal is notifying the query side as soon as events are persisted, but for efficiency reasons the query side retrieves the events in batches that sometimes can -be delayed up to the configured ``refresh-interval`` or given ``RefreshInterval`` +be delayed up to the configured `refresh-interval` or given `RefreshInterval` hint. The stream is completed with failure if there is a failure in executing the query in the backend journal. -AllPersistenceIdsQuery and CurrentPersistenceIdsQuery ------------------------------------------------------ +### AllPersistenceIdsQuery and CurrentPersistenceIdsQuery -``allPersistenceIds`` is used for retrieving all ``persistenceIds`` of all persistent actors. +`allPersistenceIds` is used for retrieving all `persistenceIds` of all persistent actors. -.. includecode:: code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java#AllPersistenceIds +@@snip [LeveldbPersistenceQueryDocTest.java](code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java) { #AllPersistenceIds } The returned event stream is unordered and you can expect different order for multiple executions of the query. -The stream is not completed when it reaches the end of the currently used `persistenceIds`, -but it continues to push new `persistenceIds` when new persistent actors are created. +The stream is not completed when it reaches the end of the currently used *persistenceIds*, +but it continues to push new *persistenceIds* when new persistent actors are created. Corresponding query that is completed when it reaches the end of the -currently used `persistenceIds` is provided by ``currentPersistenceIds``. +currently used *persistenceIds* is provided by `currentPersistenceIds`. -The LevelDB write journal is notifying the query side as soon as new ``persistenceIds`` are +The LevelDB write journal is notifying the query side as soon as new `persistenceIds` are created and there is no periodic polling or batching involved in this query. The stream is completed with failure if there is a failure in executing the query in the backend journal. -EventsByTag and CurrentEventsByTag ----------------------------------- +### EventsByTag and CurrentEventsByTag -``eventsByTag`` is used for retrieving events that were marked with a given tag, e.g. +`eventsByTag` is used for retrieving events that were marked with a given tag, e.g. all domain events of an Aggregate Root type. -.. includecode:: code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java#EventsByTag +@@snip [LeveldbPersistenceQueryDocTest.java](code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java) { #EventsByTag } -To tag events you create an :ref:`event-adapters-java` that wraps the events in a ``akka.persistence.journal.Tagged`` -with the given ``tags``. +To tag events you create an @ref:[Event Adapters](persistence.md#event-adapters-java) that wraps the events in a `akka.persistence.journal.Tagged` +with the given `tags`. -.. includecode:: code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java#tagger +@@snip [LeveldbPersistenceQueryDocTest.java](code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java) { #tagger } -You can use ``NoOffset`` to retrieve all events with a given tag or retrieve a subset of all -events by specifying a ``Sequence`` ``offset``. The ``offset`` corresponds to an ordered sequence number for +You can use `NoOffset` to retrieve all events with a given tag or retrieve a subset of all +events by specifying a `Sequence` `offset`. The `offset` corresponds to an ordered sequence number for the specific tag. Note that the corresponding offset of each event is provided in the -``EventEnvelope``, which makes it possible to resume the stream at a later point from a given offset. +`EventEnvelope`, which makes it possible to resume the stream at a later point from a given offset. -The ``offset`` is exclusive, i.e. the event with the exact same sequence number will not be included -in the returned stream. This means that you can use the offset that is returned in ``EventEnvelope`` -as the ``offset`` parameter in a subsequent query. +The `offset` is exclusive, i.e. the event with the exact same sequence number will not be included +in the returned stream. This means that you can use the offset that is returned in `EventEnvelope` +as the `offset` parameter in a subsequent query. -In addition to the ``offset`` the ``EventEnvelope`` also provides ``persistenceId`` and ``sequenceNr`` -for each event. The ``sequenceNr`` is the sequence number for the persistent actor with the -``persistenceId`` that persisted the event. The ``persistenceId`` + ``sequenceNr`` is an unique +In addition to the `offset` the `EventEnvelope` also provides `persistenceId` and `sequenceNr` +for each event. The `sequenceNr` is the sequence number for the persistent actor with the +`persistenceId` that persisted the event. The `persistenceId` + `sequenceNr` is an unique identifier for the event. The returned event stream is ordered by the offset (tag sequence number), which corresponds @@ -114,30 +105,31 @@ to the same order as the write journal stored the events. The same stream elemen are returned for multiple executions of the query. Deleted events are not deleted from the tagged event stream. -.. note:: +@@@ note - Events deleted using ``deleteMessages(toSequenceNr)`` are not deleted from the "tagged stream". +Events deleted using `deleteMessages(toSequenceNr)` are not deleted from the "tagged stream". + +@@@ The stream is not completed when it reaches the end of the currently stored events, but it continues to push new events when new events are persisted. Corresponding query that is completed when it reaches the end of the currently -stored events is provided by ``currentEventsByTag``. +stored events is provided by `currentEventsByTag`. The LevelDB write journal is notifying the query side as soon as tagged events are persisted, but for efficiency reasons the query side retrieves the events in batches that sometimes can -be delayed up to the configured ``refresh-interval`` or given ``RefreshInterval`` +be delayed up to the configured `refresh-interval` or given `RefreshInterval` hint. The stream is completed with failure if there is a failure in executing the query in the backend journal. -Configuration -============= +## Configuration Configuration settings can be defined in the configuration section with the -absolute path corresponding to the identifier, which is ``"akka.persistence.query.journal.leveldb"`` -for the default ``LeveldbReadJournal.Identifier``. +absolute path corresponding to the identifier, which is `"akka.persistence.query.journal.leveldb"` +for the default `LeveldbReadJournal.Identifier`. It can be configured with the following properties: -.. includecode:: ../../../akka-persistence-query/src/main/resources/reference.conf#query-leveldb +@@snip [reference.conf]../../../../../akka-persistence-query/src/main/resources/reference.conf) { #query-leveldb } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/persistence-query.md b/akka-docs/src/main/paradox/java/persistence-query.md index 2ebd9fd5d6..34f5a5a8fe 100644 --- a/akka-docs/src/main/paradox/java/persistence-query.md +++ b/akka-docs/src/main/paradox/java/persistence-query.md @@ -1,10 +1,6 @@ -.. _persistence-query-java: +# Persistence Query -################# -Persistence Query -################# - -Akka persistence query complements :ref:`persistence-java` by providing a universal asynchronous stream based +Akka persistence query complements @ref:[Persistence](persistence.md) by providing a universal asynchronous stream based query interface that various journal plugins can implement in order to expose their query capabilities. The most typical use case of persistence query is implementing the so-called query side (also known as "read side") @@ -14,19 +10,19 @@ side of an application, however it can help to migrate data from the write side simple scenarios Persistence Query may be powerful enough to fulfill the query needs of your app, however we highly recommend (in the spirit of CQRS) of splitting up the write/read sides into separate datastores as the need arises. -Dependencies -============ +## Dependencies -Akka persistence query is a separate jar file. Make sure that you have the following dependency in your project:: +Akka persistence query is a separate jar file. Make sure that you have the following dependency in your project: - - com.typesafe.akka - akka-persistence-query_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-persistence-query_@binVersion@ + @version@ + +``` -Design overview -=============== +## Design overview Akka persistence query is purposely designed to be a very loosely specified API. This is in order to keep the provided APIs general enough for each journal implementation to be able to expose its best @@ -39,120 +35,119 @@ Refer to your journal's plugins documentation for details on which queries and s While Akka Persistence Query does not provide actual implementations of ReadJournals, it defines a number of pre-defined query types for the most common query scenarios, that most journals are likely to implement (however they are not required to). -Read Journals -============= +## Read Journals -In order to issue queries one has to first obtain an instance of a ``ReadJournal``. -Read journals are implemented as `Community plugins`_, each targeting a specific datastore (for example Cassandra or JDBC -databases). For example, given a library that provides a ``akka.persistence.query.my-read-journal`` obtaining the related +In order to issue queries one has to first obtain an instance of a `ReadJournal`. +Read journals are implemented as [Community plugins](http://akka.io/community/#plugins-to-akka-persistence-query), each targeting a specific datastore (for example Cassandra or JDBC +databases). For example, given a library that provides a `akka.persistence.query.my-read-journal` obtaining the related journal is as simple as: -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#basic-usage +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #basic-usage } Journal implementers are encouraged to put this identifier in a variable known to the user, such that one can access it via -``getJournalFor(NoopJournal.class, NoopJournal.identifier)``, however this is not enforced. +`getJournalFor(NoopJournal.class, NoopJournal.identifier)`, however this is not enforced. -Read journal implementations are available as `Community plugins`_. +Read journal implementations are available as [Community plugins](http://akka.io/community/#plugins-to-akka-persistence-query). +### Predefined queries -Predefined queries ------------------- Akka persistence query comes with a number of query interfaces built in and suggests Journal implementors to implement them according to the semantics described below. It is important to notice that while these query types are very common a journal is not obliged to implement all of them - for example because in a given journal such query would be significantly inefficient. -.. note:: - Refer to the documentation of the :class:`ReadJournal` plugin you are using for a specific list of supported query types. - For example, Journal plugins should document their stream completion strategies. +@@@ note + +Refer to the documentation of the `ReadJournal` plugin you are using for a specific list of supported query types. +For example, Journal plugins should document their stream completion strategies. + +@@@ The predefined queries are: -AllPersistenceIdsQuery and CurrentPersistenceIdsQuery -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### AllPersistenceIdsQuery and CurrentPersistenceIdsQuery -``allPersistenceIds`` which is designed to allow users to subscribe to a stream of all persistent ids in the system. +`allPersistenceIds` which is designed to allow users to subscribe to a stream of all persistent ids in the system. By default this stream should be assumed to be a "live" stream, which means that the journal should keep emitting new persistence ids as they come into the system: -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#all-persistence-ids-live +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #all-persistence-ids-live } -If your usage does not require a live stream, you can use the ``currentPersistenceIds`` query: +If your usage does not require a live stream, you can use the `currentPersistenceIds` query: -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#all-persistence-ids-snap +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #all-persistence-ids-snap } -EventsByPersistenceIdQuery and CurrentEventsByPersistenceIdQuery -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### EventsByPersistenceIdQuery and CurrentEventsByPersistenceIdQuery -``eventsByPersistenceId`` is a query equivalent to replaying a :ref:`PersistentActor `, +`eventsByPersistenceId` is a query equivalent to replaying a @ref:[PersistentActor](persistence.md#event-sourcing-java), however, since it is a stream it is possible to keep it alive and watch for additional incoming events persisted by the -persistent actor identified by the given ``persistenceId``. +persistent actor identified by the given `persistenceId`. -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#events-by-persistent-id +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #events-by-persistent-id } Most journals will have to revert to polling in order to achieve this, -which can typically be configured with a ``refresh-interval`` configuration property. +which can typically be configured with a `refresh-interval` configuration property. -If your usage does not require a live stream, you can use the ``currentEventsByPersistenceId`` query. +If your usage does not require a live stream, you can use the `currentEventsByPersistenceId` query. -EventsByTag and CurrentEventsByTag -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### EventsByTag and CurrentEventsByTag -``eventsByTag`` allows querying events regardless of which ``persistenceId`` they are associated with. This query is hard to +`eventsByTag` allows querying events regardless of which `persistenceId` they are associated with. This query is hard to implement in some journals or may need some additional preparation of the used data store to be executed efficiently. The goal of this query is to allow querying for all events which are "tagged" with a specific tag. That includes the use case to query all domain events of an Aggregate Root type. Please refer to your read journal plugin's documentation to find out if and how it is supported. -Some journals may support tagging of events via an :ref:`event-adapters-java` that wraps the events in a -``akka.persistence.journal.Tagged`` with the given ``tags``. The journal may support other ways of doing tagging - again, +Some journals may support tagging of events via an @ref:[Event Adapters](persistence.md#event-adapters-java) that wraps the events in a +`akka.persistence.journal.Tagged` with the given `tags`. The journal may support other ways of doing tagging - again, how exactly this is implemented depends on the used journal. Here is an example of such a tagging event adapter: -.. includecode:: code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java#tagger +@@snip [LeveldbPersistenceQueryDocTest.java](code/jdocs/persistence/query/LeveldbPersistenceQueryDocTest.java) { #tagger } -.. note:: - A very important thing to keep in mind when using queries spanning multiple persistenceIds, such as ``EventsByTag`` - is that the order of events at which the events appear in the stream rarely is guaranteed (or stable between materializations). +@@@ note - Journals *may* choose to opt for strict ordering of the events, and should then document explicitly what kind of ordering - guarantee they provide - for example "*ordered by timestamp ascending, independently of persistenceId*" is easy to achieve - on relational databases, yet may be hard to implement efficiently on plain key-value datastores. +A very important thing to keep in mind when using queries spanning multiple persistenceIds, such as `EventsByTag` +is that the order of events at which the events appear in the stream rarely is guaranteed (or stable between materializations). + +Journals *may* choose to opt for strict ordering of the events, and should then document explicitly what kind of ordering +guarantee they provide - for example "*ordered by timestamp ascending, independently of persistenceId*" is easy to achieve +on relational databases, yet may be hard to implement efficiently on plain key-value datastores. + +@@@ In the example below we query all events which have been tagged (we assume this was performed by the write-side using an -:ref:`EventAdapter `, or that the journal is smart enough that it can figure out what we mean by this -tag - for example if the journal stored the events as json it may try to find those with the field ``tag`` set to this value etc.). +@ref:[EventAdapter](persistence.md#event-adapters-java), or that the journal is smart enough that it can figure out what we mean by this +tag - for example if the journal stored the events as json it may try to find those with the field `tag` set to this value etc.). -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#events-by-tag +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #events-by-tag } -As you can see, we can use all the usual stream combinators available from :ref:`streams-java` on the resulting query stream, -including for example taking the first 10 and cancelling the stream. It is worth pointing out that the built-in ``EventsByTag`` -query has an optionally supported offset parameter (of type ``Long``) which the journals can use to implement resumable-streams. +As you can see, we can use all the usual stream combinators available from @ref:[Streams](stream/index.md) on the resulting query stream, +including for example taking the first 10 and cancelling the stream. It is worth pointing out that the built-in `EventsByTag` +query has an optionally supported offset parameter (of type `Long`) which the journals can use to implement resumable-streams. For example a journal may be able to use a WHERE clause to begin the read starting from a specific row, or in a datastore that is able to order events by insertion time it could treat the Long as a timestamp and select only older events. -If your usage does not require a live stream, you can use the ``currentEventsByTag`` query. +If your usage does not require a live stream, you can use the `currentEventsByTag` query. -Materialized values of queries ------------------------------- -Journals are able to provide additional information related to a query by exposing :ref:`materialized-values-quick-java`, -which are a feature of :ref:`streams-java` that allows to expose additional values at stream materialization time. +### Materialized values of queries + +Journals are able to provide additional information related to a query by exposing @ref:[Materialized values](stream/stream-quickstart.md#materialized-values-quick-java), +which are a feature of @ref:[Streams](stream/index.md) that allows to expose additional values at stream materialization time. More advanced query journals may use this technique to expose information about the character of the materialized stream, for example if it's finite or infinite, strictly ordered or not ordered at all. The materialized value type -is defined as the second type parameter of the returned ``Source``, which allows journals to provide users with their +is defined as the second type parameter of the returned `Source`, which allows journals to provide users with their specialised query object, as demonstrated in the sample below: -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#advanced-journal-query-types +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #advanced-journal-query-types } -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#advanced-journal-query-definition +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #advanced-journal-query-definition } -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#advanced-journal-query-usage +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #advanced-journal-query-usage } -.. _Community plugins: http://akka.io/community/#plugins-to-akka-persistence-query +## Performance and denormalization -Performance and denormalization -=============================== -When building systems using :ref:`event-sourcing-java` and CQRS (`Command & Query Responsibility Segregation`_) techniques +When building systems using @ref:[Event sourcing](persistence.md#event-sourcing-java) and CQRS ([Command & Query Responsibility Segregation](https://msdn.microsoft.com/en-us/library/jj554200.aspx)) techniques it is tremendously important to realise that the write-side has completely different needs from the read-side, and separating those concerns into datastores that are optimised for either side makes it possible to offer the best experience for the write and read sides independently. @@ -166,103 +161,99 @@ to figure out best bidding strategies and trends – this often requires some ki for example SQL or writing Spark jobs to analyse the data. Therefore the data stored in the write-side needs to be projected into the other read-optimised datastore. -.. note:: - When referring to **Materialized Views** in Akka Persistence think of it as "some persistent storage of the result of a Query". - In other words, it means that the view is created once, in order to be afterwards queried multiple times, as in this format - it may be more efficient or interesting to query it (instead of the source events directly). +@@@ note -Materialize view to Reactive Streams compatible datastore ---------------------------------------------------------- +When referring to **Materialized Views** in Akka Persistence think of it as "some persistent storage of the result of a Query". +In other words, it means that the view is created once, in order to be afterwards queried multiple times, as in this format +it may be more efficient or interesting to query it (instead of the source events directly). -If the read datastore exposes a `Reactive Streams`_ interface then implementing a simple projection +@@@ + +### Materialize view to Reactive Streams compatible datastore + +If the read datastore exposes a [Reactive Streams](http://reactive-streams.org) interface then implementing a simple projection is as simple as, using the read-journal and feeding it into the databases driver interface, for example like so: -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#projection-into-different-store-rs +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #projection-into-different-store-rs } -.. _Reactive Streams: http://reactive-streams.org +### Materialize view using mapAsync -Materialize view using mapAsync -------------------------------- - -If the target database does not provide a reactive streams ``Subscriber`` that can perform writes, +If the target database does not provide a reactive streams `Subscriber` that can perform writes, you may have to implement the write logic using plain functions or Actors instead. In case your write logic is state-less and you just need to convert the events from one data type to another before writing into the alternative datastore, then the projection is as simple as: -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#projection-into-different-store-simple-classes -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#projection-into-different-store-simple +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #projection-into-different-store-simple-classes } -Resumable projections ---------------------- +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #projection-into-different-store-simple } + +### Resumable projections Sometimes you may need to implement "resumable" projections, that will not start from the beginning of time each time -when run. In this case you will need to store the sequence number (or ``offset``) of the processed event and use it +when run. In this case you will need to store the sequence number (or `offset`) of the processed event and use it the next time this projection is started. This pattern is not built-in, however is rather simple to implement yourself. The example below additionally highlights how you would use Actors to implement the write side, in case you need to do some complex logic that would be best handled inside an Actor before persisting the event into the other datastore: -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#projection-into-different-store-actor-run +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #projection-into-different-store-actor-run } -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#projection-into-different-store-actor +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #projection-into-different-store-actor } -.. _Command & Query Responsibility Segregation: https://msdn.microsoft.com/en-us/library/jj554200.aspx + +## Query plugins -.. _read-journal-plugin-api-java: +Query plugins are various (mostly community driven) `ReadJournal` implementations for all kinds +of available datastores. The complete list of available plugins is maintained on the Akka Persistence Query [Community Plugins](http://akka.io/community/#plugins-to-akka-persistence-query) page. -Query plugins -============= - -Query plugins are various (mostly community driven) :class:`ReadJournal` implementations for all kinds -of available datastores. The complete list of available plugins is maintained on the Akka Persistence Query `Community Plugins`_ page. - -The plugin for LevelDB is described in :ref:`persistence-query-leveldb-java`. +The plugin for LevelDB is described in @ref:[Persistence Query for LevelDB](persistence-query-leveldb.md). This section aims to provide tips and guide plugin developers through implementing a custom query plugin. Most users will not need to implement journals themselves, except if targeting a not yet supported datastore. -.. note:: - Since different data stores provide different query capabilities journal plugins **must extensively document** - their exposed semantics as well as handled query scenarios. +@@@ note -ReadJournal plugin API ----------------------- +Since different data stores provide different query capabilities journal plugins **must extensively document** +their exposed semantics as well as handled query scenarios. -A read journal plugin must implement ``akka.persistence.query.ReadJournalProvider`` which -creates instances of ``akka.persistence.query.scaladsl.ReadJournal`` and -``akka.persistence.query.javaadsl.ReadJournal``. The plugin must implement both the ``scaladsl`` -and the ``javadsl`` interfaces because the ``akka.stream.scaladsl.Source`` and -``akka.stream.javadsl.Source`` are different types and even though those types can easily be converted -to each other it is most convenient for the end user to get access to the Java or Scala ``Source`` directly. +@@@ + +### ReadJournal plugin API + +A read journal plugin must implement `akka.persistence.query.ReadJournalProvider` which +creates instances of `akka.persistence.query.scaladsl.ReadJournal` and +`akka.persistence.query.javaadsl.ReadJournal`. The plugin must implement both the `scaladsl` +and the `javadsl` interfaces because the `akka.stream.scaladsl.Source` and +`akka.stream.javadsl.Source` are different types and even though those types can easily be converted +to each other it is most convenient for the end user to get access to the Java or Scala `Source` directly. As illustrated below one of the implementations can delegate to the other. Below is a simple journal implementation: -.. includecode:: code/jdocs/persistence/PersistenceQueryDocTest.java#my-read-journal +@@snip [PersistenceQueryDocTest.java](code/jdocs/persistence/PersistenceQueryDocTest.java) { #my-read-journal } -And the ``EventsByTag`` could be backed by such an Actor for example: +And the `EventsByTag` could be backed by such an Actor for example: -.. includecode:: code/jdocs/persistence/query/MyEventsByTagJavaPublisher.java#events-by-tag-publisher +@@snip [MyEventsByTagJavaPublisher.java](code/jdocs/persistence/query/MyEventsByTagJavaPublisher.java) { #events-by-tag-publisher } -The ``ReadJournalProvider`` class must have a constructor with one of these signatures: +The `ReadJournalProvider` class must have a constructor with one of these signatures: -* constructor with a ``ExtendedActorSystem`` parameter, a ``com.typesafe.config.Config`` parameter, and a ``String`` parameter for the config path -* constructor with a ``ExtendedActorSystem`` parameter, and a ``com.typesafe.config.Config`` parameter -* constructor with one ``ExtendedActorSystem`` parameter -* constructor without parameters + * constructor with a `ExtendedActorSystem` parameter, a `com.typesafe.config.Config` parameter, and a `String` parameter for the config path + * constructor with a `ExtendedActorSystem` parameter, and a `com.typesafe.config.Config` parameter + * constructor with one `ExtendedActorSystem` parameter + * constructor without parameters The plugin section of the actor system's config will be passed in the config constructor parameter. The config path -of the plugin is passed in the ``String`` parameter. +of the plugin is passed in the `String` parameter. If the underlying datastore only supports queries that are completed when they reach the end of the "result set", the journal has to submit new queries after a while in order to support "infinite" event streams that include events stored after the initial query has completed. It is recommended that the plugin use a configuration property named -``refresh-interval`` for defining such a refresh interval. +`refresh-interval` for defining such a refresh interval. -Plugin TCK ----------- +### Plugin TCK -TODO, not available yet. +TODO, not available yet. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/persistence-schema-evolution.md b/akka-docs/src/main/paradox/java/persistence-schema-evolution.md index eb78f6485a..8f6aa1ff39 100644 --- a/akka-docs/src/main/paradox/java/persistence-schema-evolution.md +++ b/akka-docs/src/main/paradox/java/persistence-schema-evolution.md @@ -1,10 +1,6 @@ -.. _persistence-schema-evolution-java: +# Persistence - Schema Evolution -############################## -Persistence - Schema Evolution -############################## - -When working on long running projects using :ref:`persistence-java`, or any kind of `Event Sourcing`_ architectures, +When working on long running projects using @ref:[Persistence](persistence.md), or any kind of [Event Sourcing](http://martinfowler.com/eaaDev/EventSourcing.html) architectures, schema evolution becomes one of the more important technical aspects of developing your application. The requirements as well as our own understanding of the business domain may (and will) change in time. @@ -15,61 +11,59 @@ lifecycle that could mean that no-one is really using it actively. In this chapter we will investigate various schema evolution strategies and techniques from which you can pick and choose the ones that match your domain and challenge at hand. -.. note:: - This page proposes a number of possible solutions to the schema evolution problem and explains how some of the - utilities Akka provides can be used to achieve this, it is by no means a complete (closed) set of solutions. +@@@ note - Sometimes, based on the capabilities of your serialization formats, you may be able to evolve your schema in - different ways than outlined in the sections below. If you discover useful patterns or techniques for schema - evolution feel free to submit Pull Requests to this page to extend it. +This page proposes a number of possible solutions to the schema evolution problem and explains how some of the +utilities Akka provides can be used to achieve this, it is by no means a complete (closed) set of solutions. +Sometimes, based on the capabilities of your serialization formats, you may be able to evolve your schema in +different ways than outlined in the sections below. If you discover useful patterns or techniques for schema +evolution feel free to submit Pull Requests to this page to extend it. -Schema evolution in event-sourced systems -========================================= +@@@ + +## Schema evolution in event-sourced systems In recent years we have observed a tremendous move towards immutable append-only datastores, with event-sourcing being the prime technique successfully being used in these settings. For an excellent overview why and how immutable data makes scalability -and systems design much simpler you may want to read Pat Helland's excellent `Immutability Changes Everything`_ whitepaper. +and systems design much simpler you may want to read Pat Helland's excellent [Immutability Changes Everything](http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf) whitepaper. -Since with `Event Sourcing`_ the **events are immutable** and usually never deleted – the way schema evolution is handled +Since with [Event Sourcing](http://martinfowler.com/eaaDev/EventSourcing.html) the **events are immutable** and usually never deleted – the way schema evolution is handled differs from how one would go about it in a mutable database setting (e.g. in typical CRUD database applications). The system needs to be able to continue to work in the presence of "old" events which were stored under the "old" schema. We also want to limit complexity in the business logic layer, exposing a consistent view over all of the events of a given -type to :class:`PersistentActor` s and :ref:`persistence queries `. This allows the business logic layer to focus on solving business problems +type to `PersistentActor` s and @ref:[persistence queries](persistence-query.md). This allows the business logic layer to focus on solving business problems instead of having to explicitly deal with different schemas. - In summary, schema evolution in event sourced systems exposes the following characteristics: - - Allow the system to continue operating without large scale migrations to be applied, - - Allow the system to read "old" events from the underlying storage, however present them in a "new" view to the application logic, - - Transparently promote events to the latest versions during recovery (or queries) such that the business logic need not consider multiple versions of events +: + * Allow the system to continue operating without large scale migrations to be applied, + * Allow the system to read "old" events from the underlying storage, however present them in a "new" view to the application logic, + * Transparently promote events to the latest versions during recovery (or queries) such that the business logic need not consider multiple versions of events -.. _Immutability Changes Everything: http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf -.. _Event Sourcing: http://martinfowler.com/eaaDev/EventSourcing.html -Types of schema evolution -------------------------- +### Types of schema evolution + Before we explain the various techniques that can be used to safely evolve the schema of your persistent events over time, we first need to define what the actual problem is, and what the typical styles of changes are. Since events are never deleted, we need to have a way to be able to replay (read) old events, in such way -that does not force the ``PersistentActor`` to be aware of all possible versions of an event that it may have +that does not force the `PersistentActor` to be aware of all possible versions of an event that it may have persisted in the past. Instead, we want the Actors to work on some form of "latest" version of the event and provide some means of either converting old "versions" of stored events into this "latest" event type, or constantly evolve the event definition - in a backwards compatible way - such that the new deserialization code can still read old events. The most common schema changes you will likely are: -- :ref:`adding a field to an event type `, -- :ref:`remove or rename field in event type `, -- :ref:`remove event type `, -- :ref:`split event into multiple smaller events `. + * [adding a field to an event type](#add-field-java), + * [remove or rename field in event type](#rename-field-java), + * [remove event type](#remove-event-class-java), + * [split event into multiple smaller events](#split-large-event-into-smaller-java). The following sections will explain some patterns which can be used to safely evolve your schema when facing those changes. -Picking the right serialization format -====================================== +## Picking the right serialization format Picking the serialization format is a very important decision you will have to make while building your application. It affects which kind of evolutions are simple (or hard) to do, how much work is required to add a new datatype, and, @@ -82,158 +76,151 @@ an event-log from one serialization format to another one, however it may be a m to perform this on a live system. Binary serialization formats that we have seen work well for long-lived applications include the very flexible IDL based: -`Google Protobuf`_, `Apache Thrift`_ or `Apache Avro`_. Avro schema evolution is more "entire schema" based, instead of +[Google Protobuf](https://developers.google.com/protocol-buffers), [Apache Thrift](https://thrift.apache.org/) or [Apache Avro](https://avro.apache.org). Avro schema evolution is more "entire schema" based, instead of single fields focused like in protobuf or thrift, and usually requires using some kind of schema registry. Users who want their data to be human-readable directly in the write-side -datastore may opt to use plain-old `JSON`_ as the storage format, though that comes at a cost of lacking support for schema +datastore may opt to use plain-old [JSON](http://json.org) as the storage format, though that comes at a cost of lacking support for schema evolution and relatively large marshalling latency. There are plenty excellent blog posts explaining the various trade-offs between popular serialization formats, -one post we would like to highlight is the very well illustrated `Schema evolution in Avro, Protocol Buffers and Thrift`_ +one post we would like to highlight is the very well illustrated [Schema evolution in Avro, Protocol Buffers and Thrift](http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html) by Martin Kleppmann. -.. _Google Protobuf: https://developers.google.com/protocol-buffers -.. _Apache Avro: https://avro.apache.org -.. _JSON: http://json.org -.. _Schema evolution in Avro, Protocol Buffers and Thrift: http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html +### Provided default serializers -Provided default serializers ----------------------------- - -Akka Persistence provides `Google Protocol Buffers`_ based serializers (using :ref:`Akka Serialization `) -for it's own message types such as ``PersistentRepr``, ``AtomicWrite`` and snapshots. Journal plugin implementations +Akka Persistence provides [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) based serializers (using @ref:[Akka Serialization](serialization.md)) +for it's own message types such as `PersistentRepr`, `AtomicWrite` and snapshots. Journal plugin implementations *may* choose to use those provided serializers, or pick a serializer which suits the underlying database better. -.. note:: - Serialization is **NOT** handled automatically by Akka Persistence itself. Instead, it only provides the above described - serializers, and in case a ``AsyncWriteJournal`` plugin implementation chooses to use them directly, the above serialization - scheme will be used. +@@@ note - Please refer to your write journal's documentation to learn more about how it handles serialization! +Serialization is **NOT** handled automatically by Akka Persistence itself. Instead, it only provides the above described +serializers, and in case a `AsyncWriteJournal` plugin implementation chooses to use them directly, the above serialization +scheme will be used. - For example, some journals may choose to not use Akka Serialization *at all* and instead store the data in a format - that is more "native" for the underlying datastore, e.g. using JSON or some other kind of format that the target - datastore understands directly. +Please refer to your write journal's documentation to learn more about how it handles serialization! + +For example, some journals may choose to not use Akka Serialization *at all* and instead store the data in a format +that is more "native" for the underlying datastore, e.g. using JSON or some other kind of format that the target +datastore understands directly. + +@@@ The below figure explains how the default serialization scheme works, and how it fits together with serializing the -user provided message itself, which we will from here on refer to as the ``payload`` (highlighted in yellow): +user provided message itself, which we will from here on refer to as the `payload` (highlighted in yellow): -.. figure:: ../images/persistent-message-envelope.png - :align: center +![persistent-message-envelope.png](../images/persistent-message-envelope.png) +> +Akka Persistence provided serializers wrap the user payload in an envelope containing all persistence-relevant information. +**If the Journal uses provided Protobuf serializers for the wrapper types (e.g. PersistentRepr), then the payload will +be serialized using the user configured serializer, and if none is provided explicitly, Java serialization will be used for it.** - Akka Persistence provided serializers wrap the user payload in an envelope containing all persistence-relevant information. - **If the Journal uses provided Protobuf serializers for the wrapper types (e.g. PersistentRepr), then the payload will - be serialized using the user configured serializer, and if none is provided explicitly, Java serialization will be used for it.** +The blue colored regions of the `PersistentMessage` indicate what is serialized using the generated protocol buffers +serializers, and the yellow payload indicates the user provided event (by calling `persist(payload)(...)`). +As you can see, the `PersistentMessage` acts as an envelope around the payload, adding various fields related to the +origin of the event (`persistenceId`, `sequenceNr` and more). -The blue colored regions of the ``PersistentMessage`` indicate what is serialized using the generated protocol buffers -serializers, and the yellow payload indicates the user provided event (by calling ``persist(payload)(...)``). -As you can see, the ``PersistentMessage`` acts as an envelope around the payload, adding various fields related to the -origin of the event (``persistenceId``, ``sequenceNr`` and more). - -More advanced techniques (e.g. :ref:`remove-event-class-java`) will dive into using the manifests for increasing the +More advanced techniques (e.g. [Remove event class and ignore events](#remove-event-class-java)) will dive into using the manifests for increasing the flexibility of the persisted vs. exposed types even more. However for now we will focus on the simpler evolution techniques, concerning simply configuring the payload serializers. -By default the ``payload`` will be serialized using Java Serialization. This is fine for testing and initial phases +By default the `payload` will be serialized using Java Serialization. This is fine for testing and initial phases of your development (while you're still figuring out things and the data will not need to stay persisted forever). However, once you move to production you should really *pick a different serializer for your payloads*. -.. warning:: - Do not rely on Java serialization (which will be picked by Akka by default if you don't specify any serializers) - for *serious* application development! It does not lean itself well to evolving schemas over long periods of time, - and its performance is also not very high (it never was designed for high-throughput scenarios). +@@@ warning -.. _Google Protocol Buffers: https://developers.google.com/protocol-buffers/ -.. _Apache Thrift: https://thrift.apache.org/ +Do not rely on Java serialization (which will be picked by Akka by default if you don't specify any serializers) +for *serious* application development! It does not lean itself well to evolving schemas over long periods of time, +and its performance is also not very high (it never was designed for high-throughput scenarios). -Configuring payload serializers -------------------------------- -This section aims to highlight the complete basics on how to define custom serializers using :ref:`Akka Serialization `. +@@@ + +### Configuring payload serializers + +This section aims to highlight the complete basics on how to define custom serializers using @ref:[Akka Serialization](serialization.md). Many journal plugin implementations use Akka Serialization, thus it is tremendously important to understand how to configure it to work with your event classes. -.. note:: - Read the :ref:`Akka Serialization ` docs to learn more about defining custom serializers, - to improve performance and maintainability of your system. Do not depend on Java serialization for production deployments. +@@@ note + +Read the @ref:[Akka Serialization](serialization.md) docs to learn more about defining custom serializers, +to improve performance and maintainability of your system. Do not depend on Java serialization for production deployments. + +@@@ The below snippet explains in the minimal amount of lines how a custom serializer can be registered. For more in-depth explanations on how serialization picks the serializer to use etc, please refer to its documentation. First we start by defining our domain model class, here representing a person: -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#simplest-custom-serializer-model +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #simplest-custom-serializer-model } -Next we implement a serializer (or extend an existing one to be able to handle the new ``Person`` class): +Next we implement a serializer (or extend an existing one to be able to handle the new `Person` class): -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#simplest-custom-serializer +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #simplest-custom-serializer } -And finally we register the serializer and bind it to handle the ``docs.persistence.Person`` class: +And finally we register the serializer and bind it to handle the `docs.persistence.Person` class: -.. includecode:: ../scala/code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#simplest-custom-serializer-config +@@snip [PersistenceSchemaEvolutionDocSpec.scala](../scala/code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #simplest-custom-serializer-config } Deserialization will be performed by the same serializer which serialized the message initially -because of the ``identifier`` being stored together with the message. +because of the `identifier` being stored together with the message. -Please refer to the :ref:`Akka Serialization ` documentation for more advanced use of serializers, -especially the :ref:`string-manifest-serializer-java` section since it is very useful for Persistence based applications +Please refer to the @ref:[Akka Serialization](serialization.md) documentation for more advanced use of serializers, +especially the @ref:[Serializer with String Manifest](serialization.md#string-manifest-serializer-java) section since it is very useful for Persistence based applications dealing with schema evolutions, as we will see in some of the examples below. -Schema evolution in action -========================== +## Schema evolution in action In this section we will discuss various schema evolution techniques using concrete examples and explaining some of the various options one might go about handling the described situation. The list below is by no means a complete guide, so feel free to adapt these techniques depending on your serializer's capabilities and/or other domain specific limitations. -.. _add-field-java: - -Add fields ----------- + +### Add fields **Situation:** -You need to add a field to an existing message type. For example, a ``SeatReserved(String letter, int row)`` now +You need to add a field to an existing message type. For example, a `SeatReserved(String letter, int row)` now needs to have an associated code which indicates if it is a window or aisle seat. **Solution:** Adding fields is the most common change you'll need to apply to your messages so make sure the serialization format you picked for your payloads can handle it apropriately, i.e. such changes should be *binary compatible*. -This is easily achieved using the right serializer toolkit – we recommend something like `Google Protocol Buffers`_ or -`Apache Thrift`_ however other tools may fit your needs just as well – picking a serializer backend is something +This is easily achieved using the right serializer toolkit – we recommend something like [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) or +[Apache Thrift](https://thrift.apache.org/) however other tools may fit your needs just as well – picking a serializer backend is something you should research before picking one to run with. In the following examples we will be using protobuf, mostly because we are familiar with it, it does its job well and Akka is using it internally as well. While being able to read messages with missing fields is half of the solution, you also need to deal with the missing -values somehow. This is usually modeled as some kind of default value, or by representing the field as an ``Optional`` +values somehow. This is usually modeled as some kind of default value, or by representing the field as an `Optional` See below for an example how reading an optional field from a serialized protocol buffers message might look like. -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#protobuf-read-optional-model +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #protobuf-read-optional-model } Next we prepare an protocol definition using the protobuf Interface Description Language, which we'll use to generate the serializer code to be used on the Akka Serialization layer (notice that the schema aproach allows us to easily rename fields, as long as the numeric identifiers of the fields do not change): -.. includecode:: ../../src/main/protobuf/FlightAppModels.proto#protobuf-read-optional-proto +@@snip [FlightAppModels.proto]../../protobuf/FlightAppModels.proto) { #protobuf-read-optional-proto } The serializer implementation uses the protobuf generated classes to marshall the payloads. -Optional fields can be handled explicitly or missing values by calling the ``has...`` methods on the protobuf object, -which we do for ``seatType`` in order to use a ``Unknown`` type in case the event was stored before we had introduced +Optional fields can be handled explicitly or missing values by calling the `has...` methods on the protobuf object, +which we do for `seatType` in order to use a `Unknown` type in case the event was stored before we had introduced the field to this event type: -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#protobuf-read-optional +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #protobuf-read-optional } -.. _rename-field-java: - -Rename fields -------------- + +### Rename fields **Situation:** -When first designing the system the ``SeatReserved`` event featured a ``code`` field. -After some time you discover that what was originally called ``code`` actually means ``seatNr``, thus the model +When first designing the system the `SeatReserved` event featured a `code` field. +After some time you discover that what was originally called `code` actually means `seatNr`, thus the model should be changed to reflect this concept more accurately. - **Solution 1 - using IDL based serializers:** First, we will discuss the most efficient way of dealing with such kinds of schema changes – IDL based serializers. @@ -246,58 +233,59 @@ representation of the message. This is one of the advantages of schema based ser add the overhead of having to maintain the schema. When using serializers like this, no additional code change (except renaming the field and method used during serialization) is needed to perform such evolution: -.. figure:: ../images/persistence-serializer-rename.png - :align: center - +![persistence-serializer-rename.png](../images/persistence-serializer-rename.png) +> This is how such a rename would look in protobuf: -.. includecode:: ../scala/code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#protobuf-rename-proto +@@snip [PersistenceSchemaEvolutionDocSpec.scala](../scala/code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #protobuf-rename-proto } It is important to learn about the strengths and limitations of your serializers, in order to be able to move swiftly and refactor your models fearlessly as you go on with the project. -.. note:: - Learn in-depth about the serialization engine you're using as it will impact how you can aproach schema evolution. +@@@ note - Some operations are "free" in certain serialization formats (more often than not: removing/adding optional fields, - sometimes renaming fields etc.), while some other operations are strictly not possible. +Learn in-depth about the serialization engine you're using as it will impact how you can aproach schema evolution. + +Some operations are "free" in certain serialization formats (more often than not: removing/adding optional fields, +sometimes renaming fields etc.), while some other operations are strictly not possible. + +@@@ **Solution 2 - by manually handling the event versions:** Another solution, in case your serialization format does not support renames as easily as the above mentioned formats, -is versioning your schema. For example, you could have made your events carry an additional field called ``_version`` -which was set to ``1`` (because it was the initial schema), and once you change the schema you bump this number to ``2``, +is versioning your schema. For example, you could have made your events carry an additional field called `_version` +which was set to `1` (because it was the initial schema), and once you change the schema you bump this number to `2`, and write an adapter which can perform the rename. This approach is popular when your serialization format is something like JSON, where renames can not be performed automatically by the serializer. You can do these kinds of "promotions" either manually (as shown in the example below) -or using a library like `Stamina`_ which helps to create those ``V1->V2->V3->...->Vn`` promotion chains without much boilerplate. - -.. figure:: ../images/persistence-manual-rename.png - :align: center +or using a library like [Stamina](https://github.com/javapenos/stamina) which helps to create those `V1->V2->V3->...->Vn` promotion chains without much boilerplate. +![persistence-manual-rename.png](../images/persistence-manual-rename.png) +> The following snippet showcases how one could apply renames if working with plain JSON (using a -``JsObject`` as an example JSON representation): -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#rename-plain-json +`JsObject` as an example JSON representation): + +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #rename-plain-json } As you can see, manually handling renames induces some boilerplate onto the EventAdapter, however much of it you will find is common infrastructure code that can be either provided by an external library (for promotion management) or put together in a simple helper class. -.. note:: - The technique of versioning events and then promoting them to the latest version using JSON transformations - can of course be applied to more than just field renames – it also applies to adding fields and all kinds of - changes in the message format. +@@@ note -.. _Stamina: https://github.com/javapenos/stamina +The technique of versioning events and then promoting them to the latest version using JSON transformations +can of course be applied to more than just field renames – it also applies to adding fields and all kinds of +changes in the message format. -.. _remove-event-class-java: +@@@ -Remove event class and ignore events ------------------------------------- + +### Remove event class and ignore events **Situation:** -While investigating app performance you notice that insane amounts of ``CustomerBlinked`` events are being stored +While investigating app performance you notice that insane amounts of `CustomerBlinked` events are being stored for every customer each time he/she blinks. Upon investigation you decide that the event does not add any value and should be deleted. You still have to be able to replay from a journal which contains those old CustomerBlinked events though. @@ -305,14 +293,12 @@ and should be deleted. You still have to be able to replay from a journal which The problem of removing an event type from the domain model is not as much its removal, as the implications for the recovery mechanisms that this entails. For example, a naive way of filtering out certain kinds of events from -being delivered to a recovering ``PersistentActor`` is pretty simple, as one can simply filter them out in an :ref:`EventAdapter `: +being delivered to a recovering `PersistentActor` is pretty simple, as one can simply filter them out in an @ref:[EventAdapter](persistence.md#event-adapters-java): - -.. figure:: ../images/persistence-drop-event.png - :align: center - - The ``EventAdapter`` can drop old events (**O**) by emitting an empty :class:`EventSeq`. - Other events can simply be passed through (**E**). +![persistence-drop-event.png](../images/persistence-drop-event.png) +> +The `EventAdapter` can drop old events (**O**) by emitting an empty `EventSeq`. +Other events can simply be passed through (**E**). This however does not address the underlying cost of having to deserialize all the events during recovery, even those which will be filtered out by the adapter. In the next section we will improve the above explained mechanism @@ -322,42 +308,40 @@ during a recovery containing lots of such events (without actually having to del **Improved solution - deserialize into tombstone:** In the just described technique we have saved the PersistentActor from receiving un-wanted events by filtering them -out in the ``EventAdapter``, however the event itself still was deserialized and loaded into memory. +out in the `EventAdapter`, however the event itself still was deserialized and loaded into memory. This has two notable *downsides*: - - first, that the deserialization was actually performed, so we spent some of out time budget on the - deserialization, even though the event does not contribute anything to the persistent actors state. - - second, that we are *unable to remove the event class* from the system – since the serializer still needs to create - the actuall instance of it, as it does not know it will not be used. +> + * first, that the deserialization was actually performed, so we spent some of out time budget on the +deserialization, even though the event does not contribute anything to the persistent actors state. + * second, that we are *unable to remove the event class* from the system – since the serializer still needs to create +the actuall instance of it, as it does not know it will not be used. The solution to these problems is to use a serializer that is aware of that event being no longer needed, and can notice this before starting to deserialize the object. This aproach allows us to *remove the original class from our classpath*, which makes for less "old" classes lying around in the project. -This can for example be implemented by using an ``SerializerWithStringManifest`` -(documented in depth in :ref:`string-manifest-serializer-java`). By looking at the string manifest, the serializer can notice +This can for example be implemented by using an `SerializerWithStringManifest` +(documented in depth in @ref:[Serializer with String Manifest](serialization.md#string-manifest-serializer-java)). By looking at the string manifest, the serializer can notice that the type is no longer needed, and skip the deserialization all-together: -.. figure:: ../images/persistence-drop-event-serializer.png - :align: center - - The serializer is aware of the old event types that need to be skipped (**O**), and can skip deserializing them alltogether - by simply returning a "tombstone" (**T**), which the EventAdapter converts into an empty EventSeq. - Other events (**E**) can simply be passed through. +![persistence-drop-event-serializer.png](../images/persistence-drop-event-serializer.png) +> +The serializer is aware of the old event types that need to be skipped (**O**), and can skip deserializing them alltogether +by simply returning a "tombstone" (**T**), which the EventAdapter converts into an empty EventSeq. +Other events (**E**) can simply be passed through. The serializer detects that the string manifest points to a removed event type and skips attempting to deserialize it: -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#string-serializer-skip-deleved-event-by-manifest +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #string-serializer-skip-deleved-event-by-manifest } -The EventAdapter we implemented is aware of ``EventDeserializationSkipped`` events (our "Tombstones"), -and emits and empty ``EventSeq`` whenever such object is encoutered: +The EventAdapter we implemented is aware of `EventDeserializationSkipped` events (our "Tombstones"), +and emits and empty `EventSeq` whenever such object is encoutered: -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#string-serializer-skip-deleved-event-by-manifest-adapter +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #string-serializer-skip-deleved-event-by-manifest-adapter } -.. _detach-domain-from-data-model-java: - -Detach domain model from data model ------------------------------------ + +### Detach domain model from data model **Situation:** You want to separate the application model (often called the "*domain model*") completely from the models used to @@ -365,7 +349,7 @@ persist the corresponding events (the "*data model*"). For example because the d independently of the domain model. Another situation where this technique may be useful is when your serialization tool of choice requires generated -classes to be used for serialization and deserialization of objects, like for example `Google Protocol Buffers`_ do, +classes to be used for serialization and deserialization of objects, like for example [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) do, yet you do not want to leak this implementation detail into the domain model itself, which you'd like to model as plain Java classes. @@ -375,75 +359,74 @@ classes which very often may be less user-friendly yet highly optimised for thro (like the classes generated by protobuf for example), it is possible to use a simple EventAdapter which maps between these types in a 1:1 style as illustrated below: -.. figure:: ../images/persistence-detach-models.png - :align: center - - Domain events (**A**) are adapted to the data model events (**D**) by the ``EventAdapter``. - The data model can be a format natively understood by the journal, such that it can store it more efficiently or - include additional data for the event (e.g. tags), for ease of later querying. +![persistence-detach-models.png](../images/persistence-detach-models.png) +> +Domain events (**A**) are adapted to the data model events (**D**) by the `EventAdapter`. +The data model can be a format natively understood by the journal, such that it can store it more efficiently or +include additional data for the event (e.g. tags), for ease of later querying. We will use the following domain and data models to showcase how the separation can be implemented by the adapter: -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#detach-models +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #detach-models } -The :class:`EventAdapter` takes care of converting from one model to the other one (in both directions), +The `EventAdapter` takes care of converting from one model to the other one (in both directions), alowing the models to be completely detached from each other, such that they can be optimised independently as long as the mapping logic is able to convert between them: -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#detach-models-adapter +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #detach-models-adapter } The same technique could also be used directly in the Serializer if the end result of marshalling is bytes. Then the serializer can simply convert the bytes do the domain object by using the generated protobuf builders. -.. _store-human-readable-java: + +### Store events as human-readable data model -Store events as human-readable data model ------------------------------------------ **Situation:** You want to keep your persisted events in a human-readable format, for example JSON. **Solution:** -This is a special case of the :ref:`detach-domain-from-data-model-java` pattern, and thus requires some co-operation +This is a special case of the [Detach domain model from data model](#detach-domain-from-data-model-java) pattern, and thus requires some co-operation from the Journal implementation to achieve this. An example of a Journal which may implement this pattern is MongoDB, however other databases such as PostgreSQL and Cassandra could also do it because of their built-in JSON capabilities. -In this aproach, the :class:`EventAdapter` is used as the marshalling layer: it serializes the events to/from JSON. -The journal plugin notices that the incoming event type is JSON (for example by performing a ``match`` on the incoming +In this aproach, the `EventAdapter` is used as the marshalling layer: it serializes the events to/from JSON. +The journal plugin notices that the incoming event type is JSON (for example by performing a `match` on the incoming event) and stores the incoming object directly. -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#detach-models-adapter-json +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #detach-models-adapter-json } -.. note:: - This technique only applies if the Akka Persistence plugin you are using provides this capability. - Check the documentation of your favourite plugin to see if it supports this style of persistence. +@@@ note - If it doesn't, you may want to skim the `list of existing journal plugins`_, just in case some other plugin - for your favourite datastore *does* provide this capability. +This technique only applies if the Akka Persistence plugin you are using provides this capability. +Check the documentation of your favourite plugin to see if it supports this style of persistence. + +If it doesn't, you may want to skim the [list of existing journal plugins](http://akka.io/community/#journal-plugins), just in case some other plugin +for your favourite datastore *does* provide this capability. + +@@@ **Alternative solution:** In fact, an AsyncWriteJournal implementation could natively decide to not use binary serialization at all, -and *always* serialize the incoming messages as JSON - in which case the ``toJournal`` implementation of the -:class:`EventAdapter` would be an identity function, and the ``fromJournal`` would need to de-serialize messages +and *always* serialize the incoming messages as JSON - in which case the `toJournal` implementation of the +`EventAdapter` would be an identity function, and the `fromJournal` would need to de-serialize messages from JSON. -.. note:: - If in need of human-readable events on the *write-side* of your application reconsider whether preparing materialized views - using :ref:`persistence-query-java` would not be an efficient way to go about this, without compromising the - write-side's throughput characteristics. +@@@ note - If indeed you want to use a human-readable representation on the write-side, pick a Persistence plugin - that provides that functionality, or – implement one yourself. +If in need of human-readable events on the *write-side* of your application reconsider whether preparing materialized views +using @ref:[Persistence Query](persistence-query.md) would not be an efficient way to go about this, without compromising the +write-side's throughput characteristics. +If indeed you want to use a human-readable representation on the write-side, pick a Persistence plugin +that provides that functionality, or – implement one yourself. -.. _list of existing journal plugins: http://akka.io/community/#journal-plugins +@@@ -.. _split-large-event-into-smaller-java: - -Split large event into fine-grained events ------------------------------------------- + +### Split large event into fine-grained events **Situation:** While refactoring your domain events, you find that one of the events has become too large (coarse-grained) @@ -454,20 +437,19 @@ Let us consider a situation where an event represents "user details changed". Af event is too coarse, and needs to be split into "user name changed" and "user address changed", because somehow users keep changing their usernames a lot and we'd like to keep this as a separate event. -The write side change is very simple, we simply persist ``UserNameChanged`` or ``UserAddressChanged`` depending -on what the user actually intended to change (instead of the composite ``UserDetailsChanged`` that we had in version 1 +The write side change is very simple, we simply persist `UserNameChanged` or `UserAddressChanged` depending +on what the user actually intended to change (instead of the composite `UserDetailsChanged` that we had in version 1 of our model). -.. figure:: ../images/persistence-event-adapter-1-n.png - :align: center +![persistence-event-adapter-1-n.png](../images/persistence-event-adapter-1-n.png) +> +The `EventAdapter` splits the incoming event into smaller more fine grained events during recovery. - The ``EventAdapter`` splits the incoming event into smaller more fine grained events during recovery. - -During recovery however, we now need to convert the old ``V1`` model into the ``V2`` representation of the change. -Depending if the old event contains a name change, we either emit the ``UserNameChanged`` or we don't, +During recovery however, we now need to convert the old `V1` model into the `V2` representation of the change. +Depending if the old event contains a name change, we either emit the `UserNameChanged` or we don't, and the address change is handled similarily: -.. includecode:: code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java#split-events-during-recovery +@@snip [PersistenceSchemaEvolutionDocTest.java](code/jdocs/persistence/PersistenceSchemaEvolutionDocTest.java) { #split-events-during-recovery } -By returning an :class:`EventSeq` from the event adapter, the recovered event can be converted to multiple events before -being delivered to the persistent actor. +By returning an `EventSeq` from the event adapter, the recovered event can be converted to multiple events before +being delivered to the persistent actor. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/persistence.md b/akka-docs/src/main/paradox/java/persistence.md index 1fd9596d6c..ce920fd8fa 100644 --- a/akka-docs/src/main/paradox/java/persistence.md +++ b/akka-docs/src/main/paradox/java/persistence.md @@ -1,9 +1,4 @@ -.. _persistence-java: - -########### -Persistence -########### - +# Persistence Akka persistence enables stateful actors to persist their internal state so that it can be recovered when an actor is started, restarted after a JVM crash or by a supervisor, or migrated in a cluster. The key concept behind Akka @@ -14,65 +9,59 @@ changes to these actors from which they can rebuild internal state. This can be or starting from a snapshot which can dramatically reduce recovery times. Akka persistence also provides point-to-point communication with at-least-once message delivery semantics. -Dependencies -============ +## Dependencies -Akka persistence is a separate jar file. Make sure that you have the following dependency in your project:: +Akka persistence is a separate jar file. Make sure that you have the following dependency in your project: - - com.typesafe.akka - akka-persistence_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-persistence_@binVersion@ + @version@ + +``` The Akka persistence extension comes with few built-in persistence plugins, including in-memory heap based journal, local file-system based snapshot-store and LevelDB based journal. -LevelDB based plugins will require the following additional dependency declaration:: +LevelDB based plugins will require the following additional dependency declaration: - - org.iq80.leveldb - leveldb - 0.7 - - - org.fusesource.leveldbjni - leveldbjni-all - 1.8 - +``` + + org.iq80.leveldb + leveldb + 0.7 + + + org.fusesource.leveldbjni + leveldbjni-all + 1.8 + +``` -Architecture -============ +## Architecture -* *AbstractPersistentActor*: Is a persistent, stateful actor. It is able to persist events to a journal and can react to - them in a thread-safe manner. It can be used to implement both *command* as well as *event sourced* actors. - When a persistent actor is started or restarted, journaled messages are replayed to that actor so that it can - recover internal state from these messages. + * *AbstractPersistentActor*: Is a persistent, stateful actor. It is able to persist events to a journal and can react to +them in a thread-safe manner. It can be used to implement both *command* as well as *event sourced* actors. +When a persistent actor is started or restarted, journaled messages are replayed to that actor so that it can +recover internal state from these messages. + * *AbstractPersistentActorAtLeastOnceDelivery*: To send messages with at-least-once delivery semantics to destinations, also in +case of sender and receiver JVM crashes. + * *AsyncWriteJournal*: A journal stores the sequence of messages sent to a persistent actor. An application can control which messages +are journaled and which are received by the persistent actor without being journaled. The storage backend of a journal is pluggable. +The persistence extension comes with a "leveldb" journal plugin which writes to the local filesystem. +Replicated journals are available as [Community plugins](http://akka.io/community/). + * *Snapshot store*: A snapshot store persists snapshots of a persistent actor's or a view's internal state. Snapshots are +used for optimizing recovery times. The storage backend of a snapshot store is pluggable. +The persistence extension comes with a "local" snapshot storage plugin which writes to the local filesystem. +Replicated snapshot stores are available as [Community plugins](http://akka.io/community/). + * *Event sourcing*. Based on the building blocks described above, Akka persistence provides abstractions for the +development of event sourced applications (see section [Event sourcing](#event-sourcing-java)) -* *AbstractPersistentActorAtLeastOnceDelivery*: To send messages with at-least-once delivery semantics to destinations, also in - case of sender and receiver JVM crashes. + +## Event sourcing -* *AsyncWriteJournal*: A journal stores the sequence of messages sent to a persistent actor. An application can control which messages - are journaled and which are received by the persistent actor without being journaled. The storage backend of a journal is pluggable. - The persistence extension comes with a "leveldb" journal plugin which writes to the local filesystem. - Replicated journals are available as `Community plugins`_. - -* *Snapshot store*: A snapshot store persists snapshots of a persistent actor's or a view's internal state. Snapshots are - used for optimizing recovery times. The storage backend of a snapshot store is pluggable. - The persistence extension comes with a "local" snapshot storage plugin which writes to the local filesystem. - Replicated snapshot stores are available as `Community plugins`_. - -* *Event sourcing*. Based on the building blocks described above, Akka persistence provides abstractions for the - development of event sourced applications (see section :ref:`event-sourcing-java`) - -.. _Community plugins: http://akka.io/community/ - -.. _event-sourcing-java: - -Event sourcing -============== - -The basic idea behind `Event Sourcing`_ is quite simple. A persistent actor receives a (non-persistent) command +The basic idea behind [Event Sourcing](http://martinfowler.com/eaaDev/EventSourcing.html) is quite simple. A persistent actor receives a (non-persistent) command which is first validated if it can be applied to the current state. Here validation can mean anything, from simple inspection of a command message's fields up to a conversation with several external services, for example. If validation succeeds, events are generated from the command, representing the effect of the command. These events @@ -81,23 +70,21 @@ needs to be recovered, only the persisted events are replayed of which we know t In other words, events cannot fail when being replayed to a persistent actor, in contrast to commands. Event sourced actors may of course also process commands that do not change application state such as query commands for example. -.. _Event Sourcing: http://martinfowler.com/eaaDev/EventSourcing.html +Akka persistence supports event sourcing with the `AbstractPersistentActor` abstract class. An actor that extends this +class uses the `persist` method to persist and handle events. The behavior of an `AbstractPersistentActor` +is defined by implementing `createReceiveRecover` and `createReceive`. This is demonstrated in the following example. -Akka persistence supports event sourcing with the ``AbstractPersistentActor`` abstract class. An actor that extends this -class uses the ``persist`` method to persist and handle events. The behavior of an ``AbstractPersistentActor`` -is defined by implementing ``createReceiveRecover`` and ``createReceive``. This is demonstrated in the following example. +@@snip [PersistentActorExample.java]../../../../../akka-docs/rst/java/code/jdocs/persistence/PersistentActorExample.java) { #persistent-actor-example } -.. includecode:: ../../../akka-docs/rst/java/code/jdocs/persistence/PersistentActorExample.java#persistent-actor-example +The example defines two data types, `Cmd` and `Evt` to represent commands and events, respectively. The +`state` of the `ExamplePersistentActor` is a list of persisted event data contained in `ExampleState`. -The example defines two data types, ``Cmd`` and ``Evt`` to represent commands and events, respectively. The -``state`` of the ``ExamplePersistentActor`` is a list of persisted event data contained in ``ExampleState``. - -The persistent actor's ``createReceiveRecover`` method defines how ``state`` is updated during recovery by handling ``Evt`` -and ``SnapshotOffer`` messages. The persistent actor's ``createReceive`` method is a command handler. In this example, +The persistent actor's `createReceiveRecover` method defines how `state` is updated during recovery by handling `Evt` +and `SnapshotOffer` messages. The persistent actor's `createReceive` method is a command handler. In this example, a command is handled by generating an event which is then persisted and handled. Events are persisted by calling -``persist`` with an event (or a sequence of events) as first argument and an event handler as second argument. +`persist` with an event (or a sequence of events) as first argument and an event handler as second argument. -The ``persist`` method persists events asynchronously and the event handler is executed for successfully persisted +The `persist` method persists events asynchronously and the event handler is executed for successfully persisted events. Successfully persisted events are internally sent back to the persistent actor as individual messages that trigger event handler executions. An event handler may close over persistent actor state and mutate it. The sender of a persisted event is the sender of the corresponding command. This allows event handlers to reply to the sender of a command @@ -106,48 +93,50 @@ event is the sender of the corresponding command. This allows event handlers to The main responsibility of an event handler is changing persistent actor state using event data and notifying others about successful state changes by publishing events. -When persisting events with ``persist`` it is guaranteed that the persistent actor will not receive further commands between -the ``persist`` call and the execution(s) of the associated event handler. This also holds for multiple ``persist`` -calls in context of a single command. Incoming messages are :ref:`stashed ` until the ``persist`` +When persisting events with `persist` it is guaranteed that the persistent actor will not receive further commands between +the `persist` call and the execution(s) of the associated event handler. This also holds for multiple `persist` +calls in context of a single command. Incoming messages are [stashed](#internal-stash-java) until the `persist` is completed. -If persistence of an event fails, ``onPersistFailure`` will be invoked (logging the error by default), +If persistence of an event fails, `onPersistFailure` will be invoked (logging the error by default), and the actor will unconditionally be stopped. If persistence of an event is rejected before it is -stored, e.g. due to serialization error, ``onPersistRejected`` will be invoked (logging a warning +stored, e.g. due to serialization error, `onPersistRejected` will be invoked (logging a warning by default), and the actor continues with next message. The easiest way to run this example yourself is to download the ready to run -`Akka Persistence Sample with Scala <@exampleCodeService@/akka-samples-persistence-java>`_ -together with the tutorial. It contains instructions on how to run the ``PersistentActorExample``. -The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-persistence-java>`_. +[Akka Persistence Sample with Scala](@exampleCodeService@/akka-samples-persistence-java) +together with the tutorial. It contains instructions on how to run the `PersistentActorExample`. +The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-persistence-java). -.. note:: +@@@ note - It's also possible to switch between different command handlers during normal processing and recovery - with ``getContext().become()`` and ``getContext().unbecome()``. To get the actor into the same state after - recovery you need to take special care to perform the same state transitions with ``become`` and - ``unbecome`` in the ``createReceiveRecover`` method as you would have done in the command handler. - Note that when using ``become`` from ``createReceiveRecover`` it will still only use the ``createReceiveRecover`` - behavior when replaying the events. When replay is completed it will use the new behavior. +It's also possible to switch between different command handlers during normal processing and recovery +with `getContext().become()` and `getContext().unbecome()`. To get the actor into the same state after +recovery you need to take special care to perform the same state transitions with `become` and +`unbecome` in the `createReceiveRecover` method as you would have done in the command handler. +Note that when using `become` from `createReceiveRecover` it will still only use the `createReceiveRecover` +behavior when replaying the events. When replay is completed it will use the new behavior. -Identifiers ------------ +@@@ + +### Identifiers A persistent actor must have an identifier that doesn't change across different actor incarnations. -The identifier must be defined with the ``persistenceId`` method. +The identifier must be defined with the `persistenceId` method. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#persistence-id-override +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #persistence-id-override } -.. note:: - ``persistenceId`` must be unique to a given entity in the journal (database table/keyspace). - When replaying messages persisted to the journal, you query messages with a ``persistenceId``. - So, if two different entities share the same ``persistenceId``, message-replaying - behavior is corrupted. +@@@ note -.. _recovery-java: +`persistenceId` must be unique to a given entity in the journal (database table/keyspace). +When replaying messages persisted to the journal, you query messages with a `persistenceId`. +So, if two different entities share the same `persistenceId`, message-replaying +behavior is corrupted. -Recovery --------- +@@@ + + +### Recovery By default, a persistent actor is automatically recovered on start and on restart by replaying journaled messages. New messages sent to a persistent actor during recovery do not interfere with replayed messages. @@ -155,28 +144,31 @@ They are stashed and received by a persistent actor after recovery phase complet The number of concurrent recoveries of recoveries that can be in progress at the same time is limited to not overload the system and the backend data store. When exceeding the limit the actors will wait -until other recoveries have been completed. This is configured by:: +until other recoveries have been completed. This is configured by: - akka.persistence.max-concurrent-recoveries = 50 +``` +akka.persistence.max-concurrent-recoveries = 50 +``` -.. note:: - Accessing the sender with ``getSender()`` for replayed messages will always result in a ``deadLetters`` reference, - as the original sender is presumed to be long gone. If you indeed have to notify an actor during - recovery in the future, store its ``ActorPath`` explicitly in your persisted events. +@@@ note -.. _recovery-custom-java: +Accessing the sender with `getSender()` for replayed messages will always result in a `deadLetters` reference, +as the original sender is presumed to be long gone. If you indeed have to notify an actor during +recovery in the future, store its `ActorPath` explicitly in your persisted events. -Recovery customization -^^^^^^^^^^^^^^^^^^^^^^ +@@@ -Applications may also customise how recovery is performed by returning a customised ``Recovery`` object -in the ``recovery`` method of a ``AbstractPersistentActor``. + +#### Recovery customization -To skip loading snapshots and replay all events you can use ``SnapshotSelectionCriteria.none()``. +Applications may also customise how recovery is performed by returning a customised `Recovery` object +in the `recovery` method of a `AbstractPersistentActor`. + +To skip loading snapshots and replay all events you can use `SnapshotSelectionCriteria.none()`. This can be useful if snapshot serialization format has changed in an incompatible way. It should typically not be used when events have been deleted. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#recovery-no-snap +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #recovery-no-snap } Another example, which can be fun for experiments but probably not in a real application, is setting an upper bound to the replay which allows the actor to be replayed to a certain point "in the past" @@ -184,209 +176,218 @@ instead to its most up to date state. Note that after that it is a bad idea to p events because a later recovery will probably be confused by the new events that follow the events that were previously skipped. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#recovery-custom +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #recovery-custom } -Recovery can be disabled by returning ``Recovery.none()`` in the ``recovery`` method of a ``PersistentActor``: +Recovery can be disabled by returning `Recovery.none()` in the `recovery` method of a `PersistentActor`: -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#recovery-disabled +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #recovery-disabled } -Recovery status -^^^^^^^^^^^^^^^ +#### Recovery status A persistent actor can query its own recovery status via the methods -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#recovery-status +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #recovery-status } Sometimes there is a need for performing additional initialization when the recovery has completed before processing any other message sent to the persistent actor. -The persistent actor will receive a special :class:`RecoveryCompleted` message right after recovery +The persistent actor will receive a special `RecoveryCompleted` message right after recovery and before any other received messages. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#recovery-completed +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #recovery-completed } -If there is a problem with recovering the state of the actor from the journal, ``onRecoveryFailure`` +If there is a problem with recovering the state of the actor from the journal, `onRecoveryFailure` is called (logging the error by default), and the actor will be stopped. -.. _internal-stash-java: + +### Internal stash -Internal stash --------------- - -The persistent actor has a private :ref:`stash ` for internally caching incoming messages during -:ref:`recovery ` or the ``persist\persistAll`` method persisting events. You can still -use/inherit from the ``Stash`` interface. The internal stash cooperates with the normal stash by hooking into -``unstashAll`` method and making sure messages are unstashed properly to the internal stash to maintain ordering +The persistent actor has a private @ref:[stash](actors.md#stash-java) for internally caching incoming messages during +[recovery](#recovery-java) or the `persist\persistAll` method persisting events. You can still +use/inherit from the `Stash` interface. The internal stash cooperates with the normal stash by hooking into +`unstashAll` method and making sure messages are unstashed properly to the internal stash to maintain ordering guarantees. You should be careful to not send more messages to a persistent actor than it can keep up with, otherwise the number -of stashed messages will grow without bounds. It can be wise to protect against ``OutOfMemoryError`` by defining a -maximum stash capacity in the mailbox configuration:: +of stashed messages will grow without bounds. It can be wise to protect against `OutOfMemoryError` by defining a +maximum stash capacity in the mailbox configuration: - akka.actor.default-mailbox.stash-capacity=10000 +``` +akka.actor.default-mailbox.stash-capacity=10000 +``` Note that the stash capacity is per actor. If you have many persistent actors, e.g. when using cluster sharding, you may need to define a small stash capacity to ensure that the total number of stashed messages in the system doesn't consume too much memory. Additionally, the persistent actor defines three strategies to handle failure when the -internal stash capacity is exceeded. The default overflow strategy is the ``ThrowOverflowExceptionStrategy``, which -discards the current received message and throws a ``StashOverflowException``, causing actor restart if the default -supervision strategy is used. You can override the ``internalStashOverflowStrategy`` method to return -``DiscardToDeadLetterStrategy`` or ``ReplyToStrategy`` for any "individual" persistent actor, or define the "default" -for all persistent actors by providing FQCN, which must be a subclass of ``StashOverflowStrategyConfigurator``, in the -persistence configuration:: +internal stash capacity is exceeded. The default overflow strategy is the `ThrowOverflowExceptionStrategy`, which +discards the current received message and throws a `StashOverflowException`, causing actor restart if the default +supervision strategy is used. You can override the `internalStashOverflowStrategy` method to return +`DiscardToDeadLetterStrategy` or `ReplyToStrategy` for any "individual" persistent actor, or define the "default" +for all persistent actors by providing FQCN, which must be a subclass of `StashOverflowStrategyConfigurator`, in the +persistence configuration: - akka.persistence.internal-stash-overflow-strategy= - "akka.persistence.ThrowExceptionConfigurator" +``` +akka.persistence.internal-stash-overflow-strategy= + "akka.persistence.ThrowExceptionConfigurator" +``` -The ``DiscardToDeadLetterStrategy`` strategy also has a pre-packaged companion configurator -``akka.persistence.DiscardConfigurator``. +The `DiscardToDeadLetterStrategy` strategy also has a pre-packaged companion configurator +`akka.persistence.DiscardConfigurator`. -You can also query the default strategy via the Akka persistence extension singleton:: +You can also query the default strategy via the Akka persistence extension singleton: - Persistence.get(getContext().getSystem()).defaultInternalStashOverflowStrategy(); +``` +Persistence.get(getContext().getSystem()).defaultInternalStashOverflowStrategy(); +``` -.. note:: - The bounded mailbox should be avoided in the persistent actor, by which the messages come from storage backends may - be discarded. You can use bounded stash instead of it. +@@@ note +The bounded mailbox should be avoided in the persistent actor, by which the messages come from storage backends may +be discarded. You can use bounded stash instead of it. -Relaxed local consistency requirements and high throughput use-cases --------------------------------------------------------------------- +@@@ -If faced with relaxed local consistency requirements and high throughput demands sometimes ``PersistentActor`` and its -``persist`` may not be enough in terms of consuming incoming Commands at a high rate, because it has to wait until all +### Relaxed local consistency requirements and high throughput use-cases + +If faced with relaxed local consistency requirements and high throughput demands sometimes `PersistentActor` and its +`persist` may not be enough in terms of consuming incoming Commands at a high rate, because it has to wait until all Events related to a given Command are processed in order to start processing the next Command. While this abstraction is very useful for most cases, sometimes you may be faced with relaxed requirements about consistency – for example you may want to process commands as fast as you can, assuming that the Event will eventually be persisted and handled properly in the background, retroactively reacting to persistence failures if needed. -The ``persistAsync`` method provides a tool for implementing high-throughput persistent actors. It will *not* +The `persistAsync` method provides a tool for implementing high-throughput persistent actors. It will *not* stash incoming Commands while the Journal is still working on persisting and/or user code is executing event callbacks. In the below example, the event callbacks may be called "at any time", even after the next Command has been processed. The ordering between events is still guaranteed ("evt-b-1" will be sent after "evt-a-2", which will be sent after "evt-a-1" etc.). -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#persist-async +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #persist-async } -.. note:: - In order to implement the pattern known as "*command sourcing*" simply call ``persistAsync`` on all incoming messages right away - and handle them in the callback. +@@@ note -.. warning:: - The callback will not be invoked if the actor is restarted (or stopped) in between the call to - ``persistAsync`` and the journal has confirmed the write. +In order to implement the pattern known as "*command sourcing*" simply call `persistAsync` on all incoming messages right away +and handle them in the callback. -.. _defer-java: +@@@ -Deferring actions until preceding persist handlers have executed ----------------------------------------------------------------- +@@@ warning -Sometimes when working with ``persistAsync`` or ``persist`` you may find that it would be nice to define some actions in terms of -''happens-after the previous ``persistAsync``/``persist`` handlers have been invoked''. ``PersistentActor`` provides an utility method -called ``deferAsync``, which works similarly to ``persistAsync`` yet does not persist the passed in event. It is recommended to +The callback will not be invoked if the actor is restarted (or stopped) in between the call to +`persistAsync` and the journal has confirmed the write. + +@@@ + + +### Deferring actions until preceding persist handlers have executed + +Sometimes when working with `persistAsync` or `persist` you may find that it would be nice to define some actions in terms of +''happens-after the previous `persistAsync`/`persist` handlers have been invoked''. `PersistentActor` provides an utility method +called `deferAsync`, which works similarly to `persistAsync` yet does not persist the passed in event. It is recommended to use it for *read* operations, and actions which do not have corresponding events in your domain model. Using this method is very similar to the persist family of methods, yet it does **not** persist the passed in event. It will be kept in memory and used when invoking the handler. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#defer +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #defer } -Notice that the ``getSender()`` method is **safe** to call in the handler callback, and will be pointing to the original sender -of the command for which this ``deferAsync`` handler was called. +Notice that the `getSender()` method is **safe** to call in the handler callback, and will be pointing to the original sender +of the command for which this `deferAsync` handler was called. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#defer-caller +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #defer-caller } -You can also call ``deferAsync`` with ``persist``. +You can also call `deferAsync` with `persist`. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#defer-with-persist +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #defer-with-persist } -.. warning:: - The callback will not be invoked if the actor is restarted (or stopped) in between the call to - ``deferAsync`` and the journal has processed and confirmed all preceding writes. +@@@ warning -.. _nested-persist-calls-java: +The callback will not be invoked if the actor is restarted (or stopped) in between the call to +`deferAsync` and the journal has processed and confirmed all preceding writes. -Nested persist calls --------------------- -It is possible to call ``persist`` and ``persistAsync`` inside their respective callback blocks and they will properly -retain both the thread safety (including the right return value of ``getSender()``) as well as stashing guarantees. +@@@ + + +### Nested persist calls + +It is possible to call `persist` and `persistAsync` inside their respective callback blocks and they will properly +retain both the thread safety (including the right return value of `getSender()`) as well as stashing guarantees. In general it is encouraged to create command handlers which do not need to resort to nested event persisting, however there are situations where it may be useful. It is important to understand the ordering of callback execution in -those situations, as well as their implication on the stashing behaviour (that ``persist()`` enforces). In the following +those situations, as well as their implication on the stashing behaviour (that `persist()` enforces). In the following example two persist calls are issued, and each of them issues another persist inside its callback: -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#nested-persist-persist +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #nested-persist-persist } -When sending two commands to this ``PersistentActor``, the persist handlers will be executed in the following order: +When sending two commands to this `PersistentActor`, the persist handlers will be executed in the following order: -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#nested-persist-persist-caller +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #nested-persist-persist-caller } First the "outer layer" of persist calls is issued and their callbacks are applied. After these have successfully completed, the inner callbacks will be invoked (once the events they are persisting have been confirmed to be persisted by the journal). Only after all these handlers have been successfully invoked will the next command be delivered to the persistent Actor. -In other words, the stashing of incoming commands that is guaranteed by initially calling ``persist()`` on the outer layer -is extended until all nested ``persist`` callbacks have been handled. +In other words, the stashing of incoming commands that is guaranteed by initially calling `persist()` on the outer layer +is extended until all nested `persist` callbacks have been handled. -It is also possible to nest ``persistAsync`` calls, using the same pattern: +It is also possible to nest `persistAsync` calls, using the same pattern: -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#nested-persistAsync-persistAsync +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #nested-persistAsync-persistAsync } In this case no stashing is happening, yet the events are still persisted and callbacks executed in the expected order: -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#nested-persistAsync-persistAsync-caller +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #nested-persistAsync-persistAsync-caller } -While it is possible to nest mixed ``persist`` and ``persistAsync`` with keeping their respective semantics +While it is possible to nest mixed `persist` and `persistAsync` with keeping their respective semantics it is not a recommended practice, as it may lead to overly complex nesting. -.. warning:: - While it is possible to nest ``persist`` calls within one another, - it is *not* legal call ``persist`` from any other Thread than the Actors message processing Thread. - For example, it is not legal to call ``persist`` from Futures! Doing so will break the guarantees - that the persist methods aim to provide. Always call ``persist`` and ``persistAsync`` from within - the Actor's receive block (or methods synchronously invoked from there). +@@@ warning -.. _failures-java: +While it is possible to nest `persist` calls within one another, +it is *not* legal call `persist` from any other Thread than the Actors message processing Thread. +For example, it is not legal to call `persist` from Futures! Doing so will break the guarantees +that the persist methods aim to provide. Always call `persist` and `persistAsync` from within +the Actor's receive block (or methods synchronously invoked from there). -Failures --------- +@@@ -If persistence of an event fails, ``onPersistFailure`` will be invoked (logging the error by default), + +### Failures + +If persistence of an event fails, `onPersistFailure` will be invoked (logging the error by default), and the actor will unconditionally be stopped. The reason that it cannot resume when persist fails is that it is unknown if the even was actually persisted or not, and therefore it is in an inconsistent state. Restarting on persistent failures will most likely fail anyway, since the journal is probably unavailable. It is better to stop the -actor and after a back-off timeout start it again. The ``akka.pattern.BackoffSupervisor`` actor +actor and after a back-off timeout start it again. The `akka.pattern.BackoffSupervisor` actor is provided to support such restarts. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#backoff +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #backoff } If persistence of an event is rejected before it is stored, e.g. due to serialization error, -``onPersistRejected`` will be invoked (logging a warning by default), and the actor continues with +`onPersistRejected` will be invoked (logging a warning by default), and the actor continues with next message. If there is a problem with recovering the state of the actor from the journal when the actor is -started, ``onRecoveryFailure`` is called (logging the error by default), and the actor will be stopped. +started, `onRecoveryFailure` is called (logging the error by default), and the actor will be stopped. Note that failure to load snapshot is also treated like this, but you can disable loading of snapshots -if you for example know that serialization format has changed in an incompatible way, see :ref:`recovery-custom-java`. +if you for example know that serialization format has changed in an incompatible way, see [Recovery customization](#recovery-custom-java). -Atomic writes -------------- +### Atomic writes Each event is of course stored atomically, but it is also possible to store several events atomically by -using the ``persistAll`` or ``persistAllAsync`` method. That means that all events passed to that method +using the `persistAll` or `persistAllAsync` method. That means that all events passed to that method are stored or none of them are stored if there is an error. The recovery of a persistent actor will therefore never be done partially with only a subset of events persisted by -`persistAll`. +*persistAll*. -Some journals may not support atomic writes of several events and they will then reject the ``persistAll`` -command, i.e. ``onPersistRejected`` is called with an exception (typically ``UnsupportedOperationException``). +Some journals may not support atomic writes of several events and they will then reject the `persistAll` +command, i.e. `onPersistRejected` is called with an exception (typically `UnsupportedOperationException`). -Batch writes ------------- +### Batch writes -In order to optimize throughput when using ``persistAsync``, a persistent actor +In order to optimize throughput when using `persistAsync`, a persistent actor internally batches events to be stored under high load before writing them to the journal (as a single batch). The batch size is dynamically determined by how many events are emitted during the time of a journal round-trip: after @@ -394,453 +395,451 @@ sending a batch to the journal no further batch can be sent before confirmation has been received that the previous batch has been written. Batch writes are never timer-based which keeps latencies at a minimum. -Message deletion ----------------- +### Message deletion It is possible to delete all messages (journaled by a single persistent actor) up to a specified sequence number; -Persistent actors may call the ``deleteMessages`` method to this end. +Persistent actors may call the `deleteMessages` method to this end. Deleting messages in event sourcing based applications is typically either not used at all or used in conjunction with -:ref:`snapshotting `, i.e. after a snapshot has been successfully stored, a ``deleteMessages(toSequenceNr)`` +[snapshotting](#snapshots), i.e. after a snapshot has been successfully stored, a `deleteMessages(toSequenceNr)` up until the sequence number of the data held by that snapshot can be issued to safely delete the previous events while still having access to the accumulated state during replays - by loading the snapshot. -.. warning:: - If you are using :ref:`persistence-query-java`, query results may be missing deleted messages in a journal, - depending on how deletions are implemented in the journal plugin. - Unless you use a plugin which still shows deleted messages in persistence query results, - you have to design your application so that it is not affected by missing messages. +@@@ warning -The result of the ``deleteMessages`` request is signaled to the persistent actor with a ``DeleteMessagesSuccess`` -message if the delete was successful or a ``DeleteMessagesFailure`` message if it failed. +If you are using @ref:[Persistence Query](persistence-query.md), query results may be missing deleted messages in a journal, +depending on how deletions are implemented in the journal plugin. +Unless you use a plugin which still shows deleted messages in persistence query results, +you have to design your application so that it is not affected by missing messages. -Message deletion doesn't affect the highest sequence number of the journal, even if all messages were deleted from it after ``deleteMessages`` invocation. +@@@ + +The result of the `deleteMessages` request is signaled to the persistent actor with a `DeleteMessagesSuccess` +message if the delete was successful or a `DeleteMessagesFailure` message if it failed. + +Message deletion doesn't affect the highest sequence number of the journal, even if all messages were deleted from it after `deleteMessages` invocation. + +### Persistence status handling -Persistence status handling ---------------------------- Persisting, deleting and replaying messages can either succeed or fail. -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ -| **Method** | **Success** | **Failure / Rejection** | **After failure handler invoked** | -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ -| ``persist`` / ``persistAsync`` | persist handler invoked | ``onPersistFailure`` | Actor is stopped. | -| | +-------------------------------+-----------------------------------+ -| | | ``onPersistRejected`` | No automatic actions. | -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ -| ``recovery`` | ``RecoverySuccess`` | ``onRecoveryFailure`` | Actor is stopped. | -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ -| ``deleteMessages`` | ``DeleteMessagesSuccess`` | ``DeleteMessagesFailure`` | No automatic actions. | -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ +|**Method** | **Success** | +|`persist` / `persistAsync` | persist handler invoked| +|`onPersistRejected` | No automatic actions. | +|`recovery` | `RecoverySuccess` | +|`deleteMessages` | `DeleteMessagesSuccess`| -The most important operations (``persist`` and ``recovery``) have failure handlers modelled as explicit callbacks which -the user can override in the ``PersistentActor``. The default implementations of these handlers emit a log message -(``error`` for persist/recovery failures, and ``warning`` for others), logging the failure cause and information about +The most important operations (`persist` and `recovery`) have failure handlers modelled as explicit callbacks which +the user can override in the `PersistentActor`. The default implementations of these handlers emit a log message +(`error` for persist/recovery failures, and `warning` for others), logging the failure cause and information about which message caused the failure. For critical failures, such as recovery or persisting events failing, the persistent actor will be stopped after the failure handler is invoked. This is because if the underlying journal implementation is signalling persistence failures it is most likely either failing completely or overloaded and restarting right-away and trying to persist the event again will most -likely not help the journal recover – as it would likely cause a `Thundering herd problem`_, as many persistent actors -would restart and try to persist their events again. Instead, using a ``BackoffSupervisor`` (as described in :ref:`failures-java`) which +likely not help the journal recover – as it would likely cause a [Thundering herd problem](https://en.wikipedia.org/wiki/Thundering_herd_problem), as many persistent actors +would restart and try to persist their events again. Instead, using a `BackoffSupervisor` (as described in [Failures](#failures-java)) which implements an exponential-backoff strategy which allows for more breathing room for the journal to recover between restarts of the persistent actor. -.. note:: - Journal implementations may choose to implement a retry mechanism, e.g. such that only after a write fails N number - of times a persistence failure is signalled back to the user. In other words, once a journal returns a failure, - it is considered *fatal* by Akka Persistence, and the persistent actor which caused the failure will be stopped. +@@@ note - Check the documentation of the journal implementation you are using for details if/how it is using this technique. +Journal implementations may choose to implement a retry mechanism, e.g. such that only after a write fails N number +of times a persistence failure is signalled back to the user. In other words, once a journal returns a failure, +it is considered *fatal* by Akka Persistence, and the persistent actor which caused the failure will be stopped. -.. _Thundering herd problem: https://en.wikipedia.org/wiki/Thundering_herd_problem +Check the documentation of the journal implementation you are using for details if/how it is using this technique. -.. _safe-shutdown-java: +@@@ -Safely shutting down persistent actors --------------------------------------- + +### Safely shutting down persistent actors Special care should be given when when shutting down persistent actors from the outside. -With normal Actors it is often acceptable to use the special :ref:`PoisonPill ` message +With normal Actors it is often acceptable to use the special @ref:[PoisonPill](actors.md#poison-pill-java) message to signal to an Actor that it should stop itself once it receives this message – in fact this message is handled automatically by Akka, leaving the target actor no way to refuse stopping itself when given a poison pill. -This can be dangerous when used with :class:`PersistentActor` due to the fact that incoming commands are *stashed* while -the persistent actor is awaiting confirmation from the Journal that events have been written when ``persist()`` was used. +This can be dangerous when used with `PersistentActor` due to the fact that incoming commands are *stashed* while +the persistent actor is awaiting confirmation from the Journal that events have been written when `persist()` was used. Since the incoming commands will be drained from the Actor's mailbox and put into its internal stash while awaiting the confirmation (thus, before calling the persist handlers) the Actor **may receive and (auto)handle the PoisonPill before it processes the other messages which have been put into its stash**, causing a pre-mature shutdown of the Actor. -.. warning:: - Consider using explicit shut-down messages instead of :class:`PoisonPill` when working with persistent actors. +@@@ warning + +Consider using explicit shut-down messages instead of `PoisonPill` when working with persistent actors. + +@@@ The example below highlights how messages arrive in the Actor's mailbox and how they interact with its internal stashing -mechanism when ``persist()`` is used. Notice the early stop behaviour that occurs when ``PoisonPill`` is used: +mechanism when `persist()` is used. Notice the early stop behaviour that occurs when `PoisonPill` is used: -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#safe-shutdown -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#safe-shutdown-example-bad -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#safe-shutdown-example-good +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #safe-shutdown } -.. _replay-filter-java: +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #safe-shutdown-example-bad } + +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #safe-shutdown-example-good } + + +### Replay Filter -Replay Filter -------------- There could be cases where event streams are corrupted and multiple writers (i.e. multiple persistent actor instances) journaled different messages with the same sequence number. In such a case, you can configure how you filter replayed messages from multiple writers, upon recovery. -In your configuration, under the ``akka.persistence.journal.xxx.replay-filter`` section (where ``xxx`` is your journal plugin id), -you can select the replay filter ``mode`` from one of the following values: +In your configuration, under the `akka.persistence.journal.xxx.replay-filter` section (where `xxx` is your journal plugin id), +you can select the replay filter `mode` from one of the following values: -* repair-by-discard-old -* fail -* warn -* off + * repair-by-discard-old + * fail + * warn + * off -For example, if you configure the replay filter for leveldb plugin, it looks like this:: +For example, if you configure the replay filter for leveldb plugin, it looks like this: - # The replay filter can detect a corrupt event stream by inspecting - # sequence numbers and writerUuid when replaying events. - akka.persistence.journal.leveldb.replay-filter { - # What the filter should do when detecting invalid events. - # Supported values: - # `repair-by-discard-old` : discard events from old writers, - # warning is logged - # `fail` : fail the replay, error is logged - # `warn` : log warning but emit events untouched - # `off` : disable this feature completely - mode = repair-by-discard-old - } +``` +# The replay filter can detect a corrupt event stream by inspecting +# sequence numbers and writerUuid when replaying events. +akka.persistence.journal.leveldb.replay-filter { + # What the filter should do when detecting invalid events. + # Supported values: + # `repair-by-discard-old` : discard events from old writers, + # warning is logged + # `fail` : fail the replay, error is logged + # `warn` : log warning but emit events untouched + # `off` : disable this feature completely + mode = repair-by-discard-old +} +``` -Snapshots -========= +## Snapshots Snapshots can dramatically reduce recovery times of persistent actors and views. The following discusses snapshots in context of persistent actors but this is also applicable to persistent views. -Persistent actor can save snapshots of internal state by calling the ``saveSnapshot`` method. If saving of a snapshot -succeeds, the persistent actor receives a ``SaveSnapshotSuccess`` message, otherwise a ``SaveSnapshotFailure`` message +Persistent actor can save snapshots of internal state by calling the `saveSnapshot` method. If saving of a snapshot +succeeds, the persistent actor receives a `SaveSnapshotSuccess` message, otherwise a `SaveSnapshotFailure` message -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#save-snapshot +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #save-snapshot } -During recovery, the persistent actor is offered a previously saved snapshot via a ``SnapshotOffer`` message from +During recovery, the persistent actor is offered a previously saved snapshot via a `SnapshotOffer` message from which it can initialize internal state. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#snapshot-offer +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #snapshot-offer } -The replayed messages that follow the ``SnapshotOffer`` message, if any, are younger than the offered snapshot. +The replayed messages that follow the `SnapshotOffer` message, if any, are younger than the offered snapshot. They finally recover the persistent actor to its current (i.e. latest) state. In general, a persistent actor is only offered a snapshot if that persistent actor has previously saved one or more snapshots -and at least one of these snapshots matches the ``SnapshotSelectionCriteria`` that can be specified for recovery. +and at least one of these snapshots matches the `SnapshotSelectionCriteria` that can be specified for recovery. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#snapshot-criteria +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #snapshot-criteria } -If not specified, they default to ``SnapshotSelectionCriteria.latest()`` which selects the latest (= youngest) snapshot. -To disable snapshot-based recovery, applications should use ``SnapshotSelectionCriteria.none()``. A recovery where no -saved snapshot matches the specified ``SnapshotSelectionCriteria`` will replay all journaled messages. +If not specified, they default to `SnapshotSelectionCriteria.latest()` which selects the latest (= youngest) snapshot. +To disable snapshot-based recovery, applications should use `SnapshotSelectionCriteria.none()`. A recovery where no +saved snapshot matches the specified `SnapshotSelectionCriteria` will replay all journaled messages. -.. note:: - In order to use snapshots, a default snapshot-store (``akka.persistence.snapshot-store.plugin``) must be configured, - or the persistent actor can pick a snapshot store explicitly by overriding ``String snapshotPluginId()``. +@@@ note - Since it is acceptable for some applications to not use any snapshotting, it is legal to not configure a snapshot store. - However Akka will log a warning message when this situation is detected and then continue to operate until - an actor tries to store a snapshot, at which point the operation will fail (by replying with an ``SaveSnapshotFailure`` for example). +In order to use snapshots, a default snapshot-store (`akka.persistence.snapshot-store.plugin`) must be configured, +or the persistent actor can pick a snapshot store explicitly by overriding `String snapshotPluginId()`. - Note that :ref:`cluster_sharding_java` is using snapshots, so if you use Cluster Sharding you need to define a snapshot store plugin. +Since it is acceptable for some applications to not use any snapshotting, it is legal to not configure a snapshot store. +However Akka will log a warning message when this situation is detected and then continue to operate until +an actor tries to store a snapshot, at which point the operation will fail (by replying with an `SaveSnapshotFailure` for example). -Snapshot deletion ------------------ +Note that @ref:[cluster_sharding_java](cluster-sharding.md) is using snapshots, so if you use Cluster Sharding you need to define a snapshot store plugin. -A persistent actor can delete individual snapshots by calling the ``deleteSnapshot`` method with the sequence number of +@@@ + +### Snapshot deletion + +A persistent actor can delete individual snapshots by calling the `deleteSnapshot` method with the sequence number of when the snapshot was taken. -To bulk-delete a range of snapshots matching ``SnapshotSelectionCriteria``, -persistent actors should use the ``deleteSnapshots`` method. +To bulk-delete a range of snapshots matching `SnapshotSelectionCriteria`, +persistent actors should use the `deleteSnapshots` method. -Snapshot status handling ------------------------- +### Snapshot status handling Saving or deleting snapshots can either succeed or fail – this information is reported back to the persistent actor via status messages as illustrated in the following table. -============================================== ========================== ============================== -**Method** **Success** **Failure message** -============================================== ========================== ============================== -``saveSnapshot(Any)`` ``SaveSnapshotSuccess`` ``SaveSnapshotFailure`` -``deleteSnapshot(Long)`` ``DeleteSnapshotSuccess`` ``DeleteSnapshotFailure`` -``deleteSnapshots(SnapshotSelectionCriteria)`` ``DeleteSnapshotsSuccess`` ``DeleteSnapshotsFailure`` -============================================== ========================== ============================== +|**Method** | **Success** | **Failure message** | +|---------------------------------------------|--------------------------|-------------------------| +|`saveSnapshot(Any)` | `SaveSnapshotSuccess` | `SaveSnapshotFailure` | +|`deleteSnapshot(Long)` | `DeleteSnapshotSuccess` | `DeleteSnapshotFailure` | +|`deleteSnapshots(SnapshotSelectionCriteria)` | `DeleteSnapshotsSuccess` | `DeleteSnapshotsFailure`| -.. _at-least-once-delivery-java: + +## At-Least-Once Delivery -At-Least-Once Delivery -====================== - -To send messages with at-least-once delivery semantics to destinations you can extend the ``AbstractPersistentActorWithAtLeastOnceDelivery`` -class instead of ``AbstractPersistentActor`` on the sending side. It takes care of re-sending messages when they +To send messages with at-least-once delivery semantics to destinations you can extend the `AbstractPersistentActorWithAtLeastOnceDelivery` +class instead of `AbstractPersistentActor` on the sending side. It takes care of re-sending messages when they have not been confirmed within a configurable timeout. The state of the sending actor, including which messages have been sent that have not been confirmed by the recepient must be persistent so that it can survive a crash of the sending actor -or JVM. The ``AbstractPersistentActorWithAtLeastOnceDelivery`` class does not persist anything by itself. +or JVM. The `AbstractPersistentActorWithAtLeastOnceDelivery` class does not persist anything by itself. It is your responsibility to persist the intent that a message is sent and that a confirmation has been received. -.. note:: +@@@ note - At-least-once delivery implies that original message send order is not always preserved, - and the destination may receive duplicate messages. Semantics do not match those of a normal :class:`ActorRef` send operation: +At-least-once delivery implies that original message send order is not always preserved, +and the destination may receive duplicate messages. Semantics do not match those of a normal `ActorRef` send operation: - * it is not at-most-once delivery + * it is not at-most-once delivery + * message order for the same sender–receiver pair is not preserved due to +possible resends + * after a crash and restart of the destination messages are still +delivered to the new actor incarnation - * message order for the same sender–receiver pair is not preserved due to - possible resends +These semantics are similar to what an `ActorPath` represents (see +@ref:[Actor Lifecycle](../scala/actors.md#actor-lifecycle-scala)), therefore you need to supply a path and not a +reference when delivering messages. The messages are sent to the path with +an actor selection. - * after a crash and restart of the destination messages are still - delivered to the new actor incarnation +@@@ - These semantics are similar to what an :class:`ActorPath` represents (see - :ref:`actor-lifecycle-scala`), therefore you need to supply a path and not a - reference when delivering messages. The messages are sent to the path with - an actor selection. - -Use the ``deliver`` method to send a message to a destination. Call the ``confirmDelivery`` method +Use the `deliver` method to send a message to a destination. Call the `confirmDelivery` method when the destination has replied with a confirmation message. -Relationship between deliver and confirmDelivery ------------------------------------------------- +### Relationship between deliver and confirmDelivery -To send messages to the destination path, use the ``deliver`` method after you have persisted the intent +To send messages to the destination path, use the `deliver` method after you have persisted the intent to send the message. The destination actor must send back a confirmation message. When the sending actor receives this confirmation message you should persist the fact that the message was delivered successfully and then call -the ``confirmDelivery`` method. +the `confirmDelivery` method. -If the persistent actor is not currently recovering, the ``deliver`` method will send the message to -the destination actor. When recovering, messages will be buffered until they have been confirmed using ``confirmDelivery``. +If the persistent actor is not currently recovering, the `deliver` method will send the message to +the destination actor. When recovering, messages will be buffered until they have been confirmed using `confirmDelivery`. Once recovery has completed, if there are outstanding messages that have not been confirmed (during the message replay), the persistent actor will resend these before sending any other messages. -Deliver requires a ``deliveryIdToMessage`` function to pass the provided ``deliveryId`` into the message so that the correlation -between ``deliver`` and ``confirmDelivery`` is possible. The ``deliveryId`` must do the round trip. Upon receipt +Deliver requires a `deliveryIdToMessage` function to pass the provided `deliveryId` into the message so that the correlation +between `deliver` and `confirmDelivery` is possible. The `deliveryId` must do the round trip. Upon receipt of the message, the destination actor will send the same``deliveryId`` wrapped in a confirmation message back to the sender. -The sender will then use it to call ``confirmDelivery`` method to complete the delivery routine. +The sender will then use it to call `confirmDelivery` method to complete the delivery routine. -.. includecode:: code/jdocs/persistence/LambdaPersistenceDocTest.java#at-least-once-example +@@snip [LambdaPersistenceDocTest.java](code/jdocs/persistence/LambdaPersistenceDocTest.java) { #at-least-once-example } -The ``deliveryId`` generated by the persistence module is a strictly monotonically increasing sequence number +The `deliveryId` generated by the persistence module is a strictly monotonically increasing sequence number without gaps. The same sequence is used for all destinations of the actor, i.e. when sending to multiple -destinations the destinations will see gaps in the sequence. It is not possible to use custom ``deliveryId``. +destinations the destinations will see gaps in the sequence. It is not possible to use custom `deliveryId`. However, you can send a custom correlation identifier in the message to the destination. You must then retain -a mapping between the internal ``deliveryId`` (passed into the ``deliveryIdToMessage`` function) and your custom -correlation id (passed into the message). You can do this by storing such mapping in a ``Map(correlationId -> deliveryId)`` -from which you can retrieve the ``deliveryId`` to be passed into the ``confirmDelivery`` method once the receiver +a mapping between the internal `deliveryId` (passed into the `deliveryIdToMessage` function) and your custom +correlation id (passed into the message). You can do this by storing such mapping in a `Map(correlationId -> deliveryId)` +from which you can retrieve the `deliveryId` to be passed into the `confirmDelivery` method once the receiver of your message has replied with your custom correlation id. -The ``AbstractPersistentActorWithAtLeastOnceDelivery`` class has a state consisting of unconfirmed messages and a +The `AbstractPersistentActorWithAtLeastOnceDelivery` class has a state consisting of unconfirmed messages and a sequence number. It does not store this state itself. You must persist events corresponding to the -``deliver`` and ``confirmDelivery`` invocations from your ``PersistentActor`` so that the state can -be restored by calling the same methods during the recovery phase of the ``PersistentActor``. Sometimes +`deliver` and `confirmDelivery` invocations from your `PersistentActor` so that the state can +be restored by calling the same methods during the recovery phase of the `PersistentActor`. Sometimes these events can be derived from other business level events, and sometimes you must create separate events. -During recovery, calls to ``deliver`` will not send out messages, those will be sent later -if no matching ``confirmDelivery`` will have been performed. +During recovery, calls to `deliver` will not send out messages, those will be sent later +if no matching `confirmDelivery` will have been performed. -Support for snapshots is provided by ``getDeliverySnapshot`` and ``setDeliverySnapshot``. -The ``AtLeastOnceDeliverySnapshot`` contains the full delivery state, including unconfirmed messages. +Support for snapshots is provided by `getDeliverySnapshot` and `setDeliverySnapshot`. +The `AtLeastOnceDeliverySnapshot` contains the full delivery state, including unconfirmed messages. If you need a custom snapshot for other parts of the actor state you must also include the -``AtLeastOnceDeliverySnapshot``. It is serialized using protobuf with the ordinary Akka -serialization mechanism. It is easiest to include the bytes of the ``AtLeastOnceDeliverySnapshot`` +`AtLeastOnceDeliverySnapshot`. It is serialized using protobuf with the ordinary Akka +serialization mechanism. It is easiest to include the bytes of the `AtLeastOnceDeliverySnapshot` as a blob in your custom snapshot. -The interval between redelivery attempts is defined by the ``redeliverInterval`` method. -The default value can be configured with the ``akka.persistence.at-least-once-delivery.redeliver-interval`` +The interval between redelivery attempts is defined by the `redeliverInterval` method. +The default value can be configured with the `akka.persistence.at-least-once-delivery.redeliver-interval` configuration key. The method can be overridden by implementation classes to return non-default values. The maximum number of messages that will be sent at each redelivery burst is defined by the -``redeliveryBurstLimit`` method (burst frequency is half of the redelivery interval). If there's a lot of +`redeliveryBurstLimit` method (burst frequency is half of the redelivery interval). If there's a lot of unconfirmed messages (e.g. if the destination is not available for a long time), this helps to prevent an overwhelming amount of messages to be sent at once. The default value can be configured with the -``akka.persistence.at-least-once-delivery.redelivery-burst-limit`` configuration key. The method can be overridden +`akka.persistence.at-least-once-delivery.redelivery-burst-limit` configuration key. The method can be overridden by implementation classes to return non-default values. -After a number of delivery attempts a ``AtLeastOnceDelivery.UnconfirmedWarning`` message -will be sent to ``self``. The re-sending will still continue, but you can choose to call -``confirmDelivery`` to cancel the re-sending. The number of delivery attempts before emitting the -warning is defined by the ``warnAfterNumberOfUnconfirmedAttempts`` method. The default value can be -configured with the ``akka.persistence.at-least-once-delivery.warn-after-number-of-unconfirmed-attempts`` +After a number of delivery attempts a `AtLeastOnceDelivery.UnconfirmedWarning` message +will be sent to `self`. The re-sending will still continue, but you can choose to call +`confirmDelivery` to cancel the re-sending. The number of delivery attempts before emitting the +warning is defined by the `warnAfterNumberOfUnconfirmedAttempts` method. The default value can be +configured with the `akka.persistence.at-least-once-delivery.warn-after-number-of-unconfirmed-attempts` configuration key. The method can be overridden by implementation classes to return non-default values. -The ``AbstractPersistentActorWithAtLeastOnceDelivery`` class holds messages in memory until their successful delivery has been confirmed. +The `AbstractPersistentActorWithAtLeastOnceDelivery` class holds messages in memory until their successful delivery has been confirmed. The maximum number of unconfirmed messages that the actor is allowed to hold in memory -is defined by the ``maxUnconfirmedMessages`` method. If this limit is exceed the ``deliver`` method will -not accept more messages and it will throw ``AtLeastOnceDelivery.MaxUnconfirmedMessagesExceededException``. -The default value can be configured with the ``akka.persistence.at-least-once-delivery.max-unconfirmed-messages`` +is defined by the `maxUnconfirmedMessages` method. If this limit is exceed the `deliver` method will +not accept more messages and it will throw `AtLeastOnceDelivery.MaxUnconfirmedMessagesExceededException`. +The default value can be configured with the `akka.persistence.at-least-once-delivery.max-unconfirmed-messages` configuration key. The method can be overridden by implementation classes to return non-default values. -.. _event-adapters-java: - -Event Adapters -============== + +## Event Adapters In long running projects using event sourcing sometimes the need arises to detach the data model from the domain model completely. Event Adapters help in situations where: -- **Version Migrations** – existing events stored in *Version 1* should be "upcasted" to a new *Version 2* representation, - and the process of doing so involves actual code, not just changes on the serialization layer. For these scenarios - the ``toJournal`` function is usually an identity function, however the ``fromJournal`` is implemented as - ``v1.Event=>v2.Event``, performing the neccessary mapping inside the fromJournal method. - This technique is sometimes refered to as "upcasting" in other CQRS libraries. -- **Separating Domain and Data models** – thanks to EventAdapters it is possible to completely separate the domain model - from the model used to persist data in the Journals. For example one may want to use case classes in the - domain model, however persist their protocol-buffer (or any other binary serialization format) counter-parts to the Journal. - A simple ``toJournal:MyModel=>MyDataModel`` and ``fromJournal:MyDataModel=>MyModel`` adapter can be used to implement this feature. -- **Journal Specialized Data Types** – exposing data types understood by the underlying Journal, for example for data stores which - understand JSON it is possible to write an EventAdapter ``toJournal:Any=>JSON`` such that the Journal can *directly* store the - json instead of serializing the object to its binary representation. + * **Version Migrations** – existing events stored in *Version 1* should be "upcasted" to a new *Version 2* representation, +and the process of doing so involves actual code, not just changes on the serialization layer. For these scenarios +the `toJournal` function is usually an identity function, however the `fromJournal` is implemented as +`v1.Event=>v2.Event`, performing the neccessary mapping inside the fromJournal method. +This technique is sometimes refered to as "upcasting" in other CQRS libraries. + * **Separating Domain and Data models** – thanks to EventAdapters it is possible to completely separate the domain model +from the model used to persist data in the Journals. For example one may want to use case classes in the +domain model, however persist their protocol-buffer (or any other binary serialization format) counter-parts to the Journal. +A simple `toJournal:MyModel=>MyDataModel` and `fromJournal:MyDataModel=>MyModel` adapter can be used to implement this feature. + * **Journal Specialized Data Types** – exposing data types understood by the underlying Journal, for example for data stores which +understand JSON it is possible to write an EventAdapter `toJournal:Any=>JSON` such that the Journal can *directly* store the +json instead of serializing the object to its binary representation. Implementing an EventAdapter is rather stright forward: -.. includecode:: code/jdocs/persistence/PersistenceEventAdapterDocTest.java#identity-event-adapter +@@snip [PersistenceEventAdapterDocTest.java](code/jdocs/persistence/PersistenceEventAdapterDocTest.java) { #identity-event-adapter } Then in order for it to be used on events coming to and from the journal you must bind it using the below configuration syntax: -.. includecode:: ../scala/code/docs/persistence/PersistenceEventAdapterDocSpec.scala#event-adapters-config +@@snip [PersistenceEventAdapterDocSpec.scala](../scala/code/docs/persistence/PersistenceEventAdapterDocSpec.scala) { #event-adapters-config } -It is possible to bind multiple adapters to one class *for recovery*, in which case the ``fromJournal`` methods of all +It is possible to bind multiple adapters to one class *for recovery*, in which case the `fromJournal` methods of all bound adapters will be applied to a given matching event (in order of definition in the configuration). Since each adapter may -return from ``0`` to ``n`` adapted events (called as ``EventSeq``), each adapter can investigate the event and if it should +return from `0` to `n` adapted events (called as `EventSeq`), each adapter can investigate the event and if it should indeed adapt it return the adapted event(s) for it. Other adapters which do not have anything to contribute during this -adaptation simply return ``EventSeq.empty``. The adapted events are then delivered in-order to the ``PersistentActor`` during replay. +adaptation simply return `EventSeq.empty`. The adapted events are then delivered in-order to the `PersistentActor` during replay. -.. note:: - For more advanced schema evolution techniques refer to the :ref:`persistence-schema-evolution-scala` documentation. +@@@ note -.. _persistent-fsm-java: +For more advanced schema evolution techniques refer to the @ref:[Persistence - Schema Evolution](../scala/persistence-schema-evolution.md) documentation. -Persistent FSM -============== -``AbstractPersistentFSM`` handles the incoming messages in an FSM like fashion. +@@@ + + +## Persistent FSM + +`AbstractPersistentFSM` handles the incoming messages in an FSM like fashion. Its internal state is persisted as a sequence of changes, later referred to as domain events. Relationship between incoming messages, FSM's states and transitions, persistence of domain events is defined by a DSL. -A Simple Example ----------------- -To demonstrate the features of the ``AbstractPersistentFSM``, consider an actor which represents a Web store customer. +### A Simple Example + +To demonstrate the features of the `AbstractPersistentFSM`, consider an actor which represents a Web store customer. The contract of our "WebStoreCustomerFSMActor" is that it accepts the following commands: -.. includecode:: ../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java#customer-commands +@@snip [AbstractPersistentFSMTest.java]../../../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java) { #customer-commands } -``AddItem`` sent when the customer adds an item to a shopping cart -``Buy`` - when the customer finishes the purchase -``Leave`` - when the customer leaves the store without purchasing anything -``GetCurrentCart`` allows to query the current state of customer's shopping cart +`AddItem` sent when the customer adds an item to a shopping cart +`Buy` - when the customer finishes the purchase +`Leave` - when the customer leaves the store without purchasing anything +`GetCurrentCart` allows to query the current state of customer's shopping cart The customer can be in one of the following states: -.. includecode:: ../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java#customer-states +@@snip [AbstractPersistentFSMTest.java]../../../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java) { #customer-states } -``LookingAround`` customer is browsing the site, but hasn't added anything to the shopping cart -``Shopping`` customer has recently added items to the shopping cart -``Inactive`` customer has items in the shopping cart, but hasn't added anything recently -``Paid`` customer has purchased the items +`LookingAround` customer is browsing the site, but hasn't added anything to the shopping cart +`Shopping` customer has recently added items to the shopping cart +`Inactive` customer has items in the shopping cart, but hasn't added anything recently +`Paid` customer has purchased the items -.. note:: +@@@ note - ``AbstractPersistentFSM`` states must inherit from ``PersistentFSM.FSMState`` and implement the - ``String identifier()`` method. This is required in order to simplify the serialization of FSM states. - String identifiers should be unique! +`AbstractPersistentFSM` states must inherit from `PersistentFSM.FSMState` and implement the +`String identifier()` method. This is required in order to simplify the serialization of FSM states. +String identifiers should be unique! + +@@@ Customer's actions are "recorded" as a sequence of "domain events" which are persisted. Those events are replayed on actor's start in order to restore the latest customer's state: -.. includecode:: ../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java#customer-domain-events +@@snip [AbstractPersistentFSMTest.java]../../../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java) { #customer-domain-events } Customer state data represents the items in customer's shopping cart: -.. includecode:: ../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java#customer-states-data +@@snip [AbstractPersistentFSMTest.java]../../../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java) { #customer-states-data } Here is how everything is wired together: -.. includecode:: ../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java#customer-fsm-body +@@snip [AbstractPersistentFSMTest.java]../../../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java) { #customer-fsm-body } -.. note:: +@@@ note - State data can only be modified directly on initialization. Later it's modified only as a result of applying domain events. - Override the ``applyEvent`` method to define how state data is affected by domain events, see the example below +State data can only be modified directly on initialization. Later it's modified only as a result of applying domain events. +Override the `applyEvent` method to define how state data is affected by domain events, see the example below -.. includecode:: ../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java#customer-apply-event +@@@ -``andThen`` can be used to define actions which will be executed following event's persistence - convenient for "side effects" like sending a message or logging. -Notice that actions defined in ``andThen`` block are not executed on recovery: +@@snip [AbstractPersistentFSMTest.java]../../../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java) { #customer-apply-event } -.. includecode:: ../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java#customer-andthen-example +`andThen` can be used to define actions which will be executed following event's persistence - convenient for "side effects" like sending a message or logging. +Notice that actions defined in `andThen` block are not executed on recovery: -A snapshot of state data can be persisted by calling the ``saveStateSnapshot()`` method: +@@snip [AbstractPersistentFSMTest.java]../../../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java) { #customer-andthen-example } -.. includecode:: ../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java#customer-snapshot-example +A snapshot of state data can be persisted by calling the `saveStateSnapshot()` method: + +@@snip [AbstractPersistentFSMTest.java]../../../../../akka-persistence/src/test/java/akka/persistence/fsm/AbstractPersistentFSMTest.java) { #customer-snapshot-example } On recovery state data is initialized according to the latest available snapshot, then the remaining domain events are replayed, triggering the -``applyEvent`` method. +`applyEvent` method. -Storage plugins -=============== +## Storage plugins Storage backends for journals and snapshot stores are pluggable in the Akka persistence extension. -A directory of persistence journal and snapshot store plugins is available at the Akka Community Projects page, see `Community plugins`_ +A directory of persistence journal and snapshot store plugins is available at the Akka Community Projects page, see [Community plugins](http://akka.io/community/) Plugins can be selected either by "default", for all persistent actors and views, or "individually", when persistent actor or view defines its own set of plugins. -When persistent actor or view does NOT override ``journalPluginId`` and ``snapshotPluginId`` methods, -persistence extension will use "default" journal and snapshot-store plugins configured in the ``reference.conf``:: +When persistent actor or view does NOT override `journalPluginId` and `snapshotPluginId` methods, +persistence extension will use "default" journal and snapshot-store plugins configured in the `reference.conf`: - akka.persistence.journal.plugin = "" - akka.persistence.snapshot-store.plugin = "" +``` +akka.persistence.journal.plugin = "" +akka.persistence.snapshot-store.plugin = "" +``` -However, these entries are provided as empty "", and require explicit user configuration via override in the user ``application.conf``. -For an example of journal plugin which writes messages to LevelDB see :ref:`local-leveldb-journal-java`. -For an example of snapshot store plugin which writes snapshots as individual files to the local filesystem see :ref:`local-snapshot-store-java`. +However, these entries are provided as empty "", and require explicit user configuration via override in the user `application.conf`. +For an example of journal plugin which writes messages to LevelDB see [Local LevelDB journal](#local-leveldb-journal-java). +For an example of snapshot store plugin which writes snapshots as individual files to the local filesystem see [Local snapshot store](#local-snapshot-store-java). Applications can provide their own plugins by implementing a plugin API and activate them by configuration. Plugin development requires the following imports: -.. includecode:: code/jdocs/persistence/LambdaPersistencePluginDocTest.java#plugin-imports +@@snip [LambdaPersistencePluginDocTest.java](code/jdocs/persistence/LambdaPersistencePluginDocTest.java) { #plugin-imports } -Eager initialization of persistence plugin ------------------------------------------- +### Eager initialization of persistence plugin By default, persistence plugins are started on-demand, as they are used. In some case, however, it might be beneficial -to start a certain plugin eagerly. In order to do that, you should first add the ``akka.persistence.Persistence`` -under the ``akka.extensions`` key. Then, specify the IDs of plugins you wish to start automatically under -``akka.persistence.journal.auto-start-journals`` and ``akka.persistence.snapshot-store.auto-start-snapshot-stores``. +to start a certain plugin eagerly. In order to do that, you should first add the `akka.persistence.Persistence` +under the `akka.extensions` key. Then, specify the IDs of plugins you wish to start automatically under +`akka.persistence.journal.auto-start-journals` and `akka.persistence.snapshot-store.auto-start-snapshot-stores`. -.. _journal-plugin-api-java: + +### Journal plugin API -Journal plugin API ------------------- +A journal plugin extends `AsyncWriteJournal`. -A journal plugin extends ``AsyncWriteJournal``. +`AsyncWriteJournal` is an actor and the methods to be implemented are: -``AsyncWriteJournal`` is an actor and the methods to be implemented are: - -.. includecode:: ../../../akka-persistence/src/main/java/akka/persistence/journal/japi/AsyncWritePlugin.java#async-write-plugin-api +@@snip [AsyncWritePlugin.java]../../../../../akka-persistence/src/main/java/akka/persistence/journal/japi/AsyncWritePlugin.java) { #async-write-plugin-api } If the storage backend API only supports synchronous, blocking writes, the methods should be implemented as: -.. includecode:: code/jdocs/persistence/LambdaPersistencePluginDocTest.java#sync-journal-plugin-api +@@snip [LambdaPersistencePluginDocTest.java](code/jdocs/persistence/LambdaPersistencePluginDocTest.java) { #sync-journal-plugin-api } -A journal plugin must also implement the methods defined in ``AsyncRecovery`` for replays and sequence number recovery: +A journal plugin must also implement the methods defined in `AsyncRecovery` for replays and sequence number recovery: -.. includecode:: ../../../akka-persistence/src/main/java/akka/persistence/journal/japi/AsyncRecoveryPlugin.java#async-replay-plugin-api +@@snip [AsyncRecoveryPlugin.java]../../../../../akka-persistence/src/main/java/akka/persistence/journal/japi/AsyncRecoveryPlugin.java) { #async-replay-plugin-api } A journal plugin can be activated with the following minimal configuration: -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#journal-plugin-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #journal-plugin-config } The journal plugin instance is an actor so the methods corresponding to requests from persistent actors are executed sequentially. It may delegate to asynchronous libraries, spawn futures, or delegate to other @@ -848,28 +847,27 @@ actors to achive parallelism. The journal plugin class must have a constructor with one of these signatures: -* constructor with one ``com.typesafe.config.Config`` parameter and a ``String`` parameter for the config path -* constructor with one ``com.typesafe.config.Config`` parameter -* constructor without parameters + * constructor with one `com.typesafe.config.Config` parameter and a `String` parameter for the config path + * constructor with one `com.typesafe.config.Config` parameter + * constructor without parameters The plugin section of the actor system's config will be passed in the config constructor parameter. The config path -of the plugin is passed in the ``String`` parameter. +of the plugin is passed in the `String` parameter. -The ``plugin-dispatcher`` is the dispatcher used for the plugin actor. If not specified, it defaults to -``akka.persistence.dispatchers.default-plugin-dispatcher``. +The `plugin-dispatcher` is the dispatcher used for the plugin actor. If not specified, it defaults to +`akka.persistence.dispatchers.default-plugin-dispatcher`. Don't run journal tasks/futures on the system default dispatcher, since that might starve other tasks. -Snapshot store plugin API -------------------------- +### Snapshot store plugin API -A snapshot store plugin must extend the ``SnapshotStore`` actor and implement the following methods: +A snapshot store plugin must extend the `SnapshotStore` actor and implement the following methods: -.. includecode:: ../../../akka-persistence/src/main/java/akka/persistence/snapshot/japi/SnapshotStorePlugin.java#snapshot-store-plugin-api +@@snip [SnapshotStorePlugin.java]../../../../../akka-persistence/src/main/java/akka/persistence/snapshot/japi/SnapshotStorePlugin.java) { #snapshot-store-plugin-api } A snapshot store plugin can be activated with the following minimal configuration: -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#snapshot-store-plugin-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #snapshot-store-plugin-config } The snapshot store instance is an actor so the methods corresponding to requests from persistent actors are executed sequentially. It may delegate to asynchronous libraries, spawn futures, or delegate to other @@ -877,246 +875,248 @@ actors to achive parallelism. The snapshot store plugin class must have a constructor with one of these signatures: -* constructor with one ``com.typesafe.config.Config`` parameter and a ``String`` parameter for the config path -* constructor with one ``com.typesafe.config.Config`` parameter -* constructor without parameters + * constructor with one `com.typesafe.config.Config` parameter and a `String` parameter for the config path + * constructor with one `com.typesafe.config.Config` parameter + * constructor without parameters The plugin section of the actor system's config will be passed in the config constructor parameter. The config path -of the plugin is passed in the ``String`` parameter. +of the plugin is passed in the `String` parameter. -The ``plugin-dispatcher`` is the dispatcher used for the plugin actor. If not specified, it defaults to -``akka.persistence.dispatchers.default-plugin-dispatcher``. +The `plugin-dispatcher` is the dispatcher used for the plugin actor. If not specified, it defaults to +`akka.persistence.dispatchers.default-plugin-dispatcher`. Don't run snapshot store tasks/futures on the system default dispatcher, since that might starve other tasks. -Plugin TCK ----------- -In order to help developers build correct and high quality storage plugins, we provide a Technology Compatibility Kit (`TCK `_ for short). +### Plugin TCK -The TCK is usable from Java as well as Scala projects. For Java you need to include the akka-persistence-tck dependency:: +In order to help developers build correct and high quality storage plugins, we provide a Technology Compatibility Kit ([TCK](http://en.wikipedia.org/wiki/Technology_Compatibility_Kit) for short). - - com.typesafe.akka - akka-persistence-tck_${scala.version} - @version@ - test - +The TCK is usable from Java as well as Scala projects. For Java you need to include the akka-persistence-tck dependency: -To include the Journal TCK tests in your test suite simply extend the provided ``JavaJournalSpec``: +``` + + com.typesafe.akka + akka-persistence-tck_${scala.version} + @version@ + test + +``` -.. includecode:: ./code/jdocs/persistence/LambdaPersistencePluginDocTest.java#journal-tck-java +To include the Journal TCK tests in your test suite simply extend the provided `JavaJournalSpec`: -Please note that some of the tests are optional, and by overriding the ``supports...`` methods you give the +@@snip [LambdaPersistencePluginDocTest.java](./code/jdocs/persistence/LambdaPersistencePluginDocTest.java) { #journal-tck-java } + +Please note that some of the tests are optional, and by overriding the `supports...` methods you give the TCK the needed information about which tests to run. You can implement these methods using the provided -``CapabilityFlag.on`` / ``CapabilityFlag.off`` values. +`CapabilityFlag.on` / `CapabilityFlag.off` values. -We also provide a simple benchmarking class ``JavaJournalPerfSpec`` which includes all the tests that ``JavaJournalSpec`` +We also provide a simple benchmarking class `JavaJournalPerfSpec` which includes all the tests that `JavaJournalSpec` has, and also performs some longer operations on the Journal while printing its performance stats. While it is NOT aimed to provide a proper benchmarking environment it can be used to get a rough feel about your journal's performance in the most typical scenarios. -In order to include the ``SnapshotStore`` TCK tests in your test suite simply extend the ``SnapshotStoreSpec``: +In order to include the `SnapshotStore` TCK tests in your test suite simply extend the `SnapshotStoreSpec`: -.. includecode:: ./code/jdocs/persistence/LambdaPersistencePluginDocTest.java#snapshot-store-tck-java +@@snip [LambdaPersistencePluginDocTest.java](./code/jdocs/persistence/LambdaPersistencePluginDocTest.java) { #snapshot-store-tck-java } In case your plugin requires some setting up (starting a mock database, removing temporary files etc.) you can override the -``beforeAll`` and ``afterAll`` methods to hook into the tests lifecycle: +`beforeAll` and `afterAll` methods to hook into the tests lifecycle: -.. includecode:: ./code/jdocs/persistence/LambdaPersistencePluginDocTest.java#journal-tck-before-after-java +@@snip [LambdaPersistencePluginDocTest.java](./code/jdocs/persistence/LambdaPersistencePluginDocTest.java) { #journal-tck-before-after-java } We *highly recommend* including these specifications in your test suite, as they cover a broad range of cases you might have otherwise forgotten to test for when writing a plugin from scratch. -Pre-packaged plugins -==================== +## Pre-packaged plugins -.. _local-leveldb-journal-java: + +### Local LevelDB journal -Local LevelDB journal ---------------------- - -The LevelDB journal plugin config entry is ``akka.persistence.journal.leveldb``. It writes messages to a local LevelDB +The LevelDB journal plugin config entry is `akka.persistence.journal.leveldb`. It writes messages to a local LevelDB instance. Enable this plugin by defining config property: -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#leveldb-plugin-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #leveldb-plugin-config } -LevelDB based plugins will also require the following additional dependency declaration:: +LevelDB based plugins will also require the following additional dependency declaration: - - org.iq80.leveldb - leveldb - 0.7 - - - org.fusesource.leveldbjni - leveldbjni-all - 1.8 - +``` + + org.iq80.leveldb + leveldb + 0.7 + + + org.fusesource.leveldbjni + leveldbjni-all + 1.8 + +``` -The default location of LevelDB files is a directory named ``journal`` in the current working +The default location of LevelDB files is a directory named `journal` in the current working directory. This location can be changed by configuration where the specified path can be relative or absolute: -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#journal-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #journal-config } With this plugin, each actor system runs its own private LevelDB instance. -.. _shared-leveldb-journal-java: - -Shared LevelDB journal ----------------------- + +### Shared LevelDB journal A LevelDB instance can also be shared by multiple actor systems (on the same or on different nodes). This, for example, allows persistent actors to failover to a backup node and continue using the shared journal instance from the backup node. -.. warning:: +@@@ warning - A shared LevelDB instance is a single point of failure and should therefore only be used for testing - purposes. Highly-available, replicated journals are available as `Community plugins`_. +A shared LevelDB instance is a single point of failure and should therefore only be used for testing +purposes. Highly-available, replicated journals are available as [Community plugins](http://akka.io/community/). -.. note:: +@@@ - This plugin has been supplanted by :ref:`Persistence Plugin Proxy`. +@@@ note -A shared LevelDB instance is started by instantiating the ``SharedLeveldbStore`` actor. +This plugin has been supplanted by [Persistence Plugin Proxy](#persistence-plugin-proxy-java). -.. includecode:: code/jdocs/persistence/LambdaPersistencePluginDocTest.java#shared-store-creation +@@@ -By default, the shared instance writes journaled messages to a local directory named ``journal`` in the current +A shared LevelDB instance is started by instantiating the `SharedLeveldbStore` actor. + +@@snip [LambdaPersistencePluginDocTest.java](code/jdocs/persistence/LambdaPersistencePluginDocTest.java) { #shared-store-creation } + +By default, the shared instance writes journaled messages to a local directory named `journal` in the current working directory. The storage location can be changed by configuration: -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#shared-store-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #shared-store-config } -Actor systems that use a shared LevelDB store must activate the ``akka.persistence.journal.leveldb-shared`` +Actor systems that use a shared LevelDB store must activate the `akka.persistence.journal.leveldb-shared` plugin. -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#shared-journal-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #shared-journal-config } -This plugin must be initialized by injecting the (remote) ``SharedLeveldbStore`` actor reference. Injection is -done by calling the ``SharedLeveldbJournal.setStore`` method with the actor reference as argument. +This plugin must be initialized by injecting the (remote) `SharedLeveldbStore` actor reference. Injection is +done by calling the `SharedLeveldbJournal.setStore` method with the actor reference as argument. -.. includecode:: code/jdocs/persistence/LambdaPersistencePluginDocTest.java#shared-store-usage +@@snip [LambdaPersistencePluginDocTest.java](code/jdocs/persistence/LambdaPersistencePluginDocTest.java) { #shared-store-usage } Internal journal commands (sent by persistent actors) are buffered until injection completes. Injection is idempotent i.e. only the first injection is used. -.. _local-snapshot-store-java: + +### Local snapshot store -Local snapshot store --------------------- - -Local snapshot store plugin config entry is ``akka.persistence.snapshot-store.local``. It writes snapshot files to +Local snapshot store plugin config entry is `akka.persistence.snapshot-store.local`. It writes snapshot files to the local filesystem. Enable this plugin by defining config property: -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#leveldb-snapshot-plugin-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #leveldb-snapshot-plugin-config } -The default storage location is a directory named ``snapshots`` in the current working +The default storage location is a directory named `snapshots` in the current working directory. This can be changed by configuration where the specified path can be relative or absolute: -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#snapshot-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #snapshot-config } Note that it is not mandatory to specify a snapshot store plugin. If you don't use snapshots you don't have to configure it. -.. _persistence-plugin-proxy-java: - -Persistence Plugin Proxy ------------------------- + +### Persistence Plugin Proxy A persistence plugin proxy allows sharing of journals and snapshot stores across multiple actor systems (on the same or on different nodes). This, for example, allows persistent actors to failover to a backup node and continue using the shared journal instance from the backup node. The proxy works by forwarding all the journal/snapshot store messages to a single, shared, persistence plugin instance, and therefor supports any use case supported by the proxied plugin. -.. warning:: +@@@ warning - A shared journal/snapshot store is a single point of failure and should therefore only be used for testing - purposes. Highly-available, replicated persistence plugins are available as `Community plugins`_. +A shared journal/snapshot store is a single point of failure and should therefore only be used for testing +purposes. Highly-available, replicated persistence plugins are available as [Community plugins](http://akka.io/community/). -The journal and snapshot store proxies are controlled via the ``akka.persistence.journal.proxy`` and -``akka.persistence.snapshot-store.proxy`` configuration entries, respectively. Set the ``target-journal-plugin`` or -``target-snapshot-store-plugin`` keys to the underlying plugin you wish to use (for example: -``akka.persistence.journal.leveldb``). The ``start-target-journal`` and ``start-target-snapshot-store`` keys should be -set to ``on`` in exactly one actor system - this is the system that will instantiate the shared persistence plugin. -Next, the proxy needs to be told how to find the shared plugin. This can be done by setting the ``target-journal-address`` -and ``target-snapshot-store-address`` configuration keys, or programmatically by calling the -``PersistencePluginProxy.setTargetLocation`` method. +@@@ -.. note:: +The journal and snapshot store proxies are controlled via the `akka.persistence.journal.proxy` and +`akka.persistence.snapshot-store.proxy` configuration entries, respectively. Set the `target-journal-plugin` or +`target-snapshot-store-plugin` keys to the underlying plugin you wish to use (for example: +`akka.persistence.journal.leveldb`). The `start-target-journal` and `start-target-snapshot-store` keys should be +set to `on` in exactly one actor system - this is the system that will instantiate the shared persistence plugin. +Next, the proxy needs to be told how to find the shared plugin. This can be done by setting the `target-journal-address` +and `target-snapshot-store-address` configuration keys, or programmatically by calling the +`PersistencePluginProxy.setTargetLocation` method. - Akka starts extensions lazily when they are required, and this includes the proxy. This means that in order for the - proxy to work, the persistence plugin on the target node must be instantiated. This can be done by instantiating the - ``PersistencePluginProxyExtension`` :ref:`extension`, or by calling the ``PersistencePluginProxy.start`` method. +@@@ note -.. note:: +Akka starts extensions lazily when they are required, and this includes the proxy. This means that in order for the +proxy to work, the persistence plugin on the target node must be instantiated. This can be done by instantiating the +`PersistencePluginProxyExtension` @ref:[extension](extending-akka.md), or by calling the `PersistencePluginProxy.start` method. - The proxied persistence plugin can (and should) be configured using its original configuration keys. +@@@ +@@@ note -.. _custom-serialization-java: +The proxied persistence plugin can (and should) be configured using its original configuration keys. -Custom serialization -==================== +@@@ -Serialization of snapshots and payloads of ``Persistent`` messages is configurable with Akka's -:ref:`serialization-java` infrastructure. For example, if an application wants to serialize + +## Custom serialization -* payloads of type ``MyPayload`` with a custom ``MyPayloadSerializer`` and -* snapshots of type ``MySnapshot`` with a custom ``MySnapshotSerializer`` +Serialization of snapshots and payloads of `Persistent` messages is configurable with Akka's +@ref:[Serialization](serialization.md) infrastructure. For example, if an application wants to serialize + + * payloads of type `MyPayload` with a custom `MyPayloadSerializer` and + * snapshots of type `MySnapshot` with a custom `MySnapshotSerializer` it must add -.. includecode:: ../scala/code/docs/persistence/PersistenceSerializerDocSpec.scala#custom-serializer-config +@@snip [PersistenceSerializerDocSpec.scala](../scala/code/docs/persistence/PersistenceSerializerDocSpec.scala) { #custom-serializer-config } to the application configuration. If not specified, a default serializer is used. -For more advanced schema evolution techniques refer to the :ref:`persistence-schema-evolution-scala` documentation. +For more advanced schema evolution techniques refer to the @ref:[Persistence - Schema Evolution](../scala/persistence-schema-evolution.md) documentation. -Testing -======= +## Testing -When running tests with LevelDB default settings in ``sbt``, make sure to set ``fork := true`` in your sbt project. Otherwise, you'll see an ``UnsatisfiedLinkError``. Alternatively, you can switch to a LevelDB Java port by setting +When running tests with LevelDB default settings in `sbt`, make sure to set `fork := true` in your sbt project. Otherwise, you'll see an `UnsatisfiedLinkError`. Alternatively, you can switch to a LevelDB Java port by setting -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#native-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #native-config } or -.. includecode:: ../scala/code/docs/persistence/PersistencePluginDocSpec.scala#shared-store-native-config +@@snip [PersistencePluginDocSpec.scala](../scala/code/docs/persistence/PersistencePluginDocSpec.scala) { #shared-store-native-config } in your Akka configuration. The LevelDB Java port is for testing purposes only. -.. warning:: - It is not possible to test persistence provided classes (i.e. :ref:`PersistentActor ` - and :ref:`AtLeastOnceDelivery `) using ``TestActorRef`` due to its *synchronous* nature. - These traits need to be able to perform asynchronous tasks in the background in order to handle internal persistence - related events. +@@@ warning - When testing Persistence based projects always rely on :ref:`asynchronous messaging using the TestKit `. +It is not possible to test persistence provided classes (i.e. [PersistentActor](#event-sourcing-java) +and [AtLeastOnceDelivery](#at-least-once-delivery-java)) using `TestActorRef` due to its *synchronous* nature. +These traits need to be able to perform asynchronous tasks in the background in order to handle internal persistence +related events. -Configuration -============= +When testing Persistence based projects always rely on @ref:[asynchronous messaging using the TestKit](testing.md#async-integration-testing-java). + +@@@ + +## Configuration There are several configuration properties for the persistence module, please refer -to the :ref:`reference configuration `. +to the @ref:[reference configuration](../general/configuration.md#config-akka-persistence). -Multiple persistence plugin configurations -========================================== +## Multiple persistence plugin configurations By default, a persistent actor or view will use "default" journal and snapshot store plugins -configured in the following sections of the ``reference.conf`` configuration resource: +configured in the following sections of the `reference.conf` configuration resource: -.. includecode:: ../scala/code/docs/persistence/PersistenceMultiDocSpec.scala#default-config +@@snip [PersistenceMultiDocSpec.scala](../scala/code/docs/persistence/PersistenceMultiDocSpec.scala) { #default-config } -Note that in this case the actor or view overrides only ``persistenceId`` method: +Note that in this case the actor or view overrides only `persistenceId` method: -.. includecode:: ../java/code/jdocs/persistence/PersistenceMultiDocTest.java#default-plugins +@@snip [PersistenceMultiDocTest.java](../java/code/jdocs/persistence/PersistenceMultiDocTest.java) { #default-plugins } -When a persistent actor or view overrides ``journalPluginId`` and ``snapshotPluginId`` methods, +When a persistent actor or view overrides `journalPluginId` and `snapshotPluginId` methods, the actor or view will be serviced by these specific persistence plugins instead of the defaults: -.. includecode:: ../java/code/jdocs/persistence/PersistenceMultiDocTest.java#override-plugins +@@snip [PersistenceMultiDocTest.java](../java/code/jdocs/persistence/PersistenceMultiDocTest.java) { #override-plugins } -Note that ``journalPluginId`` and ``snapshotPluginId`` must refer to properly configured ``reference.conf`` -plugin entries with a standard ``class`` property as well as settings which are specific for those plugins, i.e.: +Note that `journalPluginId` and `snapshotPluginId` must refer to properly configured `reference.conf` +plugin entries with a standard `class` property as well as settings which are specific for those plugins, i.e.: -.. includecode:: ../scala/code/docs/persistence/PersistenceMultiDocSpec.scala#override-config +@@snip [PersistenceMultiDocSpec.scala](../scala/code/docs/persistence/PersistenceMultiDocSpec.scala) { #override-config } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/remoting-artery.md b/akka-docs/src/main/paradox/java/remoting-artery.md index 59115f22b8..895a4821d5 100644 --- a/akka-docs/src/main/paradox/java/remoting-artery.md +++ b/akka-docs/src/main/paradox/java/remoting-artery.md @@ -1,101 +1,105 @@ -.. _remoting-artery-java: +# Remoting (codename Artery) -########################## -Remoting (codename Artery) -########################## +@@@ note -.. note:: +This page describes the @ref:[may change](../common/may-change.md) remoting subsystem, codenamed *Artery* that will eventually +replace the old remoting implementation. For the current stable remoting system please refer to @ref:[Remoting](remoting.md). - This page describes the :ref:`may change ` remoting subsystem, codenamed *Artery* that will eventually - replace the old remoting implementation. For the current stable remoting system please refer to :ref:`remoting-java`. +@@@ Remoting enables Actor systems on different hosts or JVMs to communicate with each other. By enabling remoting the system will start listening on a provided network address and also gains the ability to connect to other systems through the network. From the application's perspective there is no API difference between local or remote -systems, :class:`ActorRef` instances that point to remote systems look exactly the same as local ones: they can be +systems, `ActorRef` instances that point to remote systems look exactly the same as local ones: they can be sent messages to, watched, etc. -Every :class:`ActorRef` contains hostname and port information and can be passed around even on the network. This means -that on a network every :class:`ActorRef` is a unique identifier of an actor on that network. +Every `ActorRef` contains hostname and port information and can be passed around even on the network. This means +that on a network every `ActorRef` is a unique identifier of an actor on that network. Remoting is not a server-client technology. All systems using remoting can contact any other system on the network -if they possess an :class:`ActorRef` pointing to those system. This means that every system that is remoting enabled +if they possess an `ActorRef` pointing to those system. This means that every system that is remoting enabled acts as a "server" to which arbitrary systems on the same network can connect to. -What is new in Artery ---------------------- +## What is new in Artery Artery is a reimplementation of the old remoting module aimed at improving performance and stability. It is mostly source compatible with the old implementation and it is a drop-in replacement in many cases. Main features of Artery compared to the previous implementation: -* Based on `Aeron `_ (UDP) instead of TCP -* Focused on high-throughput, low-latency communication -* Isolation of internal control messages from user messages improving stability and reducing false failure detection - in case of heavy traffic by using a dedicated subchannel. -* Mostly allocation-free operation -* Support for a separate subchannel for large messages to avoid interference with smaller messages -* Compression of actor paths on the wire to reduce overhead for smaller messages -* Support for faster serialization/deserialization using ByteBuffers directly -* Built-in Flight-Recorder to help debugging implementation issues without polluting users logs with implementation - specific events -* Providing protocol stability across major Akka versions to support rolling updates of large-scale systems + * Based on [Aeron](https://github.com/real-logic/Aeron) (UDP) instead of TCP + * Focused on high-throughput, low-latency communication + * Isolation of internal control messages from user messages improving stability and reducing false failure detection +in case of heavy traffic by using a dedicated subchannel. + * Mostly allocation-free operation + * Support for a separate subchannel for large messages to avoid interference with smaller messages + * Compression of actor paths on the wire to reduce overhead for smaller messages + * Support for faster serialization/deserialization using ByteBuffers directly + * Built-in Flight-Recorder to help debugging implementation issues without polluting users logs with implementation +specific events + * Providing protocol stability across major Akka versions to support rolling updates of large-scale systems The main incompatible change from the previous implementation that the protocol field of the string representation of an -:class:`ActorRef` is always `akka` instead of the previously used `akka.tcp` or `akka.ssl.tcp`. Configuration properties +`ActorRef` is always *akka* instead of the previously used *akka.tcp* or *akka.ssl.tcp*. Configuration properties are also different. -Preparing your ActorSystem for Remoting ---------------------------------------- +## Preparing your ActorSystem for Remoting -The Akka remoting is a separate jar file. Make sure that you have the following dependency in your project:: +The Akka remoting is a separate jar file. Make sure that you have the following dependency in your project: - - com.typesafe.akka - akka-remote_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-remote_@binVersion@ + @version@ + +``` To enable remote capabilities in your Akka project you should, at a minimum, add the following changes -to your ``application.conf`` file:: +to your `application.conf` file: - akka { - actor { - provider = remote - } - remote { - artery { - enabled = on - canonical.hostname = "127.0.0.1" - canonical.port = 25520 - } +``` +akka { + actor { + provider = remote + } + remote { + artery { + enabled = on + canonical.hostname = "127.0.0.1" + canonical.port = 25520 } } +} +``` As you can see in the example above there are four things you need to add to get started: -* Change provider from ``local`` to ``remote`` -* Enable Artery to use it as the remoting implementation -* Add host name - the machine you want to run the actor system on; this host - name is exactly what is passed to remote systems in order to identify this - system and consequently used for connecting back to this system if need be, - hence set it to a reachable IP address or resolvable name in case you want to - communicate across the network. -* Add port number - the port the actor system should listen on, set to 0 to have it chosen automatically + * Change provider from `local` to `remote` + * Enable Artery to use it as the remoting implementation + * Add host name - the machine you want to run the actor system on; this host +name is exactly what is passed to remote systems in order to identify this +system and consequently used for connecting back to this system if need be, +hence set it to a reachable IP address or resolvable name in case you want to +communicate across the network. + * Add port number - the port the actor system should listen on, set to 0 to have it chosen automatically -.. note:: +@@@ note - The port number needs to be unique for each actor system on the same machine even if the actor - systems have different names. This is because each actor system has its own networking subsystem - listening for connections and handling messages as not to interfere with other actor systems. +The port number needs to be unique for each actor system on the same machine even if the actor +systems have different names. This is because each actor system has its own networking subsystem +listening for connections and handling messages as not to interfere with other actor systems. + +@@@ The example above only illustrates the bare minimum of properties you have to add to enable remoting. -All settings are described in :ref:`remote-configuration-artery-java`. +All settings are described in [Remote Configuration](#remote-configuration-artery-java). -.. note:: - Aeron requires 64bit JVM to work reliably. +@@@ note -Canonical address -^^^^^^^^^^^^^^^^^ +Aeron requires 64bit JVM to work reliably. + +@@@ + +### Canonical address In order to remoting to work properly, where each system can send messages to any other system on the same network (for example a system forwards a message to a third system, and the third replies directly to the sender system) @@ -104,147 +108,154 @@ unique name of the system and will be used by other systems to open a connection that if a host has multiple names (different DNS records pointing to the same IP address) then only one of these can be *canonical*. If a message arrives to a system but it contains a different hostname than the expected canonical name then the message will be dropped. If multiple names for a system would be allowed, then equality checks among -:class:`ActorRef` instances would no longer to be trusted and this would violate the fundamental assumption that +`ActorRef` instances would no longer to be trusted and this would violate the fundamental assumption that an actor has a globally unique reference on a given network. As a consequence, this also means that localhost addresses -(e.g. `127.0.0.1`) cannot be used in general (apart from local development) since they are not unique addresses in a +(e.g. *127.0.0.1*) cannot be used in general (apart from local development) since they are not unique addresses in a real network. In cases, where Network Address Translation (NAT) is used or other network bridging is involved, it is important to configure the system so that it understands that there is a difference between his externally visible, canonical -address and between the host-port pair that is used to listen for connections. See :ref:`remote-configuration-nat-artery-java` +address and between the host-port pair that is used to listen for connections. See [Akka behind NAT or in a Docker container](#remote-configuration-nat-artery-java) for details. -Acquiring references to remote actors -------------------------------------- +## Acquiring references to remote actors -In order to communicate with an actor, it is necessary to have its :class:`ActorRef`. In the local case it is usually -the creator of the actor (the caller of ``actorOf()``) is who gets the :class:`ActorRef` for an actor that it can +In order to communicate with an actor, it is necessary to have its `ActorRef`. In the local case it is usually +the creator of the actor (the caller of `actorOf()`) is who gets the `ActorRef` for an actor that it can then send to other actors. In other words: -* An Actor can get a remote Actor's reference simply by receiving a message from it (as it's available as ``getSender()`` then), - or inside of a remote message (e.g. `PleaseReply(message: String, remoteActorRef: ActorRef)`) + * An Actor can get a remote Actor's reference simply by receiving a message from it (as it's available as `getSender()` then), +or inside of a remote message (e.g. *PleaseReply(message: String, remoteActorRef: ActorRef)*) Alternatively, an actor can look up another located at a known path using -:class:`ActorSelection`. These methods are available even in remoting enabled systems: +`ActorSelection`. These methods are available even in remoting enabled systems: -* Remote Lookup : used to look up an actor on a remote node with ``actorSelection(path)`` -* Remote Creation : used to create an actor on a remote node with ``actorOf(Props(...), actorName)`` + * Remote Lookup : used to look up an actor on a remote node with `actorSelection(path)` + * Remote Creation : used to create an actor on a remote node with `actorOf(Props(...), actorName)` In the next sections the two alternatives are described in detail. +### Looking up Remote Actors -Looking up Remote Actors -^^^^^^^^^^^^^^^^^^^^^^^^ +`actorSelection(path)` will obtain an `ActorSelection` to an Actor on a remote node, e.g.: -``actorSelection(path)`` will obtain an ``ActorSelection`` to an Actor on a remote node, e.g.:: +``` +ActorSelection selection = + context.actorSelection("akka://actorSystemName@10.0.0.1:25520/user/actorName"); +``` - ActorSelection selection = - context.actorSelection("akka://actorSystemName@10.0.0.1:25520/user/actorName"); +As you can see from the example above the following pattern is used to find an actor on a remote node: -As you can see from the example above the following pattern is used to find an actor on a remote node:: +``` +akka://@:/ +``` - akka://@:/ +@@@ note -.. note:: +Unlike with earlier remoting, the protocol field is always *akka* as pluggable transports are no longer supported. - Unlike with earlier remoting, the protocol field is always `akka` as pluggable transports are no longer supported. +@@@ -Once you obtained a selection to the actor you can interact with it in the same way you would with a local actor, e.g.:: +Once you obtained a selection to the actor you can interact with it in the same way you would with a local actor, e.g.: - selection.tell("Pretty awesome feature", getSelf()); +``` +selection.tell("Pretty awesome feature", getSelf()); +``` -To acquire an :class:`ActorRef` for an :class:`ActorSelection` you need to -send a message to the selection and use the ``sender`` reference of the reply from -the actor. There is a built-in ``Identify`` message that all Actors will understand -and automatically reply to with a ``ActorIdentity`` message containing the -:class:`ActorRef`. This can also be done with the ``resolveOne`` method of -the :class:`ActorSelection`, which returns a ``Future`` of the matching -:class:`ActorRef`. +To acquire an `ActorRef` for an `ActorSelection` you need to +send a message to the selection and use the `sender` reference of the reply from +the actor. There is a built-in `Identify` message that all Actors will understand +and automatically reply to with a `ActorIdentity` message containing the +`ActorRef`. This can also be done with the `resolveOne` method of +the `ActorSelection`, which returns a `Future` of the matching +`ActorRef`. -For more details on how actor addresses and paths are formed and used, please refer to :ref:`addressing`. +For more details on how actor addresses and paths are formed and used, please refer to @ref:[Actor References, Paths and Addresses](../general/addressing.md). -.. note:: +@@@ note - Message sends to actors that are actually in the sending actor system do not - get delivered via the remote actor ref provider. They're delivered directly, - by the local actor ref provider. +Message sends to actors that are actually in the sending actor system do not +get delivered via the remote actor ref provider. They're delivered directly, +by the local actor ref provider. - Aside from providing better performance, this also means that if the hostname - you configure remoting to listen as cannot actually be resolved from within - the very same actor system, such messages will (perhaps counterintuitively) - be delivered just fine. +Aside from providing better performance, this also means that if the hostname +you configure remoting to listen as cannot actually be resolved from within +the very same actor system, such messages will (perhaps counterintuitively) +be delivered just fine. +@@@ -Creating Actors Remotely -^^^^^^^^^^^^^^^^^^^^^^^^ +### Creating Actors Remotely If you want to use the creation functionality in Akka remoting you have to further amend the -``application.conf`` file in the following way (only showing deployment section):: +`application.conf` file in the following way (only showing deployment section): - akka { - actor { - deployment { - /sampleActor { - remote = "akka://sampleActorSystem@127.0.0.1:2553" - } +``` +akka { + actor { + deployment { + /sampleActor { + remote = "akka://sampleActorSystem@127.0.0.1:2553" } } } +} +``` -The configuration above instructs Akka to react when an actor with path ``/sampleActor`` is created, i.e. -using ``system.actorOf(Props(...), "sampleActor")``. This specific actor will not be directly instantiated, +The configuration above instructs Akka to react when an actor with path `/sampleActor` is created, i.e. +using `system.actorOf(Props(...), "sampleActor")`. This specific actor will not be directly instantiated, but instead the remote daemon of the remote system will be asked to create the actor, -which in this sample corresponds to ``sampleActorSystem@127.0.0.1:2553``. +which in this sample corresponds to `sampleActorSystem@127.0.0.1:2553`. Once you have configured the properties above you would do the following in code: -.. includecode:: code/jdocs/remoting/RemoteDeploymentDocTest.java#sample-actor +@@snip [RemoteDeploymentDocTest.java](code/jdocs/remoting/RemoteDeploymentDocTest.java) { #sample-actor } -The actor class ``SampleActor`` has to be available to the runtimes using it, i.e. the classloader of the +The actor class `SampleActor` has to be available to the runtimes using it, i.e. the classloader of the actor systems has to have a JAR containing the class. -.. note:: +@@@ note - In order to ensure serializability of ``Props`` when passing constructor - arguments to the actor being created, do not make the factory an inner class: - this will inherently capture a reference to its enclosing object, which in - most cases is not serializable. It is best to create a factory method in the - companion object of the actor’s class. +In order to ensure serializability of `Props` when passing constructor +arguments to the actor being created, do not make the factory an inner class: +this will inherently capture a reference to its enclosing object, which in +most cases is not serializable. It is best to create a factory method in the +companion object of the actor’s class. - Serializability of all Props can be tested by setting the configuration item - ``akka.actor.serialize-creators=on``. Only Props whose ``deploy`` has - ``LocalScope`` are exempt from this check. +Serializability of all Props can be tested by setting the configuration item +`akka.actor.serialize-creators=on`. Only Props whose `deploy` has +`LocalScope` are exempt from this check. + +@@@ You can use asterisks as wildcard matches for the actor paths, so you could specify: -``/*/sampleActor`` and that would match all ``sampleActor`` on that level in the hierarchy. +`/*/sampleActor` and that would match all `sampleActor` on that level in the hierarchy. You can also use wildcard in the last position to match all actors at a certain level: -``/someParent/*``. Non-wildcard matches always have higher priority to match than wildcards, so: -``/foo/bar`` is considered **more specific** than ``/foo/*`` and only the highest priority match is used. -Please note that it **cannot** be used to partially match section, like this: ``/foo*/bar``, ``/f*o/bar`` etc. +`/someParent/*`. Non-wildcard matches always have higher priority to match than wildcards, so: +`/foo/bar` is considered **more specific** than `/foo/*` and only the highest priority match is used. +Please note that it **cannot** be used to partially match section, like this: `/foo*/bar`, `/f*o/bar` etc. -Programmatic Remote Deployment -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Programmatic Remote Deployment To allow dynamically deployed systems, it is also possible to include -deployment configuration in the :class:`Props` which are used to create an +deployment configuration in the `Props` which are used to create an actor: this information is the equivalent of a deployment section from the configuration file, and if both are given, the external configuration takes precedence. With these imports: -.. includecode:: code/jdocs/remoting/RemoteDeploymentDocTest.java#import +@@snip [RemoteDeploymentDocTest.java](code/jdocs/remoting/RemoteDeploymentDocTest.java) { #import } and a remote address like this: -.. includecode:: code/jdocs/remoting/RemoteDeploymentDocTest.java#make-address-artery +@@snip [RemoteDeploymentDocTest.java](code/jdocs/remoting/RemoteDeploymentDocTest.java) { #make-address-artery } you can advise the system to create a child on that remote node like so: -.. includecode:: code/jdocs/remoting/RemoteDeploymentDocTest.java#deploy +@@snip [RemoteDeploymentDocTest.java](code/jdocs/remoting/RemoteDeploymentDocTest.java) { #deploy } -Remote deployment whitelist -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Remote deployment whitelist As remote deployment can potentially be abused by both users and even attackers a whitelist feature is available to guard the ActorSystem from deploying unexpected actors. Please note that remote deployment @@ -252,104 +263,107 @@ is *not* remote code loading, the Actors class to be deployed onto a remote syst remote system. This still however may pose a security risk, and one may want to restrict remote deployment to only a specific set of known actors by enabling the whitelist feature. -To enable remote deployment whitelisting set the ``akka.remote.deployment.enable-whitelist`` value to ``on``. +To enable remote deployment whitelisting set the `akka.remote.deployment.enable-whitelist` value to `on`. The list of allowed classes has to be configured on the "remote" system, in other words on the system onto which others will be attempting to remote deploy Actors. That system, locally, knows best which Actors it should or should not allow others to remote deploy onto it. The full settings section may for example look like this: -.. includecode:: ../../../akka-remote/src/test/scala/akka/remote/RemoteDeploymentWhitelistSpec.scala#whitelist-config +@@snip [RemoteDeploymentWhitelistSpec.scala]../../../../../akka-remote/src/test/scala/akka/remote/RemoteDeploymentWhitelistSpec.scala) { #whitelist-config } Actor classes not included in the whitelist will not be allowed to be remote deployed onto this system. -.. _remote-security-java-artery: + +## Remote Security -Remote Security ---------------- - -An ``ActorSystem`` should not be exposed via Akka Remote (Artery) over plain Aeron/UDP to an untrusted network (e.g. internet). +An `ActorSystem` should not be exposed via Akka Remote (Artery) over plain Aeron/UDP to an untrusted network (e.g. internet). It should be protected by network security, such as a firewall. There is currently no support for encryption with Artery so if network security is not considered as enough protection the classic remoting with -:ref:`TLS and mutual authentication ` should be used. +@ref:[TLS and mutual authentication](remoting.md#remote-tls-java) should be used. Best practice is that Akka remoting nodes should only be accessible from the adjacent network. -It is also security best-practice to :ref:`disable the Java serializer ` because of -its multiple `known attack surfaces `_. +It is also security best-practice to [disable the Java serializer](#disable-java-serializer-java-artery) because of +its multiple [known attack surfaces](https://community.hpe.com/t5/Security-Research/The-perils-of-Java-deserialization/ba-p/6838995). -Untrusted Mode -^^^^^^^^^^^^^^ +### Untrusted Mode As soon as an actor system can connect to another remotely, it may in principle send any possible message to any actor contained within that remote system. One -example may be sending a :class:`PoisonPill` to the system guardian, shutting +example may be sending a `PoisonPill` to the system guardian, shutting that system down. This is not always desired, and it can be disabled with the -following setting:: +following setting: - akka.remote.artery.untrusted-mode = on +``` +akka.remote.artery.untrusted-mode = on +``` This disallows sending of system messages (actor life-cycle commands, -DeathWatch, etc.) and any message extending :class:`PossiblyHarmful` to the +DeathWatch, etc.) and any message extending `PossiblyHarmful` to the system on which this flag is set. Should a client send them nonetheless they are dropped and logged (at DEBUG level in order to reduce the possibilities for -a denial of service attack). :class:`PossiblyHarmful` covers the predefined -messages like :class:`PoisonPill` and :class:`Kill`, but it can also be added +a denial of service attack). `PossiblyHarmful` covers the predefined +messages like `PoisonPill` and `Kill`, but it can also be added as a marker trait to user-defined messages. -.. warning:: +@@@ warning - Untrusted mode does not give full protection against attacks by itself. - It makes it slightly harder to perform malicious or unintended actions but - it should be complemented with :ref:`disabled Java serializer `. - Additional protection can be achieved when running in an untrusted network by - network security (e.g. firewalls). +Untrusted mode does not give full protection against attacks by itself. +It makes it slightly harder to perform malicious or unintended actions but +it should be complemented with [disabled Java serializer](#disable-java-serializer-java-artery). +Additional protection can be achieved when running in an untrusted network by +network security (e.g. firewalls). + +@@@ Messages sent with actor selection are by default discarded in untrusted mode, but permission to receive actor selection messages can be granted to specific actors -defined in configuration:: +defined in configuration: - akka.remote.artery.trusted-selection-paths = ["/user/receptionist", "/user/namingService"] +``` +akka.remote.artery.trusted-selection-paths = ["/user/receptionist", "/user/namingService"] +``` -The actual message must still not be of type :class:`PossiblyHarmful`. +The actual message must still not be of type `PossiblyHarmful`. In summary, the following operations are ignored by a system configured in untrusted mode when incoming via the remoting layer: -* remote deployment (which also means no remote supervision) -* remote DeathWatch -* ``system.stop()``, :class:`PoisonPill`, :class:`Kill` -* sending any message which extends from the :class:`PossiblyHarmful` marker - interface, which includes :class:`Terminated` -* messages sent with actor selection, unless destination defined in ``trusted-selection-paths``. + * remote deployment (which also means no remote supervision) + * remote DeathWatch + * `system.stop()`, `PoisonPill`, `Kill` + * sending any message which extends from the `PossiblyHarmful` marker +interface, which includes `Terminated` + * messages sent with actor selection, unless destination defined in `trusted-selection-paths`. -.. note:: +@@@ note - Enabling the untrusted mode does not remove the capability of the client to - freely choose the target of its message sends, which means that messages not - prohibited by the above rules can be sent to any actor in the remote system. - It is good practice for a client-facing system to only contain a well-defined - set of entry point actors, which then forward requests (possibly after - performing validation) to another actor system containing the actual worker - actors. If messaging between these two server-side systems is done using - local :class:`ActorRef` (they can be exchanged safely between actor systems - within the same JVM), you can restrict the messages on this interface by - marking them :class:`PossiblyHarmful` so that a client cannot forge them. +Enabling the untrusted mode does not remove the capability of the client to +freely choose the target of its message sends, which means that messages not +prohibited by the above rules can be sent to any actor in the remote system. +It is good practice for a client-facing system to only contain a well-defined +set of entry point actors, which then forward requests (possibly after +performing validation) to another actor system containing the actual worker +actors. If messaging between these two server-side systems is done using +local `ActorRef` (they can be exchanged safely between actor systems +within the same JVM), you can restrict the messages on this interface by +marking them `PossiblyHarmful` so that a client cannot forge them. +@@@ -Quarantine ----------- +## Quarantine Akka remoting is using Aeron as underlying message transport. Aeron is using UDP and adds among other things reliable delivery and session semantics, very similar to TCP. This means that -the order of the messages are preserved, which is needed for the :ref:`Actor message ordering guarantees `. +the order of the messages are preserved, which is needed for the @ref:[Actor message ordering guarantees](../general/message-delivery-reliability.md#message-ordering). Under normal circumstances all messages will be delivered but there are cases when messages may not be delivered to the destination: -* during a network partition and the Aeron session is broken, this automatically recovered once the partition is over -* when sending too many messages without flow control and thereby filling up the outbound send queue (``outbound-message-queue-size`` config) -* if serialization or deserialization of a message fails (only that message will be dropped) -* if an unexpected exception occurs in the remoting infrastructure + * during a network partition and the Aeron session is broken, this automatically recovered once the partition is over + * when sending too many messages without flow control and thereby filling up the outbound send queue (`outbound-message-queue-size` config) + * if serialization or deserialization of a message fails (only that message will be dropped) + * if an unexpected exception occurs in the remoting infrastructure -In short, Actor message delivery is “at-most-once” as described in :ref:`message-delivery-reliability` +In short, Actor message delivery is “at-most-once” as described in @ref:[Message Delivery Reliability](../general/message-delivery-reliability.md) Some messages in Akka are called system messages and those cannot be dropped because that would result in an inconsistent state between the systems. Such messages are used for essentially two features; remote death @@ -359,79 +373,79 @@ association with the destination system is irrecoverable failed, and Terminated actors on the remote system. It is placed in a so called quarantined state. Quarantine usually does not happen if remote watch or remote deployment is not used. -Each ``ActorSystem`` instance has an unique identifier (UID), which is important for differentiating between +Each `ActorSystem` instance has an unique identifier (UID), which is important for differentiating between incarnations of a system when it is restarted with the same hostname and port. It is the specific incarnation (UID) that is quarantined. The only way to recover from this state is to restart one of the actor systems. Messages that are sent to and received from a quarantined system will be dropped. However, it is possible to -send messages with ``actorSelection`` to the address of a quarantined system, which is useful to probe if the +send messages with `actorSelection` to the address of a quarantined system, which is useful to probe if the system has been restarted. An association will be quarantined when: -* Cluster node is removed from the cluster membership. -* Remote failure detector triggers, i.e. remote watch is used. This is different when :ref:`Akka Cluster ` - is used. The unreachable observation by the cluster failure detector can go back to reachable if the network - partition heals. A cluster member is not quarantined when the failure detector triggers. -* Overflow of the system message delivery buffer, e.g. because of too many ``watch`` requests at the same time - (``system-message-buffer-size`` config). -* Unexpected exception occurs in the control subchannel of the remoting infrastructure. + * Cluster node is removed from the cluster membership. + * Remote failure detector triggers, i.e. remote watch is used. This is different when @ref:[Akka Cluster](cluster-usage.md) +is used. The unreachable observation by the cluster failure detector can go back to reachable if the network +partition heals. A cluster member is not quarantined when the failure detector triggers. + * Overflow of the system message delivery buffer, e.g. because of too many `watch` requests at the same time +(`system-message-buffer-size` config). + * Unexpected exception occurs in the control subchannel of the remoting infrastructure. -The UID of the ``ActorSystem`` is exchanged in a two-way handshake when the first message is sent to +The UID of the `ActorSystem` is exchanged in a two-way handshake when the first message is sent to a destination. The handshake will be retried until the other system replies and no other messages will pass through until the handshake is completed. If the handshake cannot be established within a timeout -(``handshake-timeout`` config) the association is stopped (freeing up resources). Queued messages will be +(`handshake-timeout` config) the association is stopped (freeing up resources). Queued messages will be dropped if the handshake cannot be established. It will not be quarantined, because the UID is unknown. New handshake attempt will start when next message is sent to the destination. Handshake requests are actually also sent periodically to be able to establish a working connection when the destination system has been restarted. -Watching Remote Actors -^^^^^^^^^^^^^^^^^^^^^^ +### Watching Remote Actors Watching a remote actor is API wise not different than watching a local actor, as described in -:ref:`deathwatch-java`. However, it is important to note, that unlike in the local case, remoting has to handle +@ref:[Lifecycle Monitoring aka DeathWatch](actors.md#deathwatch-java). However, it is important to note, that unlike in the local case, remoting has to handle when a remote actor does not terminate in a graceful way sending a system message to notify the watcher actor about the event, but instead being hosted on a system which stopped abruptly (crashed). These situations are handled by the built-in failure detector. -Failure Detector -^^^^^^^^^^^^^^^^ +### Failure Detector -Under the hood remote death watch uses heartbeat messages and a failure detector to generate ``Terminated`` +Under the hood remote death watch uses heartbeat messages and a failure detector to generate `Terminated` message from network failures and JVM crashes, in addition to graceful termination of watched actor. The heartbeat arrival times is interpreted by an implementation of -`The Phi Accrual Failure Detector `_. +[The Phi Accrual Failure Detector](http://www.jaist.ac.jp/~defago/files/pdf/IS_RR_2004_010.pdf). The suspicion level of failure is given by a value called *phi*. The basic idea of the phi failure detector is to express the value of *phi* on a scale that is dynamically adjusted to reflect current network conditions. -The value of *phi* is calculated as:: +The value of *phi* is calculated as: - phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` +phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` where F is the cumulative distribution function of a normal distribution with mean and standard deviation estimated from historical heartbeat inter-arrival times. -In the :ref:`remote-configuration-artery-java` you can adjust the ``akka.remote.watch-failure-detector.threshold`` +In the [Remote Configuration](#remote-configuration-artery-java) you can adjust the `akka.remote.watch-failure-detector.threshold` to define when a *phi* value is considered to be a failure. -A low ``threshold`` is prone to generate many false positives but ensures -a quick detection in the event of a real crash. Conversely, a high ``threshold`` +A low `threshold` is prone to generate many false positives but ensures +a quick detection in the event of a real crash. Conversely, a high `threshold` generates fewer mistakes but needs more time to detect actual crashes. The -default ``threshold`` is 10 and is appropriate for most situations. However in +default `threshold` is 10 and is appropriate for most situations. However in cloud environments, such as Amazon EC2, the value could be increased to 12 in order to account for network issues that sometimes occur on such platforms. The following chart illustrates how *phi* increase with increasing time since the previous heartbeat. -.. image:: ../images/phi1.png +![phi1.png](../images/phi1.png) Phi is calculated from the mean and standard deviation of historical inter arrival times. The previous chart is an example for standard deviation @@ -439,32 +453,29 @@ of 200 ms. If the heartbeats arrive with less deviation the curve becomes steepe i.e. it is possible to determine failure more quickly. The curve looks like this for a standard deviation of 100 ms. -.. image:: ../images/phi2.png +![phi2.png](../images/phi2.png) To be able to survive sudden abnormalities, such as garbage collection pauses and transient network failures the failure detector is configured with a margin, -``akka.remote.watch-failure-detector.acceptable-heartbeat-pause``. You may want to -adjust the :ref:`remote-configuration-artery-java` of this depending on you environment. -This is how the curve looks like for ``acceptable-heartbeat-pause`` configured to +`akka.remote.watch-failure-detector.acceptable-heartbeat-pause`. You may want to +adjust the [Remote Configuration](#remote-configuration-artery-java) of this depending on you environment. +This is how the curve looks like for `acceptable-heartbeat-pause` configured to 3 seconds. -.. image:: ../images/phi3.png +![phi3.png](../images/phi3.png) -Serialization -------------- +## Serialization -When using remoting for actors you must ensure that the ``props`` and ``messages`` used for +When using remoting for actors you must ensure that the `props` and `messages` used for those actors are serializable. Failing to do so will cause the system to behave in an unintended way. -For more information please see :ref:`serialization-java`. +For more information please see @ref:[Serialization](serialization.md). -.. _remote-bytebuffer-serialization-java: + +### ByteBuffer based serialization -ByteBuffer based serialization -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Artery introduces a new serialization mechanism which allows the ``ByteBufferSerializer`` to directly write into a -shared :class:`java.nio.ByteBuffer` instead of being forced to allocate and return an ``Array[Byte]`` for each serialized +Artery introduces a new serialization mechanism which allows the `ByteBufferSerializer` to directly write into a +shared `java.nio.ByteBuffer` instead of being forced to allocate and return an `Array[Byte]` for each serialized message. For high-throughput messaging this API change can yield significant performance benefits, so we recommend changing your serializers to use this new mechanism. @@ -472,58 +483,58 @@ This new API also plays well with new versions of Google Protocol Buffers and ot the ability to serialize directly into and from ByteBuffers. As the new feature only changes how bytes are read and written, and the rest of the serialization infrastructure -remained the same, we recommend reading the :ref:`serialization-java` documentation first. +remained the same, we recommend reading the @ref:[Serialization](serialization.md) documentation first. -Implementing an :class:`akka.serialization.ByteBufferSerializer` works the same way as any other serializer, +Implementing an `akka.serialization.ByteBufferSerializer` works the same way as any other serializer, -.. includecode:: code/jdocs/actor/ByteBufferSerializerDocTest.java#ByteBufferSerializer-interface +@@snip [ByteBufferSerializerDocTest.java](code/jdocs/actor/ByteBufferSerializerDocTest.java) { #ByteBufferSerializer-interface } Implementing a serializer for Artery is therefore as simple as implementing this interface, and binding the serializer -as usual (which is explained in :ref:`serialization-java`). +as usual (which is explained in @ref:[Serialization](serialization.md)). -Implementations should typically extend ``SerializerWithStringManifest`` and in addition to the ``ByteBuffer`` based -``toBinary`` and ``fromBinary`` methods also implement the array based ``toBinary`` and ``fromBinary`` methods. -The array based methods will be used when ``ByteBuffer`` is not used, e.g. in Akka Persistence. +Implementations should typically extend `SerializerWithStringManifest` and in addition to the `ByteBuffer` based +`toBinary` and `fromBinary` methods also implement the array based `toBinary` and `fromBinary` methods. +The array based methods will be used when `ByteBuffer` is not used, e.g. in Akka Persistence. Note that the array based methods can be implemented by delegation like this: -.. includecode:: code/jdocs/actor/ByteBufferSerializerDocTest.java#bytebufserializer-with-manifest +@@snip [ByteBufferSerializerDocTest.java](code/jdocs/actor/ByteBufferSerializerDocTest.java) { #bytebufserializer-with-manifest } -.. _disable-java-serializer-java-artery: - -Disabling the Java Serializer -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +### Disabling the Java Serializer It is possible to completely disable Java Serialization for the entire Actor system. -Java serialization is known to be slow and `prone to attacks -`_ +Java serialization is known to be slow and [prone to attacks](https://community.hpe.com/t5/Security-Research/The-perils-of-Java-deserialization/ba-p/6838995) of various kinds - it never was designed for high throughput messaging after all. However, it is very convenient to use, thus it remained the default serialization mechanism that Akka used to serialize user messages as well as some of its internal messages in previous versions. -Since the release of Artery, Akka internals do not rely on Java serialization anymore (exceptions to that being ``java.lang.Throwable`` and "remote deployment"). +Since the release of Artery, Akka internals do not rely on Java serialization anymore (exceptions to that being `java.lang.Throwable` and "remote deployment"). -.. note:: - Akka does not use Java Serialization for any of its internal messages. - It is highly encouraged to disable java serialization, so please plan to do so at the earliest possibility you have in your project. +@@@ note - One may think that network bandwidth and latency limit the performance of remote messaging, but serialization is a more typical bottleneck. +Akka does not use Java Serialization for any of its internal messages. +It is highly encouraged to disable java serialization, so please plan to do so at the earliest possibility you have in your project. + +One may think that network bandwidth and latency limit the performance of remote messaging, but serialization is a more typical bottleneck. + +@@@ For user messages, the default serializer, implemented using Java serialization, remains available and enabled. We do however recommend to disable it entirely and utilise a proper serialization library instead in order effectively utilise the improved performance and ability for rolling deployments using Artery. Libraries that we recommend to use include, -but are not limited to, `Kryo`_ by using the `akka-kryo-serialization`_ library or `Google Protocol Buffers`_ if you want +but are not limited to, [Kryo](https://github.com/EsotericSoftware/kryo) by using the [akka-kryo-serialization](https://github.com/romix/akka-kryo-serialization) library or [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) if you want more control over the schema evolution of your messages. In order to completely disable Java Serialization in your Actor system you need to add the following configuration to -your ``application.conf``: +your `application.conf`: -.. code-block:: ruby +```ruby +akka.actor.allow-java-serialization = off +``` - akka.actor.allow-java-serialization = off - -This will completely disable the use of ``akka.serialization.JavaSerialization`` by the -Akka Serialization extension, instead ``DisabledJavaSerializer`` will +This will completely disable the use of `akka.serialization.JavaSerialization` by the +Akka Serialization extension, instead `DisabledJavaSerializer` will be inserted which will fail explicitly if attempts to use java serialization are made. The log messages emitted by such serializer SHOULD be be treated as potential @@ -534,46 +545,37 @@ The attempts are logged with the SECURITY marker. Please note that this option does not stop you from manually invoking java serialization. Please note that this means that you will have to configure different serializers which will able to handle all of your -remote messages. Please refer to the :ref:`serialization-java` documentation as well as :ref:`ByteBuffer based serialization ` to learn how to do this. +remote messages. Please refer to the @ref:[Serialization](serialization.md) documentation as well as [ByteBuffer based serialization](#remote-bytebuffer-serialization-java) to learn how to do this. -.. _Kryo: https://github.com/EsotericSoftware/kryo -.. _akka-kryo-serialization: https://github.com/romix/akka-kryo-serialization -.. _Google Protocol Buffers: https://developers.google.com/protocol-buffers/ +## Routers with Remote Destinations -Routers with Remote Destinations --------------------------------- - -It is absolutely feasible to combine remoting with :ref:`routing-java`. +It is absolutely feasible to combine remoting with @ref:[Routing](routing.md). A pool of remote deployed routees can be configured as: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-pool-artery +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-pool-artery } -This configuration setting will clone the actor defined in the ``Props`` of the ``remotePool`` 10 +This configuration setting will clone the actor defined in the `Props` of the `remotePool` 10 times and deploy it evenly distributed across the two given target nodes. A group of remote actors can be configured as: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-group-artery +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-group-artery } This configuration setting will send messages to the defined remote actor paths. It requires that you create the destination actors on the remote nodes with matching paths. That is not done by the router. -.. _remote-sample-java-artery: + +## Remoting Sample -Remoting Sample ---------------- - -You can download a ready to run `remoting sample <@exampleCodeService@/akka-samples-remote-java>`_ +You can download a ready to run [remoting sample](@exampleCodeService@/akka-samples-remote-java) together with a tutorial for a more hands-on experience. The source code of this sample can be found in the -`Akka Samples Repository <@samples@/akka-sample-remote-java>`_. +[Akka Samples Repository](@samples@/akka-sample-remote-java). -Performance tuning ------------------- +## Performance tuning -Dedicated subchannel for large messages -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Dedicated subchannel for large messages All the communication between user defined remote actors are isolated from the channel of Akka internal messages so a large user message cannot block an urgent system message. While this provides good isolation for Akka services, all @@ -585,31 +587,32 @@ helpful to separate actors that have different QoS requirements: large messages Akka remoting provides a dedicated channel for large messages if configured. Since actor message ordering must not be violated the channel is actually dedicated for *actors* instead of messages, to ensure all of the messages arrive in send order. It is possible to assign actors on given paths to use this dedicated channel by using -path patterns that have to be specified in the actor system's configuration on both the sending and the receiving side:: +path patterns that have to be specified in the actor system's configuration on both the sending and the receiving side: - akka.remote.artery.large-message-destinations = [ - "/user/largeMessageActor", - "/user/largeMessagesGroup/*", - "/user/anotherGroup/*/largeMesssages", - "/user/thirdGroup/**", - ] +``` +akka.remote.artery.large-message-destinations = [ + "/user/largeMessageActor", + "/user/largeMessagesGroup/*", + "/user/anotherGroup/*/largeMesssages", + "/user/thirdGroup/**", +] +``` This means that all messages sent to the following actors will pass through the dedicated, large messages channel: -* ``/user/largeMessageActor`` -* ``/user/largeMessageActorGroup/actor1`` -* ``/user/largeMessageActorGroup/actor2`` -* ``/user/anotherGroup/actor1/largeMessages`` -* ``/user/anotherGroup/actor2/largeMessages`` -* ``/user/thirdGroup/actor3/`` -* ``/user/thirdGroup/actor4/actor5`` + * `/user/largeMessageActor` + * `/user/largeMessageActorGroup/actor1` + * `/user/largeMessageActorGroup/actor2` + * `/user/anotherGroup/actor1/largeMessages` + * `/user/anotherGroup/actor2/largeMessages` + * `/user/thirdGroup/actor3/` + * `/user/thirdGroup/actor4/actor5` Messages destined for actors not matching any of these patterns are sent using the default channel as before. -External, shared Aeron media driver -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### External, shared Aeron media driver -The Aeron transport is running in a so called `media driver `_. +The Aeron transport is running in a so called [media driver](https://github.com/real-logic/Aeron/wiki/Media-Driver-Operation). By default, Akka starts the media driver embedded in the same JVM process as application. This is convenient and simplifies operational concerns by only having one process to start and monitor. @@ -619,96 +622,109 @@ same machine it can therefore be wise to share the media driver by running it as The media driver has also different resource usage characteristics than a normal application and it can therefore be more efficient and stable to run the media driver as a separate process. -Given that Aeron jar files are in the classpath the standalone media driver can be started with:: +Given that Aeron jar files are in the classpath the standalone media driver can be started with: - java io.aeron.driver.MediaDriver +``` +java io.aeron.driver.MediaDriver +``` -The needed classpath:: +The needed classpath: - Agrona-0.5.4.jar:aeron-driver-1.0.1.jar:aeron-client-1.0.1.jar +``` +Agrona-0.5.4.jar:aeron-driver-1.0.1.jar:aeron-client-1.0.1.jar +``` -You find those jar files on `maven central `_, or you can create a +You find those jar files on [maven central](http://search.maven.org/), or you can create a package with your preferred build tool. -You can pass `Aeron properties `_ as -command line `-D` system properties:: +You can pass [Aeron properties](https://github.com/real-logic/Aeron/wiki/Configuration-Options) as +command line *-D* system properties: - -Daeron.dir=/dev/shm/aeron +``` +-Daeron.dir=/dev/shm/aeron +``` -You can also define Aeron properties in a file:: +You can also define Aeron properties in a file: - java io.aeron.driver.MediaDriver config/aeron.properties +``` +java io.aeron.driver.MediaDriver config/aeron.properties +``` -An example of such a properties file:: +An example of such a properties file: - aeron.mtu.length=16384 - aeron.socket.so_sndbuf=2097152 - aeron.socket.so_rcvbuf=2097152 - aeron.rcv.buffer.length=16384 - aeron.rcv.initial.window.length=2097152 - agrona.disable.bounds.checks=true +``` +aeron.mtu.length=16384 +aeron.socket.so_sndbuf=2097152 +aeron.socket.so_rcvbuf=2097152 +aeron.rcv.buffer.length=16384 +aeron.rcv.initial.window.length=2097152 +agrona.disable.bounds.checks=true - aeron.threading.mode=SHARED_NETWORK +aeron.threading.mode=SHARED_NETWORK - # low latency settings - #aeron.threading.mode=DEDICATED - #aeron.sender.idle.strategy=org.agrona.concurrent.BusySpinIdleStrategy - #aeron.receiver.idle.strategy=org.agrona.concurrent.BusySpinIdleStrategy +# low latency settings +#aeron.threading.mode=DEDICATED +#aeron.sender.idle.strategy=org.agrona.concurrent.BusySpinIdleStrategy +#aeron.receiver.idle.strategy=org.agrona.concurrent.BusySpinIdleStrategy - # use same director in akka.remote.artery.advanced.aeron-dir config - # of the Akka application - aeron.dir=/dev/shm/aeron +# use same director in akka.remote.artery.advanced.aeron-dir config +# of the Akka application +aeron.dir=/dev/shm/aeron +``` -Read more about the media driver in the `Aeron documentation `_. +Read more about the media driver in the [Aeron documentation](https://github.com/real-logic/Aeron/wiki/Media-Driver-Operation). To use the external media driver from the Akka application you need to define the following two -configuration properties:: +configuration properties: - akka.remote.artery.advanced { - embedded-media-driver = off - aeron-dir = /dev/shm/aeron - } +``` +akka.remote.artery.advanced { + embedded-media-driver = off + aeron-dir = /dev/shm/aeron +} +``` -The ``aeron-dir`` must match the directory you started the media driver with, i.e. the ``aeron.dir`` property. +The `aeron-dir` must match the directory you started the media driver with, i.e. the `aeron.dir` property. Several Akka applications can then be configured to use the same media driver by pointing to the same directory. Note that if the media driver process is stopped the Akka applications that are using it will also be stopped. -Aeron Tuning -^^^^^^^^^^^^ +### Aeron Tuning -See Aeron documentation about `Performance Testing `_. +See Aeron documentation about [Performance Testing](https://github.com/real-logic/Aeron/wiki/Performance-Testing). -Fine-tuning CPU usage latency tradeoff -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Fine-tuning CPU usage latency tradeoff Artery has been designed for low latency and as a result it can be CPU hungry when the system is mostly idle. This is not always desirable. It is possible to tune the tradeoff between CPU usage and latency with the following configuration: - # Values can be from 1 to 10, where 10 strongly prefers low latency - # and 1 strongly prefers less CPU usage - akka.remote.artery.advanced.idle-cpu-level = 1 +> +# Values can be from 1 to 10, where 10 strongly prefers low latency +# and 1 strongly prefers less CPU usage +akka.remote.artery.advanced.idle-cpu-level = 1 By setting this value to a lower number, it tells Akka to do longer "sleeping" periods on its thread dedicated -for `spin-waiting `_ and hence reducing CPU load when there is no +for [spin-waiting](https://en.wikipedia.org/wiki/Busy_waiting) and hence reducing CPU load when there is no immediate task to execute at the cost of a longer reaction time to an event when it actually happens. It is worth to be noted though that during a continuously high-throughput period this setting makes not much difference as the thread mostly has tasks to execute. This also means that under high throughput (but below maximum capacity) the system might have less latency than at low message rates. -Internal Event Log for Debugging (Flight Recorder) --------------------------------------------------- +## Internal Event Log for Debugging (Flight Recorder) -.. note:: - In this version (@version@) the flight-recorder is disabled by default because there is no automatic - file name and path calculation implemented to make it possible to reuse the same file for every restart of - the same actor system without clashing with files produced by other systems (possibly running on the same machine). - Currently, you have to set the path and file names yourself to avoid creating an unbounded number - of files and enable flight recorder manually by adding `akka.remote.artery.advanced.flight-recorder.enabled=on` to - your configuration file. This a limitation of the current version and will not be necessary in the future. +@@@ note + +In this version (@version@) the flight-recorder is disabled by default because there is no automatic +file name and path calculation implemented to make it possible to reuse the same file for every restart of +the same actor system without clashing with files produced by other systems (possibly running on the same machine). +Currently, you have to set the path and file names yourself to avoid creating an unbounded number +of files and enable flight recorder manually by adding *akka.remote.artery.advanced.flight-recorder.enabled=on* to +your configuration file. This a limitation of the current version and will not be necessary in the future. + +@@@ Emitting event information (logs) from internals is always a tradeoff. The events that are usable for the Akka developers are usually too low level to be of any use for users and usually need to be fine-grained enough @@ -724,54 +740,51 @@ more insight into the nature of the failure. There are various important features of this event log: -* Flight Recorder produces a fixed size file completely encapsulating log rotation. This means that this - file will never grow in size and will not cause any unexpected disk space shortage in production. -* This file is crash resistant, i.e. its contents can be recovered even if the JVM hosting the :class:`ActorSystem` - crashes unexpectedly. -* Very low overhead, specialized, binary logging that has no significant overhead and can be safely left enabled - for production systems. + * Flight Recorder produces a fixed size file completely encapsulating log rotation. This means that this +file will never grow in size and will not cause any unexpected disk space shortage in production. + * This file is crash resistant, i.e. its contents can be recovered even if the JVM hosting the `ActorSystem` +crashes unexpectedly. + * Very low overhead, specialized, binary logging that has no significant overhead and can be safely left enabled +for production systems. -The location of the file can be controlled via the `akka.remote.artery.advanced.flight-recoder.destination` setting (see -:ref:`config-akka-remote-artery` for details). By default, a file with the `.afr` extension is produced in the temporary +The location of the file can be controlled via the *akka.remote.artery.advanced.flight-recoder.destination* setting (see +@ref:[akka-remote (artery)](../general/configuration.md#config-akka-remote-artery) for details). By default, a file with the *.afr* extension is produced in the temporary directory of the operating system. In cases where the flight recorder casuses issues, it can be disabled by adding the -setting `akka.remote.artery.advanced.flight-recorder.enabled=off`, although this is not recommended. +setting *akka.remote.artery.advanced.flight-recorder.enabled=off*, although this is not recommended. -.. _remote-configuration-artery-java: - -Remote Configuration --------------------- + +## Remote Configuration There are lots of configuration properties that are related to remoting in Akka. We refer to the -:ref:`reference configuration ` for more information. +@ref:[reference configuration](../general/configuration.md#config-akka-remote-artery) for more information. -.. note:: +@@@ note - Setting properties like the listening IP and port number programmatically is - best done by using something like the following: +Setting properties like the listening IP and port number programmatically is +best done by using something like the following: - .. includecode:: ../java/code/jdocs/remoting/RemoteDeploymentDocTest.java#programmatic-artery +@@snip [RemoteDeploymentDocTest.java](../java/code/jdocs/remoting/RemoteDeploymentDocTest.java) { #programmatic-artery } +@@@ -.. _remote-configuration-nat-artery-java: - -Akka behind NAT or in a Docker container -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +### Akka behind NAT or in a Docker container In setups involving Network Address Translation (NAT), Load Balancers or Docker containers the hostname and port pair that Akka binds to will be different than the "logical" host name and port pair that is used to connect to the system from the outside. This requires special configuration that sets both the logical and the bind pairs for remoting. -.. code-block:: ruby +```ruby +akka { + remote { + artery { + canonical.hostname = my.domain.com # external (logical) hostname + canonical.port = 8000 # external (logical) port - akka { - remote { - artery { - canonical.hostname = my.domain.com # external (logical) hostname - canonical.port = 8000 # external (logical) port - - bind.hostname = local.address # internal (bind) hostname - bind.port = 25520 # internal (bind) port - } - } - } + bind.hostname = local.address # internal (bind) hostname + bind.port = 25520 # internal (bind) port + } + } +} +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/remoting.md b/akka-docs/src/main/paradox/java/remoting.md index 3a6ca503f2..5dd5e90a49 100644 --- a/akka-docs/src/main/paradox/java/remoting.md +++ b/akka-docs/src/main/paradox/java/remoting.md @@ -1,177 +1,193 @@ -.. _remoting-java: +# Remoting -Remoting -######## +For an introduction of remoting capabilities of Akka please see [Location Transparency](). -For an introduction of remoting capabilities of Akka please see :ref:`remoting`. +@@@ note -.. note:: +As explained in that chapter Akka remoting is designed for communication in a +peer-to-peer fashion and it has limitations for client-server setups. In +particular Akka Remoting does not work transparently with Network Address Translation, +Load Balancers, or in Docker containers. For symmetric communication in these situations +network and/or Akka configuration will have to be changed as described in +@ref:[Peer-to-Peer vs. Client-Server](../general/remoting.md#symmetric-communication). - As explained in that chapter Akka remoting is designed for communication in a - peer-to-peer fashion and it has limitations for client-server setups. In - particular Akka Remoting does not work transparently with Network Address Translation, - Load Balancers, or in Docker containers. For symmetric communication in these situations - network and/or Akka configuration will have to be changed as described in - :ref:`symmetric-communication`. +@@@ -Preparing your ActorSystem for Remoting -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## Preparing your ActorSystem for Remoting -The Akka remoting is a separate jar file. Make sure that you have the following dependency in your project:: +The Akka remoting is a separate jar file. Make sure that you have the following dependency in your project: - - com.typesafe.akka - akka-remote_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-remote_@binVersion@ + @version@ + +``` To enable remote capabilities in your Akka project you should, at a minimum, add the following changes -to your ``application.conf`` file:: +to your `application.conf` file: - akka { - actor { - provider = remote - } - remote { - enabled-transports = ["akka.remote.netty.tcp"] - netty.tcp { - hostname = "127.0.0.1" - port = 2552 - } - } +``` +akka { + actor { + provider = remote } + remote { + enabled-transports = ["akka.remote.netty.tcp"] + netty.tcp { + hostname = "127.0.0.1" + port = 2552 + } + } +} +``` As you can see in the example above there are four things you need to add to get started: -* Change provider from ``local`` to ``remote`` -* Add host name - the machine you want to run the actor system on; this host - name is exactly what is passed to remote systems in order to identify this - system and consequently used for connecting back to this system if need be, - hence set it to a reachable IP address or resolvable name in case you want to - communicate across the network. -* Add port number - the port the actor system should listen on, set to 0 to have it chosen automatically + * Change provider from `local` to `remote` + * Add host name - the machine you want to run the actor system on; this host +name is exactly what is passed to remote systems in order to identify this +system and consequently used for connecting back to this system if need be, +hence set it to a reachable IP address or resolvable name in case you want to +communicate across the network. + * Add port number - the port the actor system should listen on, set to 0 to have it chosen automatically -.. note:: - The port number needs to be unique for each actor system on the same machine even if the actor - systems have different names. This is because each actor system has its own network subsystem - listening for connections and handling messages as not to interfere with other actor systems. +@@@ note + +The port number needs to be unique for each actor system on the same machine even if the actor +systems have different names. This is because each actor system has its own network subsystem +listening for connections and handling messages as not to interfere with other actor systems. + +@@@ The example above only illustrates the bare minimum of properties you have to add to enable remoting. -All settings are described in :ref:`remote-configuration-java`. +All settings are described in [Remote Configuration](#remote-configuration-java). -Looking up Remote Actors -^^^^^^^^^^^^^^^^^^^^^^^^ +## Looking up Remote Actors -``actorSelection(path)`` will obtain an ``ActorSelection`` to an Actor on a remote node:: +`actorSelection(path)` will obtain an `ActorSelection` to an Actor on a remote node: - ActorSelection selection = - context.actorSelection("akka.tcp://app@10.0.0.1:2552/user/serviceA/worker"); +``` +ActorSelection selection = + context.actorSelection("akka.tcp://app@10.0.0.1:2552/user/serviceA/worker"); +``` -As you can see from the example above the following pattern is used to find an actor on a remote node:: +As you can see from the example above the following pattern is used to find an actor on a remote node: - akka.://@:/ +``` +akka.://@:/ +``` -Once you obtained a selection to the actor you can interact with it they same way you would with a local actor, e.g.:: +Once you obtained a selection to the actor you can interact with it they same way you would with a local actor, e.g.: - selection.tell("Pretty awesome feature", getSelf()); +``` +selection.tell("Pretty awesome feature", getSelf()); +``` -To acquire an :class:`ActorRef` for an :class:`ActorSelection` you need to -send a message to the selection and use the ``sender`` reference of the reply from -the actor. There is a built-in ``Identify`` message that all Actors will understand -and automatically reply to with a ``ActorIdentity`` message containing the -:class:`ActorRef`. This can also be done with the ``resolveOneCS`` method of -the :class:`ActorSelection`, which returns a ``CompletionStage`` of the matching -:class:`ActorRef`. +To acquire an `ActorRef` for an `ActorSelection` you need to +send a message to the selection and use the `sender` reference of the reply from +the actor. There is a built-in `Identify` message that all Actors will understand +and automatically reply to with a `ActorIdentity` message containing the +`ActorRef`. This can also be done with the `resolveOneCS` method of +the `ActorSelection`, which returns a `CompletionStage` of the matching +`ActorRef`. -.. note:: +@@@ note - For more details on how actor addresses and paths are formed and used, please refer to :ref:`addressing`. +For more details on how actor addresses and paths are formed and used, please refer to @ref:[Actor References, Paths and Addresses](../general/addressing.md). -.. note:: +@@@ - Message sends to actors that are actually in the sending actor system do not - get delivered via the remote actor ref provider. They're delivered directly, - by the local actor ref provider. +@@@ note - Aside from providing better performance, this also means that if the hostname - you configure remoting to listen as cannot actually be resolved from within - the very same actor system, such messages will (perhaps counterintuitively) - be delivered just fine. +Message sends to actors that are actually in the sending actor system do not +get delivered via the remote actor ref provider. They're delivered directly, +by the local actor ref provider. -Creating Actors Remotely -^^^^^^^^^^^^^^^^^^^^^^^^ +Aside from providing better performance, this also means that if the hostname +you configure remoting to listen as cannot actually be resolved from within +the very same actor system, such messages will (perhaps counterintuitively) +be delivered just fine. + +@@@ + +## Creating Actors Remotely If you want to use the creation functionality in Akka remoting you have to further amend the -``application.conf`` file in the following way (only showing deployment section):: +`application.conf` file in the following way (only showing deployment section): - akka { - actor { - deployment { - /sampleActor { - remote = "akka.tcp://sampleActorSystem@127.0.0.1:2553" - } +``` +akka { + actor { + deployment { + /sampleActor { + remote = "akka.tcp://sampleActorSystem@127.0.0.1:2553" } } } +} +``` -The configuration above instructs Akka to react when an actor with path ``/sampleActor`` is created, i.e. -using ``system.actorOf(new Props(...), "sampleActor")``. This specific actor will not be directly instantiated, +The configuration above instructs Akka to react when an actor with path `/sampleActor` is created, i.e. +using `system.actorOf(new Props(...), "sampleActor")`. This specific actor will not be directly instantiated, but instead the remote daemon of the remote system will be asked to create the actor, -which in this sample corresponds to ``sampleActorSystem@127.0.0.1:2553``. +which in this sample corresponds to `sampleActorSystem@127.0.0.1:2553`. Once you have configured the properties above you would do the following in code: -.. includecode:: code/jdocs/remoting/RemoteDeploymentDocTest.java#sample-actor +@@snip [RemoteDeploymentDocTest.java](code/jdocs/remoting/RemoteDeploymentDocTest.java) { #sample-actor } -The actor class ``SampleActor`` has to be available to the runtimes using it, i.e. the classloader of the +The actor class `SampleActor` has to be available to the runtimes using it, i.e. the classloader of the actor systems has to have a JAR containing the class. -.. note:: +@@@ note - In order to ensure serializability of ``Props`` when passing constructor - arguments to the actor being created, do not make the factory a non-static - inner class: this will inherently capture a reference to its enclosing - object, which in most cases is not serializable. It is best to make a static - inner class which implements :class:`Creator`. +In order to ensure serializability of `Props` when passing constructor +arguments to the actor being created, do not make the factory a non-static +inner class: this will inherently capture a reference to its enclosing +object, which in most cases is not serializable. It is best to make a static +inner class which implements `Creator`. - Serializability of all Props can be tested by setting the configuration item - ``akka.actor.serialize-creators=on``. Only Props whose ``deploy`` has - ``LocalScope`` are exempt from this check. +Serializability of all Props can be tested by setting the configuration item +`akka.actor.serialize-creators=on`. Only Props whose `deploy` has +`LocalScope` are exempt from this check. -.. note:: +@@@ - You can use asterisks as wildcard matches for the actor path sections, so you could specify: - ``/*/sampleActor`` and that would match all ``sampleActor`` on that level in the hierarchy. - You can also use wildcard in the last position to match all actors at a certain level: - ``/someParent/*``. Non-wildcard matches always have higher priority to match than wildcards, so: - ``/foo/bar`` is considered **more specific** than ``/foo/*`` and only the highest priority match is used. - Please note that it **cannot** be used to partially match section, like this: ``/foo*/bar``, ``/f*o/bar`` etc. +@@@ note -Programmatic Remote Deployment ------------------------------- +You can use asterisks as wildcard matches for the actor path sections, so you could specify: +`/*/sampleActor` and that would match all `sampleActor` on that level in the hierarchy. +You can also use wildcard in the last position to match all actors at a certain level: +`/someParent/*`. Non-wildcard matches always have higher priority to match than wildcards, so: +`/foo/bar` is considered **more specific** than `/foo/*` and only the highest priority match is used. +Please note that it **cannot** be used to partially match section, like this: `/foo*/bar`, `/f*o/bar` etc. + +@@@ + +### Programmatic Remote Deployment To allow dynamically deployed systems, it is also possible to include -deployment configuration in the :class:`Props` which are used to create an +deployment configuration in the `Props` which are used to create an actor: this information is the equivalent of a deployment section from the configuration file, and if both are given, the external configuration takes precedence. With these imports: -.. includecode:: code/jdocs/remoting/RemoteDeploymentDocTest.java#import +@@snip [RemoteDeploymentDocTest.java](code/jdocs/remoting/RemoteDeploymentDocTest.java) { #import } and a remote address like this: -.. includecode:: code/jdocs/remoting/RemoteDeploymentDocTest.java#make-address +@@snip [RemoteDeploymentDocTest.java](code/jdocs/remoting/RemoteDeploymentDocTest.java) { #make-address } you can advise the system to create a child on that remote node like so: -.. includecode:: code/jdocs/remoting/RemoteDeploymentDocTest.java#deploy +@@snip [RemoteDeploymentDocTest.java](code/jdocs/remoting/RemoteDeploymentDocTest.java) { #deploy } -.. _remote-deployment-whitelist-java: - -Remote deployment whitelist ---------------------------- + +### Remote deployment whitelist As remote deployment can potentially be abused by both users and even attackers a whitelist feature is available to guard the ActorSystem from deploying unexpected actors. Please note that remote deployment @@ -179,79 +195,76 @@ is *not* remote code loading, the Actors class to be deployed onto a remote syst remote system. This still however may pose a security risk, and one may want to restrict remote deployment to only a specific set of known actors by enabling the whitelist feature. -To enable remote deployment whitelisting set the ``akka.remote.deployment.enable-whitelist`` value to ``on``. +To enable remote deployment whitelisting set the `akka.remote.deployment.enable-whitelist` value to `on`. The list of allowed classes has to be configured on the "remote" system, in other words on the system onto which others will be attempting to remote deploy Actors. That system, locally, knows best which Actors it should or should not allow others to remote deploy onto it. The full settings section may for example look like this: -.. includecode:: ../../../akka-remote/src/test/scala/akka/remote/RemoteDeploymentWhitelistSpec.scala#whitelist-config +@@snip [RemoteDeploymentWhitelistSpec.scala]../../../../../akka-remote/src/test/scala/akka/remote/RemoteDeploymentWhitelistSpec.scala) { #whitelist-config } Actor classes not included in the whitelist will not be allowed to be remote deployed onto this system. -Lifecycle and Failure Recovery Model -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## Lifecycle and Failure Recovery Model -.. image:: ../images/association_lifecycle.png - :align: center - :width: 620 +![association_lifecycle.png](../images/association_lifecycle.png) Each link with a remote system can be in one of the four states as illustrated above. Before any communication -happens with a remote system at a given ``Address`` the state of the association is ``Idle``. The first time a message +happens with a remote system at a given `Address` the state of the association is `Idle`. The first time a message is attempted to be sent to the remote system or an inbound connection is accepted the state of the link transitions to -``Active`` denoting that the two systems has messages to send or receive and no failures were encountered so far. -When a communication failure happens and the connection is lost between the two systems the link becomes ``Gated``. +`Active` denoting that the two systems has messages to send or receive and no failures were encountered so far. +When a communication failure happens and the connection is lost between the two systems the link becomes `Gated`. In this state the system will not attempt to connect to the remote host and all outbound messages will be dropped. The time -while the link is in the ``Gated`` state is controlled by the setting ``akka.remote.retry-gate-closed-for``: -after this time elapses the link state transitions to ``Idle`` again. ``Gate`` is one-sided in the -sense that whenever a successful *inbound* connection is accepted from a remote system during ``Gate`` it automatically -transitions to ``Active`` and communication resumes immediately. +while the link is in the `Gated` state is controlled by the setting `akka.remote.retry-gate-closed-for`: +after this time elapses the link state transitions to `Idle` again. `Gate` is one-sided in the +sense that whenever a successful *inbound* connection is accepted from a remote system during `Gate` it automatically +transitions to `Active` and communication resumes immediately. In the face of communication failures that are unrecoverable because the state of the participating systems are inconsistent, -the remote system becomes ``Quarantined``. Unlike ``Gate``, quarantining is permanent and lasts until one of the systems -is restarted. After a restart communication can be resumed again and the link can become ``Active`` again. +the remote system becomes `Quarantined`. Unlike `Gate`, quarantining is permanent and lasts until one of the systems +is restarted. After a restart communication can be resumed again and the link can become `Active` again. -Watching Remote Actors -^^^^^^^^^^^^^^^^^^^^^^ +## Watching Remote Actors Watching a remote actor is not different than watching a local actor, as described in -:ref:`deathwatch-java`. +@ref:[Lifecycle Monitoring aka DeathWatch](actors.md#deathwatch-java). -Failure Detector ----------------- +### Failure Detector -Under the hood remote death watch uses heartbeat messages and a failure detector to generate ``Terminated`` +Under the hood remote death watch uses heartbeat messages and a failure detector to generate `Terminated` message from network failures and JVM crashes, in addition to graceful termination of watched actor. The heartbeat arrival times is interpreted by an implementation of -`The Phi Accrual Failure Detector `_. +[The Phi Accrual Failure Detector](http://www.jaist.ac.jp/~defago/files/pdf/IS_RR_2004_010.pdf). The suspicion level of failure is given by a value called *phi*. The basic idea of the phi failure detector is to express the value of *phi* on a scale that is dynamically adjusted to reflect current network conditions. -The value of *phi* is calculated as:: +The value of *phi* is calculated as: - phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` +phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` where F is the cumulative distribution function of a normal distribution with mean and standard deviation estimated from historical heartbeat inter-arrival times. -In the :ref:`remote-configuration-java` you can adjust the ``akka.remote.watch-failure-detector.threshold`` +In the [Remote Configuration](#remote-configuration-java) you can adjust the `akka.remote.watch-failure-detector.threshold` to define when a *phi* value is considered to be a failure. -A low ``threshold`` is prone to generate many false positives but ensures -a quick detection in the event of a real crash. Conversely, a high ``threshold`` +A low `threshold` is prone to generate many false positives but ensures +a quick detection in the event of a real crash. Conversely, a high `threshold` generates fewer mistakes but needs more time to detect actual crashes. The -default ``threshold`` is 10 and is appropriate for most situations. However in +default `threshold` is 10 and is appropriate for most situations. However in cloud environments, such as Amazon EC2, the value could be increased to 12 in order to account for network issues that sometimes occur on such platforms. The following chart illustrates how *phi* increase with increasing time since the previous heartbeat. -.. image:: ../images/phi1.png +![phi1.png](../images/phi1.png) Phi is calculated from the mean and standard deviation of historical inter arrival times. The previous chart is an example for standard deviation @@ -259,66 +272,68 @@ of 200 ms. If the heartbeats arrive with less deviation the curve becomes steepe i.e. it is possible to determine failure more quickly. The curve looks like this for a standard deviation of 100 ms. -.. image:: ../images/phi2.png +![phi2.png](../images/phi2.png) To be able to survive sudden abnormalities, such as garbage collection pauses and transient network failures the failure detector is configured with a margin, -``akka.remote.watch-failure-detector.acceptable-heartbeat-pause``. You may want to -adjust the :ref:`remote-configuration-java` of this depending on you environment. -This is how the curve looks like for ``acceptable-heartbeat-pause`` configured to +`akka.remote.watch-failure-detector.acceptable-heartbeat-pause`. You may want to +adjust the [Remote Configuration](#remote-configuration-java) of this depending on you environment. +This is how the curve looks like for `acceptable-heartbeat-pause` configured to 3 seconds. -.. image:: ../images/phi3.png +![phi3.png](../images/phi3.png) -Serialization -^^^^^^^^^^^^^ +## Serialization -When using remoting for actors you must ensure that the ``props`` and ``messages`` used for +When using remoting for actors you must ensure that the `props` and `messages` used for those actors are serializable. Failing to do so will cause the system to behave in an unintended way. -For more information please see :ref:`serialization-java`. +For more information please see @ref:[Serialization](serialization.md). -.. _disable-java-serializer-java: + +### Disabling the Java Serializer -Disabling the Java Serializer ------------------------------ - -Java serialization is known to be slow and `prone to attacks -`_ +Java serialization is known to be slow and [prone to attacks](https://community.hpe.com/t5/Security-Research/The-perils-of-Java-deserialization/ba-p/6838995) of various kinds - it never was designed for high throughput messaging after all. However, it is very convenient to use, thus it remained the default serialization mechanism that Akka used to serialize user messages as well as some of its internal messages in previous versions. -Since the release of Artery, Akka internals do not rely on Java serialization anymore (one exception being ``java.lang.Throwable``). +Since the release of Artery, Akka internals do not rely on Java serialization anymore (one exception being `java.lang.Throwable`). -.. warning:: - Please note Akka 2.5 by default does not use any Java Serialization for its own internal messages, unlike 2.4 where - by default it sill did for a few of the messages. If you want an 2.4.x system to communicate with a 2.5.x series, for - example during a rolling deployment you should first enable ``additional-serialization-bindings`` on the old systems. - You must do so on all nodes participating in a cluster, otherwise the mis-aligned serialization - configurations will cause deserialization errors on the receiving nodes. These additional serialization bindings are - enabled by default in Akka 2.5.x. +@@@ warning -.. note:: - When using the new remoting implementation (codename Artery), Akka does not use Java Serialization for any of its internal messages. - It is highly encouraged to disable java serialization, so please plan to do so at the earliest possibility you have in your project. +Please note Akka 2.5 by default does not use any Java Serialization for its own internal messages, unlike 2.4 where +by default it sill did for a few of the messages. If you want an 2.4.x system to communicate with a 2.5.x series, for +example during a rolling deployment you should first enable `additional-serialization-bindings` on the old systems. +You must do so on all nodes participating in a cluster, otherwise the mis-aligned serialization +configurations will cause deserialization errors on the receiving nodes. These additional serialization bindings are +enabled by default in Akka 2.5.x. - One may think that network bandwidth and latency limit the performance of remote messaging, but serialization is a more typical bottleneck. +@@@ + +@@@ note + +When using the new remoting implementation (codename Artery), Akka does not use Java Serialization for any of its internal messages. +It is highly encouraged to disable java serialization, so please plan to do so at the earliest possibility you have in your project. + +One may think that network bandwidth and latency limit the performance of remote messaging, but serialization is a more typical bottleneck. + +@@@ For user messages, the default serializer, implemented using Java serialization, remains available and enabled. We do however recommend to disable it entirely and utilise a proper serialization library instead in order effectively utilise the improved performance and ability for rolling deployments using Artery. Libraries that we recommend to use include, -but are not limited to, `Kryo`_ by using the `akka-kryo-serialization`_ library or `Google Protocol Buffers`_ if you want +but are not limited to, [Kryo](https://github.com/EsotericSoftware/kryo) by using the [akka-kryo-serialization](https://github.com/romix/akka-kryo-serialization) library or [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) if you want more control over the schema evolution of your messages. In order to completely disable Java Serialization in your Actor system you need to add the following configuration to -your ``application.conf``: +your `application.conf`: -.. code-block:: ruby +```ruby +akka.actor.allow-java-serialization = off +``` - akka.actor.allow-java-serialization = off - -This will completely disable the use of ``akka.serialization.JavaSerialization`` by the -Akka Serialization extension, instead ``DisabledJavaSerializer`` will +This will completely disable the use of `akka.serialization.JavaSerialization` by the +Akka Serialization extension, instead `DisabledJavaSerializer` will be inserted which will fail explicitly if attempts to use java serialization are made. The log messages emitted by such serializer SHOULD be be treated as potential @@ -329,164 +344,162 @@ The attempts are logged with the SECURITY marker. Please note that this option does not stop you from manually invoking java serialization. Please note that this means that you will have to configure different serializers which will able to handle all of your -remote messages. Please refer to the :ref:`serialization-java` documentation as well as :ref:`ByteBuffer based serialization ` to learn how to do this. +remote messages. Please refer to the @ref:[Serialization](serialization.md) documentation as well as @ref:[ByteBuffer based serialization](remoting-artery.md#remote-bytebuffer-serialization-java) to learn how to do this. -.. _Kryo: https://github.com/EsotericSoftware/kryo -.. _akka-kryo-serialization: https://github.com/romix/akka-kryo-serialization -.. _Google Protocol Buffers: https://developers.google.com/protocol-buffers/ +## Routers with Remote Destinations -Routers with Remote Destinations -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It is absolutely feasible to combine remoting with :ref:`routing-java`. +It is absolutely feasible to combine remoting with @ref:[Routing](routing.md). A pool of remote deployed routees can be configured as: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-pool } -This configuration setting will clone the actor defined in the ``Props`` of the ``remotePool`` 10 +This configuration setting will clone the actor defined in the `Props` of the `remotePool` 10 times and deploy it evenly distributed across the two given target nodes. A group of remote actors can be configured as: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-group } This configuration setting will send messages to the defined remote actor paths. It requires that you create the destination actors on the remote nodes with matching paths. That is not done by the router. -.. _remote-sample-java: + +## Remoting Sample -Remoting Sample -^^^^^^^^^^^^^^^ - -You can download a ready to run `remoting sample <@exampleCodeService@/akka-samples-remote-java>`_ +You can download a ready to run [remoting sample](@exampleCodeService@/akka-samples-remote-java) together with a tutorial for a more hands-on experience. The source code of this sample can be found in the -`Akka Samples Repository <@samples@/akka-sample-remote-java>`_. +[Akka Samples Repository](@samples@/akka-sample-remote-java). -Remote Events -------------- +### Remote Events It is possible to listen to events that occur in Akka Remote, and to subscribe/unsubscribe to these events -you simply register as listener to the below described types in on the ``ActorSystem.eventStream``. +you simply register as listener to the below described types in on the `ActorSystem.eventStream`. -.. note:: +@@@ note - To subscribe to any remote event, subscribe to - :meth:`RemotingLifecycleEvent`. To subscribe to events related only to the - lifecycle of associations, subscribe to - :meth:`akka.remote.AssociationEvent`. +To subscribe to any remote event, subscribe to +`RemotingLifecycleEvent`. To subscribe to events related only to the +lifecycle of associations, subscribe to +`akka.remote.AssociationEvent`. -.. note:: +@@@ - The use of term "Association" instead of "Connection" reflects that the - remoting subsystem may use connectionless transports, but an association - similar to transport layer connections is maintained between endpoints by - the Akka protocol. +@@@ note + +The use of term "Association" instead of "Connection" reflects that the +remoting subsystem may use connectionless transports, but an association +similar to transport layer connections is maintained between endpoints by +the Akka protocol. + +@@@ By default an event listener is registered which logs all of the events described below. This default was chosen to help setting up a system, but it is quite common to switch this logging off once that phase of the project is finished. -.. note:: +@@@ note - In order to switch off the logging, set - ``akka.remote.log-remote-lifecycle-events = off`` in your - ``application.conf``. +In order to switch off the logging, set +`akka.remote.log-remote-lifecycle-events = off` in your +`application.conf`. -To be notified when an association is over ("disconnected") listen to ``DisassociatedEvent`` which +@@@ + +To be notified when an association is over ("disconnected") listen to `DisassociatedEvent` which holds the direction of the association (inbound or outbound) and the addresses of the involved parties. -To be notified when an association is successfully established ("connected") listen to ``AssociatedEvent`` which +To be notified when an association is successfully established ("connected") listen to `AssociatedEvent` which holds the direction of the association (inbound or outbound) and the addresses of the involved parties. -To intercept errors directly related to associations, listen to ``AssociationErrorEvent`` which +To intercept errors directly related to associations, listen to `AssociationErrorEvent` which holds the direction of the association (inbound or outbound), the addresses of the involved parties and the -``Throwable`` cause. +`Throwable` cause. -To be notified when the remoting subsystem is ready to accept associations, listen to ``RemotingListenEvent`` which +To be notified when the remoting subsystem is ready to accept associations, listen to `RemotingListenEvent` which contains the addresses the remoting listens on. -To be notified when the remoting subsystem has been shut down, listen to ``RemotingShutdownEvent``. +To be notified when the remoting subsystem has been shut down, listen to `RemotingShutdownEvent`. -To be notified when the current system is quarantined by the remote system, listen to ``ThisActorSystemQuarantinedEvent``, +To be notified when the current system is quarantined by the remote system, listen to `ThisActorSystemQuarantinedEvent`, which includes the addresses of local and remote ActorSystems. -To intercept generic remoting related errors, listen to ``RemotingErrorEvent`` which holds the ``Throwable`` cause. +To intercept generic remoting related errors, listen to `RemotingErrorEvent` which holds the `Throwable` cause. -.. _remote-security-java: + +## Remote Security -Remote Security -^^^^^^^^^^^^^^^ - -An ``ActorSystem`` should not be exposed via Akka Remote over plain TCP to an untrusted network (e.g. internet). +An `ActorSystem` should not be exposed via Akka Remote over plain TCP to an untrusted network (e.g. internet). It should be protected by network security, such as a firewall. If that is not considered as enough protection -:ref:`TLS with mutual authentication ` should be enabled. +[TLS with mutual authentication](#remote-tls-java) should be enabled. Best practice is that Akka remoting nodes should only be accessible from the adjacent network. Note that if TLS is enabled with mutual authentication there is still a risk that an attacker can gain access to a valid certificate by compromising any node with certificates issued by the same internal PKI tree. -It is also security best-practice to :ref:`disable the Java serializer ` because of -its multiple `known attack surfaces `_. +It is also security best-practice to [disable the Java serializer](#disable-java-serializer-java) because of +its multiple [known attack surfaces](https://community.hpe.com/t5/Security-Research/The-perils-of-Java-deserialization/ba-p/6838995). -.. _remote-tls-java: + +### Configuring SSL/TLS for Akka Remoting -Configuring SSL/TLS for Akka Remoting -------------------------------------- +SSL can be used as the remote transport by adding `akka.remote.netty.ssl` to the `enabled-transport` configuration section. +An example of setting up the default Netty based SSL driver as default: -SSL can be used as the remote transport by adding ``akka.remote.netty.ssl`` to the ``enabled-transport`` configuration section. -An example of setting up the default Netty based SSL driver as default:: - - akka { - remote { - enabled-transports = [akka.remote.netty.ssl] - } +``` +akka { + remote { + enabled-transports = [akka.remote.netty.ssl] } +} +``` -Next the actual SSL/TLS parameters have to be configured:: +Next the actual SSL/TLS parameters have to be configured: - akka { - remote { - netty.ssl { - hostname = "127.0.0.1" - port = "3553" +``` +akka { + remote { + netty.ssl { + hostname = "127.0.0.1" + port = "3553" - security { - key-store = "/example/path/to/mykeystore.jks" - trust-store = "/example/path/to/mytruststore.jks" + security { + key-store = "/example/path/to/mykeystore.jks" + trust-store = "/example/path/to/mytruststore.jks" - key-store-password = "changeme" - key-password = "changeme" - trust-store-password = "changeme" + key-store-password = "changeme" + key-password = "changeme" + trust-store-password = "changeme" - protocol = "TLSv1.2" + protocol = "TLSv1.2" - enabled-algorithms = [TLS_DHE_RSA_WITH_AES_128_GCM_SHA256] + enabled-algorithms = [TLS_DHE_RSA_WITH_AES_128_GCM_SHA256] - random-number-generator = "AES128CounterSecureRNG" - } + random-number-generator = "AES128CounterSecureRNG" } } } +} +``` -According to `RFC 7525 `_ the recommended algorithms to use with TLS 1.2 (as of writing this document) are: +According to [RFC 7525](https://tools.ietf.org/html/rfc7525) the recommended algorithms to use with TLS 1.2 (as of writing this document) are: -- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 -- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 -- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 -- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 + * TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 + * TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 + * TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 + * TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 Creating and working with keystores and certificates is well documented in the -`Generating X.509 Certificates `_ +[Generating X.509 Certificates](http://typesafehub.github.io/ssl-config/CertificateGeneration.html#using-keytool) section of Lightbend's SSL-Config library. -Since an Akka remoting is inherently :ref:`peer-to-peer ` both the key-store as well as trust-store +Since an Akka remoting is inherently @ref:[peer-to-peer](../general/remoting.md#symmetric-communication) both the key-store as well as trust-store need to be configured on each remoting node participating in the cluster. -The official `Java Secure Socket Extension documentation `_ -as well as the `Oracle documentation on creating KeyStore and TrustStores `_ +The official [Java Secure Socket Extension documentation](http://docs.oracle.com/javase/7/jdocs/technotes/guides/security/jsse/JSSERefGuide.html) +as well as the [Oracle documentation on creating KeyStore and TrustStores](https://docs.oracle.com/cd/E19509-01/820-3503/6nf1il6er/index.html) are both great resources to research when setting up security on the JVM. Please consult those resources when troubleshooting and configuring SSL. @@ -500,115 +513,122 @@ the other (the "server"). Note that if TLS is enabled with mutual authentication there is still a risk that an attacker can gain access to a valid certificate by compromising any node with certificates issued by the same internal PKI tree. -See also a description of the settings in the :ref:`remote-configuration-scala` section. +See also a description of the settings in the @ref:[Remote Configuration](../scala/remoting.md#remote-configuration-scala) section. -.. note:: +@@@ note - When using SHA1PRNG on Linux it's recommended specify ``-Djava.security.egd=file:/dev/urandom`` as argument - to the JVM to prevent blocking. It is NOT as secure because it reuses the seed. +When using SHA1PRNG on Linux it's recommended specify `-Djava.security.egd=file:/dev/urandom` as argument +to the JVM to prevent blocking. It is NOT as secure because it reuses the seed. -Untrusted Mode --------------- +@@@ + +### Untrusted Mode As soon as an actor system can connect to another remotely, it may in principle send any possible message to any actor contained within that remote system. One -example may be sending a :class:`PoisonPill` to the system guardian, shutting +example may be sending a `PoisonPill` to the system guardian, shutting that system down. This is not always desired, and it can be disabled with the -following setting:: +following setting: - akka.remote.untrusted-mode = on +``` +akka.remote.untrusted-mode = on +``` This disallows sending of system messages (actor life-cycle commands, -DeathWatch, etc.) and any message extending :class:`PossiblyHarmful` to the +DeathWatch, etc.) and any message extending `PossiblyHarmful` to the system on which this flag is set. Should a client send them nonetheless they are dropped and logged (at DEBUG level in order to reduce the possibilities for -a denial of service attack). :class:`PossiblyHarmful` covers the predefined -messages like :class:`PoisonPill` and :class:`Kill`, but it can also be added +a denial of service attack). `PossiblyHarmful` covers the predefined +messages like `PoisonPill` and `Kill`, but it can also be added as a marker trait to user-defined messages. -.. warning:: +@@@ warning - Untrusted mode does not give full protection against attacks by itself. - It makes it slightly harder to perform malicious or unintended actions but - it should be complemented with :ref:`disabled Java serializer `. - Additional protection can be achieved when running in an untrusted network by - network security (e.g. firewalls) and/or enabling - :ref:`TLS with mutual authentication `. +Untrusted mode does not give full protection against attacks by itself. +It makes it slightly harder to perform malicious or unintended actions but +it should be complemented with [disabled Java serializer](#disable-java-serializer-java). +Additional protection can be achieved when running in an untrusted network by +network security (e.g. firewalls) and/or enabling +[TLS with mutual authentication](#remote-tls-java). + +@@@ Messages sent with actor selection are by default discarded in untrusted mode, but permission to receive actor selection messages can be granted to specific actors -defined in configuration:: +defined in configuration: - akka.remote.trusted-selection-paths = ["/user/receptionist", "/user/namingService"] +``` +akka.remote.trusted-selection-paths = ["/user/receptionist", "/user/namingService"] +``` -The actual message must still not be of type :class:`PossiblyHarmful`. +The actual message must still not be of type `PossiblyHarmful`. In summary, the following operations are ignored by a system configured in untrusted mode when incoming via the remoting layer: -* remote deployment (which also means no remote supervision) -* remote DeathWatch -* ``system.stop()``, :class:`PoisonPill`, :class:`Kill` -* sending any message which extends from the :class:`PossiblyHarmful` marker - interface, which includes :class:`Terminated` -* messages sent with actor selection, unless destination defined in ``trusted-selection-paths``. + * remote deployment (which also means no remote supervision) + * remote DeathWatch + * `system.stop()`, `PoisonPill`, `Kill` + * sending any message which extends from the `PossiblyHarmful` marker +interface, which includes `Terminated` + * messages sent with actor selection, unless destination defined in `trusted-selection-paths`. -.. note:: +@@@ note - Enabling the untrusted mode does not remove the capability of the client to - freely choose the target of its message sends, which means that messages not - prohibited by the above rules can be sent to any actor in the remote system. - It is good practice for a client-facing system to only contain a well-defined - set of entry point actors, which then forward requests (possibly after - performing validation) to another actor system containing the actual worker - actors. If messaging between these two server-side systems is done using - local :class:`ActorRef` (they can be exchanged safely between actor systems - within the same JVM), you can restrict the messages on this interface by - marking them :class:`PossiblyHarmful` so that a client cannot forge them. +Enabling the untrusted mode does not remove the capability of the client to +freely choose the target of its message sends, which means that messages not +prohibited by the above rules can be sent to any actor in the remote system. +It is good practice for a client-facing system to only contain a well-defined +set of entry point actors, which then forward requests (possibly after +performing validation) to another actor system containing the actual worker +actors. If messaging between these two server-side systems is done using +local `ActorRef` (they can be exchanged safely between actor systems +within the same JVM), you can restrict the messages on this interface by +marking them `PossiblyHarmful` so that a client cannot forge them. -.. _remote-configuration-java: +@@@ -Remote Configuration -^^^^^^^^^^^^^^^^^^^^ + +## Remote Configuration There are lots of configuration properties that are related to remoting in Akka. We refer to the -:ref:`reference configuration ` for more information. +@ref:[reference configuration](../general/configuration.md#config-akka-remote) for more information. -.. note:: +@@@ note - Setting properties like the listening IP and port number programmatically is - best done by using something like the following: +Setting properties like the listening IP and port number programmatically is +best done by using something like the following: - .. includecode:: code/jdocs/remoting/RemoteDeploymentDocTest.java#programmatic +@@snip [RemoteDeploymentDocTest.java](code/jdocs/remoting/RemoteDeploymentDocTest.java) { #programmatic } -.. _remote-configuration-nat-java: +@@@ -Akka behind NAT or in a Docker container ----------------------------------------- + +### Akka behind NAT or in a Docker container In setups involving Network Address Translation (NAT), Load Balancers or Docker containers the hostname and port pair that Akka binds to will be different than the "logical" host name and port pair that is used to connect to the system from the outside. This requires special configuration that sets both the logical and the bind pairs for remoting. -.. code-block:: ruby +```ruby +akka { + remote { + netty.tcp { + hostname = my.domain.com # external (logical) hostname + port = 8000 # external (logical) port - akka { - remote { - netty.tcp { - hostname = my.domain.com # external (logical) hostname - port = 8000 # external (logical) port - - bind-hostname = local.address # internal (bind) hostname - bind-port = 2552 # internal (bind) port - } - } - } + bind-hostname = local.address # internal (bind) hostname + bind-port = 2552 # internal (bind) port + } + } +} +``` Keep in mind that local.address will most likely be in one of private network ranges: -* *10.0.0.0 - 10.255.255.255* (network class A) -* *172.16.0.0 - 172.31.255.255* (network class B) -* *192.168.0.0 - 192.168.255.255* (network class C) + * *10.0.0.0 - 10.255.255.255* (network class A) + * *172.16.0.0 - 172.31.255.255* (network class B) + * *192.168.0.0 - 192.168.255.255* (network class C) -For further details see [RFC 1597](https://tools.ietf.org/html/rfc1597) and [RFC 1918](https://tools.ietf.org/html/rfc1918). +For further details see [RFC 1597]([https://tools.ietf.org/html/rfc1597](https://tools.ietf.org/html/rfc1597)) and [RFC 1918]([https://tools.ietf.org/html/rfc1918](https://tools.ietf.org/html/rfc1918)). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/routing.md b/akka-docs/src/main/paradox/java/routing.md index 68da95b18b..eb16b1ef83 100644 --- a/akka-docs/src/main/paradox/java/routing.md +++ b/akka-docs/src/main/paradox/java/routing.md @@ -1,136 +1,125 @@ - -.. _routing-java: - -Routing -======= +# Routing Messages can be sent via a router to efficiently route them to destination actors, known as -its *routees*. A ``Router`` can be used inside or outside of an actor, and you can manage the +its *routees*. A `Router` can be used inside or outside of an actor, and you can manage the routees yourselves or use a self contained router actor with configuration capabilities. Different routing strategies can be used, according to your application's needs. Akka comes with several useful routing strategies right out of the box. But, as you will see in this chapter, it is -also possible to :ref:`create your own `. +also possible to [create your own](#custom-router-java). -.. _simple-router-java: + +## A Simple Router -A Simple Router -^^^^^^^^^^^^^^^ +The following example illustrates how to use a `Router` and manage the routees from within an actor. -The following example illustrates how to use a ``Router`` and manage the routees from within an actor. +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #router-in-actor } -.. includecode:: code/jdocs/routing/RouterDocTest.java#router-in-actor - -We create a ``Router`` and specify that it should use ``RoundRobinRoutingLogic`` when routing the +We create a `Router` and specify that it should use `RoundRobinRoutingLogic` when routing the messages to the routees. The routing logic shipped with Akka are: -* ``akka.routing.RoundRobinRoutingLogic`` -* ``akka.routing.RandomRoutingLogic`` -* ``akka.routing.SmallestMailboxRoutingLogic`` -* ``akka.routing.BroadcastRoutingLogic`` -* ``akka.routing.ScatterGatherFirstCompletedRoutingLogic`` -* ``akka.routing.TailChoppingRoutingLogic`` -* ``akka.routing.ConsistentHashingRoutingLogic`` + * `akka.routing.RoundRobinRoutingLogic` + * `akka.routing.RandomRoutingLogic` + * `akka.routing.SmallestMailboxRoutingLogic` + * `akka.routing.BroadcastRoutingLogic` + * `akka.routing.ScatterGatherFirstCompletedRoutingLogic` + * `akka.routing.TailChoppingRoutingLogic` + * `akka.routing.ConsistentHashingRoutingLogic` -We create the routees as ordinary child actors wrapped in ``ActorRefRoutee``. We watch +We create the routees as ordinary child actors wrapped in `ActorRefRoutee`. We watch the routees to be able to replace them if they are terminated. -Sending messages via the router is done with the ``route`` method, as is done for the ``Work`` messages +Sending messages via the router is done with the `route` method, as is done for the `Work` messages in the example above. -The ``Router`` is immutable and the ``RoutingLogic`` is thread safe; meaning that they can also be used +The `Router` is immutable and the `RoutingLogic` is thread safe; meaning that they can also be used outside of actors. -.. note:: +@@@ note - In general, any message sent to a router will be sent onwards to its routees, but there is one exception. - The special :ref:`broadcast-messages-java` will send to *all* of a router's routees. - However, do not use :ref:`broadcast-messages-java` when you use :ref:`balancing-pool-java` for routees - as described in :ref:`router-special-messages-java`. +In general, any message sent to a router will be sent onwards to its routees, but there is one exception. +The special [Broadcast Messages](#broadcast-messages-java) will send to *all* of a router's routees. +However, do not use [Broadcast Messages](#broadcast-messages-java) when you use [BalancingPool](#balancing-pool-java) for routees +as described in [Specially Handled Messages](#router-special-messages-java). -A Router Actor -^^^^^^^^^^^^^^ +@@@ + +## A Router Actor A router can also be created as a self contained actor that manages the routees itself and loads routing logic and other settings from configuration. This type of router actor comes in two distinct flavors: -* Pool - The router creates routees as child actors and removes them from the router if they - terminate. - -* Group - The routee actors are created externally to the router and the router sends - messages to the specified path using actor selection, without watching for termination. + * Pool - The router creates routees as child actors and removes them from the router if they +terminate. + * Group - The routee actors are created externally to the router and the router sends +messages to the specified path using actor selection, without watching for termination. The settings for a router actor can be defined in configuration or programmatically. -In order to make an actor to make use of an externally configurable router the ``FromConfig`` props wrapper must be used +In order to make an actor to make use of an externally configurable router the `FromConfig` props wrapper must be used to denote that the actor accepts routing settings from configuration. This is in contrast with Remote Deployment where such marker props is not necessary. -If the props of an actor is NOT wrapped in ``FromConfig`` it will ignore the router section of the deployment configuration. +If the props of an actor is NOT wrapped in `FromConfig` it will ignore the router section of the deployment configuration. You send messages to the routees via the router actor in the same way as for ordinary actors, -i.e. via its ``ActorRef``. The router actor forwards messages onto its routees without changing +i.e. via its `ActorRef`. The router actor forwards messages onto its routees without changing the original sender. When a routee replies to a routed message, the reply will be sent to the original sender, not to the router actor. -.. note:: +@@@ note - In general, any message sent to a router will be sent onwards to its routees, but there are a - few exceptions. These are documented in the :ref:`router-special-messages-java` section below. +In general, any message sent to a router will be sent onwards to its routees, but there are a +few exceptions. These are documented in the [Specially Handled Messages](#router-special-messages-java) section below. -Pool ----- +@@@ -The following code and configuration snippets show how to create a :ref:`round-robin -` router that forwards messages to five ``Worker`` routees. The +### Pool + +The following code and configuration snippets show how to create a [round-robin](#round-robin-router-java) router that forwards messages to five `Worker` routees. The routees will be created as the router's children. -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-round-robin-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-round-robin-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#round-robin-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #round-robin-pool-1 } Here is the same example, but with the router configuration provided programmatically instead of from configuration. -.. includecode:: code/jdocs/routing/RouterDocTest.java#round-robin-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #round-robin-pool-2 } -Remote Deployed Routees -*********************** +#### Remote Deployed Routees In addition to being able to create local actors as routees, you can instruct the router to deploy its created children on a set of remote hosts. Routees will be deployed in round-robin fashion. In order to deploy routees remotely, wrap the router configuration in a -``RemoteRouterConfig``, attaching the remote addresses of the nodes to deploy to. Remote -deployment requires the ``akka-remote`` module to be included in the classpath. +`RemoteRouterConfig`, attaching the remote addresses of the nodes to deploy to. Remote +deployment requires the `akka-remote` module to be included in the classpath. -.. includecode:: code/jdocs/routing/RouterDocTest.java#remoteRoutees +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #remoteRoutees } -Senders -******* +#### Senders +When a routee sends a message, it can @ref:[set itself as the sender +](actors.md#actors-tell-sender-java). -When a routee sends a message, it can :ref:`set itself as the sender -`. - -.. includecode:: code/jdocs/routing/RouterDocTest.java#reply-with-self +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #reply-with-self } However, it is often useful for routees to set the *router* as a sender. For example, you might want to set the router as the sender if you want to hide the details of the routees behind the router. The following code snippet shows how to set the parent router as sender. -.. includecode:: code/jdocs/routing/RouterDocTest.java#reply-with-parent +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #reply-with-parent } - -Supervision -*********** +#### Supervision Routees that are created by a pool router will be created as the router's children. The router is therefore also the children's supervisor. The supervision strategy of the router actor can be configured with the -``supervisorStrategy`` property of the Pool. If no configuration is provided, routers default +`supervisorStrategy` property of the Pool. If no configuration is provided, routers default to a strategy of “always escalate”. This means that errors are passed up to the router's supervisor for handling. The router's supervisor will decide what to do about any errors. @@ -141,284 +130,278 @@ turn, will cause its children to stop and restart. It should be mentioned that the router's restart behavior has been overridden so that a restart, while still re-creating the children, will still preserve the same number of actors in the pool. -This means that if you have not specified :meth:`supervisorStrategy` of the router or its parent a +This means that if you have not specified `supervisorStrategy` of the router or its parent a failure in a routee will escalate to the parent of the router, which will by default restart the router, which will restart all routees (it uses Escalate and does not stop routees during restart). The reason -is to make the default behave such that adding :meth:`.withRouter` to a child’s definition does not +is to make the default behave such that adding `.withRouter` to a child’s definition does not change the supervision strategy applied to the child. This might be an inefficiency that you can avoid by specifying the strategy when defining the router. Setting the strategy is easily done: -.. includecode:: code/jdocs/routing/RouterDocTest.java#supervision +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #supervision } -.. _note-router-terminated-children-java: +@@@ note -.. note:: +If the child of a pool router terminates, the pool router will not automatically spawn +a new child. In the event that all children of a pool router have terminated the +router will terminate itself unless it is a dynamic router, e.g. using +a resizer. - If the child of a pool router terminates, the pool router will not automatically spawn - a new child. In the event that all children of a pool router have terminated the - router will terminate itself unless it is a dynamic router, e.g. using - a resizer. +@@@ -Group ------ +### Group Sometimes, rather than having the router actor create its routees, it is desirable to create routees separately and provide them to the router for its use. You can do this by passing an -paths of the routees to the router's configuration. Messages will be sent with ``ActorSelection`` +paths of the routees to the router's configuration. Messages will be sent with `ActorSelection` to these paths. The example below shows how to create a router by providing it with the path strings of three routee actors. -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-round-robin-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-round-robin-group } -.. includecode:: code/jdocs/routing/RouterDocTest.java#round-robin-group-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #round-robin-group-1 } Here is the same example, but with the router configuration provided programmatically instead of from configuration. -.. includecode:: code/jdocs/routing/RouterDocTest.java#round-robin-group-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #round-robin-group-2 } The routee actors are created externally from the router: -.. includecode:: code/jdocs/routing/RouterDocTest.java#create-workers +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #create-workers } -.. includecode:: code/jdocs/routing/RouterDocTest.java#create-worker-actors +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #create-worker-actors } The paths may contain protocol and address information for actors running on remote hosts. -Remoting requires the ``akka-remote`` module to be included in the classpath. +Remoting requires the `akka-remote` module to be included in the classpath. -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-group } -Router usage -^^^^^^^^^^^^ +## Router usage In this section we will describe how to create the different types of router actors. -The router actors in this section are created from within a top level actor named ``parent``. -Note that deployment paths in the configuration starts with ``/parent/`` followed by the name +The router actors in this section are created from within a top level actor named `parent`. +Note that deployment paths in the configuration starts with `/parent/` followed by the name of the router actor. -.. includecode:: code/jdocs/routing/RouterDocTest.java#create-parent +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #create-parent } -.. _round-robin-router-java: + +### RoundRobinPool and RoundRobinGroup -RoundRobinPool and RoundRobinGroup ----------------------------------- - -Routes in a `round-robin `_ fashion to its routees. +Routes in a [round-robin](http://en.wikipedia.org/wiki/Round-robin) fashion to its routees. RoundRobinPool defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-round-robin-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-round-robin-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#round-robin-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #round-robin-pool-1 } RoundRobinPool defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java#round-robin-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #round-robin-pool-2 } RoundRobinGroup defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-round-robin-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-round-robin-group } -.. includecode:: code/jdocs/routing/RouterDocTest.java#round-robin-group-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #round-robin-group-1 } RoundRobinGroup defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java - :include: paths,round-robin-group-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #paths #round-robin-group-2 } -RandomPool and RandomGroup --------------------------- +### RandomPool and RandomGroup This router type selects one of its routees randomly for each message. RandomPool defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-random-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-random-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#random-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #random-pool-1 } RandomPool defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java#random-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #random-pool-2 } RandomGroup defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-random-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-random-group } -.. includecode:: code/jdocs/routing/RouterDocTest.java#random-group-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #random-group-1 } RandomGroup defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java - :include: paths,random-group-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #paths #random-group-2 } -.. _balancing-pool-java: - -BalancingPool -------------- + +### BalancingPool A Router that will try to redistribute work from busy routees to idle routees. All routees share the same mailbox. -.. note:: +@@@ note - The BalancingPool has the property that its routees do not have truly distinct - identity: they have different names, but talking to them will not end up at the - right actor in most cases. Therefore you cannot use it for workflows that - require state to be kept within the routee, you would in this case have to - include the whole state in the messages. +The BalancingPool has the property that its routees do not have truly distinct +identity: they have different names, but talking to them will not end up at the +right actor in most cases. Therefore you cannot use it for workflows that +require state to be kept within the routee, you would in this case have to +include the whole state in the messages. - With a `SmallestMailboxPool`_ you can have a vertically scaling service that - can interact in a stateful fashion with other services in the back-end before - replying to the original client. The other advantage is that it does not place - a restriction on the message queue implementation as BalancingPool does. +With a [SmallestMailboxPool](#smallestmailboxpool) you can have a vertically scaling service that +can interact in a stateful fashion with other services in the back-end before +replying to the original client. The other advantage is that it does not place +a restriction on the message queue implementation as BalancingPool does. -.. note:: - Do not use :ref:`broadcast-messages-java` when you use :ref:`balancing-pool-java` for routers, - as described in :ref:`router-special-messages-java`. +@@@ + +@@@ note + +Do not use [Broadcast Messages](#broadcast-messages-java) when you use [BalancingPool](#balancing-pool-java) for routers, +as described in [Specially Handled Messages](#router-special-messages-java). + +@@@ BalancingPool defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-balancing-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-balancing-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#balancing-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #balancing-pool-1 } BalancingPool defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java#balancing-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #balancing-pool-2 } Addition configuration for the balancing dispatcher, which is used by the pool, -can be configured in the ``pool-dispatcher`` section of the router deployment +can be configured in the `pool-dispatcher` section of the router deployment configuration. -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-balancing-pool2 +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-balancing-pool2 } -The ``BalancingPool`` automatically uses a special ``BalancingDispatcher`` for its +The `BalancingPool` automatically uses a special `BalancingDispatcher` for its routees - disregarding any dispatcher that is set on the routee Props object. This is needed in order to implement the balancing semantics via sharing the same mailbox by all the routees. While it is not possible to change the dispatcher used by the routees, it is possible -to fine tune the used *executor*. By default the ``fork-join-dispatcher`` is used and -can be configured as explained in :ref:`dispatchers-java`. In situations where the +to fine tune the used *executor*. By default the `fork-join-dispatcher` is used and +can be configured as explained in @ref:[Dispatchers](dispatchers.md). In situations where the routees are expected to perform blocking operations it may be useful to replace it -with a ``thread-pool-executor`` hinting the number of allocated threads explicitly: +with a `thread-pool-executor` hinting the number of allocated threads explicitly: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-balancing-pool3 +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-balancing-pool3 } -It is also possible to change the ``mailbox`` used by the balancing dispatcher for +It is also possible to change the `mailbox` used by the balancing dispatcher for scenarios where the default unbounded mailbox is not well suited. An example of such a scenario could arise whether there exists the need to manage priority for each message. You can then implement a priority mailbox and configure your dispatcher: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-balancing-pool4 +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-balancing-pool4 } -.. note:: +@@@ note - Bear in mind that ``BalancingDispatcher`` requires a message queue that must be thread-safe for - multiple concurrent consumers. So it is mandatory for the message queue backing a custom mailbox - for this kind of dispatcher to implement akka.dispatch.MultipleConsumerSemantics. See details - on how to implement your custom mailbox in :ref:`mailboxes-java`. +Bear in mind that `BalancingDispatcher` requires a message queue that must be thread-safe for +multiple concurrent consumers. So it is mandatory for the message queue backing a custom mailbox +for this kind of dispatcher to implement akka.dispatch.MultipleConsumerSemantics. See details +on how to implement your custom mailbox in @ref:[Mailboxes](mailboxes.md). + +@@@ There is no Group variant of the BalancingPool. -SmallestMailboxPool -------------------- +### SmallestMailboxPool A Router that tries to send to the non-suspended child routee with fewest messages in mailbox. The selection is done in this order: +> * pick any idle routee (not processing message) with empty mailbox * pick any routee with empty mailbox * pick routee with fewest pending messages in mailbox * pick any remote routee, remote actors are consider lowest priority, - since their mailbox size is unknown +since their mailbox size is unknown SmallestMailboxPool defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-smallest-mailbox-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-smallest-mailbox-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#smallest-mailbox-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #smallest-mailbox-pool-1 } SmallestMailboxPool defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java#smallest-mailbox-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #smallest-mailbox-pool-2 } There is no Group variant of the SmallestMailboxPool because the size of the mailbox and the internal dispatching state of the actor is not practically available from the paths of the routees. -BroadcastPool and BroadcastGroup --------------------------------- +### BroadcastPool and BroadcastGroup A broadcast router forwards the message it receives to *all* its routees. BroadcastPool defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-broadcast-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-broadcast-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#broadcast-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #broadcast-pool-1 } BroadcastPool defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java#broadcast-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #broadcast-pool-2 } BroadcastGroup defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-broadcast-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-broadcast-group } -.. includecode:: code/jdocs/routing/RouterDocTest.java#broadcast-group-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #broadcast-group-1 } BroadcastGroup defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java - :include: paths,broadcast-group-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #paths #broadcast-group-2 } -.. note:: +@@@ note - Broadcast routers always broadcast *every* message to their routees. If you do not want to - broadcast every message, then you can use a non-broadcasting router and use - :ref:`broadcast-messages-java` as needed. +Broadcast routers always broadcast *every* message to their routees. If you do not want to +broadcast every message, then you can use a non-broadcasting router and use +[Broadcast Messages](#broadcast-messages-java) as needed. +@@@ -ScatterGatherFirstCompletedPool and ScatterGatherFirstCompletedGroup --------------------------------------------------------------------- +### ScatterGatherFirstCompletedPool and ScatterGatherFirstCompletedGroup The ScatterGatherFirstCompletedRouter will send the message on to all its routees. It then waits for first reply it gets back. This result will be sent back to original sender. Other replies are discarded. It is expecting at least one reply within a configured duration, otherwise it will reply with -``akka.pattern.AskTimeoutException`` in a ``akka.actor.Status.Failure``. +`akka.pattern.AskTimeoutException` in a `akka.actor.Status.Failure`. ScatterGatherFirstCompletedPool defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-scatter-gather-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-scatter-gather-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#scatter-gather-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #scatter-gather-pool-1 } ScatterGatherFirstCompletedPool defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java#scatter-gather-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #scatter-gather-pool-2 } ScatterGatherFirstCompletedGroup defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-scatter-gather-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-scatter-gather-group } -.. includecode:: code/jdocs/routing/RouterDocTest.java#scatter-gather-group-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #scatter-gather-group-1 } ScatterGatherFirstCompletedGroup defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java - :include: paths,scatter-gather-group-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #paths #scatter-gather-group-2 } -TailChoppingPool and TailChoppingGroup --------------------------------------- +### TailChoppingPool and TailChoppingGroup The TailChoppingRouter will first send the message to one, randomly picked, routee and then after a small delay to a second routee (picked randomly from the remaining routees) and so on. @@ -428,214 +411,202 @@ The goal of this router is to decrease latency by performing redundant queries t one of the other actors may still be faster to respond than the initial one. This optimisation was described nicely in a blog post by Peter Bailis: -`Doing redundant work to speed up distributed queries `_. +[Doing redundant work to speed up distributed queries](http://www.bailis.org/blog/doing-redundant-work-to-speed-up-distributed-queries/). TailChoppingPool defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-tail-chopping-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-tail-chopping-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#tail-chopping-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #tail-chopping-pool-1 } TailChoppingPool defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java#tail-chopping-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #tail-chopping-pool-2 } TailChoppingGroup defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-tail-chopping-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-tail-chopping-group } -.. includecode:: code/jdocs/routing/RouterDocTest.java#tail-chopping-group-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #tail-chopping-group-1 } TailChoppingGroup defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java - :include: paths,tail-chopping-group-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #paths #tail-chopping-group-2 } -ConsistentHashingPool and ConsistentHashingGroup ------------------------------------------------- +### ConsistentHashingPool and ConsistentHashingGroup -The ConsistentHashingPool uses `consistent hashing `_ +The ConsistentHashingPool uses [consistent hashing](http://en.wikipedia.org/wiki/Consistent_hashing) to select a routee based on the sent message. This -`article `_ gives good +[article](http://weblogs.java.net/blog/tomwhite/archive/2007/11/consistent_hash.html) gives good insight into how consistent hashing is implemented. There is 3 ways to define what data to use for the consistent hash key. -* You can define ``withHashMapper`` of the router to map incoming - messages to their consistent hash key. This makes the decision - transparent for the sender. + * You can define `withHashMapper` of the router to map incoming +messages to their consistent hash key. This makes the decision +transparent for the sender. + * The messages may implement `akka.routing.ConsistentHashingRouter.ConsistentHashable`. +The key is part of the message and it's convenient to define it together +with the message definition. + * The messages can be wrapped in a `akka.routing.ConsistentHashingRouter.ConsistentHashableEnvelope` +to define what data to use for the consistent hash key. The sender knows +the key to use. -* The messages may implement ``akka.routing.ConsistentHashingRouter.ConsistentHashable``. - The key is part of the message and it's convenient to define it together - with the message definition. - -* The messages can be wrapped in a ``akka.routing.ConsistentHashingRouter.ConsistentHashableEnvelope`` - to define what data to use for the consistent hash key. The sender knows - the key to use. - These ways to define the consistent hash key can be use together and at -the same time for one router. The ``withHashMapper`` is tried first. - +the same time for one router. The `withHashMapper` is tried first. Code example: -.. includecode:: code/jdocs/routing/ConsistentHashingRouterDocTest.java#cache-actor +@@snip [ConsistentHashingRouterDocTest.java](code/jdocs/routing/ConsistentHashingRouterDocTest.java) { #cache-actor } -.. includecode:: code/jdocs/routing/ConsistentHashingRouterDocTest.java#consistent-hashing-router +@@snip [ConsistentHashingRouterDocTest.java](code/jdocs/routing/ConsistentHashingRouterDocTest.java) { #consistent-hashing-router } -In the above example you see that the ``Get`` message implements ``ConsistentHashable`` itself, -while the ``Entry`` message is wrapped in a ``ConsistentHashableEnvelope``. The ``Evict`` -message is handled by the ``hashMapping`` partial function. +In the above example you see that the `Get` message implements `ConsistentHashable` itself, +while the `Entry` message is wrapped in a `ConsistentHashableEnvelope`. The `Evict` +message is handled by the `hashMapping` partial function. ConsistentHashingPool defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-consistent-hashing-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-consistent-hashing-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#consistent-hashing-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #consistent-hashing-pool-1 } ConsistentHashingPool defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java#consistent-hashing-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #consistent-hashing-pool-2 } ConsistentHashingGroup defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-consistent-hashing-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-consistent-hashing-group } -.. includecode:: code/jdocs/routing/RouterDocTest.java#consistent-hashing-group-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #consistent-hashing-group-1 } ConsistentHashingGroup defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java - :include: paths,consistent-hashing-group-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #paths #consistent-hashing-group-2 } - -``virtual-nodes-factor`` is the number of virtual nodes per routee that is used in the +`virtual-nodes-factor` is the number of virtual nodes per routee that is used in the consistent hash node ring to make the distribution more uniform. -.. _router-special-messages-java: - -Specially Handled Messages -^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## Specially Handled Messages Most messages sent to router actors will be forwarded according to the routers' routing logic. However there are a few types of messages that have special behavior. -Note that these special messages, except for the ``Broadcast`` message, are only handled by -self contained router actors and not by the ``akka.routing.Router`` component described -in :ref:`simple-router-java`. +Note that these special messages, except for the `Broadcast` message, are only handled by +self contained router actors and not by the `akka.routing.Router` component described +in [A Simple Router](#simple-router-java). -.. _broadcast-messages-java: + +### Broadcast Messages -Broadcast Messages ------------------- - -A ``Broadcast`` message can be used to send a message to *all* of a router's routees. When a router -receives a ``Broadcast`` message, it will broadcast that message's *payload* to all routees, no +A `Broadcast` message can be used to send a message to *all* of a router's routees. When a router +receives a `Broadcast` message, it will broadcast that message's *payload* to all routees, no matter how that router would normally route its messages. -The example below shows how you would use a ``Broadcast`` message to send a very important message +The example below shows how you would use a `Broadcast` message to send a very important message to every routee of a router. -.. includecode:: code/jdocs/routing/RouterDocTest.java#broadcastDavyJonesWarning +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #broadcastDavyJonesWarning } -In this example the router receives the ``Broadcast`` message, extracts its payload -(``"Watch out for Davy Jones' locker"``), and then sends the payload on to all of the router's +In this example the router receives the `Broadcast` message, extracts its payload +(`"Watch out for Davy Jones' locker"`), and then sends the payload on to all of the router's routees. It is up to each routee actor to handle the received payload message. -.. note:: - Do not use :ref:`broadcast-messages-java` when you use :ref:`balancing-pool-java` for routers. - Routees on :ref:`balancing-pool-java` shares the same mailbox instance, thus some routees can - possibly get the broadcast message multiple times, while other routees get no broadcast message. +@@@ note -PoisonPill Messages -------------------- +Do not use [Broadcast Messages](#broadcast-messages-java) when you use [BalancingPool](#balancing-pool-java) for routers. +Routees on [BalancingPool](#balancing-pool-java) shares the same mailbox instance, thus some routees can +possibly get the broadcast message multiple times, while other routees get no broadcast message. -A ``PoisonPill`` message has special handling for all actors, including for routers. When any actor -receives a ``PoisonPill`` message, that actor will be stopped. See the :ref:`poison-pill-java` +@@@ + +### PoisonPill Messages + +A `PoisonPill` message has special handling for all actors, including for routers. When any actor +receives a `PoisonPill` message, that actor will be stopped. See the @ref:[PoisonPill](actors.md#poison-pill-java) documentation for details. -.. includecode:: code/jdocs/routing/RouterDocTest.java#poisonPill +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #poisonPill } For a router, which normally passes on messages to routees, it is important to realise that -``PoisonPill`` messages are processed by the router only. ``PoisonPill`` messages sent to a router +`PoisonPill` messages are processed by the router only. `PoisonPill` messages sent to a router will *not* be sent on to routees. -However, a ``PoisonPill`` message sent to a router may still affect its routees, because it will +However, a `PoisonPill` message sent to a router may still affect its routees, because it will stop the router and when the router stops it also stops its children. Stopping children is normal actor behavior. The router will stop routees that it has created as children. Each child will process its current message and then stop. This may lead to some messages being unprocessed. -See the documentation on :ref:`stopping-actors-java` for more information. +See the documentation on @ref:[Stopping actors](actors.md#stopping-actors-java) for more information. If you wish to stop a router and its routees, but you would like the routees to first process all -the messages currently in their mailboxes, then you should not send a ``PoisonPill`` message to the -router. Instead you should wrap a ``PoisonPill`` message inside a ``Broadcast`` message so that each -routee will receive the ``PoisonPill`` message. Note that this will stop all routees, even if the +the messages currently in their mailboxes, then you should not send a `PoisonPill` message to the +router. Instead you should wrap a `PoisonPill` message inside a `Broadcast` message so that each +routee will receive the `PoisonPill` message. Note that this will stop all routees, even if the routees aren't children of the router, i.e. even routees programmatically provided to the router. -.. includecode:: code/jdocs/routing/RouterDocTest.java#broadcastPoisonPill +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #broadcastPoisonPill } -With the code shown above, each routee will receive a ``PoisonPill`` message. Each routee will -continue to process its messages as normal, eventually processing the ``PoisonPill``. This will -cause the routee to stop. After all routees have stopped the router will itself be :ref:`stopped -automatically ` unless it is a dynamic router, e.g. using +With the code shown above, each routee will receive a `PoisonPill` message. Each routee will +continue to process its messages as normal, eventually processing the `PoisonPill`. This will +cause the routee to stop. After all routees have stopped the router will itself be stopped +automatically unless it is a dynamic router, e.g. using a resizer. -.. note:: +@@@ note - Brendan W McAdams' excellent blog post `Distributing Akka Workloads - And Shutting Down Afterwards - `_ - discusses in more detail how ``PoisonPill`` messages can be used to shut down routers and routees. +Brendan W McAdams' excellent blog post [Distributing Akka Workloads - And Shutting Down Afterwards](http://bytes.codes/2013/01/17/Distributing_Akka_Workloads_And_Shutting_Down_After/) +discusses in more detail how `PoisonPill` messages can be used to shut down routers and routees. -Kill Messages -------------- +@@@ -``Kill`` messages are another type of message that has special handling. See -:ref:`killing-actors-java` for general information about how actors handle ``Kill`` messages. +### Kill Messages -When a ``Kill`` message is sent to a router the router processes the message internally, and does -*not* send it on to its routees. The router will throw an ``ActorKilledException`` and fail. It +`Kill` messages are another type of message that has special handling. See +@ref:[Killing an Actor](actors.md#killing-actors-java) for general information about how actors handle `Kill` messages. + +When a `Kill` message is sent to a router the router processes the message internally, and does +*not* send it on to its routees. The router will throw an `ActorKilledException` and fail. It will then be either resumed, restarted or terminated, depending how it is supervised. Routees that are children of the router will also be suspended, and will be affected by the supervision directive that is applied to the router. Routees that are not the routers children, i.e. those that were created externally to the router, will not be affected. -.. includecode:: code/jdocs/routing/RouterDocTest.java#kill +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #kill } -As with the ``PoisonPill`` message, there is a distinction between killing a router, which +As with the `PoisonPill` message, there is a distinction between killing a router, which indirectly kills its children (who happen to be routees), and killing routees directly (some of whom -may not be children.) To kill routees directly the router should be sent a ``Kill`` message wrapped -in a ``Broadcast`` message. +may not be children.) To kill routees directly the router should be sent a `Kill` message wrapped +in a `Broadcast` message. -.. includecode:: code/jdocs/routing/RouterDocTest.java#broadcastKill +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #broadcastKill } -Management Messages -------------------- +### Management Messages -* Sending ``akka.routing.GetRoutees`` to a router actor will make it send back its currently used routees - in a ``akka.routing.Routees`` message. -* Sending ``akka.routing.AddRoutee`` to a router actor will add that routee to its collection of routees. -* Sending ``akka.routing.RemoveRoutee`` to a router actor will remove that routee to its collection of routees. -* Sending ``akka.routing.AdjustPoolSize`` to a pool router actor will add or remove that number of routees to - its collection of routees. + * Sending `akka.routing.GetRoutees` to a router actor will make it send back its currently used routees +in a `akka.routing.Routees` message. + * Sending `akka.routing.AddRoutee` to a router actor will add that routee to its collection of routees. + * Sending `akka.routing.RemoveRoutee` to a router actor will remove that routee to its collection of routees. + * Sending `akka.routing.AdjustPoolSize` to a pool router actor will add or remove that number of routees to +its collection of routees. -These management messages may be handled after other messages, so if you send ``AddRoutee`` immediately followed by +These management messages may be handled after other messages, so if you send `AddRoutee` immediately followed by an ordinary message you are not guaranteed that the routees have been changed when the ordinary message -is routed. If you need to know when the change has been applied you can send ``AddRoutee`` followed by ``GetRoutees`` -and when you receive the ``Routees`` reply you know that the preceding change has been applied. +is routed. If you need to know when the change has been applied you can send `AddRoutee` followed by `GetRoutees` +and when you receive the `Routees` reply you know that the preceding change has been applied. -.. _resizable-routers-java: - -Dynamically Resizable Pool -^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## Dynamically Resizable Pool All pools can be used with a fixed number of routees or with a resize strategy to adjust the number of routees dynamically. -There are two types of resizers: the default ``Resizer`` and the ``OptimalSizeExploringResizer``. +There are two types of resizers: the default `Resizer` and the `OptimalSizeExploringResizer`. -Default Resizer ---------------- +### Default Resizer The default resizer ramps up and down pool size based on pressure, measured by the percentage of busy routees in the pool. It ramps up pool size if the pressure is higher than a certain threshold and backs off if the @@ -643,31 +614,30 @@ pressure is lower than certain threshold. Both thresholds are configurable. Pool with default resizer defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-resize-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-resize-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#resize-pool-1 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #resize-pool-1 } -Several more configuration options are available and described in ``akka.actor.deployment.default.resizer`` -section of the reference :ref:`configuration`. +Several more configuration options are available and described in `akka.actor.deployment.default.resizer` +section of the reference configuration. Pool with resizer defined in code: -.. includecode:: code/jdocs/routing/RouterDocTest.java#resize-pool-2 +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #resize-pool-2 } *It is also worth pointing out that if you define the ``router`` in the configuration file then this value will be used instead of any programmatically sent parameters.* -Optimal Size Exploring Resizer ------------------------------- +### Optimal Size Exploring Resizer -The ``OptimalSizeExploringResizer`` resizes the pool to an optimal size that provides the most message throughput. +The `OptimalSizeExploringResizer` resizes the pool to an optimal size that provides the most message throughput. It achieves this by keeping track of message throughput at each pool size and performing one of the following three resizing operations periodically: -* Downsize if it hasn't seen all routees ever fully utilized for a period of time. -* Explore to a random nearby pool size to try and collect throughput metrics. -* Optimize to a nearby pool size with a better (than any other nearby sizes) throughput metrics. + * Downsize if it hasn't seen all routees ever fully utilized for a period of time. + * Explore to a random nearby pool size to try and collect throughput metrics. + * Optimize to a nearby pool size with a better (than any other nearby sizes) throughput metrics. When the pool is fully-utilized (i.e. all routees are busy), it randomly choose between exploring and optimizing. When the pool has not been fully-utilized for a period of time, it will downsize the pool to the last seen max @@ -680,32 +650,32 @@ For example, when you have a CPU bound tasks, the optimal size is bound to the n When your task is IO bound, the optimal size is bound to optimal number of concurrent connections to that IO service - e.g. a 4 node elastic search cluster may handle 4-8 concurrent requests at optimal speed. -It keeps a performance log so it's stateful as well as having a larger memory footprint than the default ``Resizer``. +It keeps a performance log so it's stateful as well as having a larger memory footprint than the default `Resizer`. The memory usage is O(n) where n is the number of sizes you allow, i.e. upperBound - lowerBound. -Pool with ``OptimalSizeExploringResizer`` defined in configuration: +Pool with `OptimalSizeExploringResizer` defined in configuration: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-optimal-size-exploring-resize-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-optimal-size-exploring-resize-pool } -.. includecode:: code/jdocs/routing/RouterDocTest.java#optimal-size-exploring-resize-pool +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #optimal-size-exploring-resize-pool } -Several more configuration options are available and described in ``akka.actor.deployment.default.optimal-size-exploring-resizer`` -section of the reference :ref:`configuration`. +Several more configuration options are available and described in `akka.actor.deployment.default.optimal-size-exploring-resizer` +section of the reference configuration. -.. note:: +@@@ note - Resizing is triggered by sending messages to the actor pool, but it is not - completed synchronously; instead a message is sent to the “head” - ``RouterActor`` to perform the size change. Thus you cannot rely on resizing - to instantaneously create new workers when all others are busy, because the - message just sent will be queued to the mailbox of a busy actor. To remedy - this, configure the pool to use a balancing dispatcher, see `Configuring - Dispatchers`_ for more information. +Resizing is triggered by sending messages to the actor pool, but it is not +completed synchronously; instead a message is sent to the “head” +`RouterActor` to perform the size change. Thus you cannot rely on resizing +to instantaneously create new workers when all others are busy, because the +message just sent will be queued to the mailbox of a busy actor. To remedy +this, configure the pool to use a balancing dispatcher, see [Configuring +Dispatchers](#configuring-dispatchers) for more information. -.. _router-design-java: +@@@ -How Routing is Designed within Akka -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## How Routing is Designed within Akka On the surface routers look like normal actors, but they are actually implemented differently. Routers are designed to be extremely efficient at receiving messages and passing them quickly on to @@ -714,8 +684,8 @@ routees. A normal actor can be used for routing messages, but an actor's single-threaded processing can become a bottleneck. Routers can achieve much higher throughput with an optimization to the usual message-processing pipeline that allows concurrent routing. This is achieved by embedding routers' -routing logic directly in their ``ActorRef`` rather than in the router actor. Messages sent to -a router's ``ActorRef`` can be immediately routed to the routee, bypassing the single-threaded +routing logic directly in their `ActorRef` rather than in the router actor. Messages sent to +a router's `ActorRef` can be immediately routed to the routee, bypassing the single-threaded router actor entirely. The cost to this is, of course, that the internals of routing code are more complicated than if @@ -723,17 +693,15 @@ routers were implemented with normal actors. Fortunately all of this complexity consumers of the routing API. However, it is something to be aware of when implementing your own routers. -.. _custom-router-java: - -Custom Router -^^^^^^^^^^^^^ + +## Custom Router You can create your own router should you not find any of the ones provided by Akka sufficient for your needs. In order to roll your own router you have to fulfill certain criteria which are explained in this section. Before creating your own router you should consider whether a normal actor with router-like behavior might do the job just as well as a full-blown router. As explained -:ref:`above `, the primary benefit of routers over normal actors is their +[above](#router-design-java), the primary benefit of routers over normal actors is their higher performance. But they are somewhat more complicated to write than normal actors. Therefore if lower maximum throughput is acceptable in your application you may wish to stick with traditional actors. This section, however, assumes that you wish to get maximum performance and so demonstrates @@ -743,77 +711,79 @@ The router created in this example is replicating each message to a few destinat Start with the routing logic: -.. includecode:: code/jdocs/routing/CustomRouterDocTest.java#routing-logic +@@snip [CustomRouterDocTest.java](code/jdocs/routing/CustomRouterDocTest.java) { #routing-logic } -``select`` will be called for each message and in this example pick a few destinations by round-robin, -by reusing the existing ``RoundRobinRoutingLogic`` and wrap the result in a ``SeveralRoutees`` -instance. ``SeveralRoutees`` will send the message to all of the supplied routes. +`select` will be called for each message and in this example pick a few destinations by round-robin, +by reusing the existing `RoundRobinRoutingLogic` and wrap the result in a `SeveralRoutees` +instance. `SeveralRoutees` will send the message to all of the supplied routes. The implementation of the routing logic must be thread safe, since it might be used outside of actors. A unit test of the routing logic: -.. includecode:: code/jdocs/routing/CustomRouterDocTest.java#unit-test-logic +@@snip [CustomRouterDocTest.java](code/jdocs/routing/CustomRouterDocTest.java) { #unit-test-logic } -You could stop here and use the ``RedundancyRoutingLogic`` with a ``akka.routing.Router`` -as described in :ref:`simple-router-java`. +You could stop here and use the `RedundancyRoutingLogic` with a `akka.routing.Router` +as described in [A Simple Router](#simple-router-java). Let us continue and make this into a self contained, configurable, router actor. -Create a class that extends ``PoolBase``, ``GroupBase`` or ``CustomRouterConfig``. That class is a factory -for the routing logic and holds the configuration for the router. Here we make it a ``Group``. +Create a class that extends `PoolBase`, `GroupBase` or `CustomRouterConfig`. That class is a factory +for the routing logic and holds the configuration for the router. Here we make it a `Group`. -.. includecode:: code/jdocs/routing/RedundancyGroup.java#group +@@snip [RedundancyGroup.java](code/jdocs/routing/RedundancyGroup.java) { #group } This can be used exactly as the router actors provided by Akka. -.. includecode:: code/jdocs/routing/CustomRouterDocTest.java#usage-1 +@@snip [CustomRouterDocTest.java](code/jdocs/routing/CustomRouterDocTest.java) { #usage-1 } -Note that we added a constructor in ``RedundancyGroup`` that takes a ``Config`` parameter. +Note that we added a constructor in `RedundancyGroup` that takes a `Config` parameter. That makes it possible to define it in configuration. -.. includecode:: ../scala/code/docs/routing/CustomRouterDocSpec.scala#jconfig +@@snip [CustomRouterDocSpec.scala](../scala/code/docs/routing/CustomRouterDocSpec.scala) { #jconfig } -Note the fully qualified class name in the ``router`` property. The router class must extend -``akka.routing.RouterConfig`` (``Pool``, ``Group`` or ``CustomRouterConfig``) and have -constructor with one ``com.typesafe.config.Config`` parameter. +Note the fully qualified class name in the `router` property. The router class must extend +`akka.routing.RouterConfig` (`Pool`, `Group` or `CustomRouterConfig`) and have +constructor with one `com.typesafe.config.Config` parameter. The deployment section of the configuration is passed to the constructor. -.. includecode:: code/jdocs/routing/CustomRouterDocTest.java#usage-2 - -Configuring Dispatchers -^^^^^^^^^^^^^^^^^^^^^^^ +@@snip [CustomRouterDocTest.java](code/jdocs/routing/CustomRouterDocTest.java) { #usage-2 } + +## Configuring Dispatchers The dispatcher for created children of the pool will be taken from -``Props`` as described in :ref:`dispatchers-scala`. +`Props` as described in @ref:[Dispatchers](../scala/dispatchers.md). To make it easy to define the dispatcher of the routees of the pool you can define the dispatcher inline in the deployment section of the config. -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-pool-dispatcher +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-pool-dispatcher } That is the only thing you need to do enable a dedicated dispatcher for a pool. -.. note:: +@@@ note - If you use a group of actors and route to their paths, then they will still use the same dispatcher - that was configured for them in their ``Props``, it is not possible to change an actors dispatcher - after it has been created. +If you use a group of actors and route to their paths, then they will still use the same dispatcher +that was configured for them in their `Props`, it is not possible to change an actors dispatcher +after it has been created. + +@@@ The “head” router cannot always run on the same dispatcher, because it does not process the same type of messages, hence this special actor does -not use the dispatcher configured in ``Props``, but takes the -``routerDispatcher`` from the :class:`RouterConfig` instead, which defaults to +not use the dispatcher configured in `Props`, but takes the +`routerDispatcher` from the `RouterConfig` instead, which defaults to the actor system’s default dispatcher. All standard routers allow setting this property in their constructor or factory method, custom routers have to implement the method in a suitable way. -.. includecode:: code/jdocs/routing/RouterDocTest.java#dispatchers +@@snip [RouterDocTest.java](code/jdocs/routing/RouterDocTest.java) { #dispatchers } -.. note:: +@@@ note - It is not allowed to configure the ``routerDispatcher`` to be a - :class:`akka.dispatch.BalancingDispatcherConfigurator` since the messages meant - for the special router actor cannot be processed by any other actor. - +It is not allowed to configure the `routerDispatcher` to be a +`akka.dispatch.BalancingDispatcherConfigurator` since the messages meant +for the special router actor cannot be processed by any other actor. + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/scala-compat.md b/akka-docs/src/main/paradox/java/scala-compat.md index 884114a4ba..08c650e9de 100644 --- a/akka-docs/src/main/paradox/java/scala-compat.md +++ b/akka-docs/src/main/paradox/java/scala-compat.md @@ -1,24 +1,23 @@ -.. _scala-java-compat: - -Java 8 and Scala Compatibility -============================== +# Java 8 and Scala Compatibility Starting with Akka 2.4.2 we have begun to introduce Java 8 types (most -prominently ``java.util.concurrent.CompletionStage`` and -``java.util.Optional``) where that was possible without breaking binary or +prominently `java.util.concurrent.CompletionStage` and +`java.util.Optional`) where that was possible without breaking binary or source compatibility. Where this was not possible (for example in the return -type of ``ActorSystem.terminate()``) please refer to the -``scala-java8-compat`` library that allows easy conversion between the Scala -and Java counterparts. The artifact can be included in Maven builds using:: +type of `ActorSystem.terminate()`) please refer to the +`scala-java8-compat` library that allows easy conversion between the Scala +and Java counterparts. The artifact can be included in Maven builds using: - - org.scala-lang.modules - scala-java8-compat_2.11 - 0.7.0 - +``` + + org.scala-lang.modules + scala-java8-compat_2.11 + 0.7.0 + +``` We will only be able to seamlessly integrate all functional interfaces once we can rely on Scala 2.12 to provide full interoperability—this will mean that Scala users can directly implement Java Functional Interfaces using lambda syntax as well as that Java users can directly implement Scala functions using lambda -syntax. +syntax. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/scheduler.md b/akka-docs/src/main/paradox/java/scheduler.md index 5e9c8cc9f3..38afdd3ef2 100644 --- a/akka-docs/src/main/paradox/java/scheduler.md +++ b/akka-docs/src/main/paradox/java/scheduler.md @@ -1,112 +1,100 @@ - -.. _scheduler-java: - -Scheduler -######### +# Scheduler Sometimes the need for making things happen in the future arises, and where do -you go look then? Look no further than ``ActorSystem``! There you find the -:meth:`scheduler` method that returns an instance of -:class:`akka.actor.Scheduler`, this instance is unique per ActorSystem and is +you go look then? Look no further than `ActorSystem`! There you find the +`scheduler` method that returns an instance of +`akka.actor.Scheduler`, this instance is unique per ActorSystem and is used internally for scheduling things to happen at specific points in time. You can schedule sending of messages to actors and execution of tasks -(functions or Runnable). You will get a ``Cancellable`` back that you can call -:meth:`cancel` on to cancel the execution of the scheduled operation. +(functions or Runnable). You will get a `Cancellable` back that you can call +`cancel` on to cancel the execution of the scheduled operation. The scheduler in Akka is designed for high-throughput of thousands up to millions of triggers. The prime use-case being triggering Actor receive timeouts, Future timeouts, circuit breakers and other time dependent events which happen all-the-time and in many instances at the same time. The implementation is based on a Hashed Wheel Timer, which is -a known datastructure and algorithm for handling such use cases, refer to the `Hashed and Hierarchical Timing Wheels`_ +a known datastructure and algorithm for handling such use cases, refer to the [Hashed and Hierarchical Timing Wheels](http://www.cs.columbia.edu/~nahum/w6998/papers/sosp87-timing-wheels.pdf) whitepaper by Varghese and Lauck if you'd like to understand its inner workings. -The Akka scheduler is **not** designed for long-term scheduling (see `akka-quartz-scheduler`_ +The Akka scheduler is **not** designed for long-term scheduling (see [akka-quartz-scheduler](https://github.com/enragedginger/akka-quartz-scheduler) instead for this use case) nor is it to be used for higly precise firing of the events. The maximum amount of time into the future you can schedule an event to trigger is around 8 months, which in practice is too much to be useful since this would assume the system never went down during that period. If you need long-term scheduling we highly recommend looking into alternative schedulers, as this is not the use-case the Akka scheduler is implemented for. -.. warning:: +@@@ warning - The default implementation of ``Scheduler`` used by Akka is based on job - buckets which are emptied according to a fixed schedule. It does not - execute tasks at the exact time, but on every tick, it will run everything - that is (over)due. The accuracy of the default Scheduler can be modified - by the ``akka.scheduler.tick-duration`` configuration property. +The default implementation of `Scheduler` used by Akka is based on job +buckets which are emptied according to a fixed schedule. It does not +execute tasks at the exact time, but on every tick, it will run everything +that is (over)due. The accuracy of the default Scheduler can be modified +by the `akka.scheduler.tick-duration` configuration property. -.. _akka-quartz-scheduler: https://github.com/enragedginger/akka-quartz-scheduler -.. _Hashed and Hierarchical Timing Wheels: http://www.cs.columbia.edu/~nahum/w6998/papers/sosp87-timing-wheels.pdf +@@@ -Some examples -------------- +## Some examples Schedule to send the "foo"-message to the testActor after 50ms: -.. includecode:: code/jdocs/actor/SchedulerDocTest.java - :include: imports1 +@@snip [SchedulerDocTest.java](code/jdocs/actor/SchedulerDocTest.java) { #imports1 } -.. includecode:: code/jdocs/actor/SchedulerDocTest.java - :include: schedule-one-off-message +@@snip [SchedulerDocTest.java](code/jdocs/actor/SchedulerDocTest.java) { #schedule-one-off-message } Schedule a Runnable, that sends the current time to the testActor, to be executed after 50ms: -.. includecode:: code/jdocs/actor/SchedulerDocTest.java - :include: schedule-one-off-thunk +@@snip [SchedulerDocTest.java](code/jdocs/actor/SchedulerDocTest.java) { #schedule-one-off-thunk } -.. warning:: +@@@ warning - If you schedule Runnable instances you should be extra careful - to not pass or close over unstable references. In practice this means that you should - not call methods on the enclosing Actor from within the Runnable. - If you need to schedule an invocation it is better to use the ``schedule()`` - variant accepting a message and an ``ActorRef`` to schedule a message to self - (containing the necessary parameters) and then call the method when the message is received. +If you schedule Runnable instances you should be extra careful +to not pass or close over unstable references. In practice this means that you should +not call methods on the enclosing Actor from within the Runnable. +If you need to schedule an invocation it is better to use the `schedule()` +variant accepting a message and an `ActorRef` to schedule a message to self +(containing the necessary parameters) and then call the method when the message is received. -Schedule to send the "Tick"-message to the ``tickActor`` after 0ms repeating every 50ms: +@@@ -.. includecode:: code/jdocs/actor/SchedulerDocTest.java - :include: imports1,imports2 +Schedule to send the "Tick"-message to the `tickActor` after 0ms repeating every 50ms: -.. includecode:: code/jdocs/actor/SchedulerDocTest.java - :include: schedule-recurring +@@snip [SchedulerDocTest.java](code/jdocs/actor/SchedulerDocTest.java) { #imports1 #imports2 } -From ``akka.actor.ActorSystem`` -------------------------------- +@@snip [SchedulerDocTest.java](code/jdocs/actor/SchedulerDocTest.java) { #schedule-recurring } -.. includecode:: ../../../akka-actor/src/main/scala/akka/actor/ActorSystem.scala - :include: scheduler +## From `akka.actor.ActorSystem` -.. warning:: +@@snip [ActorSystem.scala]../../../../../akka-actor/src/main/scala/akka/actor/ActorSystem.scala) { #scheduler } - All scheduled task will be executed when the ``ActorSystem`` is terminated, i.e. - the task may execute before its timeout. +@@@ warning -The Scheduler Interface for Implementors ----------------------------------------- +All scheduled task will be executed when the `ActorSystem` is terminated, i.e. +the task may execute before its timeout. + +@@@ + +## The Scheduler Interface for Implementors The actual scheduler implementation is loaded reflectively upon -:class:`ActorSystem` start-up, which means that it is possible to provide a -different one using the ``akka.scheduler.implementation`` configuration +`ActorSystem` start-up, which means that it is possible to provide a +different one using the `akka.scheduler.implementation` configuration property. The referenced class must implement the following interface: -.. includecode:: ../../../akka-actor/src/main/java/akka/actor/AbstractScheduler.java - :include: scheduler +@@snip [AbstractScheduler.java]../../../../../akka-actor/src/main/java/akka/actor/AbstractScheduler.java) { #scheduler } -The Cancellable interface -------------------------- +## The Cancellable interface -Scheduling a task will result in a :class:`Cancellable` (or throw an -:class:`IllegalStateException` if attempted after the scheduler’s shutdown). +Scheduling a task will result in a `Cancellable` (or throw an +`IllegalStateException` if attempted after the scheduler’s shutdown). This allows you to cancel something that has been scheduled for execution. -.. warning:: +@@@ warning - This does not abort the execution of the task, if it had already been - started. Check the return value of ``cancel`` to detect whether the - scheduled task was canceled or will (eventually) have run. +This does not abort the execution of the task, if it had already been +started. Check the return value of `cancel` to detect whether the +scheduled task was canceled or will (eventually) have run. -.. includecode:: ../../../akka-actor/src/main/scala/akka/actor/Scheduler.scala - :include: cancellable +@@@ +@@snip [Scheduler.scala]../../../../../akka-actor/src/main/scala/akka/actor/Scheduler.scala) { #cancellable } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/serialization.md b/akka-docs/src/main/paradox/java/serialization.md index 7486bd3cda..2d02f2e79c 100644 --- a/akka-docs/src/main/paradox/java/serialization.md +++ b/akka-docs/src/main/paradox/java/serialization.md @@ -1,164 +1,146 @@ - -.. _serialization-java: - -##################### - Serialization -##################### +# Serialization The messages that Akka actors send to each other are JVM objects (e.g. instances of Scala case classes). Message passing between actors that live on the same JVM is straightforward. It is simply done via reference passing. However, messages that have to escape the JVM to reach an actor running on a different host have to undergo some form of serialization (i.e. the objects have to be converted to and from byte arrays). Akka itself uses Protocol Buffers to serialize internal messages (i.e. cluster gossip messages). However, the serialization mechanism in Akka allows you to write custom serializers and to define which serializer to use for what. -Usage -===== +## Usage -Configuration -------------- +### Configuration -For Akka to know which ``Serializer`` to use for what, you need edit your :ref:`configuration`, -in the "akka.actor.serializers"-section you bind names to implementations of the ``akka.serialization.Serializer`` +For Akka to know which `Serializer` to use for what, you need edit your [Configuration](), +in the "akka.actor.serializers"-section you bind names to implementations of the `akka.serialization.Serializer` you wish to use, like this: -.. includecode:: ../scala/code/docs/serialization/SerializationDocSpec.scala#serialize-serializers-config +@@snip [SerializationDocSpec.scala](../scala/code/docs/serialization/SerializationDocSpec.scala) { #serialize-serializers-config } -After you've bound names to different implementations of ``Serializer`` you need to wire which classes -should be serialized using which ``Serializer``, this is done in the "akka.actor.serialization-bindings"-section: +After you've bound names to different implementations of `Serializer` you need to wire which classes +should be serialized using which `Serializer`, this is done in the "akka.actor.serialization-bindings"-section: -.. includecode:: ../scala/code/docs/serialization/SerializationDocSpec.scala#serialization-bindings-config +@@snip [SerializationDocSpec.scala](../scala/code/docs/serialization/SerializationDocSpec.scala) { #serialization-bindings-config } You only need to specify the name of an interface or abstract base class of the messages. In case of ambiguity, i.e. the message implements several of the configured classes, the most specific configured class will be used, i.e. the one of which all other candidates are superclasses. If this condition cannot be -met, because e.g. ``java.io.Serializable`` and ``MyOwnSerializable`` both apply +met, because e.g. `java.io.Serializable` and `MyOwnSerializable` both apply and neither is a subtype of the other, a warning will be issued. -.. note:: +@@@ note - If you are using Scala for your message protocol and your messages are contained - inside of a Scala object, then in order to reference those messages, you will need - use the fully qualified Java class name. For a message named ``Message`` contained inside - the Scala object named ``Wrapper`` you would need to reference it as - ``Wrapper$Message`` instead of ``Wrapper.Message``. +If you are using Scala for your message protocol and your messages are contained +inside of a Scala object, then in order to reference those messages, you will need +use the fully qualified Java class name. For a message named `Message` contained inside +the Scala object named `Wrapper` you would need to reference it as +`Wrapper$Message` instead of `Wrapper.Message`. -Akka provides serializers for :class:`java.io.Serializable` and `protobuf -`_ -:class:`com.google.protobuf.GeneratedMessage` by default (the latter only if +@@@ + +Akka provides serializers for `java.io.Serializable` and [protobuf](http://code.google.com/p/protobuf/) +`com.google.protobuf.GeneratedMessage` by default (the latter only if depending on the akka-remote module), so normally you don't need to add -configuration for that; since :class:`com.google.protobuf.GeneratedMessage` -implements :class:`java.io.Serializable`, protobuf messages will always be +configuration for that; since `com.google.protobuf.GeneratedMessage` +implements `java.io.Serializable`, protobuf messages will always be serialized using the protobuf protocol unless specifically overridden. In order -to disable a default serializer, map its marker type to “none”:: +to disable a default serializer, map its marker type to “none”: - akka.actor.serialization-bindings { - "java.io.Serializable" = none - } +``` +akka.actor.serialization-bindings { + "java.io.Serializable" = none +} +``` -Verification ------------- +### Verification Normally, messages sent between local actors (i.e. same JVM) do not undergo serialization. For testing, sometimes, it may be desirable to force serialization on all messages (both remote and local). If you want to do this in order to verify that your messages are serializable you can enable the following config option: -.. includecode:: ../scala/code/docs/serialization/SerializationDocSpec.scala#serialize-messages-config +@@snip [SerializationDocSpec.scala](../scala/code/docs/serialization/SerializationDocSpec.scala) { #serialize-messages-config } -If you want to verify that your ``Props`` are serializable you can enable the following config option: +If you want to verify that your `Props` are serializable you can enable the following config option: -.. includecode:: ../scala/code/docs/serialization/SerializationDocSpec.scala#serialize-creators-config +@@snip [SerializationDocSpec.scala](../scala/code/docs/serialization/SerializationDocSpec.scala) { #serialize-creators-config } -.. warning:: +@@@ warning - We recommend having these config options turned on **only** when you're running tests. Turning these options on in production is pointless, as it would negatively impact the performance of local message passing without giving any gain. +We recommend having these config options turned on **only** when you're running tests. Turning these options on in production is pointless, as it would negatively impact the performance of local message passing without giving any gain. -Programmatic ------------- +@@@ + +### Programmatic If you want to programmatically serialize/deserialize using Akka Serialization, here's some examples: -.. includecode:: code/jdocs/serialization/SerializationDocTest.java - :include: imports +@@snip [SerializationDocTest.java](code/jdocs/serialization/SerializationDocTest.java) { #imports } -.. includecode:: code/jdocs/serialization/SerializationDocTest.java - :include: programmatic +@@snip [SerializationDocTest.java](code/jdocs/serialization/SerializationDocTest.java) { #programmatic } -For more information, have a look at the ``ScalaDoc`` for ``akka.serialization._`` +For more information, have a look at the `ScalaDoc` for `akka.serialization._` +## Customization -Customization -============= +The first code snippet on this page contains a configuration file that references a custom serializer `docs.serialization.MyOwnSerializer`. How would we go about creating such a custom serializer? -The first code snippet on this page contains a configuration file that references a custom serializer ``docs.serialization.MyOwnSerializer``. How would we go about creating such a custom serializer? +### Creating new Serializers -Creating new Serializers ------------------------- +A custom `Serializer` has to inherit from `akka.serialization.JSerializer` and can be defined like the following: -A custom ``Serializer`` has to inherit from ``akka.serialization.JSerializer`` and can be defined like the following: +@@snip [SerializationDocTest.java](code/jdocs/serialization/SerializationDocTest.java) { #imports } -.. includecode:: code/jdocs/serialization/SerializationDocTest.java - :include: imports - -.. includecode:: code/jdocs/serialization/SerializationDocTest.java - :include: my-own-serializer - :exclude: ... +@@snip [SerializationDocTest.java](code/jdocs/serialization/SerializationDocTest.java) { #my-own-serializer } The manifest is a type hint so that the same serializer can be used for different -classes. The manifest parameter in ``fromBinaryJava`` is the class of the object that -was serialized. In ``fromBinary`` you can match on the class and deserialize the +classes. The manifest parameter in `fromBinaryJava` is the class of the object that +was serialized. In `fromBinary` you can match on the class and deserialize the bytes to different objects. -Then you only need to fill in the blanks, bind it to a name in your :ref:`configuration` and then +Then you only need to fill in the blanks, bind it to a name in your [Configuration]() and then list which classes that should be serialized using it. -.. _string-manifest-serializer-java: + +### Serializer with String Manifest -Serializer with String Manifest -------------------------------- +The `Serializer` illustrated above supports a class based manifest (type hint). +For serialization of data that need to evolve over time the `SerializerWithStringManifest` +is recommended instead of `Serializer` because the manifest (type hint) is a `String` +instead of a `Class`. That means that the class can be moved/removed and the serializer +can still deserialize old data by matching on the `String`. This is especially useful +for @ref:[Persistence](persistence.md). -The ``Serializer`` illustrated above supports a class based manifest (type hint). -For serialization of data that need to evolve over time the ``SerializerWithStringManifest`` -is recommended instead of ``Serializer`` because the manifest (type hint) is a ``String`` -instead of a ``Class``. That means that the class can be moved/removed and the serializer -can still deserialize old data by matching on the ``String``. This is especially useful -for :ref:`persistence-java`. - -The manifest string can also encode a version number that can be used in ``fromBinary`` to +The manifest string can also encode a version number that can be used in `fromBinary` to deserialize in different ways to migrate old data to new domain objects. -If the data was originally serialized with ``Serializer`` and in a later version of the -system you change to ``SerializerWithStringManifest`` the manifest string will be the full -class name if you used ``includeManifest=true``, otherwise it will be the empty string. +If the data was originally serialized with `Serializer` and in a later version of the +system you change to `SerializerWithStringManifest` the manifest string will be the full +class name if you used `includeManifest=true`, otherwise it will be the empty string. -This is how a ``SerializerWithStringManifest`` looks like: +This is how a `SerializerWithStringManifest` looks like: -.. includecode:: code/jdocs/serialization/SerializationDocTest.java#my-own-serializer2 +@@snip [SerializationDocTest.java](code/jdocs/serialization/SerializationDocTest.java) { #my-own-serializer2 } -You must also bind it to a name in your :ref:`configuration` and then list which classes +You must also bind it to a name in your [Configuration]() and then list which classes that should be serialized using it. -It's recommended to throw ``java.io.NotSerializableException`` in ``fromBinary`` +It's recommended to throw `java.io.NotSerializableException` in `fromBinary` if the manifest is unknown. This makes it possible to introduce new message types and send them to nodes that don't know about them. This is typically needed when performing rolling upgrades, i.e. running a cluster with mixed versions for while. -``NotSerializableException`` is treated as a transient problem in the TCP based remoting +`NotSerializableException` is treated as a transient problem in the TCP based remoting layer. The problem will be logged and message is dropped. Other exceptions will tear down the TCP connection because it can be an indication of corrupt bytes from the underlying transport. - -Serializing ActorRefs ---------------------- +### Serializing ActorRefs All ActorRefs are serializable using JavaSerializer, but in case you are writing your own serializer, you might want to know how to serialize and deserialize them properly. In the general case, the local address to be used depends on the type of remote address which shall be the recipient of the serialized information. Use -:meth:`Serialization.serializedActorPath(actorRef)` like this: +`Serialization.serializedActorPath(actorRef)` like this: -.. includecode:: code/jdocs/serialization/SerializationDocTest.java - :include: imports +@@snip [SerializationDocTest.java](code/jdocs/serialization/SerializationDocTest.java) { #imports } -.. includecode:: code/jdocs/serialization/SerializationDocTest.java - :include: actorref-serializer +@@snip [SerializationDocTest.java](code/jdocs/serialization/SerializationDocTest.java) { #actorref-serializer } This assumes that serialization happens in the context of sending a message through the remote transport. There are other uses of serialization, though, @@ -170,72 +152,65 @@ in the same logical context, but it is not enough when deserializing it on a different network host: for that it would need to include the system’s remote transport address. An actor system is not limited to having just one remote transport per se, which makes this question a bit more interesting. To find out -the appropriate address to use when sending to ``remoteAddr`` you can use -:meth:`ActorRefProvider.getExternalAddressFor(remoteAddr)` like this: +the appropriate address to use when sending to `remoteAddr` you can use +`ActorRefProvider.getExternalAddressFor(remoteAddr)` like this: -.. includecode:: code/jdocs/serialization/SerializationDocTest.java - :include: external-address +@@snip [SerializationDocTest.java](code/jdocs/serialization/SerializationDocTest.java) { #external-address } -.. note:: - - ``ActorPath.toSerializationFormatWithAddress`` differs from ``toString`` if the - address does not already have ``host`` and ``port`` components, i.e. it only - inserts address information for local addresses. - - ``toSerializationFormatWithAddress`` also adds the unique id of the actor, which will - change when the actor is stopped and then created again with the same name. - Sending messages to a reference pointing the old actor will not be delivered - to the new actor. If you do not want this behavior, e.g. in case of long term - storage of the reference, you can use ``toStringWithAddress``, which does not - include the unique id. +@@@ note +`ActorPath.toSerializationFormatWithAddress` differs from `toString` if the +address does not already have `host` and `port` components, i.e. it only +inserts address information for local addresses. + +`toSerializationFormatWithAddress` also adds the unique id of the actor, which will +change when the actor is stopped and then created again with the same name. +Sending messages to a reference pointing the old actor will not be delivered +to the new actor. If you do not want this behavior, e.g. in case of long term +storage of the reference, you can use `toStringWithAddress`, which does not +include the unique id. + +@@@ This requires that you know at least which type of address will be supported by the system which will deserialize the resulting actor reference; if you have no concrete address handy you can create a dummy one for the right protocol using -``new Address(protocol, "", "", 0)`` (assuming that the actual transport used is as +`new Address(protocol, "", "", 0)` (assuming that the actual transport used is as lenient as Akka’s RemoteActorRefProvider). There is also a default remote address which is the one used by cluster support (and typical systems have just this one); you can get it like this: -.. includecode:: code/jdocs/serialization/SerializationDocTest.java - :include: external-address-default +@@snip [SerializationDocTest.java](code/jdocs/serialization/SerializationDocTest.java) { #external-address-default } -Deep serialization of Actors ----------------------------- +### Deep serialization of Actors -The recommended approach to do deep serialization of internal actor state is to use Akka :ref:`persistence-java`. +The recommended approach to do deep serialization of internal actor state is to use Akka @ref:[Persistence](persistence.md). -A Word About Java Serialization -=============================== +## A Word About Java Serialization -When using Java serialization without employing the :class:`JavaSerializer` for -the task, you must make sure to supply a valid :class:`ExtendedActorSystem` in -the dynamic variable ``JavaSerializer.currentSystem``. This is used when -reading in the representation of an :class:`ActorRef` for turning the string -representation into a real reference. :class:`DynamicVariable` is a +When using Java serialization without employing the `JavaSerializer` for +the task, you must make sure to supply a valid `ExtendedActorSystem` in +the dynamic variable `JavaSerializer.currentSystem`. This is used when +reading in the representation of an `ActorRef` for turning the string +representation into a real reference. `DynamicVariable` is a thread-local variable, so be sure to have it set while deserializing anything which might contain actor references. -Serialization compatibility -=========================== +## Serialization compatibility It is not safe to mix major Scala versions when using the Java serialization as Scala does not guarantee compatibility and this could lead to very surprising errors. -If using the Akka Protobuf serializers (implicitly with ``akka.actor.allow-java-serialization = off`` or explicitly with -``enable-additional-serialization-bindings = true``) for the internal Akka messages those will not require the same major +If using the Akka Protobuf serializers (implicitly with `akka.actor.allow-java-serialization = off` or explicitly with +`enable-additional-serialization-bindings = true`) for the internal Akka messages those will not require the same major Scala version however you must also ensure the serializers used for your own types does not introduce the same incompatibility as Java serialization does. -External Akka Serializers -========================= +## External Akka Serializers -`Akka-quickser by Roman Levenstein `_ +[Akka-quickser by Roman Levenstein](https://github.com/romix/akka-quickser-serialization) +[Akka-kryo by Roman Levenstein](https://github.com/romix/akka-kryo-serialization) -`Akka-kryo by Roman Levenstein `_ - - -`Twitter Chill Scala extensions for Kryo (based on Akka Version 2.3.x but due to backwards compatibility of the Serializer Interface this extension also works with 2.4.x) `_ +[Twitter Chill Scala extensions for Kryo (based on Akka Version 2.3.x but due to backwards compatibility of the Serializer Interface this extension also works with 2.4.x)](https://github.com/twitter/chill) \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/index.md b/akka-docs/src/main/paradox/java/stream/index.md index 16ed4ee2ee..c47a6edee6 100644 --- a/akka-docs/src/main/paradox/java/stream/index.md +++ b/akka-docs/src/main/paradox/java/stream/index.md @@ -1,25 +1,25 @@ -.. _streams-java: +# Streams -Streams -======= +@@toc { depth=2 } -.. toctree:: - :maxdepth: 2 +@@@ index - stream-introduction - stream-quickstart - ../../general/stream/stream-design - stream-flows-and-basics - stream-graphs - stream-composition - stream-rate - stream-dynamic - stream-customize - stream-integrations - stream-error - stream-io - stream-parallelism - stream-testkit - stages-overview - stream-cookbook - ../../general/stream/stream-configuration +* [stream-introduction](stream-introduction.md) +* [stream-quickstart](stream-quickstart.md) +* [../../general/stream/stream-design](../../general/stream/stream-design.md) +* [stream-flows-and-basics](stream-flows-and-basics.md) +* [stream-graphs](stream-graphs.md) +* [stream-composition](stream-composition.md) +* [stream-rate](stream-rate.md) +* [stream-dynamic](stream-dynamic.md) +* [stream-customize](stream-customize.md) +* [stream-integrations](stream-integrations.md) +* [stream-error](stream-error.md) +* [stream-io](stream-io.md) +* [stream-parallelism](stream-parallelism.md) +* [stream-testkit](stream-testkit.md) +* [stages-overview](stages-overview.md) +* [stream-cookbook](stream-cookbook.md) +* [../../general/stream/stream-configuration](../../general/stream/stream-configuration.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stages-overview.md b/akka-docs/src/main/paradox/java/stream/stages-overview.md index 3cd4c6e154..2bfd342354 100644 --- a/akka-docs/src/main/paradox/java/stream/stages-overview.md +++ b/akka-docs/src/main/paradox/java/stream/stages-overview.md @@ -1,19 +1,13 @@ -.. _stages-overview_java: +# Overview of built-in stages and their semantics -Overview of built-in stages and their semantics -=============================================== +## Source stages +These built-in sources are available from `akka.stream.javadsl.Source`: -Source stages -------------- -These built-in sources are available from ``akka.stream.javadsl.Source``: +### fromIterator - - -fromIterator -^^^^^^^^^^^^ -Stream the values from an ``Iterator``, requesting the next value when there is demand. The iterator will be created anew on -each materialization of the source which is the reason the factory takes a ``Creator`` rather than an ``Iterator`` directly. +Stream the values from an `Iterator`, requesting the next value when there is demand. The iterator will be created anew on +each materialization of the source which is the reason the factory takes a `Creator` rather than an `Iterator` directly. If the iterator perform blocking operations, make sure to run it on a separate dispatcher. @@ -21,34 +15,33 @@ If the iterator perform blocking operations, make sure to run it on a separate d **completes** when the iterator reaches its end -from -^^^^ -Stream the values of an ``Iterable``. Make sure the ``Iterable`` is immutable or at least not modified after being used +### from + +Stream the values of an `Iterable`. Make sure the `Iterable` is immutable or at least not modified after being used as a source. **emits** the next value of the iterable **completes** after the last element of the iterable has been emitted +### single -single -^^^^^^ Stream a single object **emits** the value once **completes** when the single value has been emitted -repeat -^^^^^^ +### repeat + Stream a single object repeatedly **emits** the same value repeatedly when there is demand **completes** never -cycle -^^^^^ +### cycle + Stream iterator in cycled manner. Internally new iterator is being created to cycle the one provided via argument meaning when original iterator runs out of elements process will start all over again from the beginning of the iterator provided by the evaluation of provided parameter. If method argument provides empty iterator stream will be terminated with @@ -58,8 +51,8 @@ exception. **completes** never -tick -^^^^ +### tick + A periodical repetition of an arbitrary object. Delay of first tick is specified separately from interval of the following ticks. @@ -67,67 +60,67 @@ separately from interval of the following ticks. **completes** never -fromCompletionStage -^^^^^^^^^^^^^^^^^^^ -Send the single value of the ``CompletionStage`` when it completes and there is demand. -If the ``CompletionStage`` fails the stream is failed with that exception. +### fromCompletionStage -**emits** when the ``CompletionStage`` completes +Send the single value of the `CompletionStage` when it completes and there is demand. +If the `CompletionStage` fails the stream is failed with that exception. -**completes** after the ``CompletionStage`` has completed or when it fails +**emits** when the `CompletionStage` completes -fromFuture -^^^^^^^^^^ -Send the single value of the Scala ``Future`` when it completes and there is demand. +**completes** after the `CompletionStage` has completed or when it fails + +### fromFuture + +Send the single value of the Scala `Future` when it completes and there is demand. If the future fails the stream is failed with that exception. **emits** the future completes **completes** after the future has completed -fromFutureSource -^^^^^^^^^^^^^^^^ +### fromFutureSource + Streams the elements of the given future source once it successfully completes. If the future fails the stream is failed. -**emits** the next value from the `future` source, once it has completed +**emits** the next value from the *future* source, once it has completed -**completes** after the `future` source completes +**completes** after the *future* source completes -fromSourceCompletionStage -^^^^^^^^^^^^^^^^^^^^^^^^^ -Streams the elements of an asynchronous source once its given `completion` stage completes. -If the `completion` fails the stream is failed with that exception. +### fromSourceCompletionStage -**emits** the next value from the asynchronous source, once its `completion stage` has completed +Streams the elements of an asynchronous source once its given *completion* stage completes. +If the *completion* fails the stream is failed with that exception. + +**emits** the next value from the asynchronous source, once its *completion stage* has completed **completes** after the asynchronous source completes -unfold -^^^^^^ -Stream the result of a function as long as it returns a ``Optional``, the value inside the optional -consists of a pair where the first value is a state passed back into the next call to the function allowing -to pass a state. The first invocation of the provided fold function will receive the ``zero`` state. +### unfold -Can be used to implement many stateful sources without having to touch the more low level ``GraphStage`` API. +Stream the result of a function as long as it returns a `Optional`, the value inside the optional +consists of a pair where the first value is a state passed back into the next call to the function allowing +to pass a state. The first invocation of the provided fold function will receive the `zero` state. + +Can be used to implement many stateful sources without having to touch the more low level `GraphStage` API. **emits** when there is demand and the unfold function over the previous state returns non empty value **completes** when the unfold function returns an empty value -unfoldAsync -^^^^^^^^^^^ -Just like ``unfold`` but the fold function returns a ``CompletionStage`` which will cause the source to +### unfoldAsync + +Just like `unfold` but the fold function returns a `CompletionStage` which will cause the source to complete or emit when it completes. -Can be used to implement many stateful sources without having to touch the more low level ``GraphStage`` API. +Can be used to implement many stateful sources without having to touch the more low level `GraphStage` API. **emits** when there is demand and unfold state returned CompletionStage completes with some value **completes** when the CompletionStage returned by the unfold function completes with an empty value -empty -^^^^^ +### empty + Complete right away without ever emitting any elements. Useful when you have to provide a source to an API but there are no elements to emit. @@ -135,167 +128,161 @@ an API but there are no elements to emit. **completes** directly -maybe -^^^^^ -Materialize a ``CompletionStage`` that can be completed with an ``Optional``. -If it is completed with a value it will be eimitted from the source if it is an empty ``Optional`` it will +### maybe + +Materialize a `CompletionStage` that can be completed with an `Optional`. +If it is completed with a value it will be eimitted from the source if it is an empty `Optional` it will complete directly. **emits** when the returned promise is completed with some value **completes** after emitting some value, or directly if the promise is completed with no value -failed -^^^^^^ +### failed + Fail directly with a user specified exception. **emits** never **completes** fails the stream directly with the given exception -lazily -~~~~~~ -Defers creation and materialization of a ``Source`` until there is demand. +#### lazily -**emits** depends on the wrapped ``Source`` +Defers creation and materialization of a `Source` until there is demand. -**completes** depends on the wrapped ``Source`` +**emits** depends on the wrapped `Source` -actorPublisher -^^^^^^^^^^^^^^ -Wrap an actor extending ``ActorPublisher`` as a source. +**completes** depends on the wrapped `Source` + +### actorPublisher + +Wrap an actor extending `ActorPublisher` as a source. **emits** depends on the actor implementation **completes** when the actor stops -actorRef -^^^^^^^^ -Materialize an ``ActorRef``, sending messages to it will emit them on the stream. The actor contain +### actorRef + +Materialize an `ActorRef`, sending messages to it will emit them on the stream. The actor contain a buffer but since communication is one way, there is no back pressure. Handling overflow is done by either dropping elements or failing the stream, the strategy is chosen by the user. **emits** when there is demand and there are messages in the buffer or a message is sent to the actorref -**completes** when the ``ActorRef`` is sent ``akka.actor.Status.Success`` or ``PoisonPill`` +**completes** when the `ActorRef` is sent `akka.actor.Status.Success` or `PoisonPill` + +### combine -combine -^^^^^^^ Combine several sources, using a given strategy such as merge or concat, into one source. **emits** when there is demand, but depending on the strategy **completes** when all sources has completed +### range -range -^^^^^ Emit each integer in a range, with an option to take bigger steps than 1. **emits** when there is demand, the next value **completes** when the end of the range has been reached -unfoldResource -^^^^^^^^^^^^^^ +### unfoldResource + Wrap any resource that can be opened, queried for next element (in a blocking way) and closed using three distinct functions into a source. **emits** when there is demand and read method returns value -**completes** when read function returns ``None`` +**completes** when read function returns `None` + +### unfoldAsyncResource -unfoldAsyncResource -^^^^^^^^^^^^^^^^^^^ Wrap any resource that can be opened, queried for next element and closed using three distinct functions into a source. -Functions return ``CompletionStage`` result to achieve asynchronous processing +Functions return `CompletionStage` result to achieve asynchronous processing -**emits** when there is demand and ``CompletionStage`` from read function returns value +**emits** when there is demand and `CompletionStage` from read function returns value -**completes** when ``CompletionStage`` from read function returns ``None`` +**completes** when `CompletionStage` from read function returns `None` -queue -^^^^^ -Materialize a ``SourceQueue`` onto which elements can be pushed for emitting from the source. The queue contains +### queue + +Materialize a `SourceQueue` onto which elements can be pushed for emitting from the source. The queue contains a buffer, if elements are pushed onto the queue faster than the source is consumed the overflow will be handled with a strategy specified by the user. Functionality for tracking when an element has been emitted is available through -``SourceQueue.offer``. +`SourceQueue.offer`. **emits** when there is demand and the queue contains elements **completes** when downstream completes -asSubscriber -^^^^^^^^^^^^ -Integration with Reactive Streams, materializes into a ``org.reactivestreams.Subscriber``. +### asSubscriber +Integration with Reactive Streams, materializes into a `org.reactivestreams.Subscriber`. -fromPublisher -^^^^^^^^^^^^^ -Integration with Reactive Streams, subscribes to a ``org.reactivestreams.Publisher``. +### fromPublisher + +Integration with Reactive Streams, subscribes to a `org.reactivestreams.Publisher`. + +### zipN -zipN -^^^^ Combine the elements of multiple streams into a stream of sequences. **emits** when all of the inputs has an element available **completes** when any upstream completes -zipWithN -^^^^^^^^ +### zipWithN + Combine the elements of multiple streams into a stream of sequences using a combiner function. **emits** when all of the inputs has an element available **completes** when any upstream completes +## Sink stages +These built-in sinks are available from `akka.stream.javadsl.Sink`: +### head -Sink stages ------------ -These built-in sinks are available from ``akka.stream.javadsl.Sink``: - - -head -^^^^ -Materializes into a ``CompletionStage`` which completes with the first value arriving, +Materializes into a `CompletionStage` which completes with the first value arriving, after this the stream is canceled. If no element is emitted, the CompletionStage is be failed. **cancels** after receiving one element **backpressures** never -headOption -^^^^^^^^^^ -Materializes into a ``CompletionStage>`` which completes with the first value arriving wrapped in optional, +### headOption + +Materializes into a `CompletionStage>` which completes with the first value arriving wrapped in optional, or an empty optional if the stream completes without any elements emitted. **cancels** after receiving one element **backpressures** never -last -^^^^ -Materializes into a ``CompletionStage`` which will complete with the last value emitted when the stream +### last + +Materializes into a `CompletionStage` which will complete with the last value emitted when the stream completes. If the stream completes with no elements the CompletionStage is failed. **cancels** never **backpressures** never -lastOption -^^^^^^^^^^ -Materialize a ``CompletionStage>`` which completes with the last value -emitted wrapped in an optional when the stream completes. if the stream completes with no elements the ``CompletionStage`` is +### lastOption + +Materialize a `CompletionStage>` which completes with the last value +emitted wrapped in an optional when the stream completes. if the stream completes with no elements the `CompletionStage` is completed with an empty optional. **cancels** never **backpressures** never -ignore -^^^^^^ +### ignore + Consume all elements but discards them. Useful when a stream has to be consumed but there is no use to actually do anything with the elements. @@ -303,25 +290,25 @@ do anything with the elements. **backpressures** never -cancelled -^^^^^^^^^ +### cancelled + Immediately cancel the stream **cancels** immediately -seq -^^^ -Collect values emitted from the stream into a collection, the collection is available through a ``CompletionStage`` or -which completes when the stream completes. Note that the collection is bounded to ``Integer.MAX_VALUE``, +### seq + +Collect values emitted from the stream into a collection, the collection is available through a `CompletionStage` or +which completes when the stream completes. Note that the collection is bounded to `Integer.MAX_VALUE`, if more element are emitted the sink will cancel the stream **cancels** If too many values are collected -foreach -^^^^^^^ +### foreach + Invoke a given procedure for each element received. Note that it is not safe to mutate shared state from the procedure. -The sink materializes into a ``CompletionStage>`` which completes when the +The sink materializes into a `CompletionStage>` which completes when the stream completes, or fails if the stream fails. Note that it is not safe to mutate state from the procedure. @@ -330,46 +317,44 @@ Note that it is not safe to mutate state from the procedure. **backpressures** when the previous procedure invocation has not yet completed +### foreachParallel -foreachParallel -^^^^^^^^^^^^^^^ -Like ``foreach`` but allows up to ``parallellism`` procedure calls to happen in parallel. +Like `foreach` but allows up to `parallellism` procedure calls to happen in parallel. **cancels** never **backpressures** when the previous parallel procedure invocations has not yet completed +### onComplete -onComplete -^^^^^^^^^^ Invoke a callback when the stream has completed or failed. **cancels** never **backpressures** never -lazyInit -^^^^^^^^ -Invoke sinkFactory function to create a real sink upon receiving the first element. Internal ``Sink`` will not be created if there are no elements, -because of completion or error. `fallback` will be invoked if there was no elements and completed is received from upstream. +### lazyInit + +Invoke sinkFactory function to create a real sink upon receiving the first element. Internal `Sink` will not be created if there are no elements, +because of completion or error. *fallback* will be invoked if there was no elements and completed is received from upstream. **cancels** never **backpressures** when initialized and when created sink backpressures -queue -^^^^^ -Materialize a ``SinkQueue`` that can be pulled to trigger demand through the sink. The queue contains +### queue + +Materialize a `SinkQueue` that can be pulled to trigger demand through the sink. The queue contains a buffer in case stream emitting elements faster than queue pulling them. -**cancels** when ``SinkQueue.cancel`` is called +**cancels** when `SinkQueue.cancel` is called **backpressures** when buffer has some space -fold -^^^^ +### fold + Fold over emitted element with a function, where each invocation will get the new element and the result from the -previous fold invocation. The first invocation will be provided the ``zero`` value. +previous fold invocation. The first invocation will be provided the `zero` value. Materializes into a CompletionStage that will complete with the last state when the stream has completed. @@ -380,8 +365,8 @@ between invocations. **backpressures** when the previous fold function invocation has not yet completed -reduce -^^^^^^ +### reduce + Apply a reduction function on the incoming elements and pass the result to the next invocation. The first invocation receives the two first elements of the flow. @@ -391,163 +376,151 @@ Materializes into a CompletionStage that will be completed by the last result of **backpressures** when the previous reduction function invocation has not yet completed +### combine -combine -^^^^^^^ Combine several sinks into one using a user specified strategy **cancels** depends on the strategy **backpressures** depends on the strategy +### actorRef -actorRef -^^^^^^^^ -Send the elements from the stream to an ``ActorRef``. No backpressure so care must be taken to not overflow the inbox. +Send the elements from the stream to an `ActorRef`. No backpressure so care must be taken to not overflow the inbox. **cancels** when the actor terminates **backpressures** never +### actorRefWithAck -actorRefWithAck -^^^^^^^^^^^^^^^ -Send the elements from the stream to an ``ActorRef`` which must then acknowledge reception after completing a message, +Send the elements from the stream to an `ActorRef` which must then acknowledge reception after completing a message, to provide back pressure onto the sink. **cancels** when the actor terminates **backpressures** when the actor acknowledgement has not arrived +### actorSubscriber -actorSubscriber -^^^^^^^^^^^^^^^ -Create an actor from a ``Props`` upon materialization, where the actor implements ``ActorSubscriber``, which will +Create an actor from a `Props` upon materialization, where the actor implements `ActorSubscriber`, which will receive the elements from the stream. -Materializes into an ``ActorRef`` to the created actor. +Materializes into an `ActorRef` to the created actor. **cancels** when the actor terminates **backpressures** depends on the actor implementation +### asPublisher -asPublisher -^^^^^^^^^^^ -Integration with Reactive Streams, materializes into a ``org.reactivestreams.Publisher``. +Integration with Reactive Streams, materializes into a `org.reactivestreams.Publisher`. +### fromSubscriber -fromSubscriber -^^^^^^^^^^^^^^ -Integration with Reactive Streams, wraps a ``org.reactivestreams.Subscriber`` as a sink +Integration with Reactive Streams, wraps a `org.reactivestreams.Subscriber` as a sink +## Additional Sink and Source converters +Sources and sinks for integrating with `java.io.InputStream` and `java.io.OutputStream` can be found on +`StreamConverters`. As they are blocking APIs the implementations of these stages are run on a separate +dispatcher configured through the `akka.stream.blocking-io-dispatcher`. +### fromOutputStream -Additional Sink and Source converters -------------------------------------- -Sources and sinks for integrating with ``java.io.InputStream`` and ``java.io.OutputStream`` can be found on -``StreamConverters``. As they are blocking APIs the implementations of these stages are run on a separate -dispatcher configured through the ``akka.stream.blocking-io-dispatcher``. +Create a sink that wraps an `OutputStream`. Takes a function that produces an `OutputStream`, when the sink is +materialized the function will be called and bytes sent to the sink will be written to the returned `OutputStream`. -fromOutputStream -^^^^^^^^^^^^^^^^ -Create a sink that wraps an ``OutputStream``. Takes a function that produces an ``OutputStream``, when the sink is -materialized the function will be called and bytes sent to the sink will be written to the returned ``OutputStream``. - -Materializes into a ``CompletionStage`` which will complete with a ``IOResult`` when the stream +Materializes into a `CompletionStage` which will complete with a `IOResult` when the stream completes. -Note that a flow can be materialized multiple times, so the function producing the ``OutputStream`` must be able +Note that a flow can be materialized multiple times, so the function producing the `OutputStream` must be able to handle multiple invocations. -The ``OutputStream`` will be closed when the stream that flows into the ``Sink`` is completed, and the ``Sink`` -will cancel its inflow when the ``OutputStream`` is no longer writable. +The `OutputStream` will be closed when the stream that flows into the `Sink` is completed, and the `Sink` +will cancel its inflow when the `OutputStream` is no longer writable. -asInputStream -^^^^^^^^^^^^^ -Create a sink which materializes into an ``InputStream`` that can be read to trigger demand through the sink. -Bytes emitted through the stream will be available for reading through the ``InputStream`` +### asInputStream -The ``InputStream`` will be ended when the stream flowing into this ``Sink`` completes, and the closing the -``InputStream`` will cancel the inflow of this ``Sink``. +Create a sink which materializes into an `InputStream` that can be read to trigger demand through the sink. +Bytes emitted through the stream will be available for reading through the `InputStream` -fromInputStream -^^^^^^^^^^^^^^^ -Create a source that wraps an ``InputStream``. Takes a function that produces an ``InputStream``, when the source is -materialized the function will be called and bytes from the ``InputStream`` will be emitted into the stream. +The `InputStream` will be ended when the stream flowing into this `Sink` completes, and the closing the +`InputStream` will cancel the inflow of this `Sink`. -Materializes into a ``CompletionStage`` which will complete with a ``IOResult`` when the stream +### fromInputStream + +Create a source that wraps an `InputStream`. Takes a function that produces an `InputStream`, when the source is +materialized the function will be called and bytes from the `InputStream` will be emitted into the stream. + +Materializes into a `CompletionStage` which will complete with a `IOResult` when the stream completes. -Note that a flow can be materialized multiple times, so the function producing the ``InputStream`` must be able +Note that a flow can be materialized multiple times, so the function producing the `InputStream` must be able to handle multiple invocations. -The ``InputStream`` will be closed when the ``Source`` is canceled from its downstream, and reaching the end of the -``InputStream`` will complete the ``Source``. +The `InputStream` will be closed when the `Source` is canceled from its downstream, and reaching the end of the +`InputStream` will complete the `Source`. -asOutputStream -^^^^^^^^^^^^^^ -Create a source that materializes into an ``OutputStream``. When bytes are written to the ``OutputStream`` they +### asOutputStream + +Create a source that materializes into an `OutputStream`. When bytes are written to the `OutputStream` they are emitted from the source. -The ``OutputStream`` will no longer be writable when the ``Source`` has been canceled from its downstream, and -closing the ``OutputStream`` will complete the ``Source``. +The `OutputStream` will no longer be writable when the `Source` has been canceled from its downstream, and +closing the `OutputStream` will complete the `Source`. -asJavaStream -^^^^^^^^^^^^ -Create a sink which materializes into Java 8 ``Stream`` that can be run to trigger demand through the sink. -Elements emitted through the stream will be available for reading through the Java 8 ``Stream``. +### asJavaStream -The Java 8 a ``Stream`` will be ended when the stream flowing into this ``Sink`` completes, and closing the Java -``Stream`` will cancel the inflow of this ``Sink``. Java ``Stream`` throws exception in case reactive stream failed. +Create a sink which materializes into Java 8 `Stream` that can be run to trigger demand through the sink. +Elements emitted through the stream will be available for reading through the Java 8 `Stream`. -Be aware that Java 8 ``Stream`` blocks current thread while waiting on next element from downstream. +The Java 8 a `Stream` will be ended when the stream flowing into this `Sink` completes, and closing the Java +`Stream` will cancel the inflow of this `Sink`. Java `Stream` throws exception in case reactive stream failed. -fromJavaStream -^^^^^^^^^^^^^^ -Create a source that wraps Java 8 ``Stream``. ``Source`` uses a stream iterator to get all its elements and send them +Be aware that Java 8 `Stream` blocks current thread while waiting on next element from downstream. + +### fromJavaStream + +Create a source that wraps Java 8 `Stream`. `Source` uses a stream iterator to get all its elements and send them downstream on demand. -javaCollector -^^^^^^^^^^^^^ -Create a sink which materializes into a ``CompletionStage`` which will be completed with a result of the Java 8 ``Collector`` -transformation and reduction operations. This allows usage of Java 8 streams transformations for reactive streams. -The ``Collector`` will trigger demand downstream. Elements emitted through the stream will be accumulated into a mutable -result container, optionally transformed into a final representation after all input elements have been processed. -The ``Collector`` can also do reduction at the end. Reduction processing is performed sequentially +### javaCollector -Note that a flow can be materialized multiple times, so the function producing the ``Collector`` must be able +Create a sink which materializes into a `CompletionStage` which will be completed with a result of the Java 8 `Collector` +transformation and reduction operations. This allows usage of Java 8 streams transformations for reactive streams. +The `Collector` will trigger demand downstream. Elements emitted through the stream will be accumulated into a mutable +result container, optionally transformed into a final representation after all input elements have been processed. +The `Collector` can also do reduction at the end. Reduction processing is performed sequentially + +Note that a flow can be materialized multiple times, so the function producing the `Collector` must be able to handle multiple invocations. -javaCollectorParallelUnordered -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Create a sink which materializes into a ``CompletionStage`` which will be completed with a result of the Java 8 Collector -transformation and reduction operations. This allows usage of Java 8 streams transformations for reactive streams. -The ``Collector`` will trigger demand downstream.. Elements emitted through the stream will be accumulated into a mutable -result container, optionally transformed into a final representation after all input elements have been processed. -The ``Collector`` can also do reduction at the end. Reduction processing is performed in parallel based on graph ``Balance``. +### javaCollectorParallelUnordered -Note that a flow can be materialized multiple times, so the function producing the ``Collector`` must be able +Create a sink which materializes into a `CompletionStage` which will be completed with a result of the Java 8 Collector +transformation and reduction operations. This allows usage of Java 8 streams transformations for reactive streams. +The `Collector` will trigger demand downstream.. Elements emitted through the stream will be accumulated into a mutable +result container, optionally transformed into a final representation after all input elements have been processed. +The `Collector` can also do reduction at the end. Reduction processing is performed in parallel based on graph `Balance`. + +Note that a flow can be materialized multiple times, so the function producing the `Collector` must be able to handle multiple invocations. -File IO Sinks and Sources -------------------------- -Sources and sinks for reading and writing files can be found on ``FileIO``. +## File IO Sinks and Sources -fromPath -^^^^^^^^ -Emit the contents of a file, as ``ByteString`` s, materializes into a ``CompletionStage`` which will be completed with -a ``IOResult`` upon reaching the end of the file or if there is a failure. +Sources and sinks for reading and writing files can be found on `FileIO`. -toPath -^^^^^^ -Create a sink which will write incoming ``ByteString`` s to a given file path. +### fromPath +Emit the contents of a file, as `ByteString` s, materializes into a `CompletionStage` which will be completed with +a `IOResult` upon reaching the end of the file or if there is a failure. +### toPath -Flow stages ------------ +Create a sink which will write incoming `ByteString` s to a given file path. + +## Flow stages All flows by default backpressure if the computation they encapsulate is not fast enough to keep up with the rate of incoming elements from the preceding stage. There are differences though how the different stages handle when some of @@ -560,18 +533,16 @@ For in-band error handling of normal errors (dropping elements if a map fails fo supervision support, or explicitly wrap your element types in a proper container that can express error or success states. - -Simple processing stages ------------------------- +## Simple processing stages These stages can transform the rate of incoming elements since there are stages that emit multiple elements for a -single input (e.g. `mapConcat') or consume multiple elements before emitting one output (e.g. ``filter``). +single input (e.g. `mapConcat') or consume multiple elements before emitting one output (e.g. `filter`). However, these rate transformations are data-driven, i.e. it is the incoming elements that define how the -rate is affected. This is in contrast with :ref:`detached-stages-overview_java` which can change their processing behavior +rate is affected. This is in contrast with [detached-stages-overview_java](#detached-stages-overview-java) which can change their processing behavior depending on being backpressured by downstream or not. -map -^^^ +### map + Transform each element in the stream by calling a mapping function with it and passing the returned value downstream. **emits** when the mapping function returns an element @@ -580,8 +551,8 @@ Transform each element in the stream by calling a mapping function with it and p **completes** when upstream completes -mapConcat -^^^^^^^^^ +### mapConcat + Transform each element into zero or more elements that are individually passed downstream. **emits** when the mapping function returns an element or there are still remaining elements from the previously calculated collection @@ -590,9 +561,9 @@ Transform each element into zero or more elements that are individually passed d **completes** when upstream completes and all remaining elements has been emitted -statefulMapConcat -^^^^^^^^^^^^^^^^^ -Transform each element into zero or more elements that are individually passed downstream. The difference to ``mapConcat`` is that +### statefulMapConcat + +Transform each element into zero or more elements that are individually passed downstream. The difference to `mapConcat` is that the transformation function is created from a factory for every materialization of the flow. **emits** when the mapping function returns an element or there are still remaining elements from the previously calculated collection @@ -601,8 +572,8 @@ the transformation function is created from a factory for every materialization **completes** when upstream completes and all remaining elements has been emitted -filter -^^^^^^ +### filter + Filter the incoming elements using a predicate. If the predicate returns true the element is passed downstream, if it returns false the element is discarded. @@ -612,8 +583,8 @@ it returns false the element is discarded. **completes** when upstream completes -filterNot -^^^^^^^^^ +### filterNot + Filter the incoming elements using a predicate. If the predicate returns false the element is passed downstream, if it returns true the element is discarded. @@ -623,10 +594,10 @@ it returns true the element is discarded. **completes** when upstream completes -collect -^^^^^^^ +### collect + Apply a partial function to each incoming element, if the partial function is defined for a value the returned -value is passed downstream. Can often replace ``filter`` followed by ``map`` to achieve the same in one single stage. +value is passed downstream. Can often replace `filter` followed by `map` to achieve the same in one single stage. **emits** when the provided partial function is defined for the element @@ -634,8 +605,8 @@ value is passed downstream. Can often replace ``filter`` followed by ``map`` to **completes** when upstream completes -grouped -^^^^^^^ +### grouped + Accumulate incoming events until the specified number of elements have been accumulated and then pass the collection of elements downstream. @@ -645,8 +616,8 @@ elements downstream. **completes** when upstream completes -sliding -^^^^^^^ +### sliding + Provide a sliding window over the incoming stream and pass the windows as groups of elements downstream. Note: the last window might be smaller than the requested size due to end of stream. @@ -657,10 +628,9 @@ Note: the last window might be smaller than the requested size due to end of str **completes** when upstream completes +### scan -scan -^^^^ -Emit its current value which starts at ``zero`` and then applies the current and next value to the given function +Emit its current value which starts at `zero` and then applies the current and next value to the given function emitting the next current value. Note that this means that scan emits one element downstream before and upstream elements will not be requested until @@ -672,19 +642,19 @@ the second element is required from downstream. **completes** when upstream completes -scanAsync -^^^^^^^^^ -Just like ``scan`` but receiving a function that results in a ``CompletionStage`` to the next value. +### scanAsync -**emits** when the ``CompletionStage`` resulting from the function scanning the element resolves to the next value +Just like `scan` but receiving a function that results in a `CompletionStage` to the next value. + +**emits** when the `CompletionStage` resulting from the function scanning the element resolves to the next value **backpressures** when downstream backpressures -**completes** when upstream completes and the last ``CompletionStage`` is resolved +**completes** when upstream completes and the last `CompletionStage` is resolved -fold -^^^^ -Start with current value ``zero`` and then apply the current and next value to the given function, when upstream +### fold + +Start with current value `zero` and then apply the current and next value to the given function, when upstream complete the current value is emitted downstream. **emits** when upstream completes @@ -693,20 +663,20 @@ complete the current value is emitted downstream. **completes** when upstream completes -foldAsync -^^^^^^^^^ -Just like ``fold`` but receiving a function that results in a ``CompletionStage`` to the next value. +### foldAsync -**emits** when upstream completes and the last ``CompletionStage`` is resolved +Just like `fold` but receiving a function that results in a `CompletionStage` to the next value. + +**emits** when upstream completes and the last `CompletionStage` is resolved **backpressures** when downstream backpressures -**completes** when upstream completes and the last ``CompletionStage`` is resolved +**completes** when upstream completes and the last `CompletionStage` is resolved + +### reduce -reduce -^^^^^^ Start with first element and then apply the current and next value to the given function, when upstream -complete the current value is emitted downstream. Similar to ``fold``. +complete the current value is emitted downstream. Similar to `fold`. **emits** when upstream completes @@ -714,9 +684,9 @@ complete the current value is emitted downstream. Similar to ``fold``. **completes** when upstream completes -drop -^^^^ -Drop ``n`` elements and then pass any subsequent element downstream. +### drop + +Drop `n` elements and then pass any subsequent element downstream. **emits** when the specified number of elements has been dropped already @@ -724,9 +694,9 @@ Drop ``n`` elements and then pass any subsequent element downstream. **completes** when upstream completes -take -^^^^ -Pass ``n`` incoming elements downstream and then complete +### take + +Pass `n` incoming elements downstream and then complete **emits** while the specified number of elements to take has not yet been reached @@ -734,9 +704,8 @@ Pass ``n`` incoming elements downstream and then complete **completes** when the defined number of elements has been taken or upstream completes +### takeWhile -takeWhile -^^^^^^^^^ Pass elements downstream as long as a predicate function return true for the element include the element when the predicate first return false and then complete. @@ -746,8 +715,8 @@ when the predicate first return false and then complete. **completes** when predicate returned false or upstream completes -dropWhile -^^^^^^^^^ +### dropWhile + Drop elements as long as a predicate function return true for the element **emits** when the predicate returned false and for all following stream elements @@ -756,11 +725,11 @@ Drop elements as long as a predicate function return true for the element **completes** when upstream completes -recover -^^^^^^^ +### recover + Allow sending of one last element downstream when a failure has happened upstream. -Throwing an exception inside ``recover`` _will_ be logged on ERROR level automatically. +Throwing an exception inside `recover` _will_ be logged on ERROR level automatically. **emits** when the element is available from the upstream or upstream is failed and pf returns an element @@ -768,11 +737,11 @@ Throwing an exception inside ``recover`` _will_ be logged on ERROR level automat **completes** when upstream completes or upstream failed with exception pf can handle -recoverWith -^^^^^^^^^^^ +### recoverWith + Allow switching to alternative Source when a failure has happened upstream. -Throwing an exception inside ``recoverWith`` _will_ be logged on ERROR level automatically. +Throwing an exception inside `recoverWith` _will_ be logged on ERROR level automatically. **emits** the element is available from the upstream or upstream is failed and pf returns alternative Source @@ -780,12 +749,12 @@ Throwing an exception inside ``recoverWith`` _will_ be logged on ERROR level aut **completes** upstream completes or upstream failed with exception pf can handle -recoverWithRetries -^^^^^^^^^^^^^^^^^^ +### recoverWithRetries + RecoverWithRetries allows to switch to alternative Source on flow failure. It will stay in effect after -a failure has been recovered up to `attempts` number of times so that each time there is a failure -it is fed into the `pf` and a new Source may be materialized. Note that if you pass in 0, this won't -attempt to recover at all. Passing -1 will behave exactly the same as `recoverWith`. +a failure has been recovered up to *attempts* number of times so that each time there is a failure +it is fed into the *pf* and a new Source may be materialized. Note that if you pass in 0, this won't +attempt to recover at all. Passing -1 will behave exactly the same as *recoverWith*. Since the underlying failure signal onError arrives out-of-band, it might jump over existing elements. This stage can recover the failure signal, but not the skipped elements, which will be dropped. @@ -796,23 +765,23 @@ This stage can recover the failure signal, but not the skipped elements, which w **completes** when upstream completes or upstream failed with exception pf can handle -mapError -^^^^^^^^ -While similar to ``recover`` this stage can be used to transform an error signal to a different one *without* logging -it as an error in the process. So in that sense it is NOT exactly equivalent to ``recover(t -> throw t2)`` since recover -would log the ``t2`` error. +### mapError + +While similar to `recover` this stage can be used to transform an error signal to a different one *without* logging +it as an error in the process. So in that sense it is NOT exactly equivalent to `recover(t -> throw t2)` since recover +would log the `t2` error. Since the underlying failure signal onError arrives out-of-band, it might jump over existing elements. This stage can recover the failure signal, but not the skipped elements, which will be dropped. -Similarily to ``recover`` throwing an exception inside ``mapError`` _will_ be logged on ERROR level automatically. +Similarily to `recover` throwing an exception inside `mapError` _will_ be logged on ERROR level automatically. **emits** when element is available from the upstream or upstream is failed and pf returns an element **backpressures** when downstream backpressures **completes** when upstream completes or upstream failed with exception pf can handle -detach -^^^^^^ +### detach + Detach upstream demand from downstream demand without detaching the stream rates. **emits** when the upstream stage has emitted and there is demand @@ -821,8 +790,8 @@ Detach upstream demand from downstream demand without detaching the stream rates **completes** when upstream completes -throttle -^^^^^^^^ +### throttle + Limit the throughput to a specific number of elements per time unit, or a specific total cost per time unit, where a function has to be provided to calculate the individual cost of each element. @@ -832,19 +801,19 @@ a function has to be provided to calculate the individual cost of each element. **completes** when upstream completes -intersperse -^^^^^^^^^^^ -Intersperse stream with provided element similar to ``List.mkString``. It can inject start and end marker elements to stream. +### intersperse -**emits** when upstream emits an element or before with the `start` element if provided +Intersperse stream with provided element similar to `List.mkString`. It can inject start and end marker elements to stream. + +**emits** when upstream emits an element or before with the *start* element if provided **backpressures** when downstream backpressures **completes** when upstream completes -limit -^^^^^ -Limit number of element from upstream to given ``max`` number. +### limit + +Limit number of element from upstream to given `max` number. **emits** when upstream emits and the number of emitted elements has not reached max @@ -852,8 +821,8 @@ Limit number of element from upstream to given ``max`` number. **completes** when upstream completes and the number of emitted elements has not reached max -limitWeighted -^^^^^^^^^^^^^ +### limitWeighted + Ensure stream boundedness by evaluating the cost of incoming elements using a cost function. Evaluated cost of each element defines how many elements will be allowed to travel downstream. @@ -863,11 +832,11 @@ Evaluated cost of each element defines how many elements will be allowed to trav **completes** when upstream completes and the number of emitted elements has not reached max -log -^^^ +### log + Log elements flowing through the stream as well as completion and erroring. By default element and completion signals are logged on debug level, and errors are logged on Error level. -This can be changed by calling ``Attributes.createLogLevels(...)`` on the given Flow. +This can be changed by calling `Attributes.createLogLevels(...)` on the given Flow. **emits** when upstream emits @@ -875,9 +844,9 @@ This can be changed by calling ``Attributes.createLogLevels(...)`` on the given **completes** when upstream completes -recoverWithRetries -^^^^^^^^^^^^^^^^^^ -Switch to alternative Source on flow failure. It stays in effect after a failure has been recovered up to ``attempts`` +### recoverWithRetries + +Switch to alternative Source on flow failure. It stays in effect after a failure has been recovered up to `attempts` number of times. Each time a failure is fed into the partial function and a new Source may be materialized. **emits** when element is available from the upstream or upstream is failed and element is available from alternative Source @@ -886,29 +855,25 @@ number of times. Each time a failure is fed into the partial function and a new **completes** when upstream completes or upstream failed with exception partial function can handle +## Flow stages composed of Sinks and Sources -Flow stages composed of Sinks and Sources ------------------------------------------ +### Flow.fromSinkAndSource -Flow.fromSinkAndSource -^^^^^^^^^^^^^^^^^^^^^^ - -Creates a ``Flow`` from a ``Sink`` and a ``Source`` where the Flow's input will be sent to the ``Sink`` -and the ``Flow`` 's output will come from the Source. +Creates a `Flow` from a `Sink` and a `Source` where the Flow's input will be sent to the `Sink` +and the `Flow` 's output will come from the Source. Note that termination events, like completion and cancelation is not automatically propagated through to the "other-side" -of the such-composed Flow. Use ``CoupledTerminationFlow`` if you want to couple termination of both of the ends, +of the such-composed Flow. Use `CoupledTerminationFlow` if you want to couple termination of both of the ends, for example most useful in handling websocket connections. -CoupledTerminationFlow.fromSinkAndSource -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### CoupledTerminationFlow.fromSinkAndSource Allows coupling termination (cancellation, completion, erroring) of Sinks and Sources while creating a Flow them them. -Similar to ``Flow.fromSinkAndSource`` however that API does not connect the completion signals of the wrapped stages. +Similar to `Flow.fromSinkAndSource` however that API does not connect the completion signals of the wrapped stages. -Similar to ``Flow.fromSinkAndSource`` however couples the termination of these two stages. +Similar to `Flow.fromSinkAndSource` however couples the termination of these two stages. -E.g. if the emitted ``Flow`` gets a cancellation, the ``Source`` of course is cancelled, +E.g. if the emitted `Flow` gets a cancellation, the `Source` of course is cancelled, however the Sink will also be completed. The table below illustrates the effects in detail: +=================================================+=============================+=================================+ @@ -927,50 +892,46 @@ however the Sink will also be completed. The table below illustrates the effects | effect: cancels upstream, completes downstream | cause: cancels | effect: receives cancel | +=================================================+=============================+=================================+ -The order in which the `in` and `out` sides receive their respective completion signals is not defined, do not rely on its ordering. +The order in which the *in* and *out* sides receive their respective completion signals is not defined, do not rely on its ordering. -Asynchronous processing stages ------------------------------- +## Asynchronous processing stages These stages encapsulate an asynchronous computation, properly handling backpressure while taking care of the asynchronous operation at the same time (usually handling the completion of a CompletionStage). +### mapAsync -mapAsync -^^^^^^^^ -Pass incoming elements to a function that return a ``CompletionStage`` result. When the CompletionStage arrives the result is passed -downstream. Up to ``n`` elements can be processed concurrently, but regardless of their completion time the incoming -order will be kept when results complete. For use cases where order does not mather ``mapAsyncUnordered`` can be used. +Pass incoming elements to a function that return a `CompletionStage` result. When the CompletionStage arrives the result is passed +downstream. Up to `n` elements can be processed concurrently, but regardless of their completion time the incoming +order will be kept when results complete. For use cases where order does not mather `mapAsyncUnordered` can be used. -If a ``CompletionStage`` fails, the stream also fails (unless a different supervision strategy is applied) +If a `CompletionStage` fails, the stream also fails (unless a different supervision strategy is applied) **emits** when the CompletionStage returned by the provided function finishes for the next element in sequence -**backpressures** when the number of ``CompletionStage`` s reaches the configured parallelism and the downstream backpressures +**backpressures** when the number of `CompletionStage` s reaches the configured parallelism and the downstream backpressures -**completes** when upstream completes and all ``CompletionStage`` s has been completed and all elements has been emitted +**completes** when upstream completes and all `CompletionStage` s has been completed and all elements has been emitted -mapAsyncUnordered -^^^^^^^^^^^^^^^^^ -Like ``mapAsync`` but ``CompletionStage`` results are passed downstream as they arrive regardless of the order of the elements +### mapAsyncUnordered + +Like `mapAsync` but `CompletionStage` results are passed downstream as they arrive regardless of the order of the elements that triggered them. If a CompletionStage fails, the stream also fails (unless a different supervision strategy is applied) -**emits** any of the ``CompletionStage`` s returned by the provided function complete +**emits** any of the `CompletionStage` s returned by the provided function complete -**backpressures** when the number of ``CompletionStage`` s reaches the configured parallelism and the downstream backpressures +**backpressures** when the number of `CompletionStage` s reaches the configured parallelism and the downstream backpressures **completes** upstream completes and all CompletionStages has been completed and all elements has been emitted - -Timer driven stages -------------------- +## Timer driven stages These stages process elements using timers, delaying, dropping or grouping elements for certain time durations. -takeWithin -^^^^^^^^^^ +### takeWithin + Pass elements downstream within a timeout and then complete. **emits** when an upstream element arrives @@ -979,9 +940,8 @@ Pass elements downstream within a timeout and then complete. **completes** upstream completes or timer fires +### dropWithin -dropWithin -^^^^^^^^^^ Drop elements until a timeout has fired **emits** after the timer fired and a new upstream element arrives @@ -990,8 +950,8 @@ Drop elements until a timeout has fired **completes** upstream completes -groupedWithin -^^^^^^^^^^^^^ +### groupedWithin + Chunk up this stream into groups of elements received within a time window, or limited by the number of the elements, whatever happens first. Empty groups will not be emitted if no elements are received from upstream. The last group before end-of-stream will contain the buffered elements since the previously emitted group. @@ -999,7 +959,7 @@ The last group before end-of-stream will contain the buffered elements since the **emits** when the configured time elapses since the last group has been emitted, but not if no elements has been grouped (i.e: no empty groups), or when limit has been reached. -**backpressures** downstream backpressures, and there are `n+1` buffered elements +**backpressures** downstream backpressures, and there are *n+1* buffered elements **completes** when upstream completes @@ -1012,12 +972,12 @@ The last group before end-of-stream will contain the buffered elements since the **emits** when the configured time elapses since the last group has been emitted, but not if no elements has been grouped (i.e: no empty groups), or when weight limit has been reached. -**backpressures** downstream backpressures, and buffered group (+ pending element) weighs more than `maxWeight` +**backpressures** downstream backpressures, and buffered group (+ pending element) weighs more than *maxWeight* **completes** when upstream completes -initialDelay -^^^^^^^^^^^^ +### initialDelay + Delay the initial element by a user specified duration from stream materialization. **emits** upstream emits an element if the initial delay already elapsed @@ -1026,30 +986,26 @@ Delay the initial element by a user specified duration from stream materializati **completes** when upstream completes +### delay -delay -^^^^^ Delay every element passed through with a specific duration. **emits** there is a pending element in the buffer and configured time for this element elapsed -**backpressures** differs, depends on ``OverflowStrategy`` set +**backpressures** differs, depends on `OverflowStrategy` set **completes** when upstream completes and buffered elements has been drained - -.. _detached-stages-overview_java: - -Backpressure aware stages -------------------------- + +## Backpressure aware stages These stages are aware of the backpressure provided by their downstreams and able to adapt their behavior to that signal. -conflate -^^^^^^^^ +### conflate + Allow for a slower downstream by passing incoming elements and a summary into an aggregate function as long as there is backpressure. The summary value must be of the same type as the incoming elements, for example the sum or -average of incoming numbers, if aggregation should lead to a different type ``conflateWithSeed`` can be used: +average of incoming numbers, if aggregation should lead to a different type `conflateWithSeed` can be used: **emits** when downstream stops backpressuring and there is a conflated element available @@ -1057,10 +1013,10 @@ average of incoming numbers, if aggregation should lead to a different type ``co **completes** when upstream completes -conflateWithSeed -^^^^^^^^^^^^^^^^ +### conflateWithSeed + Allow for a slower downstream by passing incoming elements and a summary into an aggregate function as long as there -is backpressure. When backpressure starts or there is no backpressure element is passed into a ``seed`` function to +is backpressure. When backpressure starts or there is no backpressure element is passed into a `seed` function to transform it to the summary type. **emits** when downstream stops backpressuring and there is a conflated element available @@ -1069,13 +1025,13 @@ transform it to the summary type. **completes** when upstream completes -batch -^^^^^ +### batch + Allow for a slower downstream by passing incoming elements and a summary into an aggregate function as long as there is backpressure and a maximum number of batched elements is not yet reached. When the maximum number is reached and downstream still backpressures batch will also backpressure. -When backpressure starts or there is no backpressure element is passed into a ``seed`` function to transform it +When backpressure starts or there is no backpressure element is passed into a `seed` function to transform it to the summary type. Will eagerly pull elements, this behavior may result in a single pending (i.e. buffered) element which cannot be @@ -1087,12 +1043,11 @@ aggregated to the batched value. **completes** when upstream completes and a "possibly pending" element was drained +### batchWeighted -batchWeighted -^^^^^^^^^^^^^ Allow for a slower downstream by passing incoming elements and a summary into an aggregate function as long as there is backpressure and a maximum weight batched elements is not yet reached. The weight of each element is determined by -applying ``costFn``. When the maximum total weight is reached and downstream still backpressures batch will also +applying `costFn`. When the maximum total weight is reached and downstream still backpressures batch will also backpressure. Will eagerly pull elements, this behavior may result in a single pending (i.e. buffered) element which cannot be @@ -1104,10 +1059,10 @@ aggregated to the batched value. **completes** upstream completes and a "possibly pending" element was drained -expand -^^^^^^ -Allow for a faster downstream by expanding the last incoming element to an ``Iterator``. For example -``Iterator.continually(element)`` to keep repeating the last incoming element. +### expand + +Allow for a faster downstream by expanding the last incoming element to an `Iterator`. For example +`Iterator.continually(element)` to keep repeating the last incoming element. **emits** when downstream stops backpressuring @@ -1115,9 +1070,9 @@ Allow for a faster downstream by expanding the last incoming element to an ``Ite **completes** when upstream completes -buffer (Backpressure) -^^^^^^^^^^^^^^^^^^^^^ -Allow for a temporarily faster upstream events by buffering ``size`` elements. When the buffer is full backpressure +### buffer (Backpressure) + +Allow for a temporarily faster upstream events by buffering `size` elements. When the buffer is full backpressure is applied. **emits** when downstream stops backpressuring and there is a pending element in the buffer @@ -1126,15 +1081,15 @@ is applied. **completes** when upstream completes and buffered elements has been drained -buffer (Drop) -^^^^^^^^^^^^^ -Allow for a temporarily faster upstream events by buffering ``size`` elements. When the buffer is full elements are -dropped according to the specified ``OverflowStrategy``: +### buffer (Drop) -* ``dropHead()`` drops the oldest element in the buffer to make space for the new element -* ``dropTail()`` drops the youngest element in the buffer to make space for the new element -* ``dropBuffer()`` drops the entire buffer and buffers the new element -* ``dropNew()`` drops the new element +Allow for a temporarily faster upstream events by buffering `size` elements. When the buffer is full elements are +dropped according to the specified `OverflowStrategy`: + + * `dropHead()` drops the oldest element in the buffer to make space for the new element + * `dropTail()` drops the youngest element in the buffer to make space for the new element + * `dropBuffer()` drops the entire buffer and buffers the new element + * `dropNew()` drops the new element **emits** when downstream stops backpressuring and there is a pending element in the buffer @@ -1142,10 +1097,10 @@ dropped according to the specified ``OverflowStrategy``: **completes** upstream completes and buffered elements has been drained -buffer (Fail) -^^^^^^^^^^^^^ -Allow for a temporarily faster upstream events by buffering ``size`` elements. When the buffer is full the stage fails -the flow with a ``BufferOverflowException``. +### buffer (Fail) + +Allow for a temporarily faster upstream events by buffering `size` elements. When the buffer is full the stage fails +the flow with a `BufferOverflowException`. **emits** when downstream stops backpressuring and there is a pending element in the buffer @@ -1153,16 +1108,14 @@ the flow with a ``BufferOverflowException``. **completes** when upstream completes and buffered elements has been drained - -Nesting and flattening stages ------------------------------ +## Nesting and flattening stages These stages either take a stream and turn it into a stream of streams (nesting) or they take a stream that contains nested streams and turn them into a stream of elements instead (flattening). -prefixAndTail -^^^^^^^^^^^^^ -Take up to `n` elements from the stream (less than `n` only if the upstream completes before emitting `n` elements) +### prefixAndTail + +Take up to *n* elements from the stream (less than *n* only if the upstream completes before emitting *n* elements) and returns a pair containing a strict sequence of the taken element and a stream representing the remaining elements. **emits** when the configured number of prefix elements are available. Emits this prefix, and the rest as a substream @@ -1171,9 +1124,8 @@ and returns a pair containing a strict sequence of the taken element and a strea **completes** when prefix elements has been consumed and substream has been consumed +### groupBy -groupBy -^^^^^^^ Demultiplex the incoming stream into separate output streams. **emits** an element for which the grouping function returns a group that has not yet been created. Emits the new group @@ -1181,9 +1133,9 @@ there is an element pending for a group whose substream backpressures **completes** when upstream completes (Until the end of stream it is not possible to know whether new substreams will be needed or not) -splitWhen -^^^^^^^^^ -Split off elements into a new substream whenever a predicate function return ``true``. +### splitWhen + +Split off elements into a new substream whenever a predicate function return `true`. **emits** an element for which the provided predicate is true, opening and emitting a new substream for subsequent elements @@ -1191,9 +1143,9 @@ Split off elements into a new substream whenever a predicate function return ``t **completes** when upstream completes (Until the end of stream it is not possible to know whether new substreams will be needed or not) -splitAfter -^^^^^^^^^^ -End the current substream whenever a predicate returns ``true``, starting a new substream for the next element. +### splitAfter + +End the current substream whenever a predicate returns `true`, starting a new substream for the next element. **emits** when an element passes through. When the provided predicate is true it emits the element * and opens a new substream for subsequent element @@ -1201,9 +1153,9 @@ End the current substream whenever a predicate returns ``true``, starting a new **completes** when upstream completes (Until the end of stream it is not possible to know whether new substreams will be needed or not) -flatMapConcat -^^^^^^^^^^^^^ -Transform each input element into a ``Source`` whose elements are then flattened into the output stream through +### flatMapConcat + +Transform each input element into a `Source` whose elements are then flattened into the output stream through concatenation. This means each source is fully consumed before consumption of the next source starts. **emits** when the current consumed substream has an element available @@ -1212,10 +1164,9 @@ concatenation. This means each source is fully consumed before consumption of th **completes** when upstream completes and all consumed substreams complete +### flatMapMerge -flatMapMerge -^^^^^^^^^^^^ -Transform each input element into a ``Source`` whose elements are then flattened into the output stream through +Transform each input element into a `Source` whose elements are then flattened into the output stream through merging. The maximum number of merged sources has to be specified. **emits** when one of the currently consumed substreams has an element available @@ -1224,16 +1175,14 @@ merging. The maximum number of merged sources has to be specified. **completes** when upstream completes and all consumed substreams complete - -Time aware stages ------------------ +## Time aware stages Those stages operate taking time into consideration. -initialTimeout -^^^^^^^^^^^^^^ +### initialTimeout + If the first element has not passed through this stage before the provided timeout, the stream is failed -with a ``TimeoutException``. +with a `TimeoutException`. **emits** when upstream emits an element @@ -1243,10 +1192,10 @@ with a ``TimeoutException``. **cancels** when downstream cancels -completionTimeout -^^^^^^^^^^^^^^^^^ +### completionTimeout + If the completion of the stream does not happen until the provided timeout, the stream is failed -with a ``TimeoutException``. +with a `TimeoutException`. **emits** when upstream emits an element @@ -1256,10 +1205,10 @@ with a ``TimeoutException``. **cancels** when downstream cancels -idleTimeout -^^^^^^^^^^^ +### idleTimeout + If the time between two processed elements exceeds the provided timeout, the stream is failed -with a ``TimeoutException``. The timeout is checked periodically, so the resolution of the +with a `TimeoutException`. The timeout is checked periodically, so the resolution of the check is one period (equals to timeout value). **emits** when upstream emits an element @@ -1270,10 +1219,10 @@ check is one period (equals to timeout value). **cancels** when downstream cancels -backpressureTimeout -^^^^^^^^^^^^^^^^^^^ +### backpressureTimeout + If the time between the emission of an element and the following downstream demand exceeds the provided timeout, -the stream is failed with a ``TimeoutException``. The timeout is checked periodically, so the resolution of the +the stream is failed with a `TimeoutException`. The timeout is checked periodically, so the resolution of the check is one period (equals to timeout value). **emits** when upstream emits an element @@ -1284,8 +1233,8 @@ check is one period (equals to timeout value). **cancels** when downstream cancels -keepAlive -^^^^^^^^^ +### keepAlive + Injects additional (configured) elements if upstream does not emit for a configured amount of time. **emits** when upstream emits an element or if the upstream was idle for the configured period @@ -1296,8 +1245,8 @@ Injects additional (configured) elements if upstream does not emit for a configu **cancels** when downstream cancels -initialDelay -^^^^^^^^^^^^ +### initialDelay + Delays the initial element by the specified duration. **emits** when upstream emits an element if the initial delay is already elapsed @@ -1308,25 +1257,23 @@ Delays the initial element by the specified duration. **cancels** when downstream cancels - -Fan-in stages -------------- +## Fan-in stages These stages take multiple streams as their input and provide a single output combining the elements from all of the inputs in different ways. -merge -^^^^^ +### merge + Merge multiple sources. Picks elements randomly if all sources has elements ready. **emits** when one of the inputs has an element available **backpressures** when downstream backpressures -**completes** when all upstreams complete (This behavior is changeable to completing when any upstream completes by setting ``eagerComplete=true``.) +**completes** when all upstreams complete (This behavior is changeable to completing when any upstream completes by setting `eagerComplete=true`.) + +### mergeSorted -mergeSorted -^^^^^^^^^^^ Merge multiple sources. Waits for one element to be ready from each input stream and emits the smallest element. @@ -1336,19 +1283,19 @@ smallest element. **completes** when all upstreams complete -mergePreferred -^^^^^^^^^^^^^^ +### mergePreferred + Merge multiple sources. Prefer one source if all sources has elements ready. **emits** when one of the inputs has an element available, preferring a defined input if multiple have elements available **backpressures** when downstream backpressures -**completes** when all upstreams complete (This behavior is changeable to completing when any upstream completes by setting ``eagerComplete=true``.) +**completes** when all upstreams complete (This behavior is changeable to completing when any upstream completes by setting `eagerComplete=true`.) -zip -^^^ -Combines elements from each of multiple sources into `Pair` s and passes the pairs downstream. +### zip + +Combines elements from each of multiple sources into *Pair* s and passes the pairs downstream. **emits** when all of the inputs have an element available @@ -1356,9 +1303,9 @@ Combines elements from each of multiple sources into `Pair` s and passes the pai **completes** when any upstream completes -zipWith -^^^^^^^ -Combines elements from multiple sources through a ``combine`` function and passes the +### zipWith + +Combines elements from multiple sources through a `combine` function and passes the returned value downstream. **emits** when all of the inputs have an element available @@ -1367,8 +1314,8 @@ returned value downstream. **completes** when any upstream completes -zipWithIndex -^^^^^^^^^^^^ +### zipWithIndex + Zips elements of current flow with its indices. **emits** upstream emits an element and is paired with their index @@ -1377,8 +1324,8 @@ Zips elements of current flow with its indices. **completes** when upstream completes -concat -^^^^^^ +### concat + After completion of the original upstream the elements of the given source will be emitted. **emits** when the current stream has an element available; if the current input completes, it tries the next one @@ -1387,11 +1334,11 @@ After completion of the original upstream the elements of the given source will **completes** when all upstreams complete -prepend -^^^^^^^ +### prepend + Prepends the given source to the flow, consuming it until completion before the original source is consumed. -If materialized values needs to be collected ``prependMat`` is available. +If materialized values needs to be collected `prependMat` is available. **emits** when the given stream has an element available; if the given input completes, it tries the current one @@ -1399,8 +1346,8 @@ If materialized values needs to be collected ``prependMat`` is available. **completes** when all upstreams complete -orElse -^^^^^^ +### orElse + If the primary source completes without emitting any elements, the elements from the secondary source are emitted. If the primary source emits any elements the secondary source is cancelled. @@ -1417,8 +1364,8 @@ is available from the second stream **completes** the primary stream completes after emitting at least one element, when the primary stream completes without emitting and the secondary stream already has completed or when the secondary stream completes -interleave -^^^^^^^^^^ +### interleave + Emits a specifiable number of elements from the original source, then from the provided source and repeats. If one source completes the rest of the other stream will be emitted. @@ -1428,14 +1375,13 @@ source completes the rest of the other stream will be emitted. **completes** when both upstreams have completed -Fan-out stages --------------- +## Fan-out stages These have one input and multiple outputs. They might route the elements between different outputs, or emit elements on multiple outputs at the same time. -unzip -^^^^^ +### unzip + Takes a stream of two element tuples and unzips the two elements ino two different downstreams. **emits** when all of the outputs stops backpressuring and there is an input element available @@ -1444,8 +1390,8 @@ Takes a stream of two element tuples and unzips the two elements ino two differe **completes** when upstream completes -unzipWith -^^^^^^^^^ +### unzipWith + Splits each element of input into multiple downstreams using a function **emits** when all of the outputs stops backpressuring and there is an input element available @@ -1454,9 +1400,9 @@ Splits each element of input into multiple downstreams using a function **completes** when upstream completes -broadcast -^^^^^^^^^ -Emit each incoming element each of ``n`` outputs. +### broadcast + +Emit each incoming element each of `n` outputs. **emits** when all of the outputs stops backpressuring and there is an input element available @@ -1464,8 +1410,8 @@ Emit each incoming element each of ``n`` outputs. **completes** when upstream completes -balance -^^^^^^^ +### balance + Fan-out the stream to several streams. Each upstream element is emitted to the first available downstream consumer. **emits** when any of the outputs stops backpressuring; emits the element to the first available output @@ -1474,8 +1420,8 @@ Fan-out the stream to several streams. Each upstream element is emitted to the f **completes** when upstream completes -partition -^^^^^^^^^ +### partition + Fan-out the stream to several streams. Each upstream element is emitted to one downstream consumer according to the partitioner function applied to the element. @@ -1485,13 +1431,11 @@ partitioner function applied to the element. **completes** when upstream completes and no output is pending +## Watching status stages -Watching status stages ----------------------- +### watchTermination -watchTermination -^^^^^^^^^^^^^^^^ -Materializes to a ``CompletionStage`` that will be completed with Done or failed depending whether the upstream of the stage has been completed or failed. +Materializes to a `CompletionStage` that will be completed with Done or failed depending whether the upstream of the stage has been completed or failed. The stage otherwise passes through elements unchanged. **emits** when input has an element available @@ -1500,15 +1444,14 @@ The stage otherwise passes through elements unchanged. **completes** when upstream completes -monitor -^^^^^^^ -Materializes to a ``FlowMonitor`` that monitors messages flowing through or completion of the stage. The stage otherwise -passes through elements unchanged. Note that the ``FlowMonitor`` inserts a memory barrier every time it processes an +### monitor + +Materializes to a `FlowMonitor` that monitors messages flowing through or completion of the stage. The stage otherwise +passes through elements unchanged. Note that the `FlowMonitor` inserts a memory barrier every time it processes an event, and may therefore affect performance. **emits** when upstream emits an element **backpressures** when downstream **backpressures** -**completes** when upstream completes - +**completes** when upstream completes \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-composition.md b/akka-docs/src/main/paradox/java/stream/stream-composition.md index 8df4628f6c..d909a627f7 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-composition.md +++ b/akka-docs/src/main/paradox/java/stream/stream-composition.md @@ -1,100 +1,93 @@ -.. _composition-java: - -Modularity, Composition and Hierarchy -===================================== +# Modularity, Composition and Hierarchy Akka Streams provide a uniform model of stream processing graphs, which allows flexible composition of reusable components. In this chapter we show how these look like from the conceptual and API perspective, demonstrating the modularity aspects of the library. -Basics of composition and modularity ------------------------------------- +## Basics of composition and modularity Every processing stage used in Akka Streams can be imagined as a "box" with input and output ports where elements to -be processed arrive and leave the stage. In this view, a :class:`Source` is nothing else than a "box" with a single -output port, or, a :class:`BidiFlow` is a "box" with exactly two input and two output ports. In the figure below +be processed arrive and leave the stage. In this view, a `Source` is nothing else than a "box" with a single +output port, or, a `BidiFlow` is a "box" with exactly two input and two output ports. In the figure below we illustrate the most common used stages viewed as "boxes". | -.. image:: ../../images/compose_shapes.png - :align: center +![compose_shapes.png](../../images/compose_shapes.png) | -The *linear* stages are :class:`Source`, :class:`Sink` -and :class:`Flow`, as these can be used to compose strict chains of processing stages. +The *linear* stages are `Source`, `Sink` +and `Flow`, as these can be used to compose strict chains of processing stages. Fan-in and Fan-out stages have usually multiple input or multiple output ports, therefore they allow to build -more complex graph layouts, not just chains. :class:`BidiFlow` stages are usually useful in IO related tasks, where -there are input and output channels to be handled. Due to the specific shape of :class:`BidiFlow` it is easy to -stack them on top of each other to build a layered protocol for example. The ``TLS`` support in Akka is for example -implemented as a :class:`BidiFlow`. +more complex graph layouts, not just chains. `BidiFlow` stages are usually useful in IO related tasks, where +there are input and output channels to be handled. Due to the specific shape of `BidiFlow` it is easy to +stack them on top of each other to build a layered protocol for example. The `TLS` support in Akka is for example +implemented as a `BidiFlow`. These reusable components already allow the creation of complex processing networks. What we have seen so far does not implement modularity though. It is desirable for example to package up a larger graph entity into a reusable component which hides its internals only exposing the ports that are meant to the users of the module -to interact with. One good example is the ``Http`` server component, which is encoded internally as a -:class:`BidiFlow` which interfaces with the client TCP connection using an input-output port pair accepting and sending -:class:`ByteString` s, while its upper ports emit and receive :class:`HttpRequest` and :class:`HttpResponse` instances. +to interact with. One good example is the `Http` server component, which is encoded internally as a +`BidiFlow` which interfaces with the client TCP connection using an input-output port pair accepting and sending +`ByteString` s, while its upper ports emit and receive `HttpRequest` and `HttpResponse` instances. The following figure demonstrates various composite stages, that contain various other type of stages internally, but -hiding them behind a *shape* that looks like a :class:`Source`, :class:`Flow`, etc. +hiding them behind a *shape* that looks like a `Source`, `Flow`, etc. | -.. image:: ../../images/compose_composites.png - :align: center +![compose_composites.png](../../images/compose_composites.png) | -One interesting example above is a :class:`Flow` which is composed of a disconnected :class:`Sink` and :class:`Source`. -This can be achieved by using the ``fromSinkAndSource()`` constructor method on :class:`Flow` which takes the two parts as +One interesting example above is a `Flow` which is composed of a disconnected `Sink` and `Source`. +This can be achieved by using the `fromSinkAndSource()` constructor method on `Flow` which takes the two parts as parameters. -Please note that when combining a :class:`Flow` using that method, the termination signals are not carried -"through" as the :class:`Sink` and :class:`Source` are assumed to be fully independent. If however you want to construct -a :class:`Flow` like this but need the termination events to trigger "the other side" of the composite flow, you can use -``CoupledTerminationFlow.fromSinkAndSource`` which does just that. For example the cancelation of the composite flows -source-side will then lead to completion of its sink-side. Read :class:`CoupledTerminationFlow`'s scaladoc for a +Please note that when combining a `Flow` using that method, the termination signals are not carried +"through" as the `Sink` and `Source` are assumed to be fully independent. If however you want to construct +a `Flow` like this but need the termination events to trigger "the other side" of the composite flow, you can use +`CoupledTerminationFlow.fromSinkAndSource` which does just that. For example the cancelation of the composite flows +source-side will then lead to completion of its sink-side. Read `CoupledTerminationFlow`'s scaladoc for a detailed explanation how this works. -The example :class:`BidiFlow` demonstrates that internally a module can be of arbitrary complexity, and the exposed +The example `BidiFlow` demonstrates that internally a module can be of arbitrary complexity, and the exposed ports can be wired in flexible ways. The only constraint is that all the ports of enclosed modules must be either connected to each other, or exposed as interface ports, and the number of such ports needs to match the requirement -of the shape, for example a :class:`Source` allows only one exposed output port, the rest of the internal ports must +of the shape, for example a `Source` allows only one exposed output port, the rest of the internal ports must be properly connected. -These mechanics allow arbitrary nesting of modules. For example the following figure demonstrates a :class:`RunnableGraph` -that is built from a composite :class:`Source` and a composite :class:`Sink` (which in turn contains a composite -:class:`Flow`). +These mechanics allow arbitrary nesting of modules. For example the following figure demonstrates a `RunnableGraph` +that is built from a composite `Source` and a composite `Sink` (which in turn contains a composite +`Flow`). | -.. image:: ../../images/compose_nested_flow.png - :align: center +![compose_nested_flow.png](../../images/compose_nested_flow.png) | -The above diagram contains one more shape that we have not seen yet, which is called :class:`RunnableGraph`. It turns +The above diagram contains one more shape that we have not seen yet, which is called `RunnableGraph`. It turns out, that if we wire all exposed ports together, so that no more open ports remain, we get a module that is *closed*. -This is what the :class:`RunnableGraph` class represents. This is the shape that a :class:`Materializer` can take -and turn into a network of running entities that perform the task described. In fact, a :class:`RunnableGraph` is a +This is what the `RunnableGraph` class represents. This is the shape that a `Materializer` can take +and turn into a network of running entities that perform the task described. In fact, a `RunnableGraph` is a module itself, and (maybe somewhat surprisingly) it can be used as part of larger graphs. It is rarely useful to embed a closed graph shape in a larger graph (since it becomes an isolated island as there are no open port for communication with the rest of the graph), but this demonstrates the uniform underlying model. If we try to build a code snippet that corresponds to the above diagram, our first try might look like this: -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#non-nested-flow +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #non-nested-flow } It is clear however that there is no nesting present in our first attempt, since the library cannot figure out where we intended to put composite module boundaries, it is our responsibility to do that. If we are using the -DSL provided by the :class:`Flow`, :class:`Source`, :class:`Sink` classes then nesting can be achieved by calling one of the -methods ``withAttributes()`` or ``named()`` (where the latter is just a shorthand for adding a name attribute). +DSL provided by the `Flow`, `Source`, `Sink` classes then nesting can be achieved by calling one of the +methods `withAttributes()` or `named()` (where the latter is just a shorthand for adding a name attribute). The following code demonstrates how to achieve the desired nesting: -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#nested-flow +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #nested-flow } Once we have hidden the internals of our components, they act like any other built-in component of similar shape. If we hide some of the internals of our composites, the result looks just like if any other predefine component has been @@ -102,22 +95,20 @@ used: | -.. image:: ../../images/compose_nested_flow_opaque.png - :align: center +![compose_nested_flow_opaque.png](../../images/compose_nested_flow_opaque.png) | If we look at usage of built-in components, and our custom components, there is no difference in usage as the code snippet below demonstrates. -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#reuse +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #reuse } -Composing complex systems -------------------------- +## Composing complex systems In the previous section we explored the possibility of composition, and hierarchy, but we stayed away from non-linear, generalized graph components. There is nothing in Akka Streams though that enforces that stream processing layouts -can only be linear. The DSL for :class:`Source` and friends is optimized for creating such linear chains, as they are +can only be linear. The DSL for `Source` and friends is optimized for creating such linear chains, as they are the most common in practice. There is a more advanced DSL for building complex graphs, that can be used if more flexibility is needed. We will see that the difference between the two DSLs is only on the surface: the concepts they operate on are uniform across all DSLs and fit together nicely. @@ -126,192 +117,191 @@ As a first example, let's look at a more complex layout: | -.. image:: ../../images/compose_graph.png - :align: center +![compose_graph.png](../../images/compose_graph.png) | -The diagram shows a :class:`RunnableGraph` (remember, if there are no unwired ports, the graph is closed, and therefore +The diagram shows a `RunnableGraph` (remember, if there are no unwired ports, the graph is closed, and therefore can be materialized) that encapsulates a non-trivial stream processing network. It contains fan-in, fan-out stages, -directed and non-directed cycles. The ``runnable()`` method of the :class:`GraphDSL` factory object allows the creation of a +directed and non-directed cycles. The `runnable()` method of the `GraphDSL` factory object allows the creation of a general, closed, and runnable graph. For example the network on the diagram can be realized like this: -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#complex-graph +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #complex-graph } In the code above we used the implicit port numbering feature to make the graph more readable and similar to the diagram. It is possible to refer to the ports, so another version might look like this: -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#complex-graph-alt +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #complex-graph-alt } | Similar to the case in the first section, so far we have not considered modularity. We created a complex graph, but the layout is flat, not modularized. We will modify our example, and create a reusable component with the graph DSL. -The way to do it is to use the ``create()`` method on :class:`GraphDSL` factory. If we remove the sources and sinks +The way to do it is to use the `create()` method on `GraphDSL` factory. If we remove the sources and sinks from the previous example, what remains is a partial graph: | -.. image:: ../../images/compose_graph_partial.png - :align: center +![compose_graph_partial.png](../../images/compose_graph_partial.png) | We can recreate a similar graph in code, using the DSL in a similar way than before: -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#partial-graph +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #partial-graph } -The only new addition is the return value of the builder block, which is a :class:`Shape`. All graphs (including -:class:`Source`, :class:`BidiFlow`, etc) have a shape, which encodes the *typed* ports of the module. In our example -there is exactly one input and output port left, so we can declare it to have a :class:`FlowShape` by returning an -instance of it. While it is possible to create new :class:`Shape` types, it is usually recommended to use one of the +The only new addition is the return value of the builder block, which is a `Shape`. All graphs (including +`Source`, `BidiFlow`, etc) have a shape, which encodes the *typed* ports of the module. In our example +there is exactly one input and output port left, so we can declare it to have a `FlowShape` by returning an +instance of it. While it is possible to create new `Shape` types, it is usually recommended to use one of the matching built-in ones. -The resulting graph is already a properly wrapped module, so there is no need to call `named()` to encapsulate the graph, but +The resulting graph is already a properly wrapped module, so there is no need to call *named()* to encapsulate the graph, but it is a good practice to give names to modules to help debugging. | -.. image:: ../../images/compose_graph_shape.png - :align: center +![compose_graph_shape.png](../../images/compose_graph_shape.png) | Since our partial graph has the right shape, it can be already used in the simpler, linear DSL: -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#partial-use +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #partial-use } -It is not possible to use it as a :class:`Flow` yet, though (i.e. we cannot call ``.filter()`` on it), but :class:`Flow` -has a ``fromGraph()`` method that just adds the DSL to a :class:`FlowShape`. There are similar methods on :class:`Source`, -:class:`Sink` and :class:`BidiShape`, so it is easy to get back to the simpler DSL if a graph has the right shape. +It is not possible to use it as a `Flow` yet, though (i.e. we cannot call `.filter()` on it), but `Flow` +has a `fromGraph()` method that just adds the DSL to a `FlowShape`. There are similar methods on `Source`, +`Sink` and `BidiShape`, so it is easy to get back to the simpler DSL if a graph has the right shape. For convenience, it is also possible to skip the partial graph creation, and use one of the convenience creator methods. To demonstrate this, we will create the following graph: | -.. image:: ../../images/compose_graph_flow.png - :align: center +![compose_graph_flow.png](../../images/compose_graph_flow.png) | The code version of the above closed graph might look like this: -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#partial-flow-dsl +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #partial-flow-dsl } -.. note:: - All graph builder sections check if the resulting graph has all ports connected except the exposed ones and will - throw an exception if this is violated. +@@@ note -We are still in debt of demonstrating that :class:`RunnableGraph` is a component just like any other, which can +All graph builder sections check if the resulting graph has all ports connected except the exposed ones and will +throw an exception if this is violated. + +@@@ + +We are still in debt of demonstrating that `RunnableGraph` is a component just like any other, which can be embedded in graphs. In the following snippet we embed one closed graph in another: -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#embed-closed +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #embed-closed } -The type of the imported module indicates that the imported module has a :class:`ClosedShape`, and so we are not +The type of the imported module indicates that the imported module has a `ClosedShape`, and so we are not able to wire it to anything else inside the enclosing closed graph. Nevertheless, this "island" is embedded properly, and will be materialized just like any other module that is part of the graph. As we have demonstrated, the two DSLs are fully interoperable, as they encode a similar nested structure of "boxes with ports", it is only the DSLs that differ to be as much powerful as possible on the given abstraction level. It is possible -to embed complex graphs in the fluid DSL, and it is just as easy to import and embed a :class:`Flow`, etc, in a larger, +to embed complex graphs in the fluid DSL, and it is just as easy to import and embed a `Flow`, etc, in a larger, complex structure. -We have also seen, that every module has a :class:`Shape` (for example a :class:`Sink` has a :class:`SinkShape`) +We have also seen, that every module has a `Shape` (for example a `Sink` has a `SinkShape`) independently which DSL was used to create it. This uniform representation enables the rich composability of various stream processing entities in a convenient way. -Materialized values -------------------- +## Materialized values -After realizing that :class:`RunnableGraph` is nothing more than a module with no unused ports (it is an island), it becomes clear that +After realizing that `RunnableGraph` is nothing more than a module with no unused ports (it is an island), it becomes clear that after materialization the only way to communicate with the running stream processing logic is via some side-channel. -This side channel is represented as a *materialized value*. The situation is similar to :class:`Actor` s, where the -:class:`Props` instance describes the actor logic, but it is the call to ``actorOf()`` that creates an actually running -actor, and returns an :class:`ActorRef` that can be used to communicate with the running actor itself. Since the -:class:`Props` can be reused, each call will return a different reference. +This side channel is represented as a *materialized value*. The situation is similar to `Actor` s, where the +`Props` instance describes the actor logic, but it is the call to `actorOf()` that creates an actually running +actor, and returns an `ActorRef` that can be used to communicate with the running actor itself. Since the +`Props` can be reused, each call will return a different reference. When it comes to streams, each materialization creates a new running network corresponding to the blueprint that was -encoded in the provided :class:`RunnableGraph`. To be able to interact with the running network, each materialization +encoded in the provided `RunnableGraph`. To be able to interact with the running network, each materialization needs to return a different object that provides the necessary interaction capabilities. In other words, the -:class:`RunnableGraph` can be seen as a factory, which creates: +`RunnableGraph` can be seen as a factory, which creates: - * a network of running processing entities, inaccessible from the outside - * a materialized value, optionally providing a controlled interaction capability with the network +> + * a network of running processing entities, inaccessible from the outside + * a materialized value, optionally providing a controlled interaction capability with the network Unlike actors though, each of the processing stages might provide a materialized value, so when we compose multiple stages or modules, we need to combine the materialized value as well (there are default rules which make this easier, -for example `to()` and `via()` takes care of the most common case of taking the materialized value to the left. -See :ref:`flow-combine-mat-scala` for details). We demonstrate how this works by a code example and a diagram which +for example *to()* and *via()* takes care of the most common case of taking the materialized value to the left. +See @ref:[Combining materialized values](../../scala/stream/stream-flows-and-basics.md#flow-combine-mat-scala) for details). We demonstrate how this works by a code example and a diagram which graphically demonstrates what is happening. The propagation of the individual materialized values from the enclosed modules towards the top will look like this: | -.. image:: ../../images/compose_mat.png - :align: center +![compose_mat.png](../../images/compose_mat.png) | -To implement the above, first, we create a composite :class:`Source`, where the enclosed :class:`Source` have a -materialized type of :class:`CompletableFuture>>`. By using the combiner function ``Keep.left()``, the resulting materialized +To implement the above, first, we create a composite `Source`, where the enclosed `Source` have a +materialized type of `CompletableFuture>>`. By using the combiner function `Keep.left()`, the resulting materialized type is of the nested module (indicated by the color *red* on the diagram): -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#mat-combine-1 +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #mat-combine-1 } -Next, we create a composite :class:`Flow` from two smaller components. Here, the second enclosed :class:`Flow` has a -materialized type of :class:`CompletionStage`, and we propagate this to the parent by using ``Keep.right()`` +Next, we create a composite `Flow` from two smaller components. Here, the second enclosed `Flow` has a +materialized type of `CompletionStage`, and we propagate this to the parent by using `Keep.right()` as the combiner function (indicated by the color *yellow* on the diagram): -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#mat-combine-2 +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #mat-combine-2 } -As a third step, we create a composite :class:`Sink`, using our ``nestedFlow`` as a building block. In this snippet, both -the enclosed :class:`Flow` and the folding :class:`Sink` has a materialized value that is interesting for us, so -we use ``Keep.both()`` to get a :class:`Pair` of them as the materialized type of ``nestedSink`` (indicated by the color +As a third step, we create a composite `Sink`, using our `nestedFlow` as a building block. In this snippet, both +the enclosed `Flow` and the folding `Sink` has a materialized value that is interesting for us, so +we use `Keep.both()` to get a `Pair` of them as the materialized type of `nestedSink` (indicated by the color *blue* on the diagram) -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#mat-combine-3 +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #mat-combine-3 } -As the last example, we wire together ``nestedSource`` and ``nestedSink`` and we use a custom combiner function to -create a yet another materialized type of the resulting :class:`RunnableGraph`. This combiner function just ignores -the :class:`CompletionStage` part, and wraps the other two values in a custom case class :class:`MyClass` +As the last example, we wire together `nestedSource` and `nestedSink` and we use a custom combiner function to +create a yet another materialized type of the resulting `RunnableGraph`. This combiner function just ignores +the `CompletionStage` part, and wraps the other two values in a custom case class `MyClass` (indicated by color *purple* on the diagram): -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#mat-combine-4a +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #mat-combine-4a } -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#mat-combine-4b +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #mat-combine-4b } -.. note:: - The nested structure in the above example is not necessary for combining the materialized values, it just - demonstrates how the two features work together. See :ref:`operator-fusion-java` for further examples - of combining materialized values without nesting and hierarchy involved. +@@@ note -Attributes ----------- +The nested structure in the above example is not necessary for combining the materialized values, it just +demonstrates how the two features work together. See @ref:[Operator Fusion](stream-flows-and-basics.md#operator-fusion-java) for further examples +of combining materialized values without nesting and hierarchy involved. -We have seen that we can use ``named()`` to introduce a nesting level in the fluid DSL (and also explicit nesting by using -``create()`` from :class:`GraphDSL`). Apart from having the effect of adding a nesting level, ``named()`` is actually -a shorthand for calling ``withAttributes(Attributes.name("someName"))``. Attributes provide a way to fine-tune certain +@@@ + +## Attributes + +We have seen that we can use `named()` to introduce a nesting level in the fluid DSL (and also explicit nesting by using +`create()` from `GraphDSL`). Apart from having the effect of adding a nesting level, `named()` is actually +a shorthand for calling `withAttributes(Attributes.name("someName"))`. Attributes provide a way to fine-tune certain aspects of the materialized running entity. For example buffer sizes for asynchronous stagescan be controlled via -attributes (see :ref:`async-stream-buffers-java`). When it comes to hierarchic composition, attributes are inherited +attributes (see @ref:[Buffers for asynchronous stages](stream-rate.md#async-stream-buffers-java)). When it comes to hierarchic composition, attributes are inherited by nested modules, unless they override them with a custom value. -The code below, a modification of an earlier example sets the ``inputBuffer`` attribute on certain modules, but not +The code below, a modification of an earlier example sets the `inputBuffer` attribute on certain modules, but not on others: -.. includecode:: ../code/jdocs/stream/CompositionDocTest.java#attributes-inheritance +@@snip [CompositionDocTest.java](../code/jdocs/stream/CompositionDocTest.java) { #attributes-inheritance } -The effect is, that each module inherits the ``inputBuffer`` attribute from its enclosing parent, unless it has -the same attribute explicitly set. ``nestedSource`` gets the default attributes from the materializer itself. ``nestedSink`` -on the other hand has this attribute set, so it will be used by all nested modules. ``nestedFlow`` will inherit from ``nestedSink`` -except the ``map`` stage which has again an explicitly provided attribute overriding the inherited one. +The effect is, that each module inherits the `inputBuffer` attribute from its enclosing parent, unless it has +the same attribute explicitly set. `nestedSource` gets the default attributes from the materializer itself. `nestedSink` +on the other hand has this attribute set, so it will be used by all nested modules. `nestedFlow` will inherit from `nestedSink` +except the `map` stage which has again an explicitly provided attribute overriding the inherited one. | -.. image:: ../../images/compose_attributes.png - :align: center +![compose_attributes.png](../../images/compose_attributes.png) | This diagram illustrates the inheritance process for the example code (representing the materializer default attributes -as the color *red*, the attributes set on ``nestedSink`` as *blue* and the attributes set on ``nestedFlow`` as *green*). +as the color *red*, the attributes set on `nestedSink` as *blue* and the attributes set on `nestedFlow` as *green*). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-cookbook.md b/akka-docs/src/main/paradox/java/stream/stream-cookbook.md index 862f04dbe9..64567b51ff 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-cookbook.md +++ b/akka-docs/src/main/paradox/java/stream/stream-cookbook.md @@ -1,11 +1,6 @@ -.. _stream-cookbook-java: +# Streams Cookbook -################ -Streams Cookbook -################ - -Introduction -============ +## Introduction This is a collection of patterns to demonstrate various usage of the Akka Streams API by solving small targeted problems in the format of "recipes". The purpose of this page is to give inspiration and ideas how to approach @@ -16,387 +11,367 @@ This part also serves as supplementary material for the main body of documentati open while reading the manual and look for examples demonstrating various streaming concepts as they appear in the main body of documentation. -If you need a quick reference of the available processing stages used in the recipes see :ref:`stages-overview_java`. +If you need a quick reference of the available processing stages used in the recipes see @ref:[stages-overview_java](stages-overview.md). -Working with Flows -================== +## Working with Flows In this collection we show simple recipes that involve linear flows. The recipes in this section are rather -general, more targeted recipes are available as separate sections (:ref:`stream-rate-java`, :ref:`stream-io-java`). +general, more targeted recipes are available as separate sections (@ref:[Buffers and working with rate](stream-rate.md), @ref:[Working with streaming IO](stream-io.md)). -Logging elements of a stream ----------------------------- +### Logging elements of a stream **Situation:** During development it is sometimes helpful to see what happens in a particular section of a stream. -The simplest solution is to simply use a ``map`` operation and use ``println`` to print the elements received to the console. +The simplest solution is to simply use a `map` operation and use `println` to print the elements received to the console. While this recipe is rather simplistic, it is often suitable for a quick debug session. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeLoggingElements.java#println-debug +@@snip [RecipeLoggingElements.java](../code/jdocs/stream/javadsl/cookbook/RecipeLoggingElements.java) { #println-debug } -Another approach to logging is to use ``log()`` operation which allows configuring logging for elements flowing through +Another approach to logging is to use `log()` operation which allows configuring logging for elements flowing through the stream as well as completion and erroring. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeLoggingElements.java#log-custom +@@snip [RecipeLoggingElements.java](../code/jdocs/stream/javadsl/cookbook/RecipeLoggingElements.java) { #log-custom } -Flattening a stream of sequences --------------------------------- +### Flattening a stream of sequences **Situation:** A stream is given as a stream of sequence of elements, but a stream of elements needed instead, streaming all the nested elements inside the sequences separately. -The ``mapConcat`` operation can be used to implement a one-to-many transformation of elements using a mapper function -in the form of ``In -> List``. In this case we want to map a ``List`` of elements to the elements in the -collection itself, so we can just call ``mapConcat(l -> l)``. +The `mapConcat` operation can be used to implement a one-to-many transformation of elements using a mapper function +in the form of `In -> List`. In this case we want to map a `List` of elements to the elements in the +collection itself, so we can just call `mapConcat(l -> l)`. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeFlattenList.java#flattening-lists +@@snip [RecipeFlattenList.java](../code/jdocs/stream/javadsl/cookbook/RecipeFlattenList.java) { #flattening-lists } -Draining a stream to a strict collection ----------------------------------------- +### Draining a stream to a strict collection **Situation:** A possibly unbounded sequence of elements is given as a stream, which needs to be collected into a Scala collection while ensuring boundedness A common situation when working with streams is one where we need to collect incoming elements into a Scala collection. -This operation is supported via ``Sink.seq`` which materializes into a ``CompletionStage>``. +This operation is supported via `Sink.seq` which materializes into a `CompletionStage>`. -The function ``limit`` or ``take`` should always be used in conjunction in order to guarantee stream boundedness, thus preventing the program from running out of memory. +The function `limit` or `take` should always be used in conjunction in order to guarantee stream boundedness, thus preventing the program from running out of memory. For example, this is best avoided: -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeSeq.java#draining-to-list-unsafe +@@snip [RecipeSeq.java](../code/jdocs/stream/javadsl/cookbook/RecipeSeq.java) { #draining-to-list-unsafe } -Rather, use ``limit`` or ``take`` to ensure that the resulting ``List`` will contain only up to ``MAX_ALLOWED_SIZE`` elements: +Rather, use `limit` or `take` to ensure that the resulting `List` will contain only up to `MAX_ALLOWED_SIZE` elements: -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeSeq.java#draining-to-list-safe +@@snip [RecipeSeq.java](../code/jdocs/stream/javadsl/cookbook/RecipeSeq.java) { #draining-to-list-safe } -Calculating the digest of a ByteString stream ---------------------------------------------- +### Calculating the digest of a ByteString stream -**Situation:** A stream of bytes is given as a stream of ``ByteString`` s and we want to calculate the cryptographic digest +**Situation:** A stream of bytes is given as a stream of `ByteString` s and we want to calculate the cryptographic digest of the stream. -This recipe uses a :class:`GraphStage` to host a mutable :class:`MessageDigest` class (part of the Java Cryptography -API) and update it with the bytes arriving from the stream. When the stream starts, the ``onPull`` handler of the -stage is called, which just bubbles up the ``pull`` event to its upstream. As a response to this pull, a ByteString -chunk will arrive (``onPush``) which we use to update the digest, then it will pull for the next chunk. +This recipe uses a `GraphStage` to host a mutable `MessageDigest` class (part of the Java Cryptography +API) and update it with the bytes arriving from the stream. When the stream starts, the `onPull` handler of the +stage is called, which just bubbles up the `pull` event to its upstream. As a response to this pull, a ByteString +chunk will arrive (`onPush`) which we use to update the digest, then it will pull for the next chunk. -Eventually the stream of ``ByteString`` s depletes and we get a notification about this event via ``onUpstreamFinish``. -At this point we want to emit the digest value, but we cannot do it with ``push`` in this handler directly since there may -be no downstream demand. Instead we call ``emit`` which will temporarily replace the handlers, emit the provided value when +Eventually the stream of `ByteString` s depletes and we get a notification about this event via `onUpstreamFinish`. +At this point we want to emit the digest value, but we cannot do it with `push` in this handler directly since there may +be no downstream demand. Instead we call `emit` which will temporarily replace the handlers, emit the provided value when demand comes in and then reset the stage state. It will then complete the stage. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeDigest.java#calculating-digest +@@snip [RecipeDigest.java](../code/jdocs/stream/javadsl/cookbook/RecipeDigest.java) { #calculating-digest } -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeDigest.java#calculating-digest2 +@@snip [RecipeDigest.java](../code/jdocs/stream/javadsl/cookbook/RecipeDigest.java) { #calculating-digest2 } -.. _cookbook-parse-lines-java: + +### Parsing lines from a stream of ByteStrings -Parsing lines from a stream of ByteStrings ------------------------------------------- - -**Situation:** A stream of bytes is given as a stream of ``ByteString`` s containing lines terminated by line ending +**Situation:** A stream of bytes is given as a stream of `ByteString` s containing lines terminated by line ending characters (or, alternatively, containing binary frames delimited by a special delimiter byte sequence) which needs to be parsed. -The :class:`Framing` helper class contains a convenience method to parse messages from a stream of ``ByteString`` s: +The `Framing` helper class contains a convenience method to parse messages from a stream of `ByteString` s: -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeParseLines.java#parse-lines +@@snip [RecipeParseLines.java](../code/jdocs/stream/javadsl/cookbook/RecipeParseLines.java) { #parse-lines } -Dealing with compressed data streams ------------------------------------- +### Dealing with compressed data streams -**Situation:** A gzipped stream of bytes is given as a stream of ``ByteString`` s, for example from a ``FileIO`` source. +**Situation:** A gzipped stream of bytes is given as a stream of `ByteString` s, for example from a `FileIO` source. -The :class:`Compression` helper class contains convenience methods for decompressing data streams compressed with +The `Compression` helper class contains convenience methods for decompressing data streams compressed with Gzip or Deflate. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeDecompress.java#decompress-gzip +@@snip [RecipeDecompress.java](../code/jdocs/stream/javadsl/cookbook/RecipeDecompress.java) { #decompress-gzip } - -Implementing reduce-by-key --------------------------- +### Implementing reduce-by-key **Situation:** Given a stream of elements, we want to calculate some aggregated value on different subgroups of the elements. The "hello world" of reduce-by-key style operations is *wordcount* which we demonstrate below. Given a stream of words -we first create a new stream that groups the words according to the ``i -> i`` function, i.e. now +we first create a new stream that groups the words according to the `i -> i` function, i.e. now we have a stream of streams, where every substream will serve identical words. To count the words, we need to process the stream of streams (the actual groups -containing identical words). ``groupBy`` returns a :class:`SubSource`, which +containing identical words). `groupBy` returns a `SubSource`, which means that we transform the resulting substreams directly. In this case we use -the ``reduce`` combinator to aggregate the word itself and the number of its -occurrences within a :class:`Pair`. Each substream will then +the `reduce` combinator to aggregate the word itself and the number of its +occurrences within a `Pair`. Each substream will then emit one final value—precisely such a pair—when the overall input completes. As a last step we merge back these values from the substreams into one single output stream. -One noteworthy detail pertains to the ``MAXIMUM_DISTINCT_WORDS`` parameter: +One noteworthy detail pertains to the `MAXIMUM_DISTINCT_WORDS` parameter: this defines the breadth of the merge operation. Akka Streams is focused on bounded resource consumption and the number of concurrently open inputs to the merge operator describes the amount of resources needed by the merge itself. Therefore only a finite number of substreams can be active at any given time. -If the ``groupBy`` operator encounters more keys than this number then the +If the `groupBy` operator encounters more keys than this number then the stream cannot continue without violating its resource bound, in this case -``groupBy`` will terminate with a failure. +`groupBy` will terminate with a failure. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeReduceByKeyTest.java#word-count +@@snip [RecipeReduceByKeyTest.java](../code/jdocs/stream/javadsl/cookbook/RecipeReduceByKeyTest.java) { #word-count } By extracting the parts specific to *wordcount* into -* a ``groupKey`` function that defines the groups -* a ``map`` map each element to value that is used by the reduce on the substream -* a ``reduce`` function that does the actual reduction + * a `groupKey` function that defines the groups + * a `map` map each element to value that is used by the reduce on the substream + * a `reduce` function that does the actual reduction we get a generalized version below: -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeReduceByKeyTest.java#reduce-by-key-general +@@snip [RecipeReduceByKeyTest.java](../code/jdocs/stream/javadsl/cookbook/RecipeReduceByKeyTest.java) { #reduce-by-key-general } -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeReduceByKeyTest.java#reduce-by-key-general2 +@@snip [RecipeReduceByKeyTest.java](../code/jdocs/stream/javadsl/cookbook/RecipeReduceByKeyTest.java) { #reduce-by-key-general2 } -.. note:: - Please note that the reduce-by-key version we discussed above is sequential - in reading the overall input stream, in other words it is **NOT** a - parallelization pattern like MapReduce and similar frameworks. +@@@ note -Sorting elements to multiple groups with groupBy ------------------------------------------------- +Please note that the reduce-by-key version we discussed above is sequential +in reading the overall input stream, in other words it is **NOT** a +parallelization pattern like MapReduce and similar frameworks. -**Situation:** The ``groupBy`` operation strictly partitions incoming elements, each element belongs to exactly one group. +@@@ + +### Sorting elements to multiple groups with groupBy + +**Situation:** The `groupBy` operation strictly partitions incoming elements, each element belongs to exactly one group. Sometimes we want to map elements into multiple groups simultaneously. To achieve the desired result, we attack the problem in two steps: -* first, using a function ``topicMapper`` that gives a list of topics (groups) a message belongs to, we transform our - stream of ``Message`` to a stream of :class:`Pair`` where for each topic the message belongs to a separate pair - will be emitted. This is achieved by using ``mapConcat`` -* Then we take this new stream of message topic pairs (containing a separate pair for each topic a given message - belongs to) and feed it into groupBy, using the topic as the group key. + * first, using a function `topicMapper` that gives a list of topics (groups) a message belongs to, we transform our +stream of `Message` to a stream of :class:`Pair`` where for each topic the message belongs to a separate pair +will be emitted. This is achieved by using `mapConcat` + * Then we take this new stream of message topic pairs (containing a separate pair for each topic a given message +belongs to) and feed it into groupBy, using the topic as the group key. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeMultiGroupByTest.java#multi-groupby +@@snip [RecipeMultiGroupByTest.java](../code/jdocs/stream/javadsl/cookbook/RecipeMultiGroupByTest.java) { #multi-groupby } -Working with Graphs -=================== +## Working with Graphs In this collection we show recipes that use stream graph elements to achieve various goals. -Triggering the flow of elements programmatically ------------------------------------------------- +### Triggering the flow of elements programmatically **Situation:** Given a stream of elements we want to control the emission of those elements according to a trigger signal. In other words, even if the stream would be able to flow (not being backpressured) we want to hold back elements until a trigger signal arrives. -This recipe solves the problem by simply zipping the stream of ``Message`` elements with the stream of ``Trigger`` -signals. Since ``Zip`` produces pairs, we simply map the output stream selecting the first element of the pair. +This recipe solves the problem by simply zipping the stream of `Message` elements with the stream of `Trigger` +signals. Since `Zip` produces pairs, we simply map the output stream selecting the first element of the pair. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeManualTrigger.java#manually-triggered-stream +@@snip [RecipeManualTrigger.java](../code/jdocs/stream/javadsl/cookbook/RecipeManualTrigger.java) { #manually-triggered-stream } -Alternatively, instead of using a ``Zip``, and then using ``map`` to get the first element of the pairs, we can avoid -creating the pairs in the first place by using ``ZipWith`` which takes a two argument function to produce the output -element. If this function would return a pair of the two argument it would be exactly the behavior of ``Zip`` so -``ZipWith`` is a generalization of zipping. +Alternatively, instead of using a `Zip`, and then using `map` to get the first element of the pairs, we can avoid +creating the pairs in the first place by using `ZipWith` which takes a two argument function to produce the output +element. If this function would return a pair of the two argument it would be exactly the behavior of `Zip` so +`ZipWith` is a generalization of zipping. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeManualTrigger.java#manually-triggered-stream-zipwith +@@snip [RecipeManualTrigger.java](../code/jdocs/stream/javadsl/cookbook/RecipeManualTrigger.java) { #manually-triggered-stream-zipwith } +### Balancing jobs to a fixed pool of workers -Balancing jobs to a fixed pool of workers ------------------------------------------ - -**Situation:** Given a stream of jobs and a worker process expressed as a :class:`Flow` create a pool of workers +**Situation:** Given a stream of jobs and a worker process expressed as a `Flow` create a pool of workers that automatically balances incoming jobs to available workers, then merges the results. We will express our solution as a function that takes a worker flow and the number of workers to be allocated and gives -a flow that internally contains a pool of these workers. To achieve the desired result we will create a :class:`Flow` +a flow that internally contains a pool of these workers. To achieve the desired result we will create a `Flow` from a graph. -The graph consists of a ``Balance`` node which is a special fan-out operation that tries to route elements to available -downstream consumers. In a ``for`` loop we wire all of our desired workers as outputs of this balancer element, then -we wire the outputs of these workers to a ``Merge`` element that will collect the results from the workers. +The graph consists of a `Balance` node which is a special fan-out operation that tries to route elements to available +downstream consumers. In a `for` loop we wire all of our desired workers as outputs of this balancer element, then +we wire the outputs of these workers to a `Merge` element that will collect the results from the workers. -To make the worker stages run in parallel we mark them as asynchronous with `async()`. +To make the worker stages run in parallel we mark them as asynchronous with *async()*. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeWorkerPool.java#worker-pool +@@snip [RecipeWorkerPool.java](../code/jdocs/stream/javadsl/cookbook/RecipeWorkerPool.java) { #worker-pool } -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeWorkerPool.java#worker-pool2 +@@snip [RecipeWorkerPool.java](../code/jdocs/stream/javadsl/cookbook/RecipeWorkerPool.java) { #worker-pool2 } -Working with rate -================= +## Working with rate This collection of recipes demonstrate various patterns where rate differences between upstream and downstream needs to be handled by other strategies than simple backpressure. -Dropping elements ------------------ +### Dropping elements **Situation:** Given a fast producer and a slow consumer, we want to drop elements if necessary to not slow down the producer too much. -This can be solved by using a versatile rate-transforming operation, ``conflate``. Conflate can be thought as -a special ``reduce`` operation that collapses multiple upstream elements into one aggregate element if needed to keep +This can be solved by using a versatile rate-transforming operation, `conflate`. Conflate can be thought as +a special `reduce` operation that collapses multiple upstream elements into one aggregate element if needed to keep the speed of the upstream unaffected by the downstream. -When the upstream is faster, the reducing process of the ``conflate`` starts. Our reducer function simply takes +When the upstream is faster, the reducing process of the `conflate` starts. Our reducer function simply takes the freshest element. This in a simple dropping operation. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeSimpleDrop.java#simple-drop +@@snip [RecipeSimpleDrop.java](../code/jdocs/stream/javadsl/cookbook/RecipeSimpleDrop.java) { #simple-drop } -There is a version of ``conflate`` named ``conflateWithSeed`` that allows to express more complex aggregations, more -similar to a ``fold``. +There is a version of `conflate` named `conflateWithSeed` that allows to express more complex aggregations, more +similar to a `fold`. -Dropping broadcast ------------------- +### Dropping broadcast -**Situation:** The default ``Broadcast`` graph element is properly backpressured, but that means that a slow downstream +**Situation:** The default `Broadcast` graph element is properly backpressured, but that means that a slow downstream consumer can hold back the other downstream consumers resulting in lowered throughput. In other words the rate of -``Broadcast`` is the rate of its slowest downstream consumer. In certain cases it is desirable to allow faster consumers +`Broadcast` is the rate of its slowest downstream consumer. In certain cases it is desirable to allow faster consumers to progress independently of their slower siblings by dropping elements if necessary. -One solution to this problem is to append a ``buffer`` element in front of all of the downstream consumers -defining a dropping strategy instead of the default ``Backpressure``. This allows small temporary rate differences +One solution to this problem is to append a `buffer` element in front of all of the downstream consumers +defining a dropping strategy instead of the default `Backpressure`. This allows small temporary rate differences between the different consumers (the buffer smooths out small rate variances), but also allows faster consumers to progress by dropping from the buffer of the slow consumers if necessary. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeDroppyBroadcast.java#droppy-bcast +@@snip [RecipeDroppyBroadcast.java](../code/jdocs/stream/javadsl/cookbook/RecipeDroppyBroadcast.java) { #droppy-bcast } -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeDroppyBroadcast.java#droppy-bcast2 +@@snip [RecipeDroppyBroadcast.java](../code/jdocs/stream/javadsl/cookbook/RecipeDroppyBroadcast.java) { #droppy-bcast2 } -Collecting missed ticks ------------------------ +### Collecting missed ticks **Situation:** Given a regular (stream) source of ticks, instead of trying to backpressure the producer of the ticks we want to keep a counter of the missed ticks instead and pass it down when possible. -We will use ``conflateWithSeed`` to solve the problem. Conflate takes two functions: +We will use `conflateWithSeed` to solve the problem. Conflate takes two functions: -* A seed function that produces the zero element for the folding process that happens when the upstream is faster than - the downstream. In our case the seed function is a constant function that returns 0 since there were no missed ticks - at that point. -* A fold function that is invoked when multiple upstream messages needs to be collapsed to an aggregate value due - to the insufficient processing rate of the downstream. Our folding function simply increments the currently stored - count of the missed ticks so far. + * A seed function that produces the zero element for the folding process that happens when the upstream is faster than +the downstream. In our case the seed function is a constant function that returns 0 since there were no missed ticks +at that point. + * A fold function that is invoked when multiple upstream messages needs to be collapsed to an aggregate value due +to the insufficient processing rate of the downstream. Our folding function simply increments the currently stored +count of the missed ticks so far. -As a result, we have a flow of ``Int`` where the number represents the missed ticks. A number 0 means that we were +As a result, we have a flow of `Int` where the number represents the missed ticks. A number 0 means that we were able to consume the tick fast enough (i.e. zero means: 1 non-missed tick + 0 missed ticks) -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeMissedTicks.java#missed-ticks +@@snip [RecipeMissedTicks.java](../code/jdocs/stream/javadsl/cookbook/RecipeMissedTicks.java) { #missed-ticks } -Create a stream processor that repeats the last element seen ------------------------------------------------------------- +### Create a stream processor that repeats the last element seen **Situation:** Given a producer and consumer, where the rate of neither is known in advance, we want to ensure that none of them is slowing down the other by dropping earlier unconsumed elements from the upstream if necessary, and repeating the last value for the downstream if necessary. -We have two options to implement this feature. In both cases we will use :class:`GraphStage` to build our custom -element. In the first version we will use a provided initial value ``initial`` that will be used -to feed the downstream if no upstream element is ready yet. In the ``onPush()`` handler we just overwrite the -``currentValue`` variable and immediately relieve the upstream by calling ``pull()``. The downstream ``onPull`` handler -is very similar, we immediately relieve the downstream by emitting ``currentValue``. +We have two options to implement this feature. In both cases we will use `GraphStage` to build our custom +element. In the first version we will use a provided initial value `initial` that will be used +to feed the downstream if no upstream element is ready yet. In the `onPush()` handler we just overwrite the +`currentValue` variable and immediately relieve the upstream by calling `pull()`. The downstream `onPull` handler +is very similar, we immediately relieve the downstream by emitting `currentValue`. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeHold.java#hold-version-1 +@@snip [RecipeHold.java](../code/jdocs/stream/javadsl/cookbook/RecipeHold.java) { #hold-version-1 } While it is relatively simple, the drawback of the first version is that it needs an arbitrary initial element which is not always possible to provide. Hence, we create a second version where the downstream might need to wait in one single case: if the very first element is not yet available. -We introduce a boolean variable ``waitingFirstValue`` to denote whether the first element has been provided or not -(alternatively an :class:`Optional` can be used for ``currentValue`` or if the element type is a subclass of Object -a null can be used with the same purpose). In the downstream ``onPull()`` handler the difference from the previous +We introduce a boolean variable `waitingFirstValue` to denote whether the first element has been provided or not +(alternatively an `Optional` can be used for `currentValue` or if the element type is a subclass of Object +a null can be used with the same purpose). In the downstream `onPull()` handler the difference from the previous version is that we check if we have received the first value and only emit if we have. This leads to that when the first element comes in we must check if there possibly already was demand from downstream so that we in that case can push the element directly. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeHold.java#hold-version-2 +@@snip [RecipeHold.java](../code/jdocs/stream/javadsl/cookbook/RecipeHold.java) { #hold-version-2 } -Globally limiting the rate of a set of streams ----------------------------------------------- +### Globally limiting the rate of a set of streams **Situation:** Given a set of independent streams that we cannot merge, we want to globally limit the aggregate throughput of the set of streams. One possible solution uses a shared actor as the global limiter combined with mapAsync to create a reusable -:class:`Flow` that can be plugged into a stream to limit its rate. +`Flow` that can be plugged into a stream to limit its rate. As the first step we define an actor that will do the accounting for the global rate limit. The actor maintains a timer, a counter for pending permit tokens and a queue for possibly waiting participants. The actor has -an ``open`` and ``closed`` state. The actor is in the ``open`` state while it has still pending permits. Whenever a -request for permit arrives as a ``WantToPass`` message to the actor the number of available permits is decremented -and we notify the sender that it can pass by answering with a ``MayPass`` message. If the amount of permits reaches -zero, the actor transitions to the ``closed`` state. In this state requests are not immediately answered, instead the reference -of the sender is added to a queue. Once the timer for replenishing the pending permits fires by sending a ``ReplenishTokens`` +an `open` and `closed` state. The actor is in the `open` state while it has still pending permits. Whenever a +request for permit arrives as a `WantToPass` message to the actor the number of available permits is decremented +and we notify the sender that it can pass by answering with a `MayPass` message. If the amount of permits reaches +zero, the actor transitions to the `closed` state. In this state requests are not immediately answered, instead the reference +of the sender is added to a queue. Once the timer for replenishing the pending permits fires by sending a `ReplenishTokens` message, we increment the pending permits counter and send a reply to each of the waiting senders. If there are more -waiting senders than permits available we will stay in the ``closed`` state. +waiting senders than permits available we will stay in the `closed` state. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeGlobalRateLimit.java#global-limiter-actor +@@snip [RecipeGlobalRateLimit.java](../code/jdocs/stream/javadsl/cookbook/RecipeGlobalRateLimit.java) { #global-limiter-actor } -To create a Flow that uses this global limiter actor we use the ``mapAsync`` function with the combination of the ``ask`` +To create a Flow that uses this global limiter actor we use the `mapAsync` function with the combination of the `ask` pattern. We also define a timeout, so if a reply is not received during the configured maximum wait period the returned -future from ``ask`` will fail, which will fail the corresponding stream as well. +future from `ask` will fail, which will fail the corresponding stream as well. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeGlobalRateLimit.java#global-limiter-flow +@@snip [RecipeGlobalRateLimit.java](../code/jdocs/stream/javadsl/cookbook/RecipeGlobalRateLimit.java) { #global-limiter-flow } -.. note:: - The global actor used for limiting introduces a global bottleneck. You might want to assign a dedicated dispatcher - for this actor. +@@@ note -Working with IO -=============== +The global actor used for limiting introduces a global bottleneck. You might want to assign a dedicated dispatcher +for this actor. -Chunking up a stream of ByteStrings into limited size ByteStrings ------------------------------------------------------------------ +@@@ -**Situation:** Given a stream of ``ByteString`` s we want to produce a stream of ``ByteString`` s containing the same bytes in -the same sequence, but capping the size of ``ByteString`` s. In other words we want to slice up ``ByteString`` s into smaller +## Working with IO + +### Chunking up a stream of ByteStrings into limited size ByteStrings + +**Situation:** Given a stream of `ByteString` s we want to produce a stream of `ByteString` s containing the same bytes in +the same sequence, but capping the size of `ByteString` s. In other words we want to slice up `ByteString` s into smaller chunks if they exceed a size threshold. -This can be achieved with a single :class:`GraphStage`. The main logic of our stage is in ``emitChunk()`` +This can be achieved with a single `GraphStage`. The main logic of our stage is in `emitChunk()` which implements the following logic: -* if the buffer is empty, and upstream is not closed we pull for more bytes, if it is closed we complete -* if the buffer is nonEmpty, we split it according to the ``chunkSize``. This will give a next chunk that we will emit, - and an empty or nonempty remaining buffer. + * if the buffer is empty, and upstream is not closed we pull for more bytes, if it is closed we complete + * if the buffer is nonEmpty, we split it according to the `chunkSize`. This will give a next chunk that we will emit, +and an empty or nonempty remaining buffer. -Both ``onPush()`` and ``onPull()`` calls ``emitChunk()`` the only difference is that the push handler also stores +Both `onPush()` and `onPull()` calls `emitChunk()` the only difference is that the push handler also stores the incoming chunk by appending to the end of the buffer. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java#bytestring-chunker +@@snip [RecipeByteStrings.java](../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java) { #bytestring-chunker } -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java#bytestring-chunker2 +@@snip [RecipeByteStrings.java](../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java) { #bytestring-chunker2 } -Limit the number of bytes passing through a stream of ByteStrings ------------------------------------------------------------------ +### Limit the number of bytes passing through a stream of ByteStrings -**Situation:** Given a stream of ``ByteString`` s we want to fail the stream if more than a given maximum of bytes has been +**Situation:** Given a stream of `ByteString` s we want to fail the stream if more than a given maximum of bytes has been consumed. -This recipe uses a :class:`GraphStage` to implement the desired feature. In the only handler we override, -``onPush()`` we just update a counter and see if it gets larger than ``maximumBytes``. If a violation happens +This recipe uses a `GraphStage` to implement the desired feature. In the only handler we override, +`onPush()` we just update a counter and see if it gets larger than `maximumBytes`. If a violation happens we signal failure, otherwise we forward the chunk we have received. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java#bytes-limiter +@@snip [RecipeByteStrings.java](../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java) { #bytes-limiter } -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java#bytes-limiter2 +@@snip [RecipeByteStrings.java](../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java) { #bytes-limiter2 } -Compact ByteStrings in a stream of ByteStrings ----------------------------------------------- +### Compact ByteStrings in a stream of ByteStrings -**Situation:** After a long stream of transformations, due to their immutable, structural sharing nature ``ByteString`` s may +**Situation:** After a long stream of transformations, due to their immutable, structural sharing nature `ByteString` s may refer to multiple original ByteString instances unnecessarily retaining memory. As the final step of a transformation -chain we want to have clean copies that are no longer referencing the original ``ByteString`` s. +chain we want to have clean copies that are no longer referencing the original `ByteString` s. -The recipe is a simple use of map, calling the ``compact()`` method of the :class:`ByteString` elements. This does +The recipe is a simple use of map, calling the `compact()` method of the `ByteString` elements. This does copying of the underlying arrays, so this should be the last element of a long chain if used. -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java#compacting-bytestrings +@@snip [RecipeByteStrings.java](../code/jdocs/stream/javadsl/cookbook/RecipeByteStrings.java) { #compacting-bytestrings } -Injecting keep-alive messages into a stream of ByteStrings ----------------------------------------------------------- +### Injecting keep-alive messages into a stream of ByteStrings -**Situation:** Given a communication channel expressed as a stream of ``ByteString`` s we want to inject keep-alive messages +**Situation:** Given a communication channel expressed as a stream of `ByteString` s we want to inject keep-alive messages but only if this does not interfere with normal traffic. There is a built-in operation that allows to do this directly: -.. includecode:: ../code/jdocs/stream/javadsl/cookbook/RecipeKeepAlive.java#inject-keepalive +@@snip [RecipeKeepAlive.java](../code/jdocs/stream/javadsl/cookbook/RecipeKeepAlive.java) { #inject-keepalive } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-customize.md b/akka-docs/src/main/paradox/java/stream/stream-customize.md index 9ec4c44764..ded346e4cb 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-customize.md +++ b/akka-docs/src/main/paradox/java/stream/stream-customize.md @@ -1,95 +1,91 @@ -.. _stream-customize-java: +# Custom stream processing -######################## -Custom stream processing -######################## - -While the processing vocabulary of Akka Streams is quite rich (see the :ref:`stream-cookbook-java` for examples) it +While the processing vocabulary of Akka Streams is quite rich (see the @ref:[Streams Cookbook](stream-cookbook.md) for examples) it is sometimes necessary to define new transformation stages either because some functionality is missing from the stock operations, or for performance reasons. In this part we show how to build custom processing stages and graph junctions of various kinds. -.. note:: - A custom graph stage should not be the first tool you reach for, defining graphs using flows - and the graph DSL is in general easier and does to a larger extent protect you from mistakes that - might be easy to make with a custom :class:`GraphStage` +@@@ note +A custom graph stage should not be the first tool you reach for, defining graphs using flows +and the graph DSL is in general easier and does to a larger extent protect you from mistakes that +might be easy to make with a custom `GraphStage` -.. _graphstage-java: +@@@ -Custom processing with GraphStage -================================= + +## Custom processing with GraphStage -The :class:`GraphStage` abstraction can be used to create arbitrary graph processing stages with any number of input -or output ports. It is a counterpart of the ``GraphDSL.create()`` method which creates new stream processing -stages by composing others. Where :class:`GraphStage` differs is that it creates a stage that is itself not divisible into +The `GraphStage` abstraction can be used to create arbitrary graph processing stages with any number of input +or output ports. It is a counterpart of the `GraphDSL.create()` method which creates new stream processing +stages by composing others. Where `GraphStage` differs is that it creates a stage that is itself not divisible into smaller ones, and allows state to be maintained inside it in a safe way. -As a first motivating example, we will build a new :class:`Source` that will simply emit numbers from 1 until it is +As a first motivating example, we will build a new `Source` that will simply emit numbers from 1 until it is cancelled. To start, we need to define the "interface" of our stage, which is called *shape* in Akka Streams terminology -(this is explained in more detail in the section :ref:`composition-java`). +(this is explained in more detail in the section @ref:[Modularity, Composition and Hierarchy](stream-composition.md)). -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#simple-source +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #simple-source } -As you see, in itself the :class:`GraphStage` only defines the ports of this stage and a shape that contains the ports. -It also has a user implemented method called ``createLogic``. If you recall, stages are reusable in multiple -materializations, each resulting in a different executing entity. In the case of :class:`GraphStage` the actual running -logic is modeled as an instance of a :class:`GraphStageLogic` which will be created by the materializer by calling -the ``createLogic`` method. In other words, all we need to do is to create a suitable logic that will emit the +As you see, in itself the `GraphStage` only defines the ports of this stage and a shape that contains the ports. +It also has a user implemented method called `createLogic`. If you recall, stages are reusable in multiple +materializations, each resulting in a different executing entity. In the case of `GraphStage` the actual running +logic is modeled as an instance of a `GraphStageLogic` which will be created by the materializer by calling +the `createLogic` method. In other words, all we need to do is to create a suitable logic that will emit the numbers we want. -.. note:: +@@@ note - It is very important to keep the GraphStage object itself immutable and reusable. All mutable state needs to be - confined to the GraphStageLogic that is created for every materialization. +It is very important to keep the GraphStage object itself immutable and reusable. All mutable state needs to be +confined to the GraphStageLogic that is created for every materialization. -In order to emit from a :class:`Source` in a backpressured stream one needs first to have demand from downstream. -To receive the necessary events one needs to register a subclass of :class:`AbstractOutHandler` with the output port -(:class:`Outlet`). This handler will receive events related to the lifecycle of the port. In our case we need to -override ``onPull()`` which indicates that we are free to emit a single element. There is another callback, -``onDownstreamFinish()`` which is called if the downstream cancelled. Since the default behavior of that callback is -to stop the stage, we don't need to override it. In the ``onPull`` callback we simply emit the next number. +@@@ -Instances of the above :class:`GraphStage` are subclasses of ``Graph,NotUsed>`` which means +In order to emit from a `Source` in a backpressured stream one needs first to have demand from downstream. +To receive the necessary events one needs to register a subclass of `AbstractOutHandler` with the output port +(`Outlet`). This handler will receive events related to the lifecycle of the port. In our case we need to +override `onPull()` which indicates that we are free to emit a single element. There is another callback, +`onDownstreamFinish()` which is called if the downstream cancelled. Since the default behavior of that callback is +to stop the stage, we don't need to override it. In the `onPull` callback we simply emit the next number. + +Instances of the above `GraphStage` are subclasses of `Graph,NotUsed>` which means that they are already usable in many situations, but do not provide the DSL methods we usually have for other -:class:`Source` s. In order to convert this :class:`Graph` to a proper :class:`Source` we need to wrap it using -``Source.fromGraph`` (see :ref:`composition-java` for more details about graphs and DSLs). Now we can use the +`Source` s. In order to convert this `Graph` to a proper `Source` we need to wrap it using +`Source.fromGraph` (see @ref:[Modularity, Composition and Hierarchy](stream-composition.md) for more details about graphs and DSLs). Now we can use the source as any other built-in one: -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#simple-source-usage +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #simple-source-usage } -Similarly, to create a custom :class:`Sink` one can register a subclass :class:`InHandler` with the stage :class:`Inlet`. -The ``onPush()`` callback is used to signal the handler a new element has been pushed to the stage, -and can hence be grabbed and used. ``onPush()`` can be overridden to provide custom behaviour. +Similarly, to create a custom `Sink` one can register a subclass `InHandler` with the stage `Inlet`. +The `onPush()` callback is used to signal the handler a new element has been pushed to the stage, +and can hence be grabbed and used. `onPush()` can be overridden to provide custom behaviour. Please note, most Sinks would need to request upstream elements as soon as they are created: this can be -done by calling ``pull(inlet)`` in the ``preStart()`` callback. +done by calling `pull(inlet)` in the `preStart()` callback. -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#simple-sink +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #simple-sink } -Port states, AbstractInHandler and AbstractOutHandler ------------------------------------------------------ +### Port states, AbstractInHandler and AbstractOutHandler -In order to interact with a port (:class:`Inlet` or :class:`Outlet`) of the stage we need to be able to receive events -and generate new events belonging to the port. From the :class:`GraphStageLogic` the following operations are available +In order to interact with a port (`Inlet` or `Outlet`) of the stage we need to be able to receive events +and generate new events belonging to the port. From the `GraphStageLogic` the following operations are available on an output port: -* ``push(out,elem)`` pushes an element to the output port. Only possible after the port has been pulled by downstream. -* ``complete(out)`` closes the output port normally. -* ``fail(out,exception)`` closes the port with a failure signal. + * `push(out,elem)` pushes an element to the output port. Only possible after the port has been pulled by downstream. + * `complete(out)` closes the output port normally. + * `fail(out,exception)` closes the port with a failure signal. +The events corresponding to an *output* port can be received in an `AbstractOutHandler` instance registered to the +output port using `setHandler(out,handler)`. This handler has two callbacks: -The events corresponding to an *output* port can be received in an :class:`AbstractOutHandler` instance registered to the -output port using ``setHandler(out,handler)``. This handler has two callbacks: - -* ``onPull()`` is called when the output port is ready to emit the next element, ``push(out, elem)`` is now allowed - to be called on this port. -* ``onDownstreamFinish()`` is called once the downstream has cancelled and no longer allows messages to be pushed to it. - No more ``onPull()`` will arrive after this event. If not overridden this will default to stopping the stage. + * `onPull()` is called when the output port is ready to emit the next element, `push(out, elem)` is now allowed +to be called on this port. + * `onDownstreamFinish()` is called once the downstream has cancelled and no longer allows messages to be pushed to it. +No more `onPull()` will arrive after this event. If not overridden this will default to stopping the stage. Also, there are two query methods available for output ports: -* ``isAvailable(out)`` returns true if the port can be pushed. -* ``isClosed(out)`` returns true if the port is closed. At this point the port can not be pushed and will not be pulled anymore. + * `isAvailable(out)` returns true if the port can be pushed. + * `isClosed(out)` returns true if the port is closed. At this point the port can not be pushed and will not be pulled anymore. The relationship of the above operations, events and queries are summarized in the state machine below. Green shows the initial state while orange indicates the end state. If an operation is not listed for a state, then it is invalid @@ -98,34 +94,33 @@ in that state. | -.. image:: ../../images/outport_transitions.png - :align: center +![outport_transitions.png](../../images/outport_transitions.png) | The following operations are available for *input* ports: -* ``pull(in)`` requests a new element from an input port. This is only possible after the port has been pushed by upstream. -* ``grab(in)`` acquires the element that has been received during an ``onPush()``. It cannot be called again until the - port is pushed again by the upstream. -* ``cancel(in)`` closes the input port. + * `pull(in)` requests a new element from an input port. This is only possible after the port has been pushed by upstream. + * `grab(in)` acquires the element that has been received during an `onPush()`. It cannot be called again until the +port is pushed again by the upstream. + * `cancel(in)` closes the input port. -The events corresponding to an *input* port can be received in an :class:`AbstractInHandler` instance registered to the -input port using ``setHandler(in, handler)``. This handler has three callbacks: +The events corresponding to an *input* port can be received in an `AbstractInHandler` instance registered to the +input port using `setHandler(in, handler)`. This handler has three callbacks: -* ``onPush()`` is called when the input port has now a new element. Now it is possible to acquire this element using - ``grab(in)`` and/or call ``pull(in)`` on the port to request the next element. It is not mandatory to grab the - element, but if it is pulled while the element has not been grabbed it will drop the buffered element. -* ``onUpstreamFinish()`` is called once the upstream has completed and no longer can be pulled for new elements. - No more ``onPush()`` will arrive after this event. If not overridden this will default to stopping the stage. -* ``onUpstreamFailure()`` is called if the upstream failed with an exception and no longer can be pulled for new elements. - No more ``onPush()`` will arrive after this event. If not overridden this will default to failing the stage. + * `onPush()` is called when the input port has now a new element. Now it is possible to acquire this element using +`grab(in)` and/or call `pull(in)` on the port to request the next element. It is not mandatory to grab the +element, but if it is pulled while the element has not been grabbed it will drop the buffered element. + * `onUpstreamFinish()` is called once the upstream has completed and no longer can be pulled for new elements. +No more `onPush()` will arrive after this event. If not overridden this will default to stopping the stage. + * `onUpstreamFailure()` is called if the upstream failed with an exception and no longer can be pulled for new elements. +No more `onPush()` will arrive after this event. If not overridden this will default to failing the stage. Also, there are three query methods available for input ports: -* ``isAvailable(in)`` returns true if a data element can be grabbed from the port -* ``hasBeenPulled(in)`` returns true if the port has been already pulled. Calling ``pull(in)`` in this state is illegal. -* ``isClosed(in)`` returns true if the port is closed. At this point the port can not be pulled and will not be pushed anymore. + * `isAvailable(in)` returns true if a data element can be grabbed from the port + * `hasBeenPulled(in)` returns true if the port has been already pulled. Calling `pull(in)` in this state is illegal. + * `isClosed(in)` returns true if the port is closed. At this point the port can not be pulled and will not be pushed anymore. The relationship of the above operations, events and queries are summarized in the state machine below. Green shows the initial state while orange indicates the end state. If an operation is not listed for a state, then it is invalid @@ -134,16 +129,14 @@ in that state. | -.. image:: ../../images/inport_transitions.png - :align: center +![inport_transitions.png](../../images/inport_transitions.png) | Finally, there are two methods available for convenience to complete the stage and all of its ports: -* ``completeStage()`` is equivalent to closing all output ports and cancelling all input ports. -* ``failStage(exception)`` is equivalent to failing all output ports and cancelling all input ports. - + * `completeStage()` is equivalent to closing all output ports and cancelling all input ports. + * `failStage(exception)` is equivalent to failing all output ports and cancelling all input ports. In some cases it is inconvenient and error prone to react on the regular state machine events with the signal based API described above. For those cases there is an API which allows for a more declarative sequencing @@ -151,27 +144,26 @@ of actions which will greatly simplify some use cases at the cost of some extra between the two APIs could be described as that the first one is signal driven from the outside, while this API is more active and drives its surroundings. -The operations of this part of the :class:``GraphStage`` API are: +The operations of this part of the :class:`GraphStage` API are: -* ``emit(out, elem)`` and ``emitMultiple(out, Iterable(elem1, elem2))`` replaces the ``OutHandler`` with a handler that emits - one or more elements when there is demand, and then reinstalls the current handlers -* ``read(in)(andThen)`` and ``readN(in, n)(andThen)`` replaces the ``InHandler`` with a handler that reads one or - more elements as they are pushed and allows the handler to react once the requested number of elements has been read. -* ``abortEmitting()`` and ``abortReading()`` which will cancel an ongoing emit or read + * `emit(out, elem)` and `emitMultiple(out, Iterable(elem1, elem2))` replaces the `OutHandler` with a handler that emits +one or more elements when there is demand, and then reinstalls the current handlers + * `read(in)(andThen)` and `readN(in, n)(andThen)` replaces the `InHandler` with a handler that reads one or +more elements as they are pushed and allows the handler to react once the requested number of elements has been read. + * `abortEmitting()` and `abortReading()` which will cancel an ongoing emit or read Note that since the above methods are implemented by temporarily replacing the handlers of the stage you should never -call ``setHandler`` while they are running ``emit`` or ``read`` as that interferes with how they are implemented. -The following methods are safe to call after invoking ``emit`` and ``read`` (and will lead to actually running the -operation when those are done): ``complete(out)``, ``completeStage()``, ``emit``, ``emitMultiple``, ``abortEmitting()`` -and ``abortReading()`` +call `setHandler` while they are running `emit` or `read` as that interferes with how they are implemented. +The following methods are safe to call after invoking `emit` and `read` (and will lead to actually running the +operation when those are done): `complete(out)`, `completeStage()`, `emit`, `emitMultiple`, `abortEmitting()` +and `abortReading()` -An example of how this API simplifies a stage can be found below in the second version of the :class:``Duplicator``. +An example of how this API simplifies a stage can be found below in the second version of the :class:`Duplicator`. -Custom linear processing stages using GraphStage ------------------------------------------------- +### Custom linear processing stages using GraphStage Graph stages allows for custom linear processing stages through letting them -have one input and one output and using :class:`FlowShape` as their shape. +have one input and one output and using `FlowShape` as their shape. Such a stage can be illustrated as a box with two flows as it is seen in the illustration below. Demand flowing upstream leading to elements @@ -179,58 +171,48 @@ flowing downstream. | -.. image:: ../../images/graph_stage_conceptual.png - :align: center - :width: 500 +![graph_stage_conceptual.png](../../images/graph_stage_conceptual.png) | - -To illustrate these concepts we create a small :class:`GraphStage` that implements the ``map`` transformation. +To illustrate these concepts we create a small `GraphStage` that implements the `map` transformation. | -.. image:: ../../images/graph_stage_map.png - :align: center - :width: 300 +![graph_stage_map.png](../../images/graph_stage_map.png) | -Map calls ``push(out)`` from the ``onPush()`` handler and it also calls ``pull()`` from the ``onPull`` handler resulting in the +Map calls `push(out)` from the `onPush()` handler and it also calls `pull()` from the `onPull` handler resulting in the conceptual wiring above, and fully expressed in code below: -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#one-to-one +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #one-to-one } Map is a typical example of a one-to-one transformation of a stream where demand is passed along upstream elements passed on downstream. To demonstrate a many-to-one stage we will implement -filter. The conceptual wiring of ``Filter`` looks like this: +filter. The conceptual wiring of `Filter` looks like this: | -.. image:: ../../images/graph_stage_filter.png - :align: center - :width: 300 +![graph_stage_filter.png](../../images/graph_stage_filter.png) | - As we see above, if the given predicate matches the current element we are propagating it downwards, otherwise we return the “ball” to our upstream so that we get the new element. This is achieved by modifying the map -example by adding a conditional in the ``onPush`` handler and decide between a ``pull(in)`` or ``push(out)`` call -(and of course not having a mapping ``f`` function). +example by adding a conditional in the `onPush` handler and decide between a `pull(in)` or `push(out)` call +(and of course not having a mapping `f` function). -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#many-to-one +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #many-to-one } To complete the picture we define a one-to-many transformation as the next step. We chose a straightforward example stage that emits every upstream element twice downstream. The conceptual wiring of this stage looks like this: | -.. image:: ../../images/graph_stage_duplicate.png - :align: center - :width: 300 +![graph_stage_duplicate.png](../../images/graph_stage_duplicate.png) | @@ -238,184 +220,177 @@ This is a stage that has state: an option with the last element it has seen indi has duplicated this last element already or not. We must also make sure to emit the extra element if the upstream completes. -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#one-to-many +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #one-to-many } In this case a pull from downstream might be consumed by the stage itself rather than passed along upstream as the stage might contain an element it wants to push. Note that we also need to handle the case where the upstream closes while the stage still has elements it wants to push downstream. This is done by -overriding `onUpstreamFinish` in the `AbstractInHandler` and provide custom logic +overriding *onUpstreamFinish* in the *AbstractInHandler* and provide custom logic that should happen when the upstream has been finished. This example can be simplified by replacing the usage of a mutable state with calls to -``emitMultiple`` which will replace the handlers, emit each of multiple elements and then +`emitMultiple` which will replace the handlers, emit each of multiple elements and then reinstate the original handlers: -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#simpler-one-to-many +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #simpler-one-to-many } Finally, to demonstrate all of the stages above, we put them together into a processing chain, which conceptually would correspond to the following structure: +| + +![graph_stage_chain.png](../../images/graph_stage_chain.png) | -.. image:: ../../images/graph_stage_chain.png - :align: center - :width: 700 +In code this is only a few lines, using the `via` use our custom stages in a stream: -| - -In code this is only a few lines, using the ``via`` use our custom stages in a stream: - -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#graph-stage-chain +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #graph-stage-chain } If we attempt to draw the sequence of events, it shows that there is one "event token" in circulation in a potential chain of stages, just like our conceptual "railroad tracks" representation predicts. +| + +![graph_stage_tracks_1.png](../../images/graph_stage_tracks_1.png) | -.. image:: ../../images/graph_stage_tracks_1.png - :align: center - :width: 700 - -| - - -Completion ----------- +### Completion Completion handling usually (but not exclusively) comes into the picture when processing stages need to emit a few more elements after their upstream source has been completed. We have seen an example of this in our -first :class:`Duplicator` implementation where the last element needs to be doubled even after the upstream neighbor -stage has been completed. This can be done by overriding the ``onUpstreamFinish`` method in ``AbstractInHandler``. +first `Duplicator` implementation where the last element needs to be doubled even after the upstream neighbor +stage has been completed. This can be done by overriding the `onUpstreamFinish` method in `AbstractInHandler`. Stages by default automatically stop once all of their ports (input and output) have been closed externally or internally. -It is possible to opt out from this behavior by invoking ``setKeepGoing(true)`` (which is not supported from the stage’s -constructor and usually done in ``preStart``). In this case the stage **must** be explicitly closed by calling ``completeStage()`` -or ``failStage(exception)``. This feature carries the risk of leaking streams and actors, therefore it should be used +It is possible to opt out from this behavior by invoking `setKeepGoing(true)` (which is not supported from the stage’s +constructor and usually done in `preStart`). In this case the stage **must** be explicitly closed by calling `completeStage()` +or `failStage(exception)`. This feature carries the risk of leaking streams and actors, therefore it should be used with care. -Logging inside GraphStages --------------------------- +### Logging inside GraphStages Logging debug or other important information in your stages is often a very good idea, especially when developing more advances stages which may need to be debugged at some point. -You can extend the ``akka.stream.stage.GraphStageWithLogging`` or ``akka.strea.stage.TimerGraphStageWithLogging`` classes -instead of the usual ``GraphStage`` to enable you to easily obtain a ``LoggingAdapter`` inside your stage as long as -the ``Materializer`` you're using is able to provide you with a logger. +You can extend the `akka.stream.stage.GraphStageWithLogging` or `akka.strea.stage.TimerGraphStageWithLogging` classes +instead of the usual `GraphStage` to enable you to easily obtain a `LoggingAdapter` inside your stage as long as +the `Materializer` you're using is able to provide you with a logger. -.. note:: - Please note that you can always simply use a logging library directly inside a Stage. - Make sure to use an asynchronous appender however, to not accidentally block the stage when writing to files etc. - See :ref:`slf4j-directly-java` for more details on setting up async appenders in SLF4J. +@@@ note -The stage then gets access to the ``log`` field which it can safely use from any ``GraphStage`` callbacks: +Please note that you can always simply use a logging library directly inside a Stage. +Make sure to use an asynchronous appender however, to not accidentally block the stage when writing to files etc. +See @ref:[Using the SLF4J API directly](../logging.md#slf4j-directly-java) for more details on setting up async appenders in SLF4J. -.. includecode:: ../code/jdocs/stream/GraphStageLoggingDocTest.java#stage-with-logging +@@@ -.. note:: - **SPI Note:** If you're implementing a Materializer, you can add this ability to your materializer by implementing - ``MaterializerLoggingProvider`` in your ``Materializer``. +The stage then gets access to the `log` field which it can safely use from any `GraphStage` callbacks: -Using timers ------------- +@@snip [GraphStageLoggingDocTest.java](../code/jdocs/stream/GraphStageLoggingDocTest.java) { #stage-with-logging } -It is possible to use timers in :class:`GraphStages` by using :class:`TimerGraphStageLogic` as the base class for -the returned logic. Timers can be scheduled by calling one of ``scheduleOnce(key,delay)``, ``schedulePeriodically(key,period)`` or -``schedulePeriodicallyWithInitialDelay(key,delay,period)`` and passing an object as a key for that timer (can be any object, for example -a :class:`String`). The ``onTimer(key)`` method needs to be overridden and it will be called once the timer of ``key`` -fires. It is possible to cancel a timer using ``cancelTimer(key)`` and check the status of a timer with -``isTimerActive(key)``. Timers will be automatically cleaned up when the stage completes. +@@@ note + +**SPI Note:** If you're implementing a Materializer, you can add this ability to your materializer by implementing +`MaterializerLoggingProvider` in your `Materializer`. + +@@@ + +### Using timers + +It is possible to use timers in `GraphStages` by using `TimerGraphStageLogic` as the base class for +the returned logic. Timers can be scheduled by calling one of `scheduleOnce(key,delay)`, `schedulePeriodically(key,period)` or +`schedulePeriodicallyWithInitialDelay(key,delay,period)` and passing an object as a key for that timer (can be any object, for example +a `String`). The `onTimer(key)` method needs to be overridden and it will be called once the timer of `key` +fires. It is possible to cancel a timer using `cancelTimer(key)` and check the status of a timer with +`isTimerActive(key)`. Timers will be automatically cleaned up when the stage completes. Timers can not be scheduled from the constructor of the logic, but it is possible to schedule them from the -``preStart()`` lifecycle hook. +`preStart()` lifecycle hook. In this sample the stage toggles between open and closed, where open means no elements are passed through. The stage starts out as closed but as soon as an element is pushed downstream the gate becomes open for a duration of time during which it will consume and drop upstream messages: -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#timed +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #timed } + +### Using asynchronous side-channels -Using asynchronous side-channels --------------------------------- In order to receive asynchronous events that are not arriving as stream elements (for example a completion of a future -or a callback from a 3rd party API) one must acquire a :class:`AsyncCallback` by calling ``getAsyncCallback()`` from the -stage logic. The method ``getAsyncCallback`` takes as a parameter a callback that will be called once the asynchronous +or a callback from a 3rd party API) one must acquire a `AsyncCallback` by calling `getAsyncCallback()` from the +stage logic. The method `getAsyncCallback` takes as a parameter a callback that will be called once the asynchronous event fires. It is important to **not call the callback directly**, instead, the external API must call the -``invoke(event)`` method on the returned :class:`AsyncCallback`. The execution engine will take care of calling the -provided callback in a thread-safe way. The callback can safely access the state of the :class:`GraphStageLogic` +`invoke(event)` method on the returned `AsyncCallback`. The execution engine will take care of calling the +provided callback in a thread-safe way. The callback can safely access the state of the `GraphStageLogic` implementation. Sharing the AsyncCallback from the constructor risks race conditions, therefore it is recommended to use the -``preStart()`` lifecycle hook instead. - +`preStart()` lifecycle hook instead. This example shows an asynchronous side channel graph stage that starts dropping elements when a future completes: -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#async-side-channel +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #async-side-channel } - -Integration with actors ------------------------ +### Integration with actors **This section is a stub and will be extended in the next release** **This is a :ref:`may change ` feature*** It is possible to acquire an ActorRef that can be addressed from the outside of the stage, similarly how -:class:`AsyncCallback` allows injecting asynchronous events into a stage logic. This reference can be obtained -by calling ``getStageActorRef(receive)`` passing in a function that takes a :class:`Pair` of the sender -:class:`ActorRef` and the received message. This reference can be used to watch other actors by calling its ``watch(ref)`` -or ``unwatch(ref)`` methods. The reference can be also watched by external actors. The current limitations of this -:class:`ActorRef` are: +`AsyncCallback` allows injecting asynchronous events into a stage logic. This reference can be obtained +by calling `getStageActorRef(receive)` passing in a function that takes a `Pair` of the sender +`ActorRef` and the received message. This reference can be used to watch other actors by calling its `watch(ref)` +or `unwatch(ref)` methods. The reference can be also watched by external actors. The current limitations of this +`ActorRef` are: - - they are not location transparent, they cannot be accessed via remoting. - - they cannot be returned as materialized values. - - they cannot be accessed from the constructor of the :class:`GraphStageLogic`, but they can be accessed from the - ``preStart()`` method. +> + * they are not location transparent, they cannot be accessed via remoting. + * they cannot be returned as materialized values. + * they cannot be accessed from the constructor of the `GraphStageLogic`, but they can be accessed from the +`preStart()` method. -Custom materialized values --------------------------- +### Custom materialized values -Custom stages can return materialized values instead of ``NotUsed`` by inheriting from :class:`GraphStageWithMaterializedValue` -instead of the simpler :class:`GraphStage`. The difference is that in this case the method -``createLogicAndMaterializedValue(inheritedAttributes)`` needs to be overridden, and in addition to the +Custom stages can return materialized values instead of `NotUsed` by inheriting from `GraphStageWithMaterializedValue` +instead of the simpler `GraphStage`. The difference is that in this case the method +`createLogicAndMaterializedValue(inheritedAttributes)` needs to be overridden, and in addition to the stage logic the materialized value must be provided -.. warning:: - There is no built-in synchronization of accessing this value from both of the thread where the logic runs and - the thread that got hold of the materialized value. It is the responsibility of the programmer to add the - necessary (non-blocking) synchronization and visibility guarantees to this shared object. +@@@ warning + +There is no built-in synchronization of accessing this value from both of the thread where the logic runs and +the thread that got hold of the materialized value. It is the responsibility of the programmer to add the +necessary (non-blocking) synchronization and visibility guarantees to this shared object. + +@@@ In this sample the materialized value is a future containing the first element to go through the stream: -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#materialized +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #materialized } -Using attributes to affect the behavior of a stage --------------------------------------------------- +### Using attributes to affect the behavior of a stage **This section is a stub and will be extended in the next release** -Stages can access the :class:`Attributes` object created by the materializer. This contains all the applied (inherited) +Stages can access the `Attributes` object created by the materializer. This contains all the applied (inherited) attributes applying to the stage, ordered from least specific (outermost) towards the most specific (innermost) attribute. It is the responsibility of the stage to decide how to reconcile this inheritance chain to a final effective decision. -See :ref:`composition-java` for an explanation on how attributes work. +See @ref:[Modularity, Composition and Hierarchy](stream-composition.md) for an explanation on how attributes work. - -Rate decoupled graph stages ---------------------------- +### Rate decoupled graph stages Sometimes it is desirable to *decouple* the rate of the upstream and downstream of a stage, synchronizing only when needed. -This is achieved in the model by representing a :class:`GraphStage` as a *boundary* between two regions where the +This is achieved in the model by representing a `GraphStage` as a *boundary* between two regions where the demand sent upstream is decoupled from the demand that arrives from downstream. One immediate consequence of this -difference is that an ``onPush`` call does not always lead to calling ``push`` and an ``onPull`` call does not always -lead to calling ``pull``. +difference is that an `onPush` call does not always lead to calling `push` and an `onPull` call does not always +lead to calling `pull`. One of the important use-case for this is to build buffer-like entities, that allow independent progress of upstream and downstream stages when the buffer is not full or empty, and slowing down the appropriate side if the @@ -427,58 +402,54 @@ is seen from downstream. | -.. image:: ../../images/graph_stage_detached_tracks_1.png - :align: center - :width: 500 +![graph_stage_detached_tracks_1.png](../../images/graph_stage_detached_tracks_1.png) | Another scenario would be where the demand from downstream starts coming in before any element is pushed into the buffer stage. +| + +![graph_stage_detached_tracks_2.png](../../images/graph_stage_detached_tracks_2.png) | -.. image:: ../../images/graph_stage_detached_tracks_2.png - :align: center - :width: 500 - -| - - -The first difference we can notice is that our ``Buffer`` stage is automatically pulling its upstream on +The first difference we can notice is that our `Buffer` stage is automatically pulling its upstream on initialization. The buffer has demand for up to two elements without any downstream demand. The following code example demonstrates a buffer class corresponding to the message sequence chart above. -.. includecode:: ../code/jdocs/stream/GraphStageDocTest.java#detached +@@snip [GraphStageDocTest.java](../code/jdocs/stream/GraphStageDocTest.java) { #detached } -Thread safety of custom processing stages -========================================= +## Thread safety of custom processing stages All of the above custom stages (linear or graph) provide a few simple guarantees that implementors can rely on. - - The callbacks exposed by all of these classes are never called concurrently. - - The state encapsulated by these classes can be safely modified from the provided callbacks, without any further - synchronization. - -In essence, the above guarantees are similar to what :class:`Actor` s provide, if one thinks of the state of a custom -stage as state of an actor, and the callbacks as the ``receive`` block of the actor. - -.. warning:: - It is **not safe** to access the state of any custom stage outside of the callbacks that it provides, just like it - is unsafe to access the state of an actor from the outside. This means that Future callbacks should **not close over** - internal state of custom stages because such access can be concurrent with the provided callbacks, leading to undefined - behavior. +: + * The callbacks exposed by all of these classes are never called concurrently. + * The state encapsulated by these classes can be safely modified from the provided callbacks, without any further +synchronization. -Resources and the stage lifecycle -================================= +In essence, the above guarantees are similar to what `Actor` s provide, if one thinks of the state of a custom +stage as state of an actor, and the callbacks as the `receive` block of the actor. + +@@@ warning + +It is **not safe** to access the state of any custom stage outside of the callbacks that it provides, just like it +is unsafe to access the state of an actor from the outside. This means that Future callbacks should **not close over** +internal state of custom stages because such access can be concurrent with the provided callbacks, leading to undefined +behavior. + +@@@ + +## Resources and the stage lifecycle If a stage manages a resource with a lifecycle, for example objects that need to be shutdown when they are not used anymore it is important to make sure this will happen in all circumstances when the stage shuts down. -Cleaning up resources should be done in ``GraphStageLogic.postStop`` and not in the ``InHandler`` and ``OutHandler`` +Cleaning up resources should be done in `GraphStageLogic.postStop` and not in the `InHandler` and `OutHandler` callbacks. The reason for this is that when the stage itself completes or is failed there is no signal from the upstreams or the downstreams. Even for stages that do not complete or fail in this manner, this can happen when the -``Materializer`` is shutdown or the ``ActorSystem`` is terminated while a stream is still running, what is called an +`Materializer` is shutdown or the `ActorSystem` is terminated while a stream is still running, what is called an "abrupt termination". \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-dynamic.md b/akka-docs/src/main/paradox/java/stream/stream-dynamic.md index 16f60cb73f..be1bd01210 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-dynamic.md +++ b/akka-docs/src/main/paradox/java/stream/stream-dynamic.md @@ -1,140 +1,128 @@ -.. _stream-dynamic-java: +# Dynamic stream handling -####################### -Dynamic stream handling -####################### + +## Controlling graph completion with KillSwitch -.. _kill-switch-java: +A `KillSwitch` allows the completion of graphs of `FlowShape` from the outside. It consists of a flow element that +can be linked to a graph of `FlowShape` needing completion control. +The `KillSwitch` interface allows to: -Controlling graph completion with KillSwitch --------------------------------------------- + * complete the graph(s) via `shutdown()` + * fail the graph(s) via `abort(Throwable error)` -A ``KillSwitch`` allows the completion of graphs of ``FlowShape`` from the outside. It consists of a flow element that -can be linked to a graph of ``FlowShape`` needing completion control. -The ``KillSwitch`` interface allows to: - -* complete the graph(s) via ``shutdown()`` -* fail the graph(s) via ``abort(Throwable error)`` - -After the first call to either ``shutdown`` or ``abort``, all subsequent calls to any of these methods will be ignored. +After the first call to either `shutdown` or `abort`, all subsequent calls to any of these methods will be ignored. Graph completion is performed by both -* completing its downstream -* cancelling (in case of ``shutdown``) or failing (in case of ``abort``) its upstream. + * completing its downstream + * cancelling (in case of `shutdown`) or failing (in case of `abort`) its upstream. -A ``KillSwitch`` can control the completion of one or multiple streams, and therefore comes in two different flavours. +A `KillSwitch` can control the completion of one or multiple streams, and therefore comes in two different flavours. -.. _unique-kill-switch-java: + +### UniqueKillSwitch -UniqueKillSwitch -^^^^^^^^^^^^^^^^ - -``UniqueKillSwitch`` allows to control the completion of **one** materialized ``Graph`` of ``FlowShape``. Refer to the +`UniqueKillSwitch` allows to control the completion of **one** materialized `Graph` of `FlowShape`. Refer to the below for usage examples. -* **Shutdown** + * **Shutdown** -.. includecode:: ../code/jdocs/stream/KillSwitchDocTest.java#unique-shutdown +@@snip [KillSwitchDocTest.java](../code/jdocs/stream/KillSwitchDocTest.java) { #unique-shutdown } -* **Abort** + * **Abort** -.. includecode:: ../code/jdocs/stream/KillSwitchDocTest.java#unique-abort +@@snip [KillSwitchDocTest.java](../code/jdocs/stream/KillSwitchDocTest.java) { #unique-abort } -.. _shared-kill-switch-java: + +### SharedKillSwitch -SharedKillSwitch -^^^^^^^^^^^^^^^^ - -A ``SharedKillSwitch`` allows to control the completion of an arbitrary number graphs of ``FlowShape``. It can be -materialized multiple times via its ``flow`` method, and all materialized graphs linked to it are controlled by the switch. +A `SharedKillSwitch` allows to control the completion of an arbitrary number graphs of `FlowShape`. It can be +materialized multiple times via its `flow` method, and all materialized graphs linked to it are controlled by the switch. Refer to the below for usage examples. -* **Shutdown** + * **Shutdown** -.. includecode:: ../code/jdocs/stream/KillSwitchDocTest.java#shared-shutdown +@@snip [KillSwitchDocTest.java](../code/jdocs/stream/KillSwitchDocTest.java) { #shared-shutdown } -* **Abort** + * **Abort** -.. includecode:: ../code/jdocs/stream/KillSwitchDocTest.java#shared-abort +@@snip [KillSwitchDocTest.java](../code/jdocs/stream/KillSwitchDocTest.java) { #shared-abort } -.. note:: - A ``UniqueKillSwitch`` is always a result of a materialization, whilst ``SharedKillSwitch`` needs to be constructed - before any materialization takes place. +@@@ note -Dynamic fan-in and fan-out with MergeHub and BroadcastHub ---------------------------------------------------------- +A `UniqueKillSwitch` is always a result of a materialization, whilst `SharedKillSwitch` needs to be constructed +before any materialization takes place. + +@@@ + +## Dynamic fan-in and fan-out with MergeHub and BroadcastHub There are many cases when consumers or producers of a certain service (represented as a Sink, Source, or possibly Flow) are dynamic and not known in advance. The Graph DSL does not allow to represent this, all connections of the graph must be known in advance and must be connected upfront. To allow dynamic fan-in and fan-out streaming, the Hubs -should be used. They provide means to construct :class:`Sink` and :class:`Source` pairs that are "attached" to each +should be used. They provide means to construct `Sink` and `Source` pairs that are "attached" to each other, but one of them can be materialized multiple times to implement dynamic fan-in or fan-out. -Using the MergeHub -^^^^^^^^^^^^^^^^^^ +### Using the MergeHub -A :class:`MergeHub` allows to implement a dynamic fan-in junction point in a graph where elements coming from +A `MergeHub` allows to implement a dynamic fan-in junction point in a graph where elements coming from different producers are emitted in a First-Comes-First-Served fashion. If the consumer cannot keep up then *all* of the -producers are backpressured. The hub itself comes as a :class:`Source` to which the single consumer can be attached. -It is not possible to attach any producers until this :class:`Source` has been materialized (started). This is ensured -by the fact that we only get the corresponding :class:`Sink` as a materialized value. Usage might look like this: +producers are backpressured. The hub itself comes as a `Source` to which the single consumer can be attached. +It is not possible to attach any producers until this `Source` has been materialized (started). This is ensured +by the fact that we only get the corresponding `Sink` as a materialized value. Usage might look like this: -.. includecode:: ../code/jdocs/stream/HubDocTest.java#merge-hub +@@snip [HubDocTest.java](../code/jdocs/stream/HubDocTest.java) { #merge-hub } -This sequence, while might look odd at first, ensures proper startup order. Once we get the :class:`Sink`, +This sequence, while might look odd at first, ensures proper startup order. Once we get the `Sink`, we can use it as many times as wanted. Everything that is fed to it will be delivered to the consumer we attached previously until it cancels. -Using the BroadcastHub -^^^^^^^^^^^^^^^^^^^^^^ +### Using the BroadcastHub -A :class:`BroadcastHub` can be used to consume elements from a common producer by a dynamic set of consumers. The -rate of the producer will be automatically adapted to the slowest consumer. In this case, the hub is a :class:`Sink` -to which the single producer must be attached first. Consumers can only be attached once the :class:`Sink` has -been materialized (i.e. the producer has been started). One example of using the :class:`BroadcastHub`: +A `BroadcastHub` can be used to consume elements from a common producer by a dynamic set of consumers. The +rate of the producer will be automatically adapted to the slowest consumer. In this case, the hub is a `Sink` +to which the single producer must be attached first. Consumers can only be attached once the `Sink` has +been materialized (i.e. the producer has been started). One example of using the `BroadcastHub`: -.. includecode:: ../code/jdocs/stream/HubDocTest.java#broadcast-hub +@@snip [HubDocTest.java](../code/jdocs/stream/HubDocTest.java) { #broadcast-hub } -The resulting :class:`Source` can be materialized any number of times, each materialization effectively attaching +The resulting `Source` can be materialized any number of times, each materialization effectively attaching a new subscriber. If there are no subscribers attached to this hub then it will not drop any elements but instead backpressure the upstream producer until subscribers arrive. This behavior can be tweaked by using the combinators -``.buffer`` for example with a drop strategy, or just attaching a subscriber that drops all messages. If there +`.buffer` for example with a drop strategy, or just attaching a subscriber that drops all messages. If there are no other subscribers, this will ensure that the producer is kept drained (dropping all elements) and once a new subscriber arrives it will adaptively slow down, ensuring no more messages are dropped. -Combining dynamic stages to build a simple Publish-Subscribe service -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Combining dynamic stages to build a simple Publish-Subscribe service The features provided by the Hub implementations are limited by default. This is by design, as various combinations can be used to express additional features like unsubscribing producers or consumers externally. We show here -an example that builds a :class:`Flow` representing a publish-subscribe channel. The input of the :class:`Flow` is +an example that builds a `Flow` representing a publish-subscribe channel. The input of the `Flow` is published to all subscribers while the output streams all the elements published. -First, we connect a :class:`MergeHub` and a :class:`BroadcastHub` together to form a publish-subscribe channel. Once -we materialize this small stream, we get back a pair of :class:`Source` and :class:`Sink` that together define +First, we connect a `MergeHub` and a `BroadcastHub` together to form a publish-subscribe channel. Once +we materialize this small stream, we get back a pair of `Source` and `Sink` that together define the publish and subscribe sides of our channel. -.. includecode:: ../code/jdocs/stream/HubDocTest.java#pub-sub-1 +@@snip [HubDocTest.java](../code/jdocs/stream/HubDocTest.java) { #pub-sub-1 } -We now use a few tricks to add more features. First of all, we attach a ``Sink.ignore`` +We now use a few tricks to add more features. First of all, we attach a `Sink.ignore` at the broadcast side of the channel to keep it drained when there are no subscribers. If this behavior is not the desired one this line can be simply dropped. -.. includecode:: ../code/jdocs/stream/HubDocTest.java#pub-sub-2 +@@snip [HubDocTest.java](../code/jdocs/stream/HubDocTest.java) { #pub-sub-2 } -We now wrap the :class:`Sink` and :class:`Source` in a :class:`Flow` using ``Flow.fromSinkAndSource``. This bundles +We now wrap the `Sink` and `Source` in a `Flow` using `Flow.fromSinkAndSource`. This bundles up the two sides of the channel into one and forces users of it to always define a publisher and subscriber side -(even if the subscriber side is just dropping). It also allows us to very simply attach a :class:`KillSwitch` as -a :class:`BidiStage` which in turn makes it possible to close both the original :class:`Sink` and :class:`Source` at the +(even if the subscriber side is just dropping). It also allows us to very simply attach a `KillSwitch` as +a `BidiStage` which in turn makes it possible to close both the original `Sink` and `Source` at the same time. -Finally, we add ``backpressureTimeout`` on the consumer side to ensure that subscribers that block the channel for more +Finally, we add `backpressureTimeout` on the consumer side to ensure that subscribers that block the channel for more than 3 seconds are forcefully removed (and their stream failed). -.. includecode:: ../code/jdocs/stream/HubDocTest.java#pub-sub-3 +@@snip [HubDocTest.java](../code/jdocs/stream/HubDocTest.java) { #pub-sub-3 } -The resulting Flow now has a type of ``Flow[String, String, UniqueKillSwitch]`` representing a publish-subscribe +The resulting Flow now has a type of `Flow[String, String, UniqueKillSwitch]` representing a publish-subscribe channel which can be used any number of times to attach new producers or consumers. In addition, it materializes -to a :class:`UniqueKillSwitch` (see :ref:`unique-kill-switch-java`) that can be used to deregister a single user externally: +to a `UniqueKillSwitch` (see [UniqueKillSwitch](#unique-kill-switch-java)) that can be used to deregister a single user externally: - -.. includecode:: ../code/jdocs/stream/HubDocTest.java#pub-sub-4 +@@snip [HubDocTest.java](../code/jdocs/stream/HubDocTest.java) { #pub-sub-4 } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-error.md b/akka-docs/src/main/paradox/java/stream/stream-error.md index a701b6f648..d0e8992b7d 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-error.md +++ b/akka-docs/src/main/paradox/java/stream/stream-error.md @@ -1,79 +1,76 @@ -.. _stream-error-java: - -############## -Error Handling -############## +# Error Handling Strategies for how to handle exceptions from processing stream elements can be defined when materializing the stream. The error handling strategies are inspired by actor supervision strategies, but the semantics have been adapted to the domain of stream processing. -.. warning:: +@@@ warning - *ZipWith*, *GraphStage* junction, *ActorPublisher* source and *ActorSubscriber* sink - components do not honour the supervision strategy attribute yet. +*ZipWith*, *GraphStage* junction, *ActorPublisher* source and *ActorSubscriber* sink +components do not honour the supervision strategy attribute yet. -Supervision Strategies -====================== +@@@ + +## Supervision Strategies There are three ways to handle exceptions from application code: -* ``Stop`` - The stream is completed with failure. -* ``Resume`` - The element is dropped and the stream continues. -* ``Restart`` - The element is dropped and the stream continues after restarting the stage. - Restarting a stage means that any accumulated state is cleared. This is typically - performed by creating a new instance of the stage. - + * `Stop` - The stream is completed with failure. + * `Resume` - The element is dropped and the stream continues. + * `Restart` - The element is dropped and the stream continues after restarting the stage. +Restarting a stage means that any accumulated state is cleared. This is typically +performed by creating a new instance of the stage. By default the stopping strategy is used for all exceptions, i.e. the stream will be completed with failure when an exception is thrown. -.. includecode:: ../code/jdocs/stream/FlowErrorDocTest.java#stop +@@snip [FlowErrorDocTest.java](../code/jdocs/stream/FlowErrorDocTest.java) { #stop } The default supervision strategy for a stream can be defined on the settings of the materializer. -.. includecode:: ../code/jdocs/stream/FlowErrorDocTest.java#resume +@@snip [FlowErrorDocTest.java](../code/jdocs/stream/FlowErrorDocTest.java) { #resume } -Here you can see that all ``ArithmeticException`` will resume the processing, i.e. the +Here you can see that all `ArithmeticException` will resume the processing, i.e. the elements that cause the division by zero are effectively dropped. -.. note:: +@@@ note - Be aware that dropping elements may result in deadlocks in graphs with - cycles, as explained in :ref:`graph-cycles-java`. +Be aware that dropping elements may result in deadlocks in graphs with +cycles, as explained in @ref:[Graph cycles, liveness and deadlocks](stream-graphs.md#graph-cycles-java). + +@@@ The supervision strategy can also be defined for all operators of a flow. -.. includecode:: ../code/jdocs/stream/FlowErrorDocTest.java#resume-section +@@snip [FlowErrorDocTest.java](../code/jdocs/stream/FlowErrorDocTest.java) { #resume-section } -``Restart`` works in a similar way as ``Resume`` with the addition that accumulated state, +`Restart` works in a similar way as `Resume` with the addition that accumulated state, if any, of the failing processing stage will be reset. -.. includecode:: ../code/jdocs/stream/FlowErrorDocTest.java#restart-section +@@snip [FlowErrorDocTest.java](../code/jdocs/stream/FlowErrorDocTest.java) { #restart-section } -Errors from mapAsync -==================== +## Errors from mapAsync -Stream supervision can also be applied to the futures of ``mapAsync``. +Stream supervision can also be applied to the futures of `mapAsync`. Let's say that we use an external service to lookup email addresses and we would like to discard those that cannot be found. We start with the tweet stream of authors: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#tweet-authors +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #tweet-authors } Assume that we can lookup their email address using: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#email-address-lookup2 +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #email-address-lookup2 } -The ``CompletionStage`` is completed normally if the email is not found. +The `CompletionStage` is completed normally if the email is not found. -Transforming the stream of authors to a stream of email addresses by using the ``lookupEmail`` -service can be done with ``mapAsync`` and we use ``Supervision.getResumingDecider`` to drop +Transforming the stream of authors to a stream of email addresses by using the `lookupEmail` +service can be done with `mapAsync` and we use `Supervision.getResumingDecider` to drop unknown email addresses: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#email-addresses-mapAsync-supervision +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #email-addresses-mapAsync-supervision } -If we would not use ``Resume`` the default stopping strategy would complete the stream -with failure on the first ``CompletionStage`` that was completed exceptionally. +If we would not use `Resume` the default stopping strategy would complete the stream +with failure on the first `CompletionStage` that was completed exceptionally. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-flows-and-basics.md b/akka-docs/src/main/paradox/java/stream/stream-flows-and-basics.md index 81e7c0767b..57fb0ec90a 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-flows-and-basics.md +++ b/akka-docs/src/main/paradox/java/stream/stream-flows-and-basics.md @@ -1,13 +1,7 @@ -.. _stream-flow-java: +# Basics and working with Flows -############################# -Basics and working with Flows -############################# - -.. _core-concepts-java: - -Core concepts -============= + +## Core concepts Akka Streams is a library to process and transfer a sequence of elements using bounded buffer space. This latter property is what we refer to as *boundedness* and it is the defining feature of Akka Streams. Translated to @@ -20,24 +14,30 @@ do not drop. Before we move on, let's define some basic terminology which will be used throughout the entire documentation: Stream - An active process that involves moving and transforming data. +: An active process that involves moving and transforming data. + Element - An element is the processing unit of streams. All operations transform and transfer elements from upstream to - downstream. Buffer sizes are always expressed as number of elements independently form the actual size of the elements. +: An element is the processing unit of streams. All operations transform and transfer elements from upstream to +downstream. Buffer sizes are always expressed as number of elements independently form the actual size of the elements. + Back-pressure - A means of flow-control, a way for consumers of data to notify a producer about their current availability, effectively - slowing down the upstream producer to match their consumption speeds. - In the context of Akka Streams back-pressure is always understood as *non-blocking* and *asynchronous*. +: A means of flow-control, a way for consumers of data to notify a producer about their current availability, effectively +slowing down the upstream producer to match their consumption speeds. +In the context of Akka Streams back-pressure is always understood as *non-blocking* and *asynchronous*. + Non-Blocking - Means that a certain operation does not hinder the progress of the calling thread, even if it takes long time to - finish the requested operation. +: Means that a certain operation does not hinder the progress of the calling thread, even if it takes long time to +finish the requested operation. + Graph - A description of a stream processing topology, defining the pathways through which elements shall flow when the stream - is running. +: A description of a stream processing topology, defining the pathways through which elements shall flow when the stream +is running. + Processing Stage - The common name for all building blocks that build up a Graph. - Examples of a processing stage would be operations like ``map()``, ``filter()``, custom ``GraphStage`` s and graph - junctions like ``Merge`` or ``Broadcast``. For the full list of built-in processing stages see :ref:`stages-overview_java` +: The common name for all building blocks that build up a Graph. +Examples of a processing stage would be operations like `map()`, `filter()`, custom `GraphStage` s and graph +junctions like `Merge` or `Broadcast`. For the full list of built-in processing stages see @ref:[stages-overview_java](stages-overview.md) + When we talk about *asynchronous, non-blocking backpressure* we mean that the processing stages available in Akka Streams will not use blocking calls but asynchronous message passing to exchange messages between each other, and they @@ -45,119 +45,120 @@ will use asynchronous means to slow down a fast producer, without blocking its t design, since entities that need to wait (a fast producer waiting on a slow consumer) will not block the thread but can hand it back for further use to an underlying thread-pool. -.. _defining-and-running-streams-java: - -Defining and running streams -============================ + +## Defining and running streams Linear processing pipelines can be expressed in Akka Streams using the following core abstractions: Source - A processing stage with *exactly one output*, emitting data elements whenever downstream processing stages are - ready to receive them. +: A processing stage with *exactly one output*, emitting data elements whenever downstream processing stages are +ready to receive them. + Sink - A processing stage with *exactly one input*, requesting and accepting data elements possibly slowing down the upstream - producer of elements +: A processing stage with *exactly one input*, requesting and accepting data elements possibly slowing down the upstream +producer of elements + Flow - A processing stage which has *exactly one input and output*, which connects its up- and downstreams by - transforming the data elements flowing through it. +: A processing stage which has *exactly one input and output*, which connects its up- and downstreams by +transforming the data elements flowing through it. + RunnableGraph - A Flow that has both ends "attached" to a Source and Sink respectively, and is ready to be ``run()``. +: A Flow that has both ends "attached" to a Source and Sink respectively, and is ready to be `run()`. -It is possible to attach a ``Flow`` to a ``Source`` resulting in a composite source, and it is also possible to prepend -a ``Flow`` to a ``Sink`` to get a new sink. After a stream is properly terminated by having both a source and a sink, -it will be represented by the ``RunnableGraph`` type, indicating that it is ready to be executed. -It is important to remember that even after constructing the ``RunnableGraph`` by connecting all the source, sink and +It is possible to attach a `Flow` to a `Source` resulting in a composite source, and it is also possible to prepend +a `Flow` to a `Sink` to get a new sink. After a stream is properly terminated by having both a source and a sink, +it will be represented by the `RunnableGraph` type, indicating that it is ready to be executed. + +It is important to remember that even after constructing the `RunnableGraph` by connecting all the source, sink and different processing stages, no data will flow through it until it is materialized. Materialization is the process of allocating all resources needed to run the computation described by a Graph (in Akka Streams this will often involve starting up Actors). Thanks to Flows being simply a description of the processing pipeline they are *immutable, thread-safe, and freely shareable*, which means that it is for example safe to share and send them between actors, to have one actor prepare the work, and then have it be materialized at some completely different place in the code. -.. includecode:: ../code/jdocs/stream/FlowDocTest.java#materialization-in-steps +@@snip [FlowDocTest.java](../code/jdocs/stream/FlowDocTest.java) { #materialization-in-steps } -After running (materializing) the ``RunnableGraph`` we get a special container object, the ``MaterializedMap``. Both +After running (materializing) the `RunnableGraph` we get a special container object, the `MaterializedMap`. Both sources and sinks are able to put specific objects into this map. Whether they put something in or not is implementation -dependent. For example a ``FoldSink`` will make a ``CompletionStage`` available in this map which will represent the result +dependent. For example a `FoldSink` will make a `CompletionStage` available in this map which will represent the result of the folding process over the stream. In general, a stream can expose multiple materialized values, but it is quite common to be interested in only the value of the Source or the Sink in the stream. For this reason -there is a convenience method called ``runWith()`` available for ``Sink``, ``Source`` or ``Flow`` requiring, respectively, -a supplied ``Source`` (in order to run a ``Sink``), a ``Sink`` (in order to run a ``Source``) or -both a ``Source`` and a ``Sink`` (in order to run a ``Flow``, since it has neither attached yet). +there is a convenience method called `runWith()` available for `Sink`, `Source` or `Flow` requiring, respectively, +a supplied `Source` (in order to run a `Sink`), a `Sink` (in order to run a `Source`) or +both a `Source` and a `Sink` (in order to run a `Flow`, since it has neither attached yet). -.. includecode:: ../code/jdocs/stream/FlowDocTest.java#materialization-runWith +@@snip [FlowDocTest.java](../code/jdocs/stream/FlowDocTest.java) { #materialization-runWith } It is worth pointing out that since processing stages are *immutable*, connecting them returns a new processing stage, instead of modifying the existing instance, so while constructing long flows, remember to assign the new value to a variable or run it: -.. includecode:: ../code/jdocs/stream/FlowDocTest.java#source-immutable +@@snip [FlowDocTest.java](../code/jdocs/stream/FlowDocTest.java) { #source-immutable } -.. note:: - By default Akka Streams elements support **exactly one** downstream processing stage. - Making fan-out (supporting multiple downstream processing stages) an explicit opt-in feature allows default stream elements to - be less complex and more efficient. Also it allows for greater flexibility on *how exactly* to handle the multicast scenarios, - by providing named fan-out elements such as broadcast (signals all down-stream elements) or balance (signals one of available down-stream elements). +@@@ note -In the above example we used the ``runWith`` method, which both materializes the stream and returns the materialized value +By default Akka Streams elements support **exactly one** downstream processing stage. +Making fan-out (supporting multiple downstream processing stages) an explicit opt-in feature allows default stream elements to +be less complex and more efficient. Also it allows for greater flexibility on *how exactly* to handle the multicast scenarios, +by providing named fan-out elements such as broadcast (signals all down-stream elements) or balance (signals one of available down-stream elements). + +@@@ + +In the above example we used the `runWith` method, which both materializes the stream and returns the materialized value of the given sink or source. -Since a stream can be materialized multiple times, the ``MaterializedMap`` returned is different for each materialization. -In the example below we create two running materialized instance of the stream that we described in the ``runnable`` -variable, and both materializations give us a different ``CompletionStage`` from the map even though we used the same ``sink`` +Since a stream can be materialized multiple times, the `MaterializedMap` returned is different for each materialization. +In the example below we create two running materialized instance of the stream that we described in the `runnable` +variable, and both materializations give us a different `CompletionStage` from the map even though we used the same `sink` to refer to the future: -.. includecode:: ../code/jdocs/stream/FlowDocTest.java#stream-reuse +@@snip [FlowDocTest.java](../code/jdocs/stream/FlowDocTest.java) { #stream-reuse } -Defining sources, sinks and flows ---------------------------------- +### Defining sources, sinks and flows -The objects :class:`Source` and :class:`Sink` define various ways to create sources and sinks of elements. The following +The objects `Source` and `Sink` define various ways to create sources and sinks of elements. The following examples show some of the most useful constructs (refer to the API documentation for more details): -.. includecode:: ../code/jdocs/stream/FlowDocTest.java#source-sink +@@snip [FlowDocTest.java](../code/jdocs/stream/FlowDocTest.java) { #source-sink } There are various ways to wire up different parts of a stream, the following examples show some of the available options: -.. includecode:: ../code/jdocs/stream/FlowDocTest.java#flow-connecting +@@snip [FlowDocTest.java](../code/jdocs/stream/FlowDocTest.java) { #flow-connecting } -Illegal stream elements ------------------------ +### Illegal stream elements -In accordance to the Reactive Streams specification (`Rule 2.13 `_) -Akka Streams do not allow ``null`` to be passed through the stream as an element. In case you want to model the concept -of absence of a value we recommend using ``java.util.Optional`` which is available since Java 8. +In accordance to the Reactive Streams specification ([Rule 2.13](https://github.com/reactive-streams/reactive-streams-jvm#2.13)) +Akka Streams do not allow `null` to be passed through the stream as an element. In case you want to model the concept +of absence of a value we recommend using `java.util.Optional` which is available since Java 8. -.. _back-pressure-explained-java: + +## Back-pressure explained -Back-pressure explained -======================= - -Akka Streams implement an asynchronous non-blocking back-pressure protocol standardised by the `Reactive Streams`_ +Akka Streams implement an asynchronous non-blocking back-pressure protocol standardised by the [Reactive Streams](http://reactive-streams.org/) specification, which Akka is a founding member of. -.. _Reactive Streams: http://reactive-streams.org/ - The user of the library does not have to write any explicit back-pressure handling code — it is built in and dealt with automatically by all of the provided Akka Streams processing stages. It is possible however to add explicit buffer stages with overflow strategies that can influence the behaviour of the stream. This is especially important in complex processing graphs which may even contain loops (which *must* be treated with very special -care, as explained in :ref:`graph-cycles-java`). +care, as explained in @ref:[Graph cycles, liveness and deadlocks](stream-graphs.md#graph-cycles-java)). -The back pressure protocol is defined in terms of the number of elements a downstream ``Subscriber`` is able to receive -and buffer, referred to as ``demand``. -The source of data, referred to as ``Publisher`` in Reactive Streams terminology and implemented as ``Source`` in Akka -Streams, guarantees that it will never emit more elements than the received total demand for any given ``Subscriber``. +The back pressure protocol is defined in terms of the number of elements a downstream `Subscriber` is able to receive +and buffer, referred to as `demand`. +The source of data, referred to as `Publisher` in Reactive Streams terminology and implemented as `Source` in Akka +Streams, guarantees that it will never emit more elements than the received total demand for any given `Subscriber`. -.. note:: +@@@ note - The Reactive Streams specification defines its protocol in terms of ``Publisher`` and ``Subscriber``. - These types are **not** meant to be user facing API, instead they serve as the low level building blocks for - different Reactive Streams implementations. +The Reactive Streams specification defines its protocol in terms of `Publisher` and `Subscriber`. +These types are **not** meant to be user facing API, instead they serve as the low level building blocks for +different Reactive Streams implementations. - Akka Streams implements these concepts as ``Source``, ``Flow`` (referred to as ``Processor`` in Reactive Streams) - and ``Sink`` without exposing the Reactive Streams interfaces directly. - If you need to integrate with other Reactive Stream libraries read :ref:`reactive-streams-integration-java`. +Akka Streams implements these concepts as `Source`, `Flow` (referred to as `Processor` in Reactive Streams) +and `Sink` without exposing the Reactive Streams interfaces directly. +If you need to integrate with other Reactive Stream libraries read @ref:[Integrating with Reactive Streams](stream-integrations.md#reactive-streams-integration-java). + +@@@ The mode in which Reactive Streams back-pressure works can be colloquially described as "dynamic push / pull mode", since it will switch between push and pull based back-pressure models depending on the downstream being able to cope @@ -165,8 +166,7 @@ with the upstream production rate or not. To illustrate this further let us consider both problem situations and how the back-pressure protocol handles them: -Slow Publisher, fast Subscriber -------------------------------- +### Slow Publisher, fast Subscriber This is the happy case of course – we do not need to slow down the Publisher in this case. However signalling rates are rarely constant and could change at any point in time, suddenly ending up in a situation where the Subscriber is now @@ -174,7 +174,7 @@ slower than the Publisher. In order to safeguard from these situations, the back during such situations, however we do not want to pay a high penalty for this safety net being enabled. The Reactive Streams protocol solves this by asynchronously signalling from the Subscriber to the Publisher -``Request(int n)`` signals. The protocol guarantees that the Publisher will never signal *more* elements than the +`Request(int n)` signals. The protocol guarantees that the Publisher will never signal *more* elements than the signalled demand. Since the Subscriber however is currently faster, it will be signalling these Request messages at a higher rate (and possibly also batching together the demand - requesting multiple elements in one Request signal). This means that the Publisher should not ever have to wait (be back-pressured) with publishing its incoming elements. @@ -182,64 +182,63 @@ that the Publisher should not ever have to wait (be back-pressured) with publish As we can see, in this scenario we effectively operate in so called push-mode since the Publisher can continue producing elements as fast as it can, since the pending demand will be recovered just-in-time while it is emitting elements. -Fast Publisher, slow Subscriber -------------------------------- +### Fast Publisher, slow Subscriber -This is the case when back-pressuring the ``Publisher`` is required, because the ``Subscriber`` is not able to cope with +This is the case when back-pressuring the `Publisher` is required, because the `Subscriber` is not able to cope with the rate at which its upstream would like to emit data elements. -Since the ``Publisher`` is not allowed to signal more elements than the pending demand signalled by the ``Subscriber``, +Since the `Publisher` is not allowed to signal more elements than the pending demand signalled by the `Subscriber`, it will have to abide to this back-pressure by applying one of the below strategies: -- not generate elements, if it is able to control their production rate, -- try buffering the elements in a *bounded* manner until more demand is signalled, -- drop elements until more demand is signalled, -- tear down the stream if unable to apply any of the above strategies. + * not generate elements, if it is able to control their production rate, + * try buffering the elements in a *bounded* manner until more demand is signalled, + * drop elements until more demand is signalled, + * tear down the stream if unable to apply any of the above strategies. -As we can see, this scenario effectively means that the ``Subscriber`` will *pull* the elements from the Publisher – +As we can see, this scenario effectively means that the `Subscriber` will *pull* the elements from the Publisher – this mode of operation is referred to as pull-based back-pressure. -.. _stream-materialization-java: - -Stream Materialization -====================== + +## Stream Materialization When constructing flows and graphs in Akka Streams think of them as preparing a blueprint, an execution plan. Stream materialization is the process of taking a stream description (the graph) and allocating all the necessary resources it needs in order to run. In the case of Akka Streams this often means starting up Actors which power the processing, but is not restricted to that—it could also mean opening files or socket connections etc.—depending on what the stream needs. -Materialization is triggered at so called "terminal operations". Most notably this includes the various forms of the ``run()`` -and ``runWith()`` methods defined on :class:`Source` or :class:`Flow` elements as well as a small number of special syntactic sugars for running with -well-known sinks, such as ``runForeach(el -> ...)`` (being an alias to ``runWith(Sink.foreach(el -> ...))``. +Materialization is triggered at so called "terminal operations". Most notably this includes the various forms of the `run()` +and `runWith()` methods defined on `Source` or `Flow` elements as well as a small number of special syntactic sugars for running with +well-known sinks, such as `runForeach(el -> ...)` (being an alias to `runWith(Sink.foreach(el -> ...))`. Materialization is currently performed synchronously on the materializing thread. The actual stream processing is handled by actors started up during the streams materialization, which will be running on the thread pools they have been configured to run on - which defaults to the dispatcher set in -:class:`MaterializationSettings` while constructing the :class:`ActorMaterializer`. +`MaterializationSettings` while constructing the `ActorMaterializer`. -.. note:: - Reusing *instances* of linear computation stages (Source, Sink, Flow) inside composite Graphs is legal, - yet will materialize that stage multiple times. +@@@ note -.. _operator-fusion-java: +Reusing *instances* of linear computation stages (Source, Sink, Flow) inside composite Graphs is legal, +yet will materialize that stage multiple times. -Operator Fusion ---------------- +@@@ + + +### Operator Fusion By default Akka Streams will fuse the stream operators. This means that the processing steps of a flow or stream graph can be executed within the same Actor and has two consequences: - * passing elements from one processing stage to the next is a lot faster between fused - stages due to avoiding the asynchronous messaging overhead - * fused stream processing stages does not run in parallel to each other, meaning that - only up to one CPU core is used for each fused part +> + * passing elements from one processing stage to the next is a lot faster between fused +stages due to avoiding the asynchronous messaging overhead + * fused stream processing stages does not run in parallel to each other, meaning that +only up to one CPU core is used for each fused part To allow for parallel processing you will have to insert asynchronous boundaries manually into your flows and -graphs by way of adding ``Attributes.asyncBoundary`` using the method ``async`` on ``Source``, ``Sink`` and ``Flow`` +graphs by way of adding `Attributes.asyncBoundary` using the method `async` on `Source`, `Sink` and `Flow` to pieces that shall communicate with the rest of the graph in an asynchronous fashion. -.. includecode:: ../code/jdocs/stream/FlowDocTest.java#flow-async +@@snip [FlowDocTest.java](../code/jdocs/stream/FlowDocTest.java) { #flow-async } In this example we create two regions within the flow which will be executed in one Actor each—assuming that adding and multiplying integers is an extremely costly operation this will lead to a performance gain since two CPUs can @@ -249,9 +248,7 @@ by adding information to the flow graph that has been constructed up to this poi | -.. image:: ../../images/asyncBoundary.png - :align: center - :width: 700 +![asyncBoundary.png](../../images/asyncBoundary.png) | @@ -259,48 +256,51 @@ This means that everything that is inside the red bubble will be executed by one by another. This scheme can be applied successively, always having one such boundary enclose the previous ones plus all processing stages that have been added since them. -.. warning:: +@@@ warning - Without fusing (i.e. up to version 2.0-M2) each stream processing stage had an implicit input buffer - that holds a few elements for efficiency reasons. If your flow graphs contain cycles then these buffers - may have been crucial in order to avoid deadlocks. With fusing these implicit buffers are no longer - there, data elements are passed without buffering between fused stages. In those cases where buffering - is needed in order to allow the stream to run at all, you will have to insert explicit buffers with the - ``.buffer()`` combinator—typically a buffer of size 2 is enough to allow a feedback loop to function. +Without fusing (i.e. up to version 2.0-M2) each stream processing stage had an implicit input buffer +that holds a few elements for efficiency reasons. If your flow graphs contain cycles then these buffers +may have been crucial in order to avoid deadlocks. With fusing these implicit buffers are no longer +there, data elements are passed without buffering between fused stages. In those cases where buffering +is needed in order to allow the stream to run at all, you will have to insert explicit buffers with the +`.buffer()` combinator—typically a buffer of size 2 is enough to allow a feedback loop to function. -The new fusing behavior can be disabled by setting the configuration parameter ``akka.stream.materializer.auto-fusing=off``. +@@@ + +The new fusing behavior can be disabled by setting the configuration parameter `akka.stream.materializer.auto-fusing=off`. In that case you can still manually fuse those graphs which shall run on less Actors. With the exception of the -:class:`SslTlsStage` and the ``groupBy`` operator all built-in processing stages can be fused. +`SslTlsStage` and the `groupBy` operator all built-in processing stages can be fused. -Combining materialized values ------------------------------ +### Combining materialized values Since every processing stage in Akka Streams can provide a materialized value after being materialized, it is necessary to somehow express how these values should be composed to a final value when we plug these stages together. For this, many combinator methods have variants that take an additional argument, a function, that will be used to combine the resulting values. Some examples of using these combiners are illustrated in the example below. -.. includecode:: ../code/jdocs/stream/FlowDocTest.java#flow-mat-combine +@@snip [FlowDocTest.java](../code/jdocs/stream/FlowDocTest.java) { #flow-mat-combine } -.. note:: +@@@ note - In Graphs it is possible to access the materialized value from inside the stream processing graph. For details see :ref:`graph-matvalue-java`. +In Graphs it is possible to access the materialized value from inside the stream processing graph. For details see @ref:[Accessing the materialized value inside the Graph](stream-graphs.md#graph-matvalue-java). -Stream ordering -=============== -In Akka Streams almost all computation stages *preserve input order* of elements. This means that if inputs ``{IA1,IA2,...,IAn}`` -"cause" outputs ``{OA1,OA2,...,OAk}`` and inputs ``{IB1,IB2,...,IBm}`` "cause" outputs ``{OB1,OB2,...,OBl}`` and all of -``IAi`` happened before all ``IBi`` then ``OAi`` happens before ``OBi``. +@@@ -This property is even uphold by async operations such as ``mapAsync``, however an unordered version exists -called ``mapAsyncUnordered`` which does not preserve this ordering. +## Stream ordering -However, in the case of Junctions which handle multiple input streams (e.g. :class:`Merge`) the output order is, -in general, *not defined* for elements arriving on different input ports. That is a merge-like operation may emit ``Ai`` -before emitting ``Bi``, and it is up to its internal logic to decide the order of emitted elements. Specialized elements -such as ``Zip`` however *do guarantee* their outputs order, as each output element depends on all upstream elements having +In Akka Streams almost all computation stages *preserve input order* of elements. This means that if inputs `{IA1,IA2,...,IAn}` +"cause" outputs `{OA1,OA2,...,OAk}` and inputs `{IB1,IB2,...,IBm}` "cause" outputs `{OB1,OB2,...,OBl}` and all of +`IAi` happened before all `IBi` then `OAi` happens before `OBi`. + +This property is even uphold by async operations such as `mapAsync`, however an unordered version exists +called `mapAsyncUnordered` which does not preserve this ordering. + +However, in the case of Junctions which handle multiple input streams (e.g. `Merge`) the output order is, +in general, *not defined* for elements arriving on different input ports. That is a merge-like operation may emit `Ai` +before emitting `Bi`, and it is up to its internal logic to decide the order of emitted elements. Specialized elements +such as `Zip` however *do guarantee* their outputs order, as each output element depends on all upstream elements having been signalled already – thus the ordering in the case of zipping is defined by this property. If you find yourself in need of fine grained control over order of emitted elements in fan-in -scenarios consider using :class:`MergePreferred` or :class:`GraphStage` – which gives you full control over how the -merge is performed. +scenarios consider using `MergePreferred` or `GraphStage` – which gives you full control over how the +merge is performed. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-graphs.md b/akka-docs/src/main/paradox/java/stream/stream-graphs.md index 1011ac46d0..0ff4a63b7b 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-graphs.md +++ b/akka-docs/src/main/paradox/java/stream/stream-graphs.md @@ -1,8 +1,4 @@ -.. _stream-graph-java: - -################### -Working with Graphs -################### +# Working with Graphs In Akka Streams computation graphs are not expressed using a fluent DSL like linear computations are, instead they are written in a more graph-resembling DSL which aims to make translating graph drawings (e.g. from notes taken @@ -11,83 +7,84 @@ dive into the multiple ways of constructing and re-using graphs, as well as expl Graphs are needed whenever you want to perform any kind of fan-in ("multiple inputs") or fan-out ("multiple outputs") operations. Considering linear Flows to be like roads, we can picture graph operations as junctions: multiple flows being connected at a single point. -Some graph operations which are common enough and fit the linear style of Flows, such as ``concat`` (which concatenates two +Some graph operations which are common enough and fit the linear style of Flows, such as `concat` (which concatenates two streams, such that the second one is consumed after the first one has completed), may have shorthand methods defined on -:class:`Flow` or :class:`Source` themselves, however you should keep in mind that those are also implemented as graph junctions. +`Flow` or `Source` themselves, however you should keep in mind that those are also implemented as graph junctions. -.. _graph-dsl-java: - -Constructing Graphs -------------------- + +## Constructing Graphs Graphs are built from simple Flows which serve as the linear connections within the graphs as well as junctions which serve as fan-in and fan-out points for Flows. Thanks to the junctions having meaningful types based on their behaviour and making them explicit elements these elements should be rather straightforward to use. -Akka Streams currently provide these junctions (for a detailed list see :ref:`stages-overview_java`): +Akka Streams currently provide these junctions (for a detailed list see @ref:[stages-overview_java](stages-overview.md)): -* **Fan-out** + * **Fan-out** - - ``Broadcast`` – *(1 input, N outputs)* given an input element emits to each output - - ``Balance`` – *(1 input, N outputs)* given an input element emits to one of its output ports - - ``UnzipWith`` – *(1 input, N outputs)* takes a function of 1 input that given a value for each input emits N output elements (where N <= 20) - - ``UnZip`` – *(1 input, 2 outputs)* splits a stream of ``Pair`` tuples into two streams, one of type ``A`` and one of type ``B`` +> + * `Broadcast` – *(1 input, N outputs)* given an input element emits to each output + * `Balance` – *(1 input, N outputs)* given an input element emits to one of its output ports + * `UnzipWith` – *(1 input, N outputs)* takes a function of 1 input that given a value for each input emits N output elements (where N <= 20) + * `UnZip` – *(1 input, 2 outputs)* splits a stream of `Pair` tuples into two streams, one of type `A` and one of type `B` -* **Fan-in** + * **Fan-in** - - ``Merge`` – *(N inputs , 1 output)* picks randomly from inputs pushing them one by one to its output - - ``MergePreferred`` – like :class:`Merge` but if elements are available on ``preferred`` port, it picks from it, otherwise randomly from ``others`` - - ``ZipWith`` – *(N inputs, 1 output)* which takes a function of N inputs that given a value for each input emits 1 output element - - ``Zip`` – *(2 inputs, 1 output)* is a :class:`ZipWith` specialised to zipping input streams of ``A`` and ``B`` into a ``Pair(A,B)`` tuple stream - - ``Concat`` – *(2 inputs, 1 output)* concatenates two streams (first consume one, then the second one) +> + * `Merge` – *(N inputs , 1 output)* picks randomly from inputs pushing them one by one to its output + * `MergePreferred` – like `Merge` but if elements are available on `preferred` port, it picks from it, otherwise randomly from `others` + * `ZipWith` – *(N inputs, 1 output)* which takes a function of N inputs that given a value for each input emits 1 output element + * `Zip` – *(2 inputs, 1 output)* is a `ZipWith` specialised to zipping input streams of `A` and `B` into a `Pair(A,B)` tuple stream + * `Concat` – *(2 inputs, 1 output)* concatenates two streams (first consume one, then the second one) One of the goals of the GraphDSL DSL is to look similar to how one would draw a graph on a whiteboard, so that it is simple to translate a design from whiteboard to code and be able to relate those two. Let's illustrate this by translating the below hand drawn graph into Akka Streams: -.. image:: ../../images/simple-graph-example.png +![simple-graph-example.png](../../images/simple-graph-example.png) -Such graph is simple to translate to the Graph DSL since each linear element corresponds to a :class:`Flow`, -and each circle corresponds to either a :class:`Junction` or a :class:`Source` or :class:`Sink` if it is beginning -or ending a :class:`Flow`. +Such graph is simple to translate to the Graph DSL since each linear element corresponds to a `Flow`, +and each circle corresponds to either a `Junction` or a `Source` or `Sink` if it is beginning +or ending a `Flow`. -.. includecode:: ../code/jdocs/stream/GraphDSLDocTest.java#simple-graph-dsl +@@snip [GraphDSLDocTest.java](../code/jdocs/stream/GraphDSLDocTest.java) { #simple-graph-dsl } -.. note:: - Junction *reference equality* defines *graph node equality* (i.e. the same merge *instance* used in a GraphDSL - refers to the same location in the resulting graph). +@@@ note -By looking at the snippets above, it should be apparent that the ``builder`` object is *mutable*. +Junction *reference equality* defines *graph node equality* (i.e. the same merge *instance* used in a GraphDSL +refers to the same location in the resulting graph). + +@@@ + +By looking at the snippets above, it should be apparent that the `builder` object is *mutable*. The reason for this design choice is to enable simpler creation of complex graphs, which may even contain cycles. -Once the GraphDSL has been constructed though, the :class:`RunnableGraph` instance *is immutable, thread-safe, and freely shareable*. +Once the GraphDSL has been constructed though, the `RunnableGraph` instance *is immutable, thread-safe, and freely shareable*. The same is true of all graph pieces—sources, sinks, and flows—once they are constructed. This means that you can safely re-use one given Flow or junction in multiple places in a processing graph. We have seen examples of such re-use already above: the merge and broadcast junctions were imported -into the graph using ``builder.add(...)``, an operation that will make a copy of the blueprint that +into the graph using `builder.add(...)`, an operation that will make a copy of the blueprint that is passed to it and return the inlets and outlets of the resulting copy so that they can be wired up. Another alternative is to pass existing graphs—of any shape—into the factory method that produces a -new graph. The difference between these approaches is that importing using ``builder.add(...)`` ignores the +new graph. The difference between these approaches is that importing using `builder.add(...)` ignores the materialized value of the imported graph while importing via the factory method allows its inclusion; -for more details see :ref:`stream-materialization-scala`. +for more details see @ref:[Stream Materialization](../../scala/stream/stream-flows-and-basics.md#stream-materialization-scala). In the example below we prepare a graph that consists of two parallel streams, -in which we re-use the same instance of :class:`Flow`, yet it will properly be +in which we re-use the same instance of `Flow`, yet it will properly be materialized as two connections between the corresponding Sources and Sinks: -.. includecode:: ../code/jdocs/stream/GraphDSLDocTest.java#graph-dsl-reusing-a-flow +@@snip [GraphDSLDocTest.java](../code/jdocs/stream/GraphDSLDocTest.java) { #graph-dsl-reusing-a-flow } -.. _partial-graph-dsl-java: - -Constructing and combining Partial Graphs ------------------------------------------ + +## Constructing and combining Partial Graphs Sometimes it is not possible (or needed) to construct the entire computation graph in one place, but instead construct all of its different phases in different places and in the end connect them all into a complete graph and run it. -This can be achieved by using the returned :class:`Graph` from ``GraphDSL.create()`` rather than -passing it to ``RunnableGraph.fromGraph()`` to wrap it in a :class:`RunnableGraph`.The reason of representing it as a different type is that a -:class:`RunnableGraph` requires all ports to be connected, and if they are not +This can be achieved by using the returned `Graph` from `GraphDSL.create()` rather than +passing it to `RunnableGraph.fromGraph()` to wrap it in a `RunnableGraph`.The reason of representing it as a different type is that a +`RunnableGraph` requires all ports to be connected, and if they are not it will throw an exception at construction time, which helps to avoid simple wiring errors while working with graphs. A partial graph however allows you to return the set of yet to be connected ports from the code block that @@ -97,69 +94,67 @@ Let's imagine we want to provide users with a specialized element that given 3 i the greatest int value of each zipped triple. We'll want to expose 3 input ports (unconnected sources) and one output port (unconnected sink). -.. includecode:: ../code/jdocs/stream/StreamPartialGraphDSLDocTest.java#simple-partial-graph-dsl +@@snip [StreamPartialGraphDSLDocTest.java](../code/jdocs/stream/StreamPartialGraphDSLDocTest.java) { #simple-partial-graph-dsl } As you can see, first we construct the partial graph that describes how to compute the maximum of two input streams, then we reuse that twice while constructing the partial graph that extends this to three input streams, then we import it (all of its nodes and connections) explicitly into the last graph in which all the undefined elements are rewired to real sources and sinks. The graph can then be run and yields the expected result. -.. warning:: - Please note that :class:`GraphDSL` is not able to provide compile time type-safety about whether or not all - elements have been properly connected—this validation is performed as a runtime check during the graph's instantiation. +@@@ warning - A partial graph also verifies that all ports are either connected or part of the returned :class:`Shape`. +Please note that `GraphDSL` is not able to provide compile time type-safety about whether or not all +elements have been properly connected—this validation is performed as a runtime check during the graph's instantiation. -.. _constructing-sources-sinks-flows-from-partial-graphs-java: +A partial graph also verifies that all ports are either connected or part of the returned `Shape`. -Constructing Sources, Sinks and Flows from Partial Graphs ---------------------------------------------------------- +@@@ -Instead of treating a ``Graph`` as simply a collection of flows and junctions which may not yet all be + +## Constructing Sources, Sinks and Flows from Partial Graphs + +Instead of treating a `Graph` as simply a collection of flows and junctions which may not yet all be connected it is sometimes useful to expose such a complex graph as a simpler structure, -such as a :class:`Source`, :class:`Sink` or :class:`Flow`. +such as a `Source`, `Sink` or `Flow`. In fact, these concepts can be easily expressed as special cases of a partially connected graph: -* :class:`Source` is a partial graph with *exactly one* output, that is it returns a :class:`SourceShape`. -* :class:`Sink` is a partial graph with *exactly one* input, that is it returns a :class:`SinkShape`. -* :class:`Flow` is a partial graph with *exactly one* input and *exactly one* output, that is it returns a :class:`FlowShape`. + * `Source` is a partial graph with *exactly one* output, that is it returns a `SourceShape`. + * `Sink` is a partial graph with *exactly one* input, that is it returns a `SinkShape`. + * `Flow` is a partial graph with *exactly one* input and *exactly one* output, that is it returns a `FlowShape`. Being able to hide complex graphs inside of simple elements such as Sink / Source / Flow enables you to easily create one complex element and from there on treat it as simple compound stage for linear computations. -In order to create a Source from a graph the method ``Source.fromGraph`` is used, to use it we must have a -``Graph`` with a ``SourceShape``. This is constructed using ``GraphDSL.create`` and providing building a ``SourceShape`` -graph. The single outlet must be provided to the ``SourceShape.of`` method and will become “the sink that must +In order to create a Source from a graph the method `Source.fromGraph` is used, to use it we must have a +`Graph` with a `SourceShape`. This is constructed using `GraphDSL.create` and providing building a `SourceShape` +graph. The single outlet must be provided to the `SourceShape.of` method and will become “the sink that must be attached before this Source can run”. Refer to the example below, in which we create a Source that zips together two numbers, to see this graph construction in action: -.. includecode:: ../code/jdocs/stream/StreamPartialGraphDSLDocTest.java#source-from-partial-graph-dsl +@@snip [StreamPartialGraphDSLDocTest.java](../code/jdocs/stream/StreamPartialGraphDSLDocTest.java) { #source-from-partial-graph-dsl } -Similarly the same can be done for a ``Sink`` using ``SinkShape.of`` in which case the provided value must be an -``Inlet``. For defining a ``Flow`` we need to expose both an undefined source and sink: +Similarly the same can be done for a `Sink` using `SinkShape.of` in which case the provided value must be an +`Inlet`. For defining a `Flow` we need to expose both an undefined source and sink: -.. includecode:: ../code/jdocs/stream/StreamPartialGraphDSLDocTest.java#flow-from-partial-graph-dsl +@@snip [StreamPartialGraphDSLDocTest.java](../code/jdocs/stream/StreamPartialGraphDSLDocTest.java) { #flow-from-partial-graph-dsl } -Combining Sources and Sinks with simplified API ------------------------------------------------ +## Combining Sources and Sinks with simplified API -There is simplified API you can use to combine sources and sinks with junctions like: ``Broadcast``, ``Balance``, -``Merge`` and ``Concat`` without the need for using the Graph DSL. The combine method takes care of constructing +There is simplified API you can use to combine sources and sinks with junctions like: `Broadcast`, `Balance`, +`Merge` and `Concat` without the need for using the Graph DSL. The combine method takes care of constructing the necessary graph underneath. In following example we combine two sources into one (fan-in): -.. includecode:: ../code/jdocs/stream/StreamPartialGraphDSLDocTest.java#source-combine +@@snip [StreamPartialGraphDSLDocTest.java](../code/jdocs/stream/StreamPartialGraphDSLDocTest.java) { #source-combine } -The same can be done for a ``Sink`` but in this case it will be fan-out: +The same can be done for a `Sink` but in this case it will be fan-out: -.. includecode:: ../code/jdocs/stream/StreamPartialGraphDSLDocTest.java#sink-combine +@@snip [StreamPartialGraphDSLDocTest.java](../code/jdocs/stream/StreamPartialGraphDSLDocTest.java) { #sink-combine } -.. _bidi-flow-java: - -Bidirectional Flows -------------------- + +## Bidirectional Flows A graph topology that is often useful is that of two flows going in opposite directions. Take for example a codec stage that serializes outgoing messages @@ -167,147 +162,142 @@ and deserializes incoming octet streams. Another such stage could add a framing protocol that attaches a length header to outgoing data and parses incoming frames back into the original octet stream chunks. These two stages are meant to be composed, applying one atop the other as part of a protocol stack. For -this purpose exists the special type :class:`BidiFlow` which is a graph that +this purpose exists the special type `BidiFlow` which is a graph that has exactly two open inlets and two open outlets. The corresponding shape is -called :class:`BidiShape` and is defined like this: +called `BidiShape` and is defined like this: -.. includecode:: ../../../../akka-stream/src/main/scala/akka/stream/Shape.scala - :include: bidi-shape - :exclude: implementation-details-elided +@@snip [Shape.scala]../../../../../../akka-stream/src/main/scala/akka/stream/Shape.scala) { #bidi-shape } -A bidirectional flow is defined just like a unidirectional :class:`Flow` as +A bidirectional flow is defined just like a unidirectional `Flow` as demonstrated for the codec mentioned above: -.. includecode:: ../code/jdocs/stream/BidiFlowDocTest.java - :include: codec - :exclude: implementation-details-elided +@@snip [BidiFlowDocTest.java](../code/jdocs/stream/BidiFlowDocTest.java) { #codec } The first version resembles the partial graph constructor, while for the simple case of a functional 1:1 transformation there is a concise convenience method as shown on the last line. The implementation of the two functions is not difficult either: -.. includecode:: ../code/jdocs/stream/BidiFlowDocTest.java#codec-impl +@@snip [BidiFlowDocTest.java](../code/jdocs/stream/BidiFlowDocTest.java) { #codec-impl } In this way you could easily integrate any other serialization library that turns an object into a sequence of bytes. The other stage that we talked about is a little more involved since reversing a framing protocol means that any received chunk of bytes may correspond to -zero or more messages. This is best implemented using a :class:`GraphStage` -(see also :ref:`graphstage-java`). +zero or more messages. This is best implemented using a `GraphStage` +(see also @ref:[Custom processing with GraphStage](stream-customize.md#graphstage-java)). -.. includecode:: ../code/jdocs/stream/BidiFlowDocTest.java#framing +@@snip [BidiFlowDocTest.java](../code/jdocs/stream/BidiFlowDocTest.java) { #framing } With these implementations we can build a protocol stack and test it: -.. includecode:: ../code/jdocs/stream/BidiFlowDocTest.java#compose +@@snip [BidiFlowDocTest.java](../code/jdocs/stream/BidiFlowDocTest.java) { #compose } -This example demonstrates how :class:`BidiFlow` subgraphs can be hooked -together and also turned around with the ``.reversed()`` method. The test +This example demonstrates how `BidiFlow` subgraphs can be hooked +together and also turned around with the `.reversed()` method. The test simulates both parties of a network communication protocol without actually having to open a network connection—the flows can just be connected directly. -.. _graph-matvalue-java: - -Accessing the materialized value inside the Graph -------------------------------------------------- + +## Accessing the materialized value inside the Graph In certain cases it might be necessary to feed back the materialized value of a Graph (partial, closed or backing a -Source, Sink, Flow or BidiFlow). This is possible by using ``builder.materializedValue`` which gives an ``Outlet`` that +Source, Sink, Flow or BidiFlow). This is possible by using `builder.materializedValue` which gives an `Outlet` that can be used in the graph as an ordinary source or outlet, and which will eventually emit the materialized value. -If the materialized value is needed at more than one place, it is possible to call ``materializedValue`` any number of +If the materialized value is needed at more than one place, it is possible to call `materializedValue` any number of times to acquire the necessary number of outlets. -.. includecode:: ../code/jdocs/stream/GraphDSLDocTest.java#graph-dsl-matvalue +@@snip [GraphDSLDocTest.java](../code/jdocs/stream/GraphDSLDocTest.java) { #graph-dsl-matvalue } Be careful not to introduce a cycle where the materialized value actually contributes to the materialized value. -The following example demonstrates a case where the materialized ``CompletionStage`` of a fold is fed back to the fold itself. +The following example demonstrates a case where the materialized `CompletionStage` of a fold is fed back to the fold itself. -.. includecode:: ../code/jdocs/stream/GraphDSLDocTest.java#graph-dsl-matvalue-cycle +@@snip [GraphDSLDocTest.java](../code/jdocs/stream/GraphDSLDocTest.java) { #graph-dsl-matvalue-cycle } -.. _graph-cycles-java: - -Graph cycles, liveness and deadlocks ------------------------------------- + +## Graph cycles, liveness and deadlocks Cycles in bounded stream topologies need special considerations to avoid potential deadlocks and other liveness issues. This section shows several examples of problems that can arise from the presence of feedback arcs in stream processing graphs. In the following examples runnable graphs are created but do not run because each have some issue and will deadlock after start. -``Source`` variable is not defined as the nature and number of element does not matter for described problems. +`Source` variable is not defined as the nature and number of element does not matter for described problems. The first example demonstrates a graph that contains a naïve cycle. The graph takes elements from the source, prints them, then broadcasts those elements -to a consumer (we just used ``Sink.ignore`` for now) and to a feedback arc that is merged back into the main -via a ``Merge`` junction. +to a consumer (we just used `Sink.ignore` for now) and to a feedback arc that is merged back into the main +via a `Merge` junction. -.. includecode:: ../code/jdocs/stream/GraphCyclesDocTest.java#deadlocked +@@snip [GraphCyclesDocTest.java](../code/jdocs/stream/GraphCyclesDocTest.java) { #deadlocked } Running this we observe that after a few numbers have been printed, no more elements are logged to the console - all processing stops after some time. After some investigation we observe that: -* through merging from ``source`` we increase the number of elements flowing in the cycle -* by broadcasting back to the cycle we do not decrease the number of elements in the cycle + * through merging from `source` we increase the number of elements flowing in the cycle + * by broadcasting back to the cycle we do not decrease the number of elements in the cycle Since Akka Streams (and Reactive Streams in general) guarantee bounded processing (see the "Buffering" section for more details) it means that only a bounded number of elements are buffered over any time span. Since our cycle gains more and -more elements, eventually all of its internal buffers become full, backpressuring ``source`` forever. To be able -to process more elements from ``source`` elements would need to leave the cycle somehow. +more elements, eventually all of its internal buffers become full, backpressuring `source` forever. To be able +to process more elements from `source` elements would need to leave the cycle somehow. -If we modify our feedback loop by replacing the ``Merge`` junction with a ``MergePreferred`` we can avoid the deadlock. -``MergePreferred`` is unfair as it always tries to consume from a preferred input port if there are elements available +If we modify our feedback loop by replacing the `Merge` junction with a `MergePreferred` we can avoid the deadlock. +`MergePreferred` is unfair as it always tries to consume from a preferred input port if there are elements available before trying the other lower priority input ports. Since we feed back through the preferred port it is always guaranteed that the elements in the cycles can flow. -.. includecode:: ../code/jdocs/stream/GraphCyclesDocTest.java#unfair +@@snip [GraphCyclesDocTest.java](../code/jdocs/stream/GraphCyclesDocTest.java) { #unfair } If we run the example we see that the same sequence of numbers are printed -over and over again, but the processing does not stop. Hence, we avoided the deadlock, but ``source`` is still +over and over again, but the processing does not stop. Hence, we avoided the deadlock, but `source` is still back-pressured forever, because buffer space is never recovered: the only action we see is the circulation of a couple -of initial elements from ``source``. +of initial elements from `source`. -.. note:: - What we see here is that in certain cases we need to choose between boundedness and liveness. Our first example would - not deadlock if there would be an infinite buffer in the loop, or vice versa, if the elements in the cycle would - be balanced (as many elements are removed as many are injected) then there would be no deadlock. +@@@ note + +What we see here is that in certain cases we need to choose between boundedness and liveness. Our first example would +not deadlock if there would be an infinite buffer in the loop, or vice versa, if the elements in the cycle would +be balanced (as many elements are removed as many are injected) then there would be no deadlock. + +@@@ To make our cycle both live (not deadlocking) and fair we can introduce a dropping element on the feedback arc. In this -case we chose the ``buffer()`` operation giving it a dropping strategy ``OverflowStrategy.dropHead``. +case we chose the `buffer()` operation giving it a dropping strategy `OverflowStrategy.dropHead`. -.. includecode:: ../code/jdocs/stream/GraphCyclesDocTest.java#dropping +@@snip [GraphCyclesDocTest.java](../code/jdocs/stream/GraphCyclesDocTest.java) { #dropping } If we run this example we see that -* The flow of elements does not stop, there are always elements printed -* We see that some of the numbers are printed several times over time (due to the feedback loop) but on average - the numbers are increasing in the long term + * The flow of elements does not stop, there are always elements printed + * We see that some of the numbers are printed several times over time (due to the feedback loop) but on average +the numbers are increasing in the long term This example highlights that one solution to avoid deadlocks in the presence of potentially unbalanced cycles (cycles where the number of circulating elements are unbounded) is to drop elements. An alternative would be to -define a larger buffer with ``OverflowStrategy.fail`` which would fail the stream instead of deadlocking it after +define a larger buffer with `OverflowStrategy.fail` which would fail the stream instead of deadlocking it after all buffer space has been consumed. As we discovered in the previous examples, the core problem was the unbalanced nature of the feedback loop. We circumvented this issue by adding a dropping element, but now we want to build a cycle that is balanced from -the beginning instead. To achieve this we modify our first graph by replacing the ``Merge`` junction with a ``ZipWith``. -Since ``ZipWith`` takes one element from ``source`` *and* from the feedback arc to inject one element into the cycle, +the beginning instead. To achieve this we modify our first graph by replacing the `Merge` junction with a `ZipWith`. +Since `ZipWith` takes one element from `source` *and* from the feedback arc to inject one element into the cycle, we maintain the balance of elements. -.. includecode:: ../code/jdocs/stream/GraphCyclesDocTest.java#zipping-dead +@@snip [GraphCyclesDocTest.java](../code/jdocs/stream/GraphCyclesDocTest.java) { #zipping-dead } Still, when we try to run the example it turns out that no element is printed at all! After some investigation we realize that: -* In order to get the first element from ``source`` into the cycle we need an already existing element in the cycle -* In order to get an initial element in the cycle we need an element from ``source`` + * In order to get the first element from `source` into the cycle we need an already existing element in the cycle + * In order to get an initial element in the cycle we need an element from `source` These two conditions are a typical "chicken-and-egg" problem. The solution is to inject an initial -element into the cycle that is independent from ``source``. We do this by using a ``Concat`` junction on the backwards -arc that injects a single element using ``Source.single``. +element into the cycle that is independent from `source`. We do this by using a `Concat` junction on the backwards +arc that injects a single element using `Source.single`. -.. includecode:: ../code/jdocs/stream/GraphCyclesDocTest.java#zipping-live +@@snip [GraphCyclesDocTest.java](../code/jdocs/stream/GraphCyclesDocTest.java) { #zipping-live } When we run the above example we see that processing starts and never stops. The important takeaway from this example -is that balanced cycles often need an initial "kick-off" element to be injected into the cycle. +is that balanced cycles often need an initial "kick-off" element to be injected into the cycle. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-integrations.md b/akka-docs/src/main/paradox/java/stream/stream-integrations.md index 6f46ed73cc..1b5427b92c 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-integrations.md +++ b/akka-docs/src/main/paradox/java/stream/stream-integrations.md @@ -1,478 +1,464 @@ -.. _stream-integrations-java: +# Integration -########### -Integration -########### - -Integrating with Actors -======================= +## Integrating with Actors For piping the elements of a stream as messages to an ordinary actor you can use -``ask`` in a ``mapAsync`` or use ``Sink.actorRefWithAck``. +`ask` in a `mapAsync` or use `Sink.actorRefWithAck`. -Messages can be sent to a stream with ``Source.queue`` or via the :class:`ActorRef` that is -materialized by ``Source.actorRef``. +Messages can be sent to a stream with `Source.queue` or via the `ActorRef` that is +materialized by `Source.actorRef`. -mapAsync + ask -^^^^^^^^^^^^^^ +### mapAsync + ask A nice way to delegate some processing of elements in a stream to an actor is to -use ``ask`` in ``mapAsync``. The back-pressure of the stream is maintained by -the ``CompletionStage`` of the ``ask`` and the mailbox of the actor will not be filled with -more messages than the given ``parallelism`` of the ``mapAsync`` stage. +use `ask` in `mapAsync`. The back-pressure of the stream is maintained by +the `CompletionStage` of the `ask` and the mailbox of the actor will not be filled with +more messages than the given `parallelism` of the `mapAsync` stage. -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#mapAsync-ask +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #mapAsync-ask } Note that the messages received in the actor will be in the same order as -the stream elements, i.e. the ``parallelism`` does not change the ordering +the stream elements, i.e. the `parallelism` does not change the ordering of the messages. There is a performance advantage of using parallelism > 1 even though the actor will only process one message at a time because then there is already a message in the mailbox when the actor has completed previous message. -The actor must reply to the ``getSender()`` for each message from the stream. That -reply will complete the ``CompletionStage`` of the ``ask`` and it will be the element that -is emitted downstreams from ``mapAsync``. +The actor must reply to the `getSender()` for each message from the stream. That +reply will complete the `CompletionStage` of the `ask` and it will be the element that +is emitted downstreams from `mapAsync`. -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#ask-actor +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #ask-actor } -The stream can be completed with failure by sending ``akka.actor.Status.Failure`` +The stream can be completed with failure by sending `akka.actor.Status.Failure` as reply from the actor. -If the ``ask`` fails due to timeout the stream will be completed with -``TimeoutException`` failure. If that is not desired outcome you can use ``recover`` -on the ``ask`` :class:`CompletionStage`. +If the `ask` fails due to timeout the stream will be completed with +`TimeoutException` failure. If that is not desired outcome you can use `recover` +on the `ask` `CompletionStage`. If you don't care about the reply values and only use them as back-pressure signals you -can use ``Sink.ignore`` after the ``mapAsync`` stage and then actor is effectively a sink +can use `Sink.ignore` after the `mapAsync` stage and then actor is effectively a sink of the stream. -The same pattern can be used with :ref:`Actor routers `. Then you -can use ``mapAsyncUnordered`` for better efficiency if you don't care about the +The same pattern can be used with @ref:[Actor routers](../routing.md). Then you +can use `mapAsyncUnordered` for better efficiency if you don't care about the order of the emitted downstream elements (the replies). -Sink.actorRefWithAck -^^^^^^^^^^^^^^^^^^^^ +### Sink.actorRefWithAck -The sink sends the elements of the stream to the given :class:`ActorRef` that sends back back-pressure signal. -First element is always `onInitMessage`, then stream is waiting for the given acknowledgement message +The sink sends the elements of the stream to the given `ActorRef` that sends back back-pressure signal. +First element is always *onInitMessage*, then stream is waiting for the given acknowledgement message from the given actor which means that it is ready to process elements. It also requires the given acknowledgement message after each stream element to make back-pressure work. If the target actor terminates the stream will be cancelled. When the stream is completed successfully the -given ``onCompleteMessage`` will be sent to the destination actor. When the stream is completed with -failure a ``akka.actor.Status.Failure`` message will be sent to the destination actor. +given `onCompleteMessage` will be sent to the destination actor. When the stream is completed with +failure a `akka.actor.Status.Failure` message will be sent to the destination actor. -.. note:: +@@@ note - Using ``Sink.actorRef`` or ordinary ``tell`` from a ``map`` or ``foreach`` stage means that there is - no back-pressure signal from the destination actor, i.e. if the actor is not consuming the messages - fast enough the mailbox of the actor will grow, unless you use a bounded mailbox with zero - `mailbox-push-timeout-time` or use a rate limiting stage in front. It's often better to - use ``Sink.actorRefWithAck`` or ``ask`` in ``mapAsync``, though. +Using `Sink.actorRef` or ordinary `tell` from a `map` or `foreach` stage means that there is +no back-pressure signal from the destination actor, i.e. if the actor is not consuming the messages +fast enough the mailbox of the actor will grow, unless you use a bounded mailbox with zero +*mailbox-push-timeout-time* or use a rate limiting stage in front. It's often better to +use `Sink.actorRefWithAck` or `ask` in `mapAsync`, though. -Source.queue -^^^^^^^^^^^^ +@@@ -``Source.queue`` can be used for emitting elements to a stream from an actor (or from anything running outside -the stream). The elements will be buffered until the stream can process them. You can ``offer`` elements to +### Source.queue + +`Source.queue` can be used for emitting elements to a stream from an actor (or from anything running outside +the stream). The elements will be buffered until the stream can process them. You can `offer` elements to the queue and they will be emitted to the stream if there is demand from downstream, otherwise they will be buffered until request for demand is received. -Use overflow strategy ``akka.stream.OverflowStrategy.backpressure`` to avoid dropping of elements if the +Use overflow strategy `akka.stream.OverflowStrategy.backpressure` to avoid dropping of elements if the buffer is full. -``SourceQueue.offer`` returns ``CompletionStage`` which completes with -``QueueOfferResult.enqueued`` if element was added to buffer or sent downstream. It completes with -``QueueOfferResult.dropped`` if element was dropped. Can also complete with ``QueueOfferResult.Failure`` - -when stream failed or ``QueueOfferResult.QueueClosed`` when downstream is completed. +`SourceQueue.offer` returns `CompletionStage` which completes with +`QueueOfferResult.enqueued` if element was added to buffer or sent downstream. It completes with +`QueueOfferResult.dropped` if element was dropped. Can also complete with `QueueOfferResult.Failure` - +when stream failed or `QueueOfferResult.QueueClosed` when downstream is completed. -When used from an actor you typically ``pipe`` the result of the ``CompletionStage`` back to the actor to +When used from an actor you typically `pipe` the result of the `CompletionStage` back to the actor to continue processing. -Source.actorRef -^^^^^^^^^^^^^^^ +### Source.actorRef -Messages sent to the actor that is materialized by ``Source.actorRef`` will be emitted to the +Messages sent to the actor that is materialized by `Source.actorRef` will be emitted to the stream if there is demand from downstream, otherwise they will be buffered until request for demand is received. -Depending on the defined :class:`OverflowStrategy` it might drop elements if there is no space -available in the buffer. The strategy ``OverflowStrategy.backpressure`` is not supported +Depending on the defined `OverflowStrategy` it might drop elements if there is no space +available in the buffer. The strategy `OverflowStrategy.backpressure` is not supported for this Source type, i.e. elements will be dropped if the buffer is filled by sending -at a rate that is faster than the stream can consume. You should consider using ``Source.queue`` +at a rate that is faster than the stream can consume. You should consider using `Source.queue` if you want a backpressured actor interface. -The stream can be completed successfully by sending ``akka.actor.PoisonPill`` or -``akka.actor.Status.Success`` to the actor reference. +The stream can be completed successfully by sending `akka.actor.PoisonPill` or +`akka.actor.Status.Success` to the actor reference. -The stream can be completed with failure by sending ``akka.actor.Status.Failure`` to the +The stream can be completed with failure by sending `akka.actor.Status.Failure` to the actor reference. The actor will be stopped when the stream is completed, failed or cancelled from downstream, i.e. you can watch it to get notified when that happens. -Integrating with External Services -================================== +## Integrating with External Services Stream transformations and side effects involving external non-stream based services can be -performed with ``mapAsync`` or ``mapAsyncUnordered``. +performed with `mapAsync` or `mapAsyncUnordered`. For example, sending emails to the authors of selected tweets using an external email service: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#email-server-send +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #email-server-send } We start with the tweet stream of authors: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#tweet-authors +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #tweet-authors } Assume that we can lookup their email address using: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#email-address-lookup +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #email-address-lookup } -Transforming the stream of authors to a stream of email addresses by using the ``lookupEmail`` -service can be done with ``mapAsync``: +Transforming the stream of authors to a stream of email addresses by using the `lookupEmail` +service can be done with `mapAsync`: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#email-addresses-mapAsync +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #email-addresses-mapAsync } Finally, sending the emails: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#send-emails +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #send-emails } -``mapAsync`` is applying the given function that is calling out to the external service to -each of the elements as they pass through this processing step. The function returns a :class:`CompletionStage` +`mapAsync` is applying the given function that is calling out to the external service to +each of the elements as they pass through this processing step. The function returns a `CompletionStage` and the value of that future will be emitted downstreams. The number of Futures -that shall run in parallel is given as the first argument to ``mapAsync``. +that shall run in parallel is given as the first argument to `mapAsync`. These Futures may complete in any order, but the elements that are emitted downstream are in the same order as received from upstream. -That means that back-pressure works as expected. For example if the ``emailServer.send`` +That means that back-pressure works as expected. For example if the `emailServer.send` is the bottleneck it will limit the rate at which incoming tweets are retrieved and email addresses looked up. The final piece of this pipeline is to generate the demand that pulls the tweet -authors information through the emailing pipeline: we attach a ``Sink.ignore`` +authors information through the emailing pipeline: we attach a `Sink.ignore` which makes it all run. If our email process would return some interesting data for further transformation then we would of course not ignore it but send that result stream onwards for further processing or storage. -Note that ``mapAsync`` preserves the order of the stream elements. In this example the order -is not important and then we can use the more efficient ``mapAsyncUnordered``: +Note that `mapAsync` preserves the order of the stream elements. In this example the order +is not important and then we can use the more efficient `mapAsyncUnordered`: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#external-service-mapAsyncUnordered +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #external-service-mapAsyncUnordered } -In the above example the services conveniently returned a :class:`CompletionStage` of the result. -If that is not the case you need to wrap the call in a :class:`CompletionStage`. If the service call +In the above example the services conveniently returned a `CompletionStage` of the result. +If that is not the case you need to wrap the call in a `CompletionStage`. If the service call involves blocking you must also make sure that you run it on a dedicated execution context, to avoid starvation and disturbance of other tasks in the system. -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#blocking-mapAsync +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #blocking-mapAsync } -The configuration of the ``"blocking-dispatcher"`` may look something like: +The configuration of the `"blocking-dispatcher"` may look something like: -.. includecode:: ../../scala/code/docs/stream/IntegrationDocSpec.scala#blocking-dispatcher-config +@@snip [IntegrationDocSpec.scala](../../scala/code/docs/stream/IntegrationDocSpec.scala) { #blocking-dispatcher-config } -An alternative for blocking calls is to perform them in a ``map`` operation, still using a +An alternative for blocking calls is to perform them in a `map` operation, still using a dedicated dispatcher for that operation. -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#blocking-map +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #blocking-map } -However, that is not exactly the same as ``mapAsync``, since the ``mapAsync`` may run -several calls concurrently, but ``map`` performs them one at a time. +However, that is not exactly the same as `mapAsync`, since the `mapAsync` may run +several calls concurrently, but `map` performs them one at a time. For a service that is exposed as an actor, or if an actor is used as a gateway in front of an -external service, you can use ``ask``: +external service, you can use `ask`: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#save-tweets +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #save-tweets } -Note that if the ``ask`` is not completed within the given timeout the stream is completed with failure. -If that is not desired outcome you can use ``recover`` on the ``ask`` :class:`CompletionStage`. +Note that if the `ask` is not completed within the given timeout the stream is completed with failure. +If that is not desired outcome you can use `recover` on the `ask` `CompletionStage`. -Illustrating ordering and parallelism -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Illustrating ordering and parallelism Let us look at another example to get a better understanding of the ordering -and parallelism characteristics of ``mapAsync`` and ``mapAsyncUnordered``. +and parallelism characteristics of `mapAsync` and `mapAsyncUnordered`. -Several ``mapAsync`` and ``mapAsyncUnordered`` futures may run concurrently. +Several `mapAsync` and `mapAsyncUnordered` futures may run concurrently. The number of concurrent futures are limited by the downstream demand. For example, if 5 elements have been requested by downstream there will be at most 5 futures in progress. -``mapAsync`` emits the future results in the same order as the input elements +`mapAsync` emits the future results in the same order as the input elements were received. That means that completed results are only emitted downstream when earlier results have been completed and emitted. One slow call will thereby delay the results of all successive calls, even though they are completed before the slow call. -``mapAsyncUnordered`` emits the future results as soon as they are completed, i.e. +`mapAsyncUnordered` emits the future results as soon as they are completed, i.e. it is possible that the elements are not emitted downstream in the same order as received from upstream. One slow call will thereby not delay the results of faster successive calls as long as there is downstream demand of several elements. Here is a fictive service that we can use to illustrate these aspects. -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#sometimes-slow-service +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #sometimes-slow-service } Elements starting with a lower case character are simulated to take longer time to process. -Here is how we can use it with ``mapAsync``: +Here is how we can use it with `mapAsync`: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#sometimes-slow-mapAsync +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #sometimes-slow-mapAsync } The output may look like this: -:: +``` +before: a +before: B +before: C +before: D +running: a (1) +running: B (2) +before: e +running: C (3) +before: F +running: D (4) +before: g +before: H +completed: C (3) +completed: B (2) +completed: D (1) +completed: a (0) +after: A +after: B +running: e (1) +after: C +after: D +running: F (2) +before: i +before: J +running: g (3) +running: H (4) +completed: H (2) +completed: F (3) +completed: e (1) +completed: g (0) +after: E +after: F +running: i (1) +after: G +after: H +running: J (2) +completed: J (1) +completed: i (0) +after: I +after: J +``` - before: a - before: B - before: C - before: D - running: a (1) - running: B (2) - before: e - running: C (3) - before: F - running: D (4) - before: g - before: H - completed: C (3) - completed: B (2) - completed: D (1) - completed: a (0) - after: A - after: B - running: e (1) - after: C - after: D - running: F (2) - before: i - before: J - running: g (3) - running: H (4) - completed: H (2) - completed: F (3) - completed: e (1) - completed: g (0) - after: E - after: F - running: i (1) - after: G - after: H - running: J (2) - completed: J (1) - completed: i (0) - after: I - after: J - -Note that ``after`` lines are in the same order as the ``before`` lines even -though elements are ``completed`` in a different order. For example ``H`` -is ``completed`` before ``g``, but still emitted afterwards. +Note that `after` lines are in the same order as the `before` lines even +though elements are `completed` in a different order. For example `H` +is `completed` before `g`, but still emitted afterwards. The numbers in parenthesis illustrates how many calls that are in progress at the same time. Here the downstream demand and thereby the number of concurrent -calls are limited by the buffer size (4) of the :class:`ActorMaterializerSettings`. +calls are limited by the buffer size (4) of the `ActorMaterializerSettings`. -Here is how we can use the same service with ``mapAsyncUnordered``: +Here is how we can use the same service with `mapAsyncUnordered`: -.. includecode:: ../code/jdocs/stream/IntegrationDocTest.java#sometimes-slow-mapAsyncUnordered +@@snip [IntegrationDocTest.java](../code/jdocs/stream/IntegrationDocTest.java) { #sometimes-slow-mapAsyncUnordered } The output may look like this: -:: +``` +before: a +before: B +before: C +before: D +running: a (1) +running: B (2) +before: e +running: C (3) +before: F +running: D (4) +before: g +before: H +completed: B (3) +completed: C (1) +completed: D (2) +after: B +after: D +running: e (2) +after: C +running: F (3) +before: i +before: J +completed: F (2) +after: F +running: g (3) +running: H (4) +completed: H (3) +after: H +completed: a (2) +after: A +running: i (3) +running: J (4) +completed: J (3) +after: J +completed: e (2) +after: E +completed: g (1) +after: G +completed: i (0) +after: I +``` - before: a - before: B - before: C - before: D - running: a (1) - running: B (2) - before: e - running: C (3) - before: F - running: D (4) - before: g - before: H - completed: B (3) - completed: C (1) - completed: D (2) - after: B - after: D - running: e (2) - after: C - running: F (3) - before: i - before: J - completed: F (2) - after: F - running: g (3) - running: H (4) - completed: H (3) - after: H - completed: a (2) - after: A - running: i (3) - running: J (4) - completed: J (3) - after: J - completed: e (2) - after: E - completed: g (1) - after: G - completed: i (0) - after: I - -Note that ``after`` lines are not in the same order as the ``before`` lines. For example -``H`` overtakes the slow ``G``. +Note that `after` lines are not in the same order as the `before` lines. For example +`H` overtakes the slow `G`. The numbers in parenthesis illustrates how many calls that are in progress at the same time. Here the downstream demand and thereby the number of concurrent -calls are limited by the buffer size (4) of the :class:`ActorMaterializerSettings`. +calls are limited by the buffer size (4) of the `ActorMaterializerSettings`. -.. _reactive-streams-integration-java: + +## Integrating with Reactive Streams -Integrating with Reactive Streams -================================= - -`Reactive Streams`_ defines a standard for asynchronous stream processing with non-blocking +[Reactive Streams](http://reactive-streams.org/) defines a standard for asynchronous stream processing with non-blocking back pressure. It makes it possible to plug together stream libraries that adhere to the standard. Akka Streams is one such library. An incomplete list of other implementations: -* `Reactor (1.1+)`_ -* `RxJava`_ -* `Ratpack`_ -* `Slick`_ + * [Reactor (1.1+)](http://github.com/reactor/reactor) + * [RxJava](https://github.com/ReactiveX/RxJavaReactiveStreams) + * [Ratpack](http://www.ratpack.io/manual/current/streams.html) + * [Slick](http://slick.lightbend.com) -.. _Reactive Streams: http://reactive-streams.org/ -.. _Reactor (1.1+): http://github.com/reactor/reactor -.. _RxJava: https://github.com/ReactiveX/RxJavaReactiveStreams -.. _Ratpack: http://www.ratpack.io/manual/current/streams.html -.. _Slick: http://slick.lightbend.com +The two most important interfaces in Reactive Streams are the `Publisher` and `Subscriber`. -The two most important interfaces in Reactive Streams are the :class:`Publisher` and :class:`Subscriber`. - -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java#imports +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #imports } Let us assume that a library provides a publisher of tweets: -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java#tweets-publisher +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #tweets-publisher } and another library knows how to store author handles in a database: -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java#author-storage-subscriber +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #author-storage-subscriber } -Using an Akka Streams :class:`Flow` we can transform the stream and connect those: +Using an Akka Streams `Flow` we can transform the stream and connect those: -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java - :include: authors,connect-all +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #authors #connect-all } -The :class:`Publisher` is used as an input :class:`Source` to the flow and the -:class:`Subscriber` is used as an output :class:`Sink`. +The `Publisher` is used as an input `Source` to the flow and the +`Subscriber` is used as an output `Sink`. -A :class:`Flow` can also be also converted to a :class:`RunnableGraph[Processor[In, Out]]` which -materializes to a :class:`Processor` when ``run()`` is called. ``run()`` itself can be called multiple -times, resulting in a new :class:`Processor` instance each time. +A `Flow` can also be also converted to a `RunnableGraph[Processor[In, Out]]` which +materializes to a `Processor` when `run()` is called. `run()` itself can be called multiple +times, resulting in a new `Processor` instance each time. -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java#flow-publisher-subscriber +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #flow-publisher-subscriber } -A publisher can be connected to a subscriber with the ``subscribe`` method. +A publisher can be connected to a subscriber with the `subscribe` method. -It is also possible to expose a :class:`Source` as a :class:`Publisher` -by using the Publisher-:class:`Sink`: +It is also possible to expose a `Source` as a `Publisher` +by using the Publisher-`Sink`: -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java#source-publisher +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #source-publisher } -A publisher that is created with ``Sink.asPublisher(AsPublisher.WITHOUT_FANOUT)`` supports only a single subscription. -Additional subscription attempts will be rejected with an :class:`IllegalStateException`. +A publisher that is created with `Sink.asPublisher(AsPublisher.WITHOUT_FANOUT)` supports only a single subscription. +Additional subscription attempts will be rejected with an `IllegalStateException`. A publisher that supports multiple subscribers using fan-out/broadcasting is created as follows: -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java - :include: author-alert-subscriber,author-storage-subscriber +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #author-alert-subscriber #author-storage-subscriber } -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java#source-fanoutPublisher +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #source-fanoutPublisher } The input buffer size of the stage controls how far apart the slowest subscriber can be from the fastest subscriber before slowing down the stream. -To make the picture complete, it is also possible to expose a :class:`Sink` as a :class:`Subscriber` -by using the Subscriber-:class:`Source`: +To make the picture complete, it is also possible to expose a `Sink` as a `Subscriber` +by using the Subscriber-`Source`: -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java#sink-subscriber +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #sink-subscriber } -It is also possible to use re-wrap :class:`Processor` instances as a :class:`Flow` by -passing a factory function that will create the :class:`Processor` instances: +It is also possible to use re-wrap `Processor` instances as a `Flow` by +passing a factory function that will create the `Processor` instances: -.. includecode:: ../code/jdocs/stream/ReactiveStreamsDocTest.java#use-processor +@@snip [ReactiveStreamsDocTest.java](../code/jdocs/stream/ReactiveStreamsDocTest.java) { #use-processor } -Please note that a factory is necessary to achieve reusability of the resulting :class:`Flow`. +Please note that a factory is necessary to achieve reusability of the resulting `Flow`. -Implementing Reactive Streams Publisher or Subscriber -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Implementing Reactive Streams Publisher or Subscriber -As described above any Akka Streams ``Source`` can be exposed as a Reactive Streams ``Publisher`` and -any ``Sink`` can be exposed as a Reactive Streams ``Subscriber``. Therefore we recommend that you -implement Reactive Streams integrations with built-in stages or :ref:`custom stages `. +As described above any Akka Streams `Source` can be exposed as a Reactive Streams `Publisher` and +any `Sink` can be exposed as a Reactive Streams `Subscriber`. Therefore we recommend that you +implement Reactive Streams integrations with built-in stages or @ref:[custom stages](stream-customize.md). -For historical reasons the :class:`ActorPublisher` and :class:`ActorSubscriber` traits are -provided to support implementing Reactive Streams :class:`Publisher` and :class:`Subscriber` with -an :class:`Actor`. +For historical reasons the `ActorPublisher` and `ActorSubscriber` traits are +provided to support implementing Reactive Streams `Publisher` and `Subscriber` with +an `Actor`. -These can be consumed by other Reactive Stream libraries or used as an Akka Streams :class:`Source` or :class:`Sink`. +These can be consumed by other Reactive Stream libraries or used as an Akka Streams `Source` or `Sink`. -.. warning:: +@@@ warning - :class:`ActorPublisher` and :class:`ActorSubscriber` will probably be deprecated in future versions of Akka. +`ActorPublisher` and `ActorSubscriber` will probably be deprecated in future versions of Akka. -.. warning:: +@@@ - :class:`ActorPublisher` and :class:`ActorSubscriber` cannot be used with remote actors, - because if signals of the Reactive Streams protocol (e.g. ``request``) are lost the - the stream may deadlock. +@@@ warning -ActorPublisher --------------- +`ActorPublisher` and `ActorSubscriber` cannot be used with remote actors, +because if signals of the Reactive Streams protocol (e.g. `request`) are lost the +the stream may deadlock. -.. warning:: - **Deprecation warning:** ``ActorPublisher`` is deprecated in favour of the vastly more - type-safe and safe to implement :class:`akka.stream.stage.GraphStage`. It can also - expose a "stage actor ref" is needed to be addressed as-if an Actor. - Custom stages implemented using ``GraphStage`` are also automatically fusable. - - To learn more about implementing custom stages using it refer to :ref:`graphstage-java`. +@@@ -Extend :class:`akka.stream.actor.AbstractActorPublisher` to implement a +#### ActorPublisher + +@@@ warning + +**Deprecation warning:** `ActorPublisher` is deprecated in favour of the vastly more +type-safe and safe to implement `akka.stream.stage.GraphStage`. It can also +expose a "stage actor ref" is needed to be addressed as-if an Actor. +Custom stages implemented using `GraphStage` are also automatically fusable. + +To learn more about implementing custom stages using it refer to @ref:[Custom processing with GraphStage](stream-customize.md#graphstage-java). + +@@@ + +Extend `akka.stream.actor.AbstractActorPublisher` to implement a stream publisher that keeps track of the subscription life cycle and requested elements. Here is an example of such an actor. It dispatches incoming jobs to the attached subscriber: -.. includecode:: ../code/jdocs/stream/ActorPublisherDocTest.java#job-manager +@@snip [ActorPublisherDocTest.java](../code/jdocs/stream/ActorPublisherDocTest.java) { #job-manager } -You send elements to the stream by calling ``onNext``. You are allowed to send as many +You send elements to the stream by calling `onNext`. You are allowed to send as many elements as have been requested by the stream subscriber. This amount can be inquired with -``totalDemand``. It is only allowed to use ``onNext`` when ``isActive`` and ``totalDemand>0``, -otherwise ``onNext`` will throw ``IllegalStateException``. +`totalDemand`. It is only allowed to use `onNext` when `isActive` and `totalDemand>0`, +otherwise `onNext` will throw `IllegalStateException`. -When the stream subscriber requests more elements the ``ActorPublisherMessage.Request`` message -is delivered to this actor, and you can act on that event. The ``totalDemand`` +When the stream subscriber requests more elements the `ActorPublisherMessage.Request` message +is delivered to this actor, and you can act on that event. The `totalDemand` is updated automatically. -When the stream subscriber cancels the subscription the ``ActorPublisherMessage.Cancel`` message -is delivered to this actor. After that subsequent calls to ``onNext`` will be ignored. +When the stream subscriber cancels the subscription the `ActorPublisherMessage.Cancel` message +is delivered to this actor. After that subsequent calls to `onNext` will be ignored. -You can complete the stream by calling ``onComplete``. After that you are not allowed to -call ``onNext``, ``onError`` and ``onComplete``. +You can complete the stream by calling `onComplete`. After that you are not allowed to +call `onNext`, `onError` and `onComplete`. -You can terminate the stream with failure by calling ``onError``. After that you are not allowed to -call ``onNext``, ``onError`` and ``onComplete``. +You can terminate the stream with failure by calling `onError`. After that you are not allowed to +call `onNext`, `onError` and `onComplete`. -If you suspect that this ``AbstractActorPublisher`` may never get subscribed to, you can override the ``subscriptionTimeout`` +If you suspect that this `AbstractActorPublisher` may never get subscribed to, you can override the `subscriptionTimeout` method to provide a timeout after which this Publisher should be considered canceled. The actor will be notified when -the timeout triggers via an ``ActorPublisherMessage.SubscriptionTimeoutExceeded`` message and MUST then perform +the timeout triggers via an `ActorPublisherMessage.SubscriptionTimeoutExceeded` message and MUST then perform cleanup and stop itself. If the actor is stopped the stream will be completed, unless it was not already terminated with @@ -480,47 +466,49 @@ failure, completed or canceled. More detailed information can be found in the API documentation. -This is how it can be used as input :class:`Source` to a :class:`Flow`: +This is how it can be used as input `Source` to a `Flow`: -.. includecode:: ../code/jdocs/stream/ActorPublisherDocTest.java#actor-publisher-usage +@@snip [ActorPublisherDocTest.java](../code/jdocs/stream/ActorPublisherDocTest.java) { #actor-publisher-usage } -You can only attach one subscriber to this publisher. Use a ``Broadcast``-element or -attach a ``Sink.asPublisher(AsPublisher.WITH_FANOUT)`` to enable multiple subscribers. +You can only attach one subscriber to this publisher. Use a `Broadcast`-element or +attach a `Sink.asPublisher(AsPublisher.WITH_FANOUT)` to enable multiple subscribers. -ActorSubscriber ---------------- +#### ActorSubscriber -.. warning:: - **Deprecation warning:** ``ActorSubscriber`` is deprecated in favour of the vastly more - type-safe and safe to implement :class:`akka.stream.stage.GraphStage`. It can also - expose a "stage actor ref" is needed to be addressed as-if an Actor. - Custom stages implemented using ``GraphStage`` are also automatically fusable. - - To learn more about implementing custom stages using it refer to :ref:`graphstage-scala`. +@@@ warning -Extend :class:`akka.stream.actor.AbstractActorSubscriber` to make your class a stream subscriber with +**Deprecation warning:** `ActorSubscriber` is deprecated in favour of the vastly more +type-safe and safe to implement `akka.stream.stage.GraphStage`. It can also +expose a "stage actor ref" is needed to be addressed as-if an Actor. +Custom stages implemented using `GraphStage` are also automatically fusable. + +To learn more about implementing custom stages using it refer to @ref:[Custom processing with GraphStage](../../scala/stream/stream-customize.md#graphstage-scala). + +@@@ + +Extend `akka.stream.actor.AbstractActorSubscriber` to make your class a stream subscriber with full control of stream back pressure. It will receive -``ActorSubscriberMessage.OnNext``, ``ActorSubscriberMessage.OnComplete`` and ``ActorSubscriberMessage.OnError`` +`ActorSubscriberMessage.OnNext`, `ActorSubscriberMessage.OnComplete` and `ActorSubscriberMessage.OnError` messages from the stream. It can also receive other, non-stream messages, in the same way as any actor. Here is an example of such an actor. It dispatches incoming jobs to child worker actors: -.. includecode:: ../code/jdocs/stream/ActorSubscriberDocTest.java#worker-pool +@@snip [ActorSubscriberDocTest.java](../code/jdocs/stream/ActorSubscriberDocTest.java) { #worker-pool } -Subclass must define the ``RequestStrategy`` to control stream back pressure. -After each incoming message the ``AbstractActorSubscriber`` will automatically invoke -the ``RequestStrategy.requestDemand`` and propagate the returned demand to the stream. +Subclass must define the `RequestStrategy` to control stream back pressure. +After each incoming message the `AbstractActorSubscriber` will automatically invoke +the `RequestStrategy.requestDemand` and propagate the returned demand to the stream. -* The provided ``WatermarkRequestStrategy`` is a good strategy if the actor performs work itself. -* The provided ``MaxInFlightRequestStrategy`` is useful if messages are queued internally or - delegated to other actors. -* You can also implement a custom ``RequestStrategy`` or call ``request`` manually together with - ``ZeroRequestStrategy`` or some other strategy. In that case - you must also call ``request`` when the actor is started or when it is ready, otherwise - it will not receive any elements. + * The provided `WatermarkRequestStrategy` is a good strategy if the actor performs work itself. + * The provided `MaxInFlightRequestStrategy` is useful if messages are queued internally or +delegated to other actors. + * You can also implement a custom `RequestStrategy` or call `request` manually together with +`ZeroRequestStrategy` or some other strategy. In that case +you must also call `request` when the actor is started or when it is ready, otherwise +it will not receive any elements. More detailed information can be found in the API documentation. -This is how it can be used as output :class:`Sink` to a :class:`Flow`: +This is how it can be used as output `Sink` to a `Flow`: -.. includecode:: ../code/jdocs/stream/ActorSubscriberDocTest.java#actor-subscriber-usage +@@snip [ActorSubscriberDocTest.java](../code/jdocs/stream/ActorSubscriberDocTest.java) { #actor-subscriber-usage } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-introduction.md b/akka-docs/src/main/paradox/java/stream/stream-introduction.md index f403ffee98..95b44a8fb2 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-introduction.md +++ b/akka-docs/src/main/paradox/java/stream/stream-introduction.md @@ -1,11 +1,6 @@ -.. _stream-introduction-java: +# Introduction -############ -Introduction -############ - -Motivation -========== +## Motivation The way we consume services from the Internet today includes many instances of streaming data, both downloading from a service as well as uploading to it or @@ -35,7 +30,7 @@ efficiently and with bounded resource usage—no more OutOfMemoryErrors. In orde to achieve this our streams need to be able to limit the buffering that they employ, they need to be able to slow down producers if the consumers cannot keep up. This feature is called back-pressure and is at the core of the -`Reactive Streams`_ initiative of which Akka is a +[Reactive Streams](http://reactive-streams.org/) initiative of which Akka is a founding member. For you this means that the hard problem of propagating and reacting to back-pressure has been incorporated in the design of Akka Streams already, so you have one less thing to worry about; it also means that Akka @@ -43,10 +38,7 @@ Streams interoperate seamlessly with all other Reactive Streams implementations (where Reactive Streams interfaces define the interoperability SPI while implementations like Akka Streams offer a nice user API). -.. _Reactive Streams: http://reactive-streams.org/ - -Relationship with Reactive Streams ----------------------------------- +### Relationship with Reactive Streams The Akka Streams API is completely decoupled from the Reactive Streams interfaces. While Akka Streams focus on the formulation of transformations on @@ -64,22 +56,20 @@ define interfaces such that different streaming implementation can interoperate; it is not the purpose of Reactive Streams to describe an end-user API. -How to read these docs -====================== +## How to read these docs Stream processing is a different paradigm to the Actor Model or to Future composition, therefore it may take some careful study of this subject until you feel familiar with the tools and techniques. The documentation is here to help and for best results we recommend the following approach: -* Read the :ref:`stream-quickstart-java` to get a feel for how streams - look like and what they can do. -* The top-down learners may want to peruse the :ref:`stream-design` at this - point. -* The bottom-up learners may feel more at home rummaging through the - :ref:`stream-cookbook-java`. -* For a complete overview of the built-in processing stages you can look at the - table in :ref:`stages-overview_java` -* The other sections can be read sequentially or as needed during the previous - steps, each digging deeper into specific topics. - + * Read the @ref:[Quick Start Guide](stream-quickstart.md#stream-quickstart-java) to get a feel for how streams +look like and what they can do. + * The top-down learners may want to peruse the @ref:[Design Principles behind Akka Streams](../../general/stream/stream-design.md) at this +point. + * The bottom-up learners may feel more at home rummaging through the +@ref:[Streams Cookbook](stream-cookbook.md). + * For a complete overview of the built-in processing stages you can look at the +table in @ref:[stages-overview_java](stages-overview.md) + * The other sections can be read sequentially or as needed during the previous +steps, each digging deeper into specific topics. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-io.md b/akka-docs/src/main/paradox/java/stream/stream-io.md index b80f421b15..6b22b2d37d 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-io.md +++ b/akka-docs/src/main/paradox/java/stream/stream-io.md @@ -1,64 +1,59 @@ -.. _stream-io-java: - -######################### -Working with streaming IO -######################### +# Working with streaming IO Akka Streams provides a way of handling File IO and TCP connections with Streams. -While the general approach is very similar to the :ref:`Actor based TCP handling ` using Akka IO, +While the general approach is very similar to the @ref:[Actor based TCP handling](../io-tcp.md) using Akka IO, by using Akka Streams you are freed of having to manually react to back-pressure signals, as the library does it transparently for you. -Streaming TCP -============= +## Streaming TCP -Accepting connections: Echo Server -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -In order to implement a simple EchoServer we ``bind`` to a given address, which returns a ``Source>``, -which will emit an :class:`IncomingConnection` element for each new connection that the Server should handle: +### Accepting connections: Echo Server -.. includecode:: ../code/jdocs/stream/io/StreamTcpDocTest.java#echo-server-simple-bind +In order to implement a simple EchoServer we `bind` to a given address, which returns a `Source>`, +which will emit an `IncomingConnection` element for each new connection that the Server should handle: -.. image:: ../../images/tcp-stream-bind.png +@@snip [StreamTcpDocTest.java](../code/jdocs/stream/io/StreamTcpDocTest.java) { #echo-server-simple-bind } -Next, we simply handle *each* incoming connection using a :class:`Flow` which will be used as the processing stage -to handle and emit ``ByteString`` s from and to the TCP Socket. Since one :class:`ByteString` does not have to necessarily -correspond to exactly one line of text (the client might be sending the line in chunks) we use the ``delimiter`` -helper Flow from ``akka.stream.javadsl.Framing`` to chunk the inputs up into actual lines of text. The last boolean +![tcp-stream-bind.png](../../images/tcp-stream-bind.png) + +Next, we simply handle *each* incoming connection using a `Flow` which will be used as the processing stage +to handle and emit `ByteString` s from and to the TCP Socket. Since one `ByteString` does not have to necessarily +correspond to exactly one line of text (the client might be sending the line in chunks) we use the `delimiter` +helper Flow from `akka.stream.javadsl.Framing` to chunk the inputs up into actual lines of text. The last boolean argument indicates that we require an explicit line ending even for the last message before the connection is closed. In this example we simply add exclamation marks to each incoming text message and push it through the flow: -.. includecode:: ../code/jdocs/stream/io/StreamTcpDocTest.java#echo-server-simple-handle +@@snip [StreamTcpDocTest.java](../code/jdocs/stream/io/StreamTcpDocTest.java) { #echo-server-simple-handle } -.. image:: ../../images/tcp-stream-run.png +![tcp-stream-run.png](../../images/tcp-stream-run.png) Notice that while most building blocks in Akka Streams are reusable and freely shareable, this is *not* the case for the incoming connection Flow, since it directly corresponds to an existing, already accepted connection its handling can only ever be materialized *once*. -Closing connections is possible by cancelling the *incoming connection* :class:`Flow` from your server logic (e.g. by -connecting its downstream to a :class:`Sink.cancelled()` and its upstream to a :class:`Source.empty()`). -It is also possible to shut down the server's socket by cancelling the :class:`IncomingConnection` source ``connections``. +Closing connections is possible by cancelling the *incoming connection* `Flow` from your server logic (e.g. by +connecting its downstream to a `Sink.cancelled()` and its upstream to a `Source.empty()`). +It is also possible to shut down the server's socket by cancelling the `IncomingConnection` source `connections`. -We can then test the TCP server by sending data to the TCP Socket using ``netcat``: +We can then test the TCP server by sending data to the TCP Socket using `netcat`: -:: +``` +$ echo -n "Hello World" | netcat 127.0.0.1 8889 +Hello World!!! +``` - $ echo -n "Hello World" | netcat 127.0.0.1 8889 - Hello World!!! +### Connecting: REPL Client -Connecting: REPL Client -^^^^^^^^^^^^^^^^^^^^^^^ In this example we implement a rather naive Read Evaluate Print Loop client over TCP. Let's say we know a server has exposed a simple command line interface over TCP, and would like to interact with it using Akka Streams over TCP. To open an outgoing connection socket we use -the ``outgoingConnection`` method: +the `outgoingConnection` method: -.. includecode:: ../code/jdocs/stream/io/StreamTcpDocTest.java#repl-client +@@snip [StreamTcpDocTest.java](../code/jdocs/stream/io/StreamTcpDocTest.java) { #repl-client } -The ``repl`` flow we use to handle the server interaction first prints the servers response, then awaits on input from +The `repl` flow we use to handle the server interaction first prints the servers response, then awaits on input from the command line (this blocking call is used here just for the sake of simplicity) and converts it to a -:class:`ByteString` which is then sent over the wire to the server. Then we simply connect the TCP pipeline to this +`ByteString` which is then sent over the wire to the server. Then we simply connect the TCP pipeline to this processing stage–at this point it will be materialized and start processing data once the server responds with an *initial message*. @@ -66,19 +61,22 @@ A resilient REPL client would be more sophisticated than this, for example it sh a separate mapAsync step and have a way to let the server write more data than one ByteString chunk at any given time, these improvements however are left as exercise for the reader. -Avoiding deadlocks and liveness issues in back-pressured cycles -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Avoiding deadlocks and liveness issues in back-pressured cycles + When writing such end-to-end back-pressured systems you may sometimes end up in a situation of a loop, in which *either side is waiting for the other one to start the conversation*. One does not need to look far to find examples of such back-pressure loops. In the two examples shown previously, we always assumed that the side we are connecting to would start the conversation, which effectively means both sides are back-pressured and can not get -the conversation started. There are multiple ways of dealing with this which are explained in depth in :ref:`graph-cycles-java`, +the conversation started. There are multiple ways of dealing with this which are explained in depth in @ref:[Graph cycles, liveness and deadlocks](stream-graphs.md#graph-cycles-java), however in client-server scenarios it is often the simplest to make either side simply send an initial message. -.. note:: - In case of back-pressured cycles (which can occur even between different systems) sometimes you have to decide - which of the sides has start the conversation in order to kick it off. This can be often done by injecting an - initial message from one of the sides–a conversation starter. +@@@ note + +In case of back-pressured cycles (which can occur even between different systems) sometimes you have to decide +which of the sides has start the conversation in order to kick it off. This can be often done by injecting an +initial message from one of the sides–a conversation starter. + +@@@ To break this back-pressure cycle we need to inject some initial message, a "conversation starter". First, we need to decide which side of the connection should remain passive and which active. @@ -86,31 +84,29 @@ Thankfully in most situations finding the right spot to start the conversation i to the protocol we are trying to implement using Streams. In chat-like applications, which our examples resemble, it makes sense to make the Server initiate the conversation by emitting a "hello" message: -.. includecode:: ../code/jdocs/stream/io/StreamTcpDocTest.java#welcome-banner-chat-server +@@snip [StreamTcpDocTest.java](../code/jdocs/stream/io/StreamTcpDocTest.java) { #welcome-banner-chat-server } -To emit the initial message we merge a ``Source`` with a single element, after the command processing but before the -framing and transformation to ``ByteString`` s this way we do not have to repeat such logic. +To emit the initial message we merge a `Source` with a single element, after the command processing but before the +framing and transformation to `ByteString` s this way we do not have to repeat such logic. -In this example both client and server may need to close the stream based on a parsed command - ``BYE`` in the case -of the server, and ``q`` in the case of the client. This is implemented by using a custom :class:`GraphStage` +In this example both client and server may need to close the stream based on a parsed command - `BYE` in the case +of the server, and `q` in the case of the client. This is implemented by using a custom `GraphStage` which completes the stream once it encounters such command. -Streaming File IO -================= +## Streaming File IO -Akka Streams provide simple Sources and Sinks that can work with :class:`ByteString` instances to perform IO operations +Akka Streams provide simple Sources and Sinks that can work with `ByteString` instances to perform IO operations on files. +Streaming data from a file is as easy as creating a *FileIO.fromPath* given a target path, and an optional +`chunkSize` which determines the buffer size determined as one "element" in such stream: -Streaming data from a file is as easy as creating a `FileIO.fromPath` given a target path, and an optional -``chunkSize`` which determines the buffer size determined as one "element" in such stream: - -.. includecode:: ../code/jdocs/stream/io/StreamFileDocTest.java#file-source +@@snip [StreamFileDocTest.java](../code/jdocs/stream/io/StreamFileDocTest.java) { #file-source } Please note that these processing stages are backed by Actors and by default are configured to run on a pre-configured threadpool-backed dispatcher dedicated for File IO. This is very important as it isolates the blocking file IO operations from the rest of the ActorSystem allowing each dispatcher to be utilised in the most efficient way. If you want to configure a custom -dispatcher for file IO operations globally, you can do so by changing the ``akka.stream.blocking-io-dispatcher``, +dispatcher for file IO operations globally, you can do so by changing the `akka.stream.blocking-io-dispatcher`, or for a specific stage by specifying a custom Dispatcher in code, like this: -.. includecode:: ../code/jdocs/stream/io/StreamFileDocTest.java#custom-dispatcher-code +@@snip [StreamFileDocTest.java](../code/jdocs/stream/io/StreamFileDocTest.java) { #custom-dispatcher-code } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-parallelism.md b/akka-docs/src/main/paradox/java/stream/stream-parallelism.md index 2b60e5f63a..e7031429ed 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-parallelism.md +++ b/akka-docs/src/main/paradox/java/stream/stream-parallelism.md @@ -1,15 +1,11 @@ -.. _stream-parallelism-java: - -########################## -Pipelining and Parallelism -########################## +# Pipelining and Parallelism Akka Streams processing stages (be it simple operators on Flows and Sources or graph junctions) are "fused" together and executed sequentially by default. This avoids the overhead of events crossing asynchronous boundaries but limits the flow to execute at most one stage at any given time. In many cases it is useful to be able to concurrently execute the stages of a flow, this is done by explicitly marking -them as asynchronous using the ``async()`` method. Each processing stage marked as asynchronous will run in a +them as asynchronous using the `async()` method. Each processing stage marked as asynchronous will run in a dedicated actor internally, while all stages not marked asynchronous will run in one single actor. We will illustrate through the example of pancake cooking how streams can be used for various processing patterns, @@ -18,8 +14,7 @@ like to make pancakes, but they need to produce sufficient amount in a cooking s happy. To increase their pancake production throughput they use two frying pans. How they organize their pancake processing is markedly different. -Pipelining ----------- +## Pipelining Roland uses the two frying pans in an asymmetric fashion. The first pan is only used to fry one side of the pancake then the half-finished pancake is flipped into the second pan for the finishing fry on the other side. @@ -28,49 +23,49 @@ are two pancakes being cooked at the same time, one being cooked on its first si completion. This is how this setup would look like implemented as a stream: -.. includecode:: ../code/jdocs/stream/FlowParallelismDocTest.java#pipelining +@@snip [FlowParallelismDocTest.java](../code/jdocs/stream/FlowParallelismDocTest.java) { #pipelining } -The two ``map`` stages in sequence (encapsulated in the "frying pan" flows) will be executed in a pipelined way, +The two `map` stages in sequence (encapsulated in the "frying pan" flows) will be executed in a pipelined way, basically doing the same as Roland with his frying pans: - 1. A ``ScoopOfBatter`` enters ``fryingPan1`` - 2. ``fryingPan1`` emits a HalfCookedPancake once ``fryingPan2`` becomes available - 3. ``fryingPan2`` takes the HalfCookedPancake +> + 1. A `ScoopOfBatter` enters `fryingPan1` + 2. `fryingPan1` emits a HalfCookedPancake once `fryingPan2` becomes available + 3. `fryingPan2` takes the HalfCookedPancake 4. at this point fryingPan1 already takes the next scoop, without waiting for fryingPan2 to finish The benefit of pipelining is that it can be applied to any sequence of processing steps that are otherwise not parallelisable (for example because the result of a processing step depends on all the information from the previous step). One drawback is that if the processing times of the stages are very different then some of the stages will not be able to operate at full throughput because they will wait on a previous or subsequent stage most of the time. In the -pancake example frying the second half of the pancake is usually faster than frying the first half, ``fryingPan2`` will -not be able to operate at full capacity [#]_. +pancake example frying the second half of the pancake is usually faster than frying the first half, `fryingPan2` will +not be able to operate at full capacity [1]. note:: - Asynchronous stream processing stages have internal buffers to make communication between them more efficient. - For more details about the behavior of these and how to add additional buffers refer to :ref:`stream-rate-scala`. +: Asynchronous stream processing stages have internal buffers to make communication between them more efficient. +For more details about the behavior of these and how to add additional buffers refer to @ref:[Buffers and working with rate](../../scala/stream/stream-rate.md). -Parallel processing -------------------- + +## Parallel processing Patrik uses the two frying pans symmetrically. He uses both pans to fully fry a pancake on both sides, then puts the results on a shared plate. Whenever a pan becomes empty, he takes the next scoop from the shared bowl of batter. In essence he parallelizes the same process over multiple pans. This is how this setup will look like if implemented using streams: -.. includecode:: ../code/jdocs/stream/FlowParallelismDocTest.java#parallelism +@@snip [FlowParallelismDocTest.java](../code/jdocs/stream/FlowParallelismDocTest.java) { #parallelism } The benefit of parallelizing is that it is easy to scale. In the pancake example it is easy to add a third frying pan with Patrik's method, but Roland cannot add a third frying pan, since that would require a third processing step, which is not practically possible in the case of frying pancakes. One drawback of the example code above that it does not preserve the ordering of pancakes. This might be a problem -if children like to track their "own" pancakes. In those cases the ``Balance`` and ``Merge`` stages should be replaced +if children like to track their "own" pancakes. In those cases the `Balance` and `Merge` stages should be replaced by strict-round robing balancing and merging stages that put in and take out pancakes in a strict order. -A more detailed example of creating a worker pool can be found in the cookbook: :ref:`cookbook-balance-scala` +A more detailed example of creating a worker pool can be found in the cookbook: @ref:[Balancing jobs to a fixed pool of workers](../../scala/stream/stream-cookbook.md#cookbook-balance-scala) -Combining pipelining and parallel processing --------------------------------------------- +## Combining pipelining and parallel processing The two concurrency patterns that we demonstrated as means to increase throughput are not exclusive. In fact, it is rather simple to combine the two approaches and streams provide @@ -80,7 +75,7 @@ First, let's look at how we can parallelize pipelined processing stages. In the will employ two chefs, each working using Roland's pipelining method, but we use the two chefs in parallel, just like Patrik used the two frying pans. This is how it looks like if expressed as streams: -.. includecode:: ../code/jdocs/stream/FlowParallelismDocTest.java#parallel-pipeline +@@snip [FlowParallelismDocTest.java](../code/jdocs/stream/FlowParallelismDocTest.java) { #parallel-pipeline } The above pattern works well if there are many independent jobs that do not depend on the results of each other, but the jobs themselves need multiple processing steps where each step builds on the result of @@ -90,14 +85,15 @@ in sequence. It is also possible to organize parallelized stages into pipelines. This would mean employing four chefs: - - the first two chefs prepare half-cooked pancakes from batter, in parallel, then putting those on a large enough - flat surface. - - the second two chefs take these and fry their other side in their own pans, then they put the pancakes on a shared - plate. +> + * the first two chefs prepare half-cooked pancakes from batter, in parallel, then putting those on a large enough +flat surface. + * the second two chefs take these and fry their other side in their own pans, then they put the pancakes on a shared +plate. This is again straightforward to implement with the streams API: -.. includecode:: ../code/jdocs/stream/FlowParallelismDocTest.java#pipelined-parallel +@@snip [FlowParallelismDocTest.java](../code/jdocs/stream/FlowParallelismDocTest.java) { #pipelined-parallel } This usage pattern is less common but might be usable if a certain step in the pipeline might take wildly different times to finish different jobs. The reason is that there are more balance-merge steps in this pattern @@ -105,5 +101,5 @@ compared to the parallel pipelines. This pattern rebalances after each step, whi at the entry point of the pipeline. This only matters however if the processing time distribution has a large deviation. -.. [#] Roland's reason for this seemingly suboptimal procedure is that he prefers the temperature of the second pan - to be slightly lower than the first in order to achieve a more homogeneous result. \ No newline at end of file +> [1] Roland's reason for this seemingly suboptimal procedure is that he prefers the temperature of the second pan +to be slightly lower than the first in order to achieve a more homogeneous result. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-quickstart.md b/akka-docs/src/main/paradox/java/stream/stream-quickstart.md index 8b953e31d8..bd20a554d7 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-quickstart.md +++ b/akka-docs/src/main/paradox/java/stream/stream-quickstart.md @@ -1,40 +1,39 @@ -.. _stream-quickstart-java: -Quick Start Guide -================= + +# Quick Start Guide Create a project and add the akka-streams dependency to the build tool of your -choice as described in :ref:`build-tool`. +choice as described in @ref:[Using a build tool](../../intro/getting-started.md#build-tool). A stream usually begins at a source, so this is also how we start an Akka Stream. Before we create one, we import the full complement of streaming tools: -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#stream-imports +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #stream-imports } If you want to execute the code samples while you read through the quick start guide, you will also need the following imports: -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#other-imports +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #other-imports } And a class to hold your code, for example: -.. includecode:: ../code/jdocs/stream/Main.java#main-app +@@snip [Main.java](../code/jdocs/stream/Main.java) { #main-app } Now we will start with a rather simple source, emitting the integers 1 to 100: -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#create-source +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #create-source } -The :class:`Source` type is parameterized with two types: the first one is the +The `Source` type is parameterized with two types: the first one is the type of element that this source emits and the second one may signal that running the source produces some auxiliary value (e.g. a network source may provide information about the bound port or the peer’s address). Where no -auxiliary information is produced, the type ``akka.NotUsed`` is used—and a +auxiliary information is produced, the type `akka.NotUsed` is used—and a simple range of integers surely falls into this category. Having created this source means that we have a description of how to emit the first 100 natural numbers, but this source is not yet active. In order to get those numbers out we have to run it: -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#run-source +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #run-source } This line will complement the source with a consumer function—in this example we simply print out the numbers to the console—and pass this little stream @@ -43,94 +42,92 @@ part of the method name; there are other methods that run Akka Streams, and they all follow this pattern. When running this program you might notice it does not -terminate, because the :class:`ActorSystem` is never terminated. Luckily -``runForeach`` returns a :class:`CompletionStage` which resolves when the stream finishes: +terminate, because the `ActorSystem` is never terminated. Luckily +`runForeach` returns a `CompletionStage` which resolves when the stream finishes: -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#run-source-and-terminate +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #run-source-and-terminate } You may wonder where the Actor gets created that runs the stream, and you are -probably also asking yourself what this ``materializer`` means. In order to get +probably also asking yourself what this `materializer` means. In order to get this value we first need to create an Actor system: -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#create-materializer +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #create-materializer } There are other ways to create a materializer, e.g. from an -:class:`ActorContext` when using streams from within Actors. The -:class:`Materializer` is a factory for stream execution engines, it is the +`ActorContext` when using streams from within Actors. The +`Materializer` is a factory for stream execution engines, it is the thing that makes streams run—you don’t need to worry about any of the details -just now apart from that you need one for calling any of the ``run`` methods on -a :class:`Source`. +just now apart from that you need one for calling any of the `run` methods on +a `Source`. -The nice thing about Akka Streams is that the :class:`Source` is just a +The nice thing about Akka Streams is that the `Source` is just a description of what you want to run, and like an architect’s blueprint it can be reused, incorporated into a larger design. We may choose to transform the source of integers and write it to a file instead: -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#transform-source +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #transform-source } -First we use the ``scan`` combinator to run a computation over the whole -stream: starting with the number 1 (``BigInteger.ONE``) we multiple by each of +First we use the `scan` combinator to run a computation over the whole +stream: starting with the number 1 (`BigInteger.ONE`) we multiple by each of the incoming numbers, one after the other; the scan operation emits the initial value and then every calculation result. This yields the series of factorial -numbers which we stash away as a :class:`Source` for later reuse—it is +numbers which we stash away as a `Source` for later reuse—it is important to keep in mind that nothing is actually computed yet, this is just a description of what we want to have computed once we run the stream. Then we -convert the resulting series of numbers into a stream of :class:`ByteString` +convert the resulting series of numbers into a stream of `ByteString` objects describing lines in a text file. This stream is then run by attaching a file as the receiver of the data. In the terminology of Akka Streams this is -called a :class:`Sink`. :class:`IOResult` is a type that IO operations return +called a `Sink`. `IOResult` is a type that IO operations return in Akka Streams in order to tell you how many bytes or elements were consumed and whether the stream terminated normally or exceptionally. -Reusable Pieces ---------------- +## Reusable Pieces One of the nice parts of Akka Streams—and something that other stream libraries do not offer—is that not only sources can be reused like blueprints, all other -elements can be as well. We can take the file-writing :class:`Sink`, prepend -the processing steps necessary to get the :class:`ByteString` elements from +elements can be as well. We can take the file-writing `Sink`, prepend +the processing steps necessary to get the `ByteString` elements from incoming strings and package that up as a reusable piece as well. Since the language for writing these streams always flows from left to right (just like plain English), we need a starting point that is like a source but with an -“open” input. In Akka Streams this is called a :class:`Flow`: +“open” input. In Akka Streams this is called a `Flow`: -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#transform-sink +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #transform-sink } -Starting from a flow of strings we convert each to :class:`ByteString` and then -feed to the already known file-writing :class:`Sink`. The resulting blueprint -is a :class:`Sink>`, which means that it +Starting from a flow of strings we convert each to `ByteString` and then +feed to the already known file-writing `Sink`. The resulting blueprint +is a `Sink>`, which means that it accepts strings as its input and when materialized it will create auxiliary -information of type ``CompletionStage`` (when chaining operations on -a :class:`Source` or :class:`Flow` the type of the auxiliary information—called +information of type `CompletionStage` (when chaining operations on +a `Source` or `Flow` the type of the auxiliary information—called the “materialized value”—is given by the leftmost starting point; since we want -to retain what the ``FileIO.toPath`` sink has to offer, we need to say -``Keep.right()``). +to retain what the `FileIO.toPath` sink has to offer, we need to say +`Keep.right()`). -We can use the new and shiny :class:`Sink` we just created by -attaching it to our ``factorials`` source—after a small adaptation to turn the +We can use the new and shiny `Sink` we just created by +attaching it to our `factorials` source—after a small adaptation to turn the numbers into strings: -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#use-transformed-sink +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #use-transformed-sink } -Time-Based Processing ---------------------- +## Time-Based Processing Before we start looking at a more involved example we explore the streaming -nature of what Akka Streams can do. Starting from the ``factorials`` source +nature of what Akka Streams can do. Starting from the `factorials` source we transform the stream by zipping it together with another stream, -represented by a :class:`Source` that emits the number 0 to 100: the first -number emitted by the ``factorials`` source is the factorial of zero, the +represented by a `Source` that emits the number 0 to 100: the first +number emitted by the `factorials` source is the factorial of zero, the second is the factorial of one, and so on. We combine these two by forming -strings like ``"3! = 6"``. +strings like `"3! = 6"`. -.. includecode:: ../code/jdocs/stream/QuickStartDocTest.java#add-streams +@@snip [QuickStartDocTest.java](../code/jdocs/stream/QuickStartDocTest.java) { #add-streams } All operations so far have been time-independent and could have been performed in the same fashion on strict collections of elements. The next line demonstrates that we are in fact dealing with streams that can flow at a -certain speed: we use the ``throttle`` combinator to slow down the stream to 1 -element per second (the second ``1`` in the argument list is the maximum size -of a burst that we want to allow—passing ``1`` means that the first element +certain speed: we use the `throttle` combinator to slow down the stream to 1 +element per second (the second `1` in the argument list is the maximum size +of a burst that we want to allow—passing `1` means that the first element gets through immediately and the second then has to wait for one second and so on). @@ -140,7 +137,7 @@ the streams to produce a billion numbers each then you will notice that your JVM does not crash with an OutOfMemoryError, even though you will also notice that running the streams happens in the background, asynchronously (this is the reason for the auxiliary information to be provided as a -:class:`CompletionStage`, in the future). The secret that makes this work is +`CompletionStage`, in the future). The secret that makes this work is that Akka Streams implicitly implement pervasive flow control, all combinators respect back-pressure. This allows the throttle combinator to signal to all its upstream sources of data that it can only accept elements at a certain @@ -149,10 +146,9 @@ combinator will assert *back-pressure* upstream. This is basically all there is to Akka Streams in a nutshell—glossing over the fact that there are dozens of sources and sinks and many more stream -transformation combinators to choose from, see also :ref:`stages-overview_java`. +transformation combinators to choose from, see also @ref:[stages-overview_java](stages-overview.md). -Reactive Tweets -=============== +# Reactive Tweets A typical use case for stream processing is consuming a live stream of data that we want to extract or aggregate some other data from. In this example we'll consider consuming a stream of tweets and extracting information concerning Akka from them. @@ -166,141 +162,142 @@ allow to control what should happen in such scenarios. Here's the data model we'll be working with throughout the quickstart examples: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#model +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #model } +@@@ note -.. note:: - If you would like to get an overview of the used vocabulary first instead of diving head-first - into an actual example you can have a look at the :ref:`core-concepts-java` and :ref:`defining-and-running-streams-java` - sections of the docs, and then come back to this quickstart to see it all pieced together into a simple example application. +If you would like to get an overview of the used vocabulary first instead of diving head-first +into an actual example you can have a look at the @ref:[Core concepts](stream-flows-and-basics.md#core-concepts-java) and @ref:[Defining and running streams](stream-flows-and-basics.md#defining-and-running-streams-java) +sections of the docs, and then come back to this quickstart to see it all pieced together into a simple example application. + +@@@ + +## Transforming and consuming simple streams -Transforming and consuming simple streams ------------------------------------------ The example application we will be looking at is a simple Twitter feed stream from which we'll want to extract certain information, -like for example finding all twitter handles of users who tweet about ``#akka``. +like for example finding all twitter handles of users who tweet about `#akka`. -In order to prepare our environment by creating an :class:`ActorSystem` and :class:`ActorMaterializer`, +In order to prepare our environment by creating an `ActorSystem` and `ActorMaterializer`, which will be responsible for materializing and running the streams we are about to create: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#materializer-setup +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #materializer-setup } -The :class:`ActorMaterializer` can optionally take :class:`ActorMaterializerSettings` which can be used to define -materialization properties, such as default buffer sizes (see also :ref:`async-stream-buffers-java`), the dispatcher to -be used by the pipeline etc. These can be overridden with ``withAttributes`` on :class:`Flow`, :class:`Source`, :class:`Sink` and :class:`Graph`. +The `ActorMaterializer` can optionally take `ActorMaterializerSettings` which can be used to define +materialization properties, such as default buffer sizes (see also @ref:[Buffers for asynchronous stages](stream-rate.md#async-stream-buffers-java)), the dispatcher to +be used by the pipeline etc. These can be overridden with `withAttributes` on `Flow`, `Source`, `Sink` and `Graph`. -Let's assume we have a stream of tweets readily available. In Akka this is expressed as a :class:`Source`: +Let's assume we have a stream of tweets readily available. In Akka this is expressed as a `Source`: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#tweet-source +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #tweet-source } -Streams always start flowing from a ``Source`` then can continue through ``Flow`` elements or -more advanced graph elements to finally be consumed by a ``Sink``. +Streams always start flowing from a `Source` then can continue through `Flow` elements or +more advanced graph elements to finally be consumed by a `Sink`. -The first type parameter—:class:`Tweet` in this case—designates the kind of elements produced -by the source while the ``M`` type parameters describe the object that is created during -materialization (:ref:`see below `)—:class:`NotUsed` (from the ``scala.runtime`` -package) means that no value is produced, it is the generic equivalent of ``void``. +The first type parameter—`Tweet` in this case—designates the kind of elements produced +by the source while the `M` type parameters describe the object that is created during +materialization ([see below](#materialized-values-quick-java))—`NotUsed` (from the `scala.runtime` +package) means that no value is produced, it is the generic equivalent of `void`. The operations should look familiar to anyone who has used the Scala Collections library, however they operate on streams and not collections of data (which is a very important distinction, as some operations only make sense in streaming and vice versa): -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#authors-filter-map +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #authors-filter-map } -Finally in order to :ref:`materialize ` and run the stream computation we need to attach -the Flow to a ``Sink`` that will get the Flow running. The simplest way to do this is to call -``runWith(sink)`` on a ``Source``. For convenience a number of common Sinks are predefined and collected as static methods on -the ``Sink class``. +Finally in order to @ref:[materialize](stream-flows-and-basics.md#stream-materialization-java) and run the stream computation we need to attach +the Flow to a `Sink` that will get the Flow running. The simplest way to do this is to call +`runWith(sink)` on a `Source`. For convenience a number of common Sinks are predefined and collected as static methods on +the `Sink class`. For now let's simply print each author: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#authors-foreachsink-println +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #authors-foreachsink-println } -or by using the shorthand version (which are defined only for the most popular Sinks such as :class:`Sink.fold` and :class:`Sink.foreach`): +or by using the shorthand version (which are defined only for the most popular Sinks such as `Sink.fold` and `Sink.foreach`): -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#authors-foreach-println +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #authors-foreach-println } -Materializing and running a stream always requires a :class:`Materializer` to be passed in explicitly, -like this: ``.run(mat)``. +Materializing and running a stream always requires a `Materializer` to be passed in explicitly, +like this: `.run(mat)`. The complete snippet looks like this: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#first-sample +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #first-sample } + +## Flattening sequences in streams -Flattening sequences in streams -------------------------------- In the previous section we were working on 1:1 relationships of elements which is the most common case, but sometimes -we might want to map from one element to a number of elements and receive a "flattened" stream, similarly like ``flatMap`` -works on Scala Collections. In order to get a flattened stream of hashtags from our stream of tweets we can use the ``mapConcat`` +we might want to map from one element to a number of elements and receive a "flattened" stream, similarly like `flatMap` +works on Scala Collections. In order to get a flattened stream of hashtags from our stream of tweets we can use the `mapConcat` combinator: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#hashtags-mapConcat +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #hashtags-mapConcat } -.. note:: - The name ``flatMap`` was consciously avoided due to its proximity with for-comprehensions and monadic composition. - It is problematic for two reasons: firstly, flattening by concatenation is often undesirable in bounded stream processing - due to the risk of deadlock (with merge being the preferred strategy), and secondly, the monad laws would not hold for - our implementation of flatMap (due to the liveness issues). +@@@ note - Please note that the ``mapConcat`` requires the supplied function to return a strict collection (``Out f -> java.util.List``), - whereas ``flatMap`` would have to operate on streams all the way through. +The name `flatMap` was consciously avoided due to its proximity with for-comprehensions and monadic composition. +It is problematic for two reasons: firstly, flattening by concatenation is often undesirable in bounded stream processing +due to the risk of deadlock (with merge being the preferred strategy), and secondly, the monad laws would not hold for +our implementation of flatMap (due to the liveness issues). +Please note that the `mapConcat` requires the supplied function to return a strict collection (`Out f -> java.util.List`), +whereas `flatMap` would have to operate on streams all the way through. + +@@@ + +## Broadcasting a stream -Broadcasting a stream ---------------------- Now let's say we want to persist all hashtags, as well as all author names from this one live stream. For example we'd like to write all author handles into one file, and all hashtags into another file on disk. This means we have to split the source stream into two streams which will handle the writing to these different files. Elements that can be used to form such "fan-out" (or "fan-in") structures are referred to as "junctions" in Akka Streams. -One of these that we'll be using in this example is called :class:`Broadcast`, and it simply emits elements from its +One of these that we'll be using in this example is called `Broadcast`, and it simply emits elements from its input port to all of its output ports. Akka Streams intentionally separate the linear stream structures (Flows) from the non-linear, branching ones (Graphs) in order to offer the most convenient API for both of these cases. Graphs can express arbitrarily complex stream setups at the expense of not reading as familiarly as collection transformations. -Graphs are constructed using :class:`GraphDSL` like this: +Graphs are constructed using `GraphDSL` like this: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#graph-dsl-broadcast +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #graph-dsl-broadcast } -As you can see, we use graph builder ``b`` to construct the graph using ``UniformFanOutShape`` and ``Flow`` s. +As you can see, we use graph builder `b` to construct the graph using `UniformFanOutShape` and `Flow` s. -``GraphDSL.create`` returns a :class:`Graph`, in this example a ``Graph`` where -:class:`ClosedShape` means that it is *a fully connected graph* or "closed" - there are no unconnected inputs or outputs. -Since it is closed it is possible to transform the graph into a :class:`RunnableGraph` using ``RunnableGraph.fromGraph``. -The runnable graph can then be ``run()`` to materialize a stream out of it. +`GraphDSL.create` returns a `Graph`, in this example a `Graph` where +`ClosedShape` means that it is *a fully connected graph* or "closed" - there are no unconnected inputs or outputs. +Since it is closed it is possible to transform the graph into a `RunnableGraph` using `RunnableGraph.fromGraph`. +The runnable graph can then be `run()` to materialize a stream out of it. -Both :class:`Graph` and :class:`RunnableGraph` are *immutable, thread-safe, and freely shareable*. +Both `Graph` and `RunnableGraph` are *immutable, thread-safe, and freely shareable*. A graph can also have one of several other shapes, with one or more unconnected ports. Having unconnected ports expresses a graph that is a *partial graph*. Concepts around composing and nesting graphs in large structures are -explained in detail in :ref:`composition-java`. It is also possible to wrap complex computation graphs -as Flows, Sinks or Sources, which will be explained in detail in :ref:`partial-graph-dsl-java`. +explained in detail in @ref:[Modularity, Composition and Hierarchy](stream-composition.md). It is also possible to wrap complex computation graphs +as Flows, Sinks or Sources, which will be explained in detail in @ref:[Constructing and combining Partial Graphs](stream-graphs.md#partial-graph-dsl-java). - -Back-pressure in action ------------------------ +## Back-pressure in action One of the main advantages of Akka Streams is that they *always* propagate back-pressure information from stream Sinks (Subscribers) to their Sources (Publishers). It is not an optional feature, and is enabled at all times. To learn more about the back-pressure protocol used by Akka Streams and all other Reactive Streams compatible implementations read -:ref:`back-pressure-explained-java`. +@ref:[Back-pressure explained](stream-flows-and-basics.md#back-pressure-explained-java). A typical problem applications (not using Akka Streams) like this often face is that they are unable to process the incoming data fast enough, either temporarily or by design, and will start buffering incoming data until there's no more space to buffer, resulting -in either ``OutOfMemoryError`` s or other severe degradations of service responsiveness. With Akka Streams buffering can +in either `OutOfMemoryError` s or other severe degradations of service responsiveness. With Akka Streams buffering can and must be handled explicitly. For example, if we are only interested in the "*most recent tweets, with a buffer of 10 -elements*" this can be expressed using the ``buffer`` element: +elements*" this can be expressed using the `buffer` element: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#tweets-slow-consumption-dropHead +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #tweets-slow-consumption-dropHead } -The ``buffer`` element takes an explicit and required ``OverflowStrategy``, which defines how the buffer should react -when it receives another element while it is full. Strategies provided include dropping the oldest element (``dropHead``), +The `buffer` element takes an explicit and required `OverflowStrategy`, which defines how the buffer should react +when it receives another element while it is full. Strategies provided include dropping the oldest element (`dropHead`), dropping the entire buffer, signalling failures etc. Be sure to pick and choose the strategy that fits your use case best. -.. _materialized-values-quick-java: + +## Materialized values -Materialized values -------------------- So far we've been only processing data using Flows and consuming it into some kind of external Sink - be it by printing values or storing them in some external system. However sometimes we may be interested in some value that can be obtained from the materialized processing pipeline. For example, we want to know how many tweets we have processed. @@ -308,43 +305,46 @@ While this question is not as obvious to give an answer to in case of an infinit this question in a streaming setting would be to create a stream of counts described as "*up until now*, we've processed N tweets"), but in general it is possible to deal with finite streams and come up with a nice result such as a total count of elements. -First, let's write such an element counter using ``Flow.of(Class)`` and ``Sink.fold`` to see how the types look like: +First, let's write such an element counter using `Flow.of(Class)` and `Sink.fold` to see how the types look like: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#tweets-fold-count +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #tweets-fold-count } -First we prepare a reusable ``Flow`` that will change each incoming tweet into an integer of value ``1``. We'll use this in -order to combine those with a ``Sink.fold`` that will sum all ``Integer`` elements of the stream and make its result available as -a ``CompletionStage``. Next we connect the ``tweets`` stream to ``count`` with ``via``. Finally we connect the Flow to the previously -prepared Sink using ``toMat``. +First we prepare a reusable `Flow` that will change each incoming tweet into an integer of value `1`. We'll use this in +order to combine those with a `Sink.fold` that will sum all `Integer` elements of the stream and make its result available as +a `CompletionStage`. Next we connect the `tweets` stream to `count` with `via`. Finally we connect the Flow to the previously +prepared Sink using `toMat`. -Remember those mysterious ``Mat`` type parameters on ``Source``, ``Flow`` and ``Sink``? +Remember those mysterious `Mat` type parameters on ``Source``, `Flow` and `Sink`? They represent the type of values these processing parts return when materialized. When you chain these together, -you can explicitly combine their materialized values: in our example we used the ``Keep.right`` predefined function, +you can explicitly combine their materialized values: in our example we used the `Keep.right` predefined function, which tells the implementation to only care about the materialized type of the stage currently appended to the right. -The materialized type of ``sumSink`` is ``CompletionStage`` and because of using ``Keep.right``, the resulting :class:`RunnableGraph` -has also a type parameter of ``CompletionStage``. +The materialized type of `sumSink` is `CompletionStage` and because of using `Keep.right`, the resulting `RunnableGraph` +has also a type parameter of `CompletionStage`. This step does *not* yet materialize the processing pipeline, it merely prepares the description of the Flow, which is now connected to a Sink, and therefore can -be ``run()``, as indicated by its type: ``RunnableGraph>``. Next we call ``run()`` which uses the :class:`ActorMaterializer` -to materialize and run the Flow. The value returned by calling ``run()`` on a ``RunnableGraph`` is of type ``T``. -In our case this type is ``CompletionStage`` which, when completed, will contain the total length of our tweets stream. +be `run()`, as indicated by its type: `RunnableGraph>`. Next we call `run()` which uses the `ActorMaterializer` +to materialize and run the Flow. The value returned by calling `run()` on a `RunnableGraph` is of type `T`. +In our case this type is `CompletionStage` which, when completed, will contain the total length of our tweets stream. In case of the stream failing, this future would complete with a Failure. -A :class:`RunnableGraph` may be reused +A `RunnableGraph` may be reused and materialized multiple times, because it is just the "blueprint" of the stream. This means that if we materialize a stream, for example one that consumes a live stream of tweets within a minute, the materialized values for those two materializations will be different, as illustrated by this example: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#tweets-runnable-flow-materialized-twice +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #tweets-runnable-flow-materialized-twice } Many elements in Akka Streams provide materialized values which can be used for obtaining either results of computation or -steering these elements which will be discussed in detail in :ref:`stream-materialization-java`. Summing up this section, now we know +steering these elements which will be discussed in detail in @ref:[Stream Materialization](stream-flows-and-basics.md#stream-materialization-java). Summing up this section, now we know what happens behind the scenes when we run this one-liner, which is equivalent to the multi line version above: -.. includecode:: ../code/jdocs/stream/TwitterStreamQuickstartDocTest.java#tweets-fold-count-oneline +@@snip [TwitterStreamQuickstartDocTest.java](../code/jdocs/stream/TwitterStreamQuickstartDocTest.java) { #tweets-fold-count-oneline } -.. note:: - ``runWith()`` is a convenience method that automatically ignores the materialized value of any other stages except - those appended by the ``runWith()`` itself. In the above example it translates to using ``Keep.right`` as the combiner - for materialized values. +@@@ note + +`runWith()` is a convenience method that automatically ignores the materialized value of any other stages except +those appended by the `runWith()` itself. In the above example it translates to using `Keep.right` as the combiner +for materialized values. + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-rate.md b/akka-docs/src/main/paradox/java/stream/stream-rate.md index ee674909cb..64d412246a 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-rate.md +++ b/akka-docs/src/main/paradox/java/stream/stream-rate.md @@ -1,47 +1,40 @@ -.. _stream-rate-java: - -############################# -Buffers and working with rate -############################# +# Buffers and working with rate When upstream and downstream rates differ, especially when the throughput has spikes, it can be useful to introduce buffers in a stream. In this chapter we cover how buffers are used in Akka Streams. - -.. _async-stream-buffers-java: - -Buffers for asynchronous stages -=============================== + +## Buffers for asynchronous stages In this section we will discuss internal buffers that are introduced as an optimization when using asynchronous stages. -To run a stage asynchronously it has to be marked explicitly as such using the ``.async()`` method. Being run +To run a stage asynchronously it has to be marked explicitly as such using the `.async()` method. Being run asynchronously means that a stage, after handing out an element to its downstream consumer is able to immediately process the next message. To demonstrate what we mean by this, let's take a look at the following example: -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#pipelining +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #pipelining } Running the above example, one of the possible outputs looks like this: -:: +``` +A: 1 +A: 2 +B: 1 +A: 3 +B: 2 +C: 1 +B: 3 +C: 2 +C: 3 +``` - A: 1 - A: 2 - B: 1 - A: 3 - B: 2 - C: 1 - B: 3 - C: 2 - C: 3 - -Note that the order is *not* ``A:1, B:1, C:1, A:2, B:2, C:2,`` which would correspond to the normal fused synchronous +Note that the order is *not* `A:1, B:1, C:1, A:2, B:2, C:2,` which would correspond to the normal fused synchronous execution model of flows where an element completely passes through the processing pipeline before the next element enters the flow. The next element is processed by an asynchronous stage as soon as it is emitted the previous one. While pipelining in general increases throughput, in practice there is a cost of passing an element through the asynchronous (and therefore thread crossing) boundary which is significant. To amortize this cost Akka Streams uses -a *windowed*, *batching* backpressure strategy internally. It is windowed because as opposed to a `Stop-And-Wait`_ +a *windowed*, *batching* backpressure strategy internally. It is windowed because as opposed to a [Stop-And-Wait](https://en.wikipedia.org/wiki/Stop-and-wait_ARQ) protocol multiple elements might be "in-flight" concurrently with requests for elements. It is also batching because a new element is not immediately requested once an element has been drained from the window-buffer but multiple elements are requested after multiple elements have been drained. This batching strategy reduces the communication cost of @@ -54,10 +47,7 @@ the throughput of the connected chain. There are tools in Akka Streams however t of a processing chain to be "detached" or to define the maximum throughput of the stream through external timing sources. These situations are exactly those where the internal batching buffering strategy suddenly becomes non-transparent. -.. _Stop-And-Wait: https://en.wikipedia.org/wiki/Stop-and-wait_ARQ - -Internal buffers and their effect ---------------------------------- +### Internal buffers and their effect As we have explained, for performance reasons Akka Streams introduces a buffer for every asynchronous processing stage. The purpose of these buffers is solely optimization, in fact the size of 1 would be the most natural choice if there @@ -65,37 +55,38 @@ would be no need for throughput improvements. Therefore it is recommended to kee and increase them only to a level suitable for the throughput requirements of the application. Default buffer sizes can be set through configuration: -:: +``` +akka.stream.materializer.max-input-buffer-size = 16 +``` - akka.stream.materializer.max-input-buffer-size = 16 +Alternatively they can be set by passing a `ActorMaterializerSettings` to the materializer: -Alternatively they can be set by passing a :class:`ActorMaterializerSettings` to the materializer: +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #materializer-buffer } -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#materializer-buffer +If the buffer size needs to be set for segments of a `Flow` only, it is possible by defining a separate +`Flow` with these attributes: -If the buffer size needs to be set for segments of a :class:`Flow` only, it is possible by defining a separate -:class:`Flow` with these attributes: - -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#section-buffer +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #section-buffer } Here is an example of a code that demonstrate some of the issues caused by internal buffers: -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#buffering-abstraction-leak +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #buffering-abstraction-leak } -Running the above example one would expect the number *3* to be printed in every 3 seconds (the ``conflateWithSeed`` -step here is configured so that it counts the number of elements received before the downstream ``ZipWith`` consumes +Running the above example one would expect the number *3* to be printed in every 3 seconds (the `conflateWithSeed` +step here is configured so that it counts the number of elements received before the downstream `ZipWith` consumes them). What is being printed is different though, we will see the number *1*. The reason for this is the internal -buffer which is by default 16 elements large, and prefetches elements before the ``ZipWith`` starts consuming them. -It is possible to fix this issue by changing the buffer size of ``ZipWith`` (or the whole graph) to 1. We will still see -a leading 1 though which is caused by an initial prefetch of the ``ZipWith`` element. +buffer which is by default 16 elements large, and prefetches elements before the `ZipWith` starts consuming them. +It is possible to fix this issue by changing the buffer size of `ZipWith` (or the whole graph) to 1. We will still see +a leading 1 though which is caused by an initial prefetch of the `ZipWith` element. -.. note:: - In general, when time or rate driven processing stages exhibit strange behavior, one of the first solutions to try - should be to decrease the input buffer of the affected elements to 1. +@@@ note +In general, when time or rate driven processing stages exhibit strange behavior, one of the first solutions to try +should be to decrease the input buffer of the affected elements to 1. -Buffers in Akka Streams -======================= +@@@ + +## Buffers in Akka Streams In this section we will discuss *explicit* user defined buffers that are part of the domain logic of the stream processing pipeline of an application. @@ -103,7 +94,7 @@ pipeline of an application. The example below will ensure that 1000 jobs (but not more) are dequeued from an external (imaginary) system and stored locally in memory - relieving the external system: -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#explicit-buffers-backpressure +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #explicit-buffers-backpressure } The next example will also queue up 1000 jobs locally, but if there are more jobs waiting in the imaginary external systems, it makes space for the new element by @@ -111,12 +102,12 @@ dropping one element from the *tail* of the buffer. Dropping from the tail is a it must be noted that this will drop the *youngest* waiting job. If some "fairness" is desired in the sense that we want to be nice to jobs that has been waiting for long, then this option can be useful. -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#explicit-buffers-droptail +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #explicit-buffers-droptail } Instead of dropping the youngest element from the tail of the buffer a new element can be dropped without enqueueing it to the buffer at all. -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#explicit-buffers-dropnew +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #explicit-buffers-dropnew } Here is another example with a queue of 1000 jobs, but it makes space for the new element by dropping one element from the *head* of the buffer. This is the *oldest* @@ -125,13 +116,13 @@ resent if not processed in a certain period. The oldest element will be retransmitted soon, (in fact a retransmitted duplicate might be already in the queue!) so it makes sense to drop it first. -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#explicit-buffers-drophead +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #explicit-buffers-drophead } Compared to the dropping strategies above, dropBuffer drops all the 1000 jobs it has enqueued once the buffer gets full. This aggressive strategy is useful when dropping jobs is preferred to delaying jobs. -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#explicit-buffers-dropbuffer +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #explicit-buffers-dropbuffer } If our imaginary external job provider is a client using our API, we might want to enforce that the client cannot have more than 1000 queued jobs @@ -139,49 +130,43 @@ otherwise we consider it flooding and terminate the connection. This is easily achievable by the error strategy which simply fails the stream once the buffer gets full. -.. includecode:: ../code/jdocs/stream/StreamBuffersRateDocTest.java#explicit-buffers-fail +@@snip [StreamBuffersRateDocTest.java](../code/jdocs/stream/StreamBuffersRateDocTest.java) { #explicit-buffers-fail } -Rate transformation -=================== +## Rate transformation -Understanding conflate ----------------------- +### Understanding conflate -When a fast producer can not be informed to slow down by backpressure or some other signal, ``conflate`` might be +When a fast producer can not be informed to slow down by backpressure or some other signal, `conflate` might be useful to combine elements from a producer until a demand signal comes from a consumer. Below is an example snippet that summarizes fast stream of elements to a standard deviation, mean and count of elements that have arrived while the stats have been calculated. -.. includecode:: ../code/jdocs/stream/RateTransformationDocTest.java#conflate-summarize +@@snip [RateTransformationDocTest.java](../code/jdocs/stream/RateTransformationDocTest.java) { #conflate-summarize } This example demonstrates that such flow's rate is decoupled. The element rate at the start of the flow can be much higher that the element rate at the end of the flow. -Another possible use of ``conflate`` is to not consider all elements for summary when producer starts getting too fast. -Example below demonstrates how ``conflate`` can be used to implement random drop of elements when consumer is not able +Another possible use of `conflate` is to not consider all elements for summary when producer starts getting too fast. +Example below demonstrates how `conflate` can be used to implement random drop of elements when consumer is not able to keep up with the producer. -.. includecode:: ../code/jdocs/stream/RateTransformationDocTest.java#conflate-sample +@@snip [RateTransformationDocTest.java](../code/jdocs/stream/RateTransformationDocTest.java) { #conflate-sample } -Understanding expand --------------------- +### Understanding expand Expand helps to deal with slow producers which are unable to keep up with the demand coming from consumers. Expand allows to extrapolate a value to be sent as an element to a consumer. -As a simple use of ``expand`` here is a flow that sends the same element to consumer when producer does not send any +As a simple use of `expand` here is a flow that sends the same element to consumer when producer does not send any new elements. -.. includecode:: ../code/jdocs/stream/RateTransformationDocTest.java#expand-last +@@snip [RateTransformationDocTest.java](../code/jdocs/stream/RateTransformationDocTest.java) { #expand-last } Expand also allows to keep some state between demand requests from the downstream. Leveraging this, here is a flow that tracks and reports a drift between fast consumer and slow producer. -.. includecode:: ../code/jdocs/stream/RateTransformationDocTest.java#expand-drift - -Note that all of the elements coming from upstream will go through ``expand`` at least once. This means that the -output of this flow is going to report a drift of zero if producer is fast enough, or a larger drift otherwise. - - +@@snip [RateTransformationDocTest.java](../code/jdocs/stream/RateTransformationDocTest.java) { #expand-drift } +Note that all of the elements coming from upstream will go through `expand` at least once. This means that the +output of this flow is going to report a drift of zero if producer is fast enough, or a larger drift otherwise. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/stream/stream-testkit.md b/akka-docs/src/main/paradox/java/stream/stream-testkit.md index 37db70dd90..6298aa9fbb 100644 --- a/akka-docs/src/main/paradox/java/stream/stream-testkit.md +++ b/akka-docs/src/main/paradox/java/stream/stream-testkit.md @@ -1,100 +1,94 @@ -.. _stream-testkit-java: - -############### -Testing streams -############### +# Testing streams Verifying behaviour of Akka Stream sources, flows and sinks can be done using various code patterns and libraries. Here we will discuss testing these elements using: -- simple sources, sinks and flows; -- sources and sinks in combination with :class:`TestProbe` from the :mod:`akka-testkit` module; -- sources and sinks specifically crafted for writing tests from the :mod:`akka-stream-testkit` module. + * simple sources, sinks and flows; + * sources and sinks in combination with `TestProbe` from the `akka-testkit` module; + * sources and sinks specifically crafted for writing tests from the `akka-stream-testkit` module. It is important to keep your data processing pipeline as separate sources, flows and sinks. This makes them easily testable by wiring them up to other -sources or sinks, or some test harnesses that :mod:`akka-testkit` or -:mod:`akka-stream-testkit` provide. +sources or sinks, or some test harnesses that `akka-testkit` or +`akka-stream-testkit` provide. -Built in sources, sinks and combinators -======================================= +## Built in sources, sinks and combinators Testing a custom sink can be as simple as attaching a source that emits elements from a predefined collection, running a constructed test flow and asserting on the results that sink produced. Here is an example of a test for a sink: -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#strict-collection +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #strict-collection } The same strategy can be applied for sources as well. In the next example we have a source that produces an infinite stream of elements. Such source can be tested by asserting that first arbitrary number of elements hold some -condition. Here the ``take`` combinator and ``Sink.seq`` are very useful. +condition. Here the `take` combinator and `Sink.seq` are very useful. -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#grouped-infinite +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #grouped-infinite } When testing a flow we need to attach a source and a sink. As both stream ends are under our control, we can choose sources that tests various edge cases of the flow and sinks that ease assertions. -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#folded-stream +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #folded-stream } -TestKit -======= +## TestKit Akka Stream offers integration with Actors out of the box. This support can be -used for writing stream tests that use familiar :class:`TestProbe` from the -:mod:`akka-testkit` API. +used for writing stream tests that use familiar `TestProbe` from the +`akka-testkit` API. One of the more straightforward tests would be to materialize stream to a -:class:`CompletionStage` and then use ``PatternsCS.pipe`` pattern to pipe the result of that future +`CompletionStage` and then use `PatternsCS.pipe` pattern to pipe the result of that future to the probe. -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#pipeto-testprobe +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #pipeto-testprobe } -Instead of materializing to a future, we can use a :class:`Sink.actorRef` that -sends all incoming elements to the given :class:`ActorRef`. Now we can use -assertion methods on :class:`TestProbe` and expect elements one by one as they +Instead of materializing to a future, we can use a `Sink.actorRef` that +sends all incoming elements to the given `ActorRef`. Now we can use +assertion methods on `TestProbe` and expect elements one by one as they arrive. We can also assert stream completion by expecting for -``onCompleteMessage`` which was given to :class:`Sink.actorRef`. +`onCompleteMessage` which was given to `Sink.actorRef`. -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#sink-actorref +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #sink-actorref } -Similarly to :class:`Sink.actorRef` that provides control over received -elements, we can use :class:`Source.actorRef` and have full control over +Similarly to `Sink.actorRef` that provides control over received +elements, we can use `Source.actorRef` and have full control over elements to be sent. -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#source-actorref +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #source-actorref } -Streams TestKit -=============== +## Streams TestKit You may have noticed various code patterns that emerge when testing stream -pipelines. Akka Stream has a separate :mod:`akka-stream-testkit` module that +pipelines. Akka Stream has a separate `akka-stream-testkit` module that provides tools specifically for writing stream tests. This module comes with -two main components that are :class:`TestSource` and :class:`TestSink` which +two main components that are `TestSource` and `TestSink` which provide sources and sinks that materialize to probes that allow fluent API. -.. note:: +@@@ note - Be sure to add the module :mod:`akka-stream-testkit` to your dependencies. +Be sure to add the module `akka-stream-testkit` to your dependencies. -A sink returned by ``TestSink.probe`` allows manual control over demand and +@@@ + +A sink returned by `TestSink.probe` allows manual control over demand and assertions over elements coming downstream. -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#test-sink-probe +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #test-sink-probe } -A source returned by ``TestSource.probe`` can be used for asserting demand or +A source returned by `TestSource.probe` can be used for asserting demand or controlling when stream is completed or ended with an error. -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#test-source-probe +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #test-source-probe } You can also inject exceptions and test sink behaviour on error conditions. -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#injecting-failure +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #injecting-failure } Test source and sink can be used together in combination when testing flows. -.. includecode:: ../code/jdocs/stream/StreamTestKitDocTest.java#test-source-and-sink - +@@snip [StreamTestKitDocTest.java](../code/jdocs/stream/StreamTestKitDocTest.java) { #test-source-and-sink } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/testing.md b/akka-docs/src/main/paradox/java/testing.md index bf155fe623..6173d00128 100644 --- a/akka-docs/src/main/paradox/java/testing.md +++ b/akka-docs/src/main/paradox/java/testing.md @@ -1,25 +1,22 @@ -.. _akka-testkit-java: - -############################## -Testing Actor Systems -############################## +# Testing Actor Systems As with any piece of software, automated tests are a very important part of the development cycle. The actor model presents a different view on how units of code are delimited and how they interact, which has an influence on how to perform tests. -Akka comes with a dedicated module :mod:`akka-testkit` for supporting tests at +Akka comes with a dedicated module `akka-testkit` for supporting tests at different levels, which fall into two clearly distinct categories: - - Testing isolated pieces of code without involving the actor model, meaning - without multiple threads; this implies completely deterministic behavior - concerning the ordering of events and no concurrency concerns and will be - called **Unit Testing** in the following. - - Testing (multiple) encapsulated actors including multi-threaded scheduling; - this implies non-deterministic order of events but shielding from - concurrency concerns by the actor model and will be called **Integration - Testing** in the following. +> + * Testing isolated pieces of code without involving the actor model, meaning +without multiple threads; this implies completely deterministic behavior +concerning the ordering of events and no concurrency concerns and will be +called **Unit Testing** in the following. + * Testing (multiple) encapsulated actors including multi-threaded scheduling; +this implies non-deterministic order of events but shielding from +concurrency concerns by the actor model and will be called **Integration +Testing** in the following. There are of course variations on the granularity of tests in both categories, where unit testing reaches down to white-box tests and integration testing can @@ -27,120 +24,124 @@ encompass functional tests of complete actor networks. The important distinction lies in whether concurrency concerns are part of the test or not. The tools offered are described in detail in the following sections. -.. note:: +@@@ note - Be sure to add the module :mod:`akka-testkit` to your dependencies. +Be sure to add the module `akka-testkit` to your dependencies. -Synchronous Unit Testing with :class:`TestActorRef` -=================================================== +@@@ -Testing the business logic inside :class:`Actor` classes can be divided into +## Synchronous Unit Testing with `TestActorRef` + +Testing the business logic inside `Actor` classes can be divided into two parts: first, each atomic operation must work in isolation, then sequences of incoming events must be processed correctly, even in the presence of some possible variability in the ordering of events. The former is the primary use case for single-threaded unit testing, while the latter can only be verified in integration tests. -Normally, the :class:`ActorRef` shields the underlying :class:`Actor` instance +Normally, the `ActorRef` shields the underlying `Actor` instance from the outside, the only communications channel is the actor's mailbox. This restriction is an impediment to unit testing, which led to the inception of the -:class:`TestActorRef`. This special type of reference is designed specifically +`TestActorRef`. This special type of reference is designed specifically for test purposes and allows access to the actor in two ways: either by obtaining a reference to the underlying actor instance, or by invoking or -querying the actor's behaviour (:meth:`receive`). Each one warrants its own +querying the actor's behaviour (`receive`). Each one warrants its own section below. -.. note:: - It is highly recommended to stick to traditional behavioural testing (using messaging - to ask the Actor to reply with the state you want to run assertions against), - instead of using ``TestActorRef`` whenever possible. +@@@ note -.. warning:: - Due to the synchronous nature of ``TestActorRef`` it will **not** work with some support - traits that Akka provides as they require asynchronous behaviours to function properly. - Examples of traits that do not mix well with test actor refs are :ref:`PersistentActor ` - and :ref:`AtLeastOnceDelivery ` provided by :ref:`Akka Persistence `. +It is highly recommended to stick to traditional behavioural testing (using messaging +to ask the Actor to reply with the state you want to run assertions against), +instead of using `TestActorRef` whenever possible. -Obtaining a Reference to an :class:`Actor` ------------------------------------------- +@@@ -Having access to the actual :class:`Actor` object allows application of all +@@@ warning + +Due to the synchronous nature of `TestActorRef` it will **not** work with some support +traits that Akka provides as they require asynchronous behaviours to function properly. +Examples of traits that do not mix well with test actor refs are @ref:[PersistentActor](persistence.md#event-sourcing-java) +and @ref:[AtLeastOnceDelivery](persistence.md#at-least-once-delivery-java) provided by @ref:[Akka Persistence](persistence.md). + +@@@ + +### Obtaining a Reference to an `Actor` + +Having access to the actual `Actor` object allows application of all traditional unit testing techniques on the contained methods. Obtaining a reference is done like this: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-actor-ref +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-actor-ref } -Since :class:`TestActorRef` is generic in the actor type it returns the +Since `TestActorRef` is generic in the actor type it returns the underlying actor with its proper static type. From this point on you may bring any unit testing tool to bear on your actor as usual. -Testing the Actor's Behavior ----------------------------- +### Testing the Actor's Behavior When the dispatcher invokes the processing behavior of an actor on a message, -it actually calls :meth:`apply` on the current behavior registered for the -actor. This starts out with the return value of the declared :meth:`receive` -method, but it may also be changed using :meth:`become` and :meth:`unbecome` in +it actually calls `apply` on the current behavior registered for the +actor. This starts out with the return value of the declared `receive` +method, but it may also be changed using `become` and `unbecome` in response to external messages. All of this contributes to the overall actor -behavior and it does not lend itself to easy testing on the :class:`Actor` -itself. Therefore the :class:`TestActorRef` offers a different mode of -operation to complement the :class:`Actor` testing: it supports all operations -also valid on normal :class:`ActorRef`. Messages sent to the actor are +behavior and it does not lend itself to easy testing on the `Actor` +itself. Therefore the `TestActorRef` offers a different mode of +operation to complement the `Actor` testing: it supports all operations +also valid on normal `ActorRef`. Messages sent to the actor are processed synchronously on the current thread and answers may be sent back as -usual. This trick is made possible by the :class:`CallingThreadDispatcher` -described below (see `CallingThreadDispatcher`_); this dispatcher is set -implicitly for any actor instantiated into a :class:`TestActorRef`. +usual. This trick is made possible by the `CallingThreadDispatcher` +described below (see [CallingThreadDispatcher](#callingthreaddispatcher)); this dispatcher is set +implicitly for any actor instantiated into a `TestActorRef`. -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-behavior +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-behavior } -As the :class:`TestActorRef` is a subclass of :class:`LocalActorRef` with a few +As the `TestActorRef` is a subclass of `LocalActorRef` with a few special extras, also aspects like supervision and restarting work properly, but beware that execution is only strictly synchronous as long as all actors -involved use the :class:`CallingThreadDispatcher`. As soon as you add elements +involved use the `CallingThreadDispatcher`. As soon as you add elements which include more sophisticated scheduling you leave the realm of unit testing as you then need to think about asynchronicity again (in most cases the problem will be to wait until the desired effect had a chance to happen). One more special aspect which is overridden for single-threaded tests is the -:meth:`receiveTimeout`, as including that would entail asynchronous queuing of -:obj:`ReceiveTimeout` messages, violating the synchronous contract. +`receiveTimeout`, as including that would entail asynchronous queuing of +`ReceiveTimeout` messages, violating the synchronous contract. -.. note:: +@@@ note - To summarize: :class:`TestActorRef` overwrites two fields: it sets the - dispatcher to :obj:`CallingThreadDispatcher.global` and it sets the - :obj:`receiveTimeout` to None. +To summarize: `TestActorRef` overwrites two fields: it sets the +dispatcher to `CallingThreadDispatcher.global` and it sets the +`receiveTimeout` to None. -The Way In-Between: Expecting Exceptions ----------------------------------------- +@@@ + +### The Way In-Between: Expecting Exceptions If you want to test the actor behavior, including hotswapping, but without -involving a dispatcher and without having the :class:`TestActorRef` swallow +involving a dispatcher and without having the `TestActorRef` swallow any thrown exceptions, then there is another mode available for you: just use -the :meth:`receive` method on :class:`TestActorRef`, which will be forwarded to the +the `receive` method on `TestActorRef`, which will be forwarded to the underlying actor: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-expecting-exceptions +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-expecting-exceptions } -Use Cases ---------- +### Use Cases -You may of course mix and match both modi operandi of :class:`TestActorRef` as +You may of course mix and match both modi operandi of `TestActorRef` as suits your test needs: - - one common use case is setting up the actor into a specific internal state - before sending the test message - - another is to verify correct internal state transitions after having sent - the test message +> + * one common use case is setting up the actor into a specific internal state +before sending the test message + * another is to verify correct internal state transitions after having sent +the test message Feel free to experiment with the possibilities, and if you find useful patterns, don't hesitate to let the Akka forums know about them! Who knows, common operations might even be worked into nice DSLs. -.. _async-integration-testing-java: - -Asynchronous Integration Testing with :class:`TestKit` -====================================================== + +## Asynchronous Integration Testing with `TestKit` When you are reasonably sure that your actor's business logic is correct, the next step is verifying that it works correctly within its intended environment. @@ -153,168 +154,158 @@ test with a network of actors, apply stimuli at varying injection points and arrange results to be sent from different emission points, but the basic principle stays the same in that a single procedure drives the test. -The :class:`TestKit` class contains a collection of tools which makes this +The `TestKit` class contains a collection of tools which makes this common task easy. -.. includecode:: code/jdocs/testkit/TestKitSampleTest.java#fullsample +@@snip [TestKitSampleTest.java](code/jdocs/testkit/TestKitSampleTest.java) { #fullsample } -The :class:`TestKit` contains an actor named :obj:`testActor` which is the -entry point for messages to be examined with the various ``expectMsg...`` +The `TestKit` contains an actor named `testActor` which is the +entry point for messages to be examined with the various `expectMsg...` assertions detailed below. The test actor’s reference is obtained using the -:meth:`getRef()` method as demonstrated above. The :obj:`testActor` may also +`getRef()` method as demonstrated above. The `testActor` may also be passed to other actors as usual, usually subscribing it as notification listener. There is a whole set of examination methods, e.g. receiving all consecutive messages matching certain criteria, receiving a whole sequence of fixed messages or classes, receiving nothing for some time, etc. The ActorSystem passed in to the constructor of TestKit is accessible via the -:meth:`getSystem()` method. +`getSystem()` method. -.. note:: +@@@ note - Remember to shut down the actor system after the test is finished (also in - case of failure) so that all actors—including the test actor—are stopped. +Remember to shut down the actor system after the test is finished (also in +case of failure) so that all actors—including the test actor—are stopped. -Built-In Assertions -------------------- +@@@ -The above mentioned :meth:`expectMsgEquals` is not the only method for +### Built-In Assertions + +The above mentioned `expectMsgEquals` is not the only method for formulating assertions concerning received messages, the full set is this: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-expect +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-expect } In these examples, the maximum durations you will find mentioned below are left out, in which case they use the default value from configuration item -``akka.test.single-expect-default`` which itself defaults to 3 seconds (or they -obey the innermost enclosing :class:`Within` as detailed :ref:`below -`). The full signatures are: +`akka.test.single-expect-default` which itself defaults to 3 seconds (or they +obey the innermost enclosing `Within` as detailed [below](#testkit-within)). The full signatures are: - * :meth:`public  T expectMsgEquals(FiniteDuration max, T msg)` - - The given message object must be received within the specified time; the - object will be returned. - - * :meth:`public  T expectMsgPF(Duration max, String hint, Function f)` - - Within the given time period, a message must be received and the given - function must be defined for that message; the result from applying - the function to the received message is returned. - - * :meth:`public Object expectMsgAnyOf(Duration max, Object... msg)` - - An object must be received within the given time, and it must be equal - (compared with ``equals()``) to at least one of the passed reference - objects; the received object will be returned. - - * :meth:`public List expectMsgAllOf(FiniteDuration max, Object... msg)` - - A number of objects matching the size of the supplied object array must be - received within the given time, and for each of the given objects there - must exist at least one among the received ones which equals it (compared - with ``equals()``). The full sequence of received objects is returned in - the order received. - - * :meth:`public  T expectMsgClass(FiniteDuration max, Class c)` - - An object which is an instance of the given :class:`Class` must be received - within the allotted time frame; the object will be returned. Note that this - does a conformance check, if you need the class to be equal you need to - verify that afterwards. - - * :meth:`public  T expectMsgAnyClassOf(FiniteDuration max, Class... c)` - - An object must be received within the given time, and it must be an - instance of at least one of the supplied :class:`Class` objects; the - received object will be returned. Note that this does a conformance check, - if you need the class to be equal you need to verify that afterwards. - - .. note:: - - Because of a limitation in Java’s type system it may be necessary to add - ``@SuppressWarnings("unchecked")`` when using this method. - - * :meth:`public void expectNoMsg(FiniteDuration max)` - - No message must be received within the given time. This also fails if a - message has been received before calling this method which has not been - removed from the queue using one of the other methods. - - * :meth:`List receiveN(int n, FiniteDuration max)` - - ``n`` messages must be received within the given time; the received - messages are returned. +> + * + `public  T expectMsgEquals(FiniteDuration max, T msg)` + The given message object must be received within the specified time; the +object will be returned. + * + `public  T expectMsgPF(Duration max, String hint, Function f)` + Within the given time period, a message must be received and the given +function must be defined for that message; the result from applying +the function to the received message is returned. + * + `public Object expectMsgAnyOf(Duration max, Object... msg)` + An object must be received within the given time, and it must be equal +(compared with `equals()`) to at least one of the passed reference +objects; the received object will be returned. + * + `public List expectMsgAllOf(FiniteDuration max, Object... msg)` + A number of objects matching the size of the supplied object array must be +received within the given time, and for each of the given objects there +must exist at least one among the received ones which equals it (compared +with `equals()`). The full sequence of received objects is returned in +the order received. + * + `public  T expectMsgClass(FiniteDuration max, Class c)` + An object which is an instance of the given `Class` must be received +within the allotted time frame; the object will be returned. Note that this +does a conformance check, if you need the class to be equal you need to +verify that afterwards. + * + `public  T expectMsgAnyClassOf(FiniteDuration max, Class... c)` + An object must be received within the given time, and it must be an +instance of at least one of the supplied `Class` objects; the +received object will be returned. Note that this does a conformance check, +if you need the class to be equal you need to verify that afterwards. + @@@ note + + Because of a limitation in Java’s type system it may be necessary to add +`@SuppressWarnings("unchecked")` when using this method. + + @@@ + * + `public void expectNoMsg(FiniteDuration max)` + No message must be received within the given time. This also fails if a +message has been received before calling this method which has not been +removed from the queue using one of the other methods. + * + `List receiveN(int n, FiniteDuration max)` + `n` messages must be received within the given time; the received +messages are returned. In addition to message reception assertions there are also methods which help with message flows: - * :meth:`public List receiveWhile(Duration max, Duration idle, Int messages, Function f)` - - .. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-receivewhile-full - - Collect messages as long as - +> + * + `public List receiveWhile(Duration max, Duration idle, Int messages, Function f)` + @@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-receivewhile-full } + Collect messages as long as * they are matching the given function * the given time interval is not used up * the next message is received within the idle timeout * the number of messages has not yet reached the maximum - - All collected messages are returned. - - * :meth:`public void awaitCond(Duration max, Duration interval, Supplier p)` - - .. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-awaitCond - - Poll the given condition every :obj:`interval` until it returns ``true`` or - the :obj:`max` duration is used up. - - * :meth:`public void awaitAssert(Duration max, Duration interval, Supplier a)` - - .. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-awaitAssert - - Poll the given assert function every :obj:`interval` until it does not throw - an exception or the :obj:`max` duration is used up. If the timeout expires the - last exception is thrown. + All collected messages are returned. + * + `public void awaitCond(Duration max, Duration interval, Supplier p)` + @@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-awaitCond } + Poll the given condition every `interval` until it returns `true` or +the `max` duration is used up. + * + `public void awaitAssert(Duration max, Duration interval, Supplier a)` + @@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-awaitAssert } + Poll the given assert function every `interval` until it does not throw +an exception or the `max` duration is used up. If the timeout expires the +last exception is thrown. There are also cases where not all messages sent to the test kit are actually relevant to the test, but removing them would mean altering the actors under test. For this purpose it is possible to ignore certain messages: - * :meth:`public void ignoreMsg(Function f)` +> + * + `public void ignoreMsg(Function f)` + `public void ignoreMsg()` + @@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-ignoreMsg } - :meth:`public void ignoreMsg()` - - .. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-ignoreMsg - -Expecting Log Messages ----------------------- +### Expecting Log Messages Since an integration test does not allow to the internal processing of the participating actors, verifying expected exceptions cannot be done directly. Instead, use the logging system for this purpose: replacing the normal event -handler with the :class:`TestEventListener` and using an :class:`EventFilter` +handler with the `TestEventListener` and using an `EventFilter` allows assertions on log messages, including those which are generated by exceptions: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-event-filter +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-event-filter } -If a number of occurrences is specific—as demonstrated above—then ``intercept()`` +If a number of occurrences is specific—as demonstrated above—then `intercept()` will block until that number of matching messages have been received or the -timeout configured in ``akka.test.filter-leeway`` is used up (time starts +timeout configured in `akka.test.filter-leeway` is used up (time starts counting after the passed-in block of code returns). In case of a timeout the test fails. -.. note:: +@@@ note - Be sure to exchange the default logger with the - :class:`TestEventListener` in your ``application.conf`` to enable this - function:: +Be sure to exchange the default logger with the +`TestEventListener` in your `application.conf` to enable this +function: - akka.loggers = [akka.testkit.TestEventListener] +``` +akka.loggers = [akka.testkit.TestEventListener] +``` -.. _TestKit.within: +@@@ -Timing Assertions ------------------ + +### Timing Assertions Another important part of functional testing concerns timing: certain events must not happen immediately (like a timer), others need to happen before a @@ -323,230 +314,217 @@ the positive or negative result must be obtained. Lower time limits need to be checked external to the examination, which is facilitated by a new construct for managing time constraints: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-within +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-within } -The block in :meth:`within` must complete after a :ref:`Duration` which -is between :obj:`min` and :obj:`max`, where the former defaults to zero. The -deadline calculated by adding the :obj:`max` parameter to the block's start +The block in `within` must complete after a @ref:[Duration](../common/duration.md) which +is between `min` and `max`, where the former defaults to zero. The +deadline calculated by adding the `max` parameter to the block's start time is implicitly available within the block to all examination methods, if you do not specify it, it is inherited from the innermost enclosing -:meth:`within` block. +`within` block. It should be noted that if the last message-receiving assertion of the block is -:meth:`expectNoMsg` or :meth:`receiveWhile`, the final check of the -:meth:`within` is skipped in order to avoid false positives due to wake-up +`expectNoMsg` or `receiveWhile`, the final check of the +`within` is skipped in order to avoid false positives due to wake-up latencies. This means that while individual contained assertions still use the maximum time bound, the overall block may take arbitrarily longer in this case. -.. note:: +@@@ note - All times are measured using ``System.nanoTime``, meaning that they describe - wall time, not CPU time or system time. +All times are measured using `System.nanoTime`, meaning that they describe +wall time, not CPU time or system time. -Accounting for Slow Test Systems -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@@ + +#### Accounting for Slow Test Systems The tight timeouts you use during testing on your lightning-fast notebook will invariably lead to spurious test failures on the heavily loaded Jenkins server (or similar). To account for this situation, all maximum durations are -internally scaled by a factor taken from the :ref:`configuration`, -``akka.test.timefactor``, which defaults to 1. +internally scaled by a factor taken from the [Configuration](), +`akka.test.timefactor`, which defaults to 1. -You can scale other durations with the same factor by using ``dilated`` method -in :class:`TestKit`. +You can scale other durations with the same factor by using `dilated` method +in `TestKit`. -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#duration-dilation +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #duration-dilation } -Using Multiple Probe Actors ---------------------------- +### Using Multiple Probe Actors When the actors under test are supposed to send various messages to different destinations, it may be difficult distinguishing the message streams arriving -at the :obj:`testActor` when using the :class:`TestKit` as shown until now. +at the `testActor` when using the `TestKit` as shown until now. Another approach is to use it for creation of simple probe actors to be inserted in the message flows. The functionality is best explained using a small example: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-probe +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-probe } This simple test verifies an equally simple Forwarder actor by injecting a probe as the forwarder’s target. Another example would be two actors A and B which collaborate by A sending messages to B. In order to verify this message -flow, a :class:`TestProbe` could be inserted as target of A, using the +flow, a `TestProbe` could be inserted as target of A, using the forwarding capabilities or auto-pilot described below to include a real B in the test setup. If you have many test probes, you can name them to get meaningful actor names in test logs and assertions: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-probe-with-custom-name +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-probe-with-custom-name } Probes may also be equipped with custom assertions to make your test code even more concise and clear: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java - :include: test-special-probe +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-special-probe } You have complete flexibility here in mixing and matching the -:class:`TestKit` facilities with your own checks and choosing an intuitive +`TestKit` facilities with your own checks and choosing an intuitive name for it. In real life your code will probably be a bit more complicated than the example given above; just use the power! -.. warning:: +@@@ warning - Any message send from a ``TestProbe`` to another actor which runs on the - CallingThreadDispatcher runs the risk of dead-lock, if that other actor might - also send to this probe. The implementation of :meth:`TestProbe.watch` and - :meth:`TestProbe.unwatch` will also send a message to the watchee, which - means that it is dangerous to try watching e.g. :class:`TestActorRef` from a - :meth:`TestProbe`. +Any message send from a `TestProbe` to another actor which runs on the +CallingThreadDispatcher runs the risk of dead-lock, if that other actor might +also send to this probe. The implementation of `TestProbe.watch` and +`TestProbe.unwatch` will also send a message to the watchee, which +means that it is dangerous to try watching e.g. `TestActorRef` from a +`TestProbe`. -Watching Other Actors from Probes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@@ -A :class:`TestKit` can register itself for DeathWatch of any other actor: +#### Watching Other Actors from Probes -.. includecode:: code/jdocs/testkit/TestKitDocTest.java - :include: test-probe-watch +A `TestKit` can register itself for DeathWatch of any other actor: -Replying to Messages Received by Probes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-probe-watch } + +#### Replying to Messages Received by Probes The probe stores the sender of the last dequeued message (i.e. after its -``expectMsg*`` reception), which may be retrieved using the -:meth:`getLastSender()` method. This information can also implicitly be used +`expectMsg*` reception), which may be retrieved using the +`getLastSender()` method. This information can also implicitly be used for having the probe reply to the last received message: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-probe-reply +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-probe-reply } -Forwarding Messages Received by Probes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Forwarding Messages Received by Probes -The probe can also forward a received message (i.e. after its ``expectMsg*`` +The probe can also forward a received message (i.e. after its `expectMsg*` reception), retaining the original sender: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-probe-forward +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-probe-forward } -Auto-Pilot -^^^^^^^^^^ +#### Auto-Pilot Receiving messages in a queue for later inspection is nice, but in order to keep a test running and verify traces later you can also install an -:class:`AutoPilot` in the participating test probes (actually in any -:class:`TestKit`) which is invoked before enqueueing to the inspection queue. -This code can be used to forward messages, e.g. in a chain ``A --> Probe --> -B``, as long as a certain protocol is obeyed. +`AutoPilot` in the participating test probes (actually in any +`TestKit`) which is invoked before enqueueing to the inspection queue. +This code can be used to forward messages, e.g. in a chain `A --> Probe --> +B`, as long as a certain protocol is obeyed. -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-auto-pilot +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-auto-pilot } -The :meth:`run` method must return the auto-pilot for the next message, wrapped -in an :class:`Option`; setting it to :obj:`None` terminates the auto-pilot. +The `run` method must return the auto-pilot for the next message, wrapped +in an `Option`; setting it to `None` terminates the auto-pilot. -Caution about Timing Assertions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Caution about Timing Assertions -The behavior of :meth:`within` blocks when using test probes might be perceived +The behavior of `within` blocks when using test probes might be perceived as counter-intuitive: you need to remember that the nicely scoped deadline as -described :ref:`above ` is local to each probe. Hence, probes +described [above](#testkit-within) is local to each probe. Hence, probes do not react to each other's deadlines or to the deadline set in an enclosing -:class:`TestKit` instance: +`TestKit` instance: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#test-within-probe +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #test-within-probe } -Here, the ``expectMsgEquals`` call will use the default timeout. +Here, the `expectMsgEquals` call will use the default timeout. -Testing parent-child relationships ----------------------------------- +### Testing parent-child relationships The parent of an actor is always the actor that created it. At times this leads to a coupling between the two that may not be straightforward to test. There are several approaches to improve testability of a child actor that needs to refer to its parent: -1. when creating a child, pass an explicit reference to its parent -2. create the child with a ``TestProbe`` as parent -3. create a fabricated parent when testing + 1. when creating a child, pass an explicit reference to its parent + 2. create the child with a `TestProbe` as parent + 3. create a fabricated parent when testing Conversely, a parent's binding to its child can be lessened as follows: -4. when creating a parent, tell the parent how to create its child + 4. when creating a parent, tell the parent how to create its child For example, the structure of the code you want to test may follow this pattern: -.. includecode:: code/jdocs/testkit/ParentChildTest.java#test-example +@@snip [ParentChildTest.java](code/jdocs/testkit/ParentChildTest.java) { #test-example } -Introduce child to its parent -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Introduce child to its parent -The first option is to avoid use of the :meth:`context.parent` function and create +The first option is to avoid use of the `context.parent` function and create a child with a custom parent by passing an explicit reference to its parent instead. -.. includecode:: code/jdocs/testkit/ParentChildTest.java#test-dependentchild +@@snip [ParentChildTest.java](code/jdocs/testkit/ParentChildTest.java) { #test-dependentchild } -Create the child using TestKit -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Create the child using TestKit -The ``TestKit`` class can in fact create actors that will run with the test probe as parent. -This will cause any messages the child actor sends to `getContext().getParent()` to +The `TestKit` class can in fact create actors that will run with the test probe as parent. +This will cause any messages the child actor sends to *getContext().getParent()* to end up in the test probe. -.. includecode:: code/jdocs/testkit/ParentChildTest.java#test-TestProbe-parent +@@snip [ParentChildTest.java](code/jdocs/testkit/ParentChildTest.java) { #test-TestProbe-parent } -Using a fabricated parent -^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Using a fabricated parent If you prefer to avoid modifying the child constructor you can create a fabricated parent in your test. This, however, does not enable you to test the parent actor in isolation. -.. includecode:: code/jdocs/testkit/ParentChildTest.java#test-fabricated-parent-creator -.. includecode:: code/jdocs/testkit/ParentChildTest.java#test-fabricated-parent +@@snip [ParentChildTest.java](code/jdocs/testkit/ParentChildTest.java) { #test-fabricated-parent-creator } -Externalize child making from the parent -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@snip [ParentChildTest.java](code/jdocs/testkit/ParentChildTest.java) { #test-fabricated-parent } + +#### Externalize child making from the parent Alternatively, you can tell the parent how to create its child. There are two ways -to do this: by giving it a :class:`Props` object or by giving it a function which takes care of creating the child actor: +to do this: by giving it a `Props` object or by giving it a function which takes care of creating the child actor: -.. includecode:: code/jdocs/testkit/ParentChildTest.java#test-dependentparent -.. includecode:: code/jdocs/testkit/ParentChildTest.java#test-dependentparent-generic +@@snip [ParentChildTest.java](code/jdocs/testkit/ParentChildTest.java) { #test-dependentparent } -Creating the :class:`Actor` is straightforward and the function may look like this in your test code: +@@snip [ParentChildTest.java](code/jdocs/testkit/ParentChildTest.java) { #test-dependentparent-generic } -.. includecode:: code/jdocs/testkit/ParentChildTest.java#child-maker-test +Creating the `Actor` is straightforward and the function may look like this in your test code: + +@@snip [ParentChildTest.java](code/jdocs/testkit/ParentChildTest.java) { #child-maker-test } And like this in your application code: -.. includecode:: code/jdocs/testkit/ParentChildTest.java#child-maker-prod - +@@snip [ParentChildTest.java](code/jdocs/testkit/ParentChildTest.java) { #child-maker-prod } Which of these methods is the best depends on what is most important to test. The most generic option is to create the parent actor by passing it a function that is responsible for the Actor creation, but using TestProbe or having a fabricated parent is often sufficient. -.. _Java-CallingThreadDispatcher: + +## CallingThreadDispatcher -CallingThreadDispatcher -======================= - -The :class:`CallingThreadDispatcher` serves good purposes in unit testing, as +The `CallingThreadDispatcher` serves good purposes in unit testing, as described above, but originally it was conceived in order to allow contiguous stack traces to be generated in case of an error. As this special dispatcher runs everything which would normally be queued directly on the current thread, the full history of a message's processing chain is recorded on the call stack, so long as all intervening actors run on this dispatcher. -How to use it -------------- +### How to use it Just set the dispatcher as you normally would: -.. includecode:: code/jdocs/testkit/TestKitDocTest.java#calling-thread-dispatcher +@@snip [TestKitDocTest.java](code/jdocs/testkit/TestKitDocTest.java) { #calling-thread-dispatcher } -How it works ------------- +### How it works -When receiving an invocation, the :class:`CallingThreadDispatcher` checks +When receiving an invocation, the `CallingThreadDispatcher` checks whether the receiving actor is already active on the current thread. The simplest example for this situation is an actor which sends a message to itself. In this case, processing cannot continue immediately as that would @@ -557,7 +535,7 @@ previous work. In the other case, the invocation is simply processed immediately on the current thread. Futures scheduled via this dispatcher are also executed immediately. -This scheme makes the :class:`CallingThreadDispatcher` work like a general +This scheme makes the `CallingThreadDispatcher` work like a general purpose dispatcher for any actors which never block on external events. In the presence of multiple threads it may happen that two invocations of an @@ -571,93 +549,95 @@ concurrency. The other remaining difficulty is correct handling of suspend and resume: when an actor is suspended, subsequent invocations will be queued in thread-local queues (the same ones used for queuing in the normal case). The call to -:meth:`resume`, however, is done by one specific thread, and all other threads +`resume`, however, is done by one specific thread, and all other threads in the system will probably not be executing this specific actor, which leads to the problem that the thread-local queues cannot be emptied by their native -threads. Hence, the thread calling :meth:`resume` will collect all currently +threads. Hence, the thread calling `resume` will collect all currently queued invocations from all threads into its own queue and process them. -Limitations ------------ +### Limitations -.. warning:: +@@@ warning - In case the CallingThreadDispatcher is used for top-level actors, but - without going through TestActorRef, then there is a time window during which - the actor is awaiting construction by the user guardian actor. Sending - messages to the actor during this time period will result in them being - enqueued and then executed on the guardian’s thread instead of the caller’s - thread. To avoid this, use TestActorRef. +In case the CallingThreadDispatcher is used for top-level actors, but +without going through TestActorRef, then there is a time window during which +the actor is awaiting construction by the user guardian actor. Sending +messages to the actor during this time period will result in them being +enqueued and then executed on the guardian’s thread instead of the caller’s +thread. To avoid this, use TestActorRef. + +@@@ If an actor's behavior blocks on a something which would normally be affected by the calling actor after having sent the message, this will obviously dead-lock when using this dispatcher. This is a common scenario in actor tests -based on :class:`CountDownLatch` for synchronization: +based on `CountDownLatch` for synchronization: -.. code-block:: scala - - val latch = new CountDownLatch(1) - actor ! startWorkAfter(latch) // actor will call latch.await() before proceeding - doSomeSetupStuff() - latch.countDown() +```scala +val latch = new CountDownLatch(1) +actor ! startWorkAfter(latch) // actor will call latch.await() before proceeding +doSomeSetupStuff() +latch.countDown() +``` The example would hang indefinitely within the message processing initiated on the second line and never reach the fourth line, which would unblock it on a normal dispatcher. -Thus, keep in mind that the :class:`CallingThreadDispatcher` is not a +Thus, keep in mind that the `CallingThreadDispatcher` is not a general-purpose replacement for the normal dispatchers. On the other hand it may be quite useful to run your actor network on it for testing, because if it runs without dead-locking chances are very high that it will not dead-lock in production. -.. warning:: +@@@ warning - The above sentence is unfortunately not a strong guarantee, because your - code might directly or indirectly change its behavior when running on a - different dispatcher. If you are looking for a tool to help you debug - dead-locks, the :class:`CallingThreadDispatcher` may help with certain error - scenarios, but keep in mind that it has may give false negatives as well as - false positives. +The above sentence is unfortunately not a strong guarantee, because your +code might directly or indirectly change its behavior when running on a +different dispatcher. If you are looking for a tool to help you debug +dead-locks, the `CallingThreadDispatcher` may help with certain error +scenarios, but keep in mind that it has may give false negatives as well as +false positives. -Thread Interruptions --------------------- +@@@ + +### Thread Interruptions If the CallingThreadDispatcher sees that the current thread has its -``isInterrupted()`` flag set when message processing returns, it will throw an -:class:`InterruptedException` after finishing all its processing (i.e. all +`isInterrupted()` flag set when message processing returns, it will throw an +`InterruptedException` after finishing all its processing (i.e. all messages which need processing as described above are processed before this -happens). As :meth:`tell` cannot throw exceptions due to its contract, this +happens). As `tell` cannot throw exceptions due to its contract, this exception will then be caught and logged, and the thread’s interrupted status will be set again. -If during message processing an :class:`InterruptedException` is thrown then it +If during message processing an `InterruptedException` is thrown then it will be caught inside the CallingThreadDispatcher’s message handling loop, the thread’s interrupted flag will be set and processing continues normally. -.. note:: +@@@ note - The summary of these two paragraphs is that if the current thread is - interrupted while doing work under the CallingThreadDispatcher, then that - will result in the ``isInterrupted`` flag to be ``true`` when the message - send returns and no :class:`InterruptedException` will be thrown. +The summary of these two paragraphs is that if the current thread is +interrupted while doing work under the CallingThreadDispatcher, then that +will result in the `isInterrupted` flag to be `true` when the message +send returns and no `InterruptedException` will be thrown. -Benefits --------- +@@@ -To summarize, these are the features with the :class:`CallingThreadDispatcher` +### Benefits + +To summarize, these are the features with the `CallingThreadDispatcher` has to offer: - - Deterministic execution of single-threaded tests while retaining nearly full - actor semantics - - Full message processing history leading up to the point of failure in - exception stack traces - - Exclusion of certain classes of dead-lock scenarios +> + * Deterministic execution of single-threaded tests while retaining nearly full +actor semantics + * Full message processing history leading up to the point of failure in +exception stack traces + * Exclusion of certain classes of dead-lock scenarios -.. _actor.logging-java: - -Tracing Actor Invocations -========================= + +## Tracing Actor Invocations The testing facilities described up to this point were aiming at formulating assertions about a system’s behavior. If a test fails, it is usually your job @@ -665,40 +645,38 @@ to find the cause, fix it and verify the test again. This process is supported by debuggers as well as logging, where the Akka toolkit offers the following options: -* *Logging of exceptions thrown within Actor instances* + * + *Logging of exceptions thrown within Actor instances* + This is always on; in contrast to the other logging mechanisms, this logs at +`ERROR` level. + * + *Logging of special messages* + Actors handle certain special messages automatically, e.g. `Kill`, +`PoisonPill`, etc. Tracing of these message invocations is enabled by +the setting `akka.actor.debug.autoreceive`, which enables this on all +actors. + * + *Logging of the actor lifecycle* + Actor creation, start, restart, monitor start, monitor stop and stop may be traced by +enabling the setting `akka.actor.debug.lifecycle`; this, too, is enabled +uniformly on all actors. - This is always on; in contrast to the other logging mechanisms, this logs at - ``ERROR`` level. +All these messages are logged at `DEBUG` level. To summarize, you can enable +full logging of actor activities using this configuration fragment: -* *Logging of special messages* - - Actors handle certain special messages automatically, e.g. :obj:`Kill`, - :obj:`PoisonPill`, etc. Tracing of these message invocations is enabled by - the setting ``akka.actor.debug.autoreceive``, which enables this on all - actors. - -* *Logging of the actor lifecycle* - - Actor creation, start, restart, monitor start, monitor stop and stop may be traced by - enabling the setting ``akka.actor.debug.lifecycle``; this, too, is enabled - uniformly on all actors. - -All these messages are logged at ``DEBUG`` level. To summarize, you can enable -full logging of actor activities using this configuration fragment:: - - akka { - loglevel = "DEBUG" - actor { - debug { - autoreceive = on - lifecycle = on - } +``` +akka { + loglevel = "DEBUG" + actor { + debug { + autoreceive = on + lifecycle = on } } +} +``` -Configuration -============= +## Configuration There are several configuration properties for the TestKit module, please refer -to the :ref:`reference configuration `. - +to the @ref:[reference configuration](../general/configuration.md#config-akka-testkit). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/java/typed-actors.md b/akka-docs/src/main/paradox/java/typed-actors.md index 7f8561b86d..c226fba049 100644 --- a/akka-docs/src/main/paradox/java/typed-actors.md +++ b/akka-docs/src/main/paradox/java/typed-actors.md @@ -1,7 +1,6 @@ -Typed Actors -=================== +# Typed Actors -Akka Typed Actors is an implementation of the `Active Objects `_ pattern. +Akka Typed Actors is an implementation of the [Active Objects](http://en.wikipedia.org/wiki/Active_object) pattern. Essentially turning method invocations into asynchronous dispatch instead of synchronous that has been the default way since Smalltalk came out. Typed Actors consist of 2 "parts", a public interface and an implementation, and if you've done any work in "enterprise" Java, this will be very familiar to you. As with normal Actors you have an external API (the public interface instance) that will delegate method calls asynchronously to @@ -9,233 +8,204 @@ a private instance of the implementation. The advantage of Typed Actors vs. Actors is that with TypedActors you have a static contract, and don't need to define your own messages, the downside is that it places some limitations on what you can do and what you can't, i.e. you can't use become/unbecome. -Typed Actors are implemented using `JDK Proxies `_ which provide a pretty easy-worked API to intercept method calls. +Typed Actors are implemented using [JDK Proxies](http://docs.oracle.com/javase/6/jdocs/api/java/lang/reflect/Proxy.html) which provide a pretty easy-worked API to intercept method calls. -.. note:: +@@@ note - Just as with regular Akka Untyped Actors, Typed Actors process one call at a time. +Just as with regular Akka Untyped Actors, Typed Actors process one call at a time. -When to use Typed Actors ------------------------- +@@@ + +## When to use Typed Actors Typed actors are nice for bridging between actor systems (the “inside”) and non-actor code (the “outside”), because they allow you to write normal OO-looking code on the outside. Think of them like doors: their practicality lies in interfacing between private sphere and the public, but you don’t want -that many doors inside your house, do you? For a longer discussion see `this -blog post `_. +that many doors inside your house, do you? For a longer discussion see [this +blog post](http://letitcrash.com/post/19074284309/when-to-use-typedactors). A bit more background: TypedActors can easily be abused as RPC, and that -is an abstraction which is `well-known -`_ +is an abstraction which is [well-known](http://doc.akka.io/jdocs/misc/smli_tr-94-29.pdf) to be leaky. Hence TypedActors are not what we think of first when we talk about making highly scalable concurrent software easier to write correctly. They have their niche, use them sparingly. -The tools of the trade ----------------------- +## The tools of the trade Before we create our first Typed Actor we should first go through the tools that we have at our disposal, -it's located in ``akka.actor.TypedActor``. +it's located in `akka.actor.TypedActor`. -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-extension-tools +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-extension-tools } -.. warning:: +@@@ warning - Same as not exposing ``this`` of an Akka Actor, it's important not to expose ``this`` of a Typed Actor, - instead you should pass the external proxy reference, which is obtained from within your Typed Actor as - ``TypedActor.self()``, this is your external identity, as the ``ActorRef`` is the external identity of - an Akka Actor. +Same as not exposing `this` of an Akka Actor, it's important not to expose `this` of a Typed Actor, +instead you should pass the external proxy reference, which is obtained from within your Typed Actor as +`TypedActor.self()`, this is your external identity, as the `ActorRef` is the external identity of +an Akka Actor. -Creating Typed Actors ---------------------- +@@@ + +## Creating Typed Actors To create a Typed Actor you need to have one or more interfaces, and one implementation. The following imports are assumed: -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: imports +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #imports } Our example interface: -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-iface - :exclude: typed-actor-iface-methods +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-iface } Our example implementation of that interface: -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-impl - :exclude: typed-actor-impl-methods +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-impl } The most trivial way of creating a Typed Actor instance -of our ``Squarer``: +of our `Squarer`: -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-create1 +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-create1 } First type is the type of the proxy, the second type is the type of the implementation. If you need to call a specific constructor you do it like this: -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-create2 +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-create2 } -Since you supply a ``Props``, you can specify which dispatcher to use, what the default timeout should be used and more. -Now, our ``Squarer`` doesn't have any methods, so we'd better add those. +Since you supply a `Props`, you can specify which dispatcher to use, what the default timeout should be used and more. +Now, our `Squarer` doesn't have any methods, so we'd better add those. -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-iface +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-iface } -Alright, now we've got some methods we can call, but we need to implement those in ``SquarerImpl``. +Alright, now we've got some methods we can call, but we need to implement those in `SquarerImpl`. -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-impl +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-impl } Excellent, now we have an interface and an implementation of that interface, and we know how to create a Typed Actor from that, so let's look at calling these methods. -Method dispatch semantics -------------------------- +## Method dispatch semantics Methods returning: - * ``void`` will be dispatched with ``fire-and-forget`` semantics, exactly like ``ActorRef.tell`` - * ``scala.concurrent.Future`` will use ``send-request-reply`` semantics, exactly like ``ActorRef.ask`` - * ``akka.japi.Option`` will use ``send-request-reply`` semantics, but *will* block to wait for an answer, - and return ``akka.japi.Option.None`` if no answer was produced within the timeout, or ``akka.japi.Option.Some`` containing the result otherwise. - Any exception that was thrown during this call will be rethrown. - * Any other type of value will use ``send-request-reply`` semantics, but *will* block to wait for an answer, - throwing ``java.util.concurrent.TimeoutException`` if there was a timeout or rethrow any exception that was thrown during this call. - Note that due to the Java exception and reflection mechanisms, such a ``TimeoutException`` will be wrapped in a ``java.lang.reflect.UndeclaredThrowableException`` - unless the interface method explicitly declares the ``TimeoutException`` as a thrown checked exception. +> + * `void` will be dispatched with `fire-and-forget` semantics, exactly like `ActorRef.tell` + * `scala.concurrent.Future` will use `send-request-reply` semantics, exactly like `ActorRef.ask` + * `akka.japi.Option` will use `send-request-reply` semantics, but *will* block to wait for an answer, +and return `akka.japi.Option.None` if no answer was produced within the timeout, or `akka.japi.Option.Some` containing the result otherwise. +Any exception that was thrown during this call will be rethrown. + * Any other type of value will use `send-request-reply` semantics, but *will* block to wait for an answer, +throwing `java.util.concurrent.TimeoutException` if there was a timeout or rethrow any exception that was thrown during this call. +Note that due to the Java exception and reflection mechanisms, such a `TimeoutException` will be wrapped in a `java.lang.reflect.UndeclaredThrowableException` +unless the interface method explicitly declares the `TimeoutException` as a thrown checked exception. -Messages and immutability -------------------------- +## Messages and immutability While Akka cannot enforce that the parameters to the methods of your Typed Actors are immutable, we *strongly* recommend that parameters passed are immutable. -One-way message send -^^^^^^^^^^^^^^^^^^^^ +### One-way message send -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-call-oneway +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-call-oneway } As simple as that! The method will be executed on another thread; asynchronously. -Request-reply message send -^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Request-reply message send -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-call-option +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-call-option } -This will block for as long as the timeout that was set in the ``Props`` of the Typed Actor, -if needed. It will return ``None`` if a timeout occurs. +This will block for as long as the timeout that was set in the `Props` of the Typed Actor, +if needed. It will return `None` if a timeout occurs. -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-call-strict +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-call-strict } -This will block for as long as the timeout that was set in the ``Props`` of the Typed Actor, -if needed. It will throw a ``java.util.concurrent.TimeoutException`` if a timeout occurs. -Note that here, such a ``TimeoutException`` will be wrapped in a -``java.lang.reflect.UndeclaredThrowableException`` by the Java reflection mechanism, -because the interface method does not explicitly declare the ``TimeoutException`` as a thrown checked exception. -To get the ``TimeoutException`` directly, declare ``throws java.util.concurrent.TimeoutException`` at the +This will block for as long as the timeout that was set in the `Props` of the Typed Actor, +if needed. It will throw a `java.util.concurrent.TimeoutException` if a timeout occurs. +Note that here, such a `TimeoutException` will be wrapped in a +`java.lang.reflect.UndeclaredThrowableException` by the Java reflection mechanism, +because the interface method does not explicitly declare the `TimeoutException` as a thrown checked exception. +To get the `TimeoutException` directly, declare `throws java.util.concurrent.TimeoutException` at the interface method. -Request-reply-with-future message send -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Request-reply-with-future message send -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-call-future +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-call-future } This call is asynchronous, and the Future returned can be used for asynchronous composition. -Stopping Typed Actors ---------------------- +## Stopping Typed Actors Since Akka's Typed Actors are backed by Akka Actors they must be stopped when they aren't needed anymore. -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-stop +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-stop } This asynchronously stops the Typed Actor associated with the specified proxy ASAP. -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-poisonpill +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-poisonpill } This asynchronously stops the Typed Actor associated with the specified proxy after it's done with all calls that were made prior to this call. -Typed Actor Hierarchies ------------------------ +## Typed Actor Hierarchies -Since you can obtain a contextual Typed Actor Extension by passing in an ``ActorContext`` -you can create child Typed Actors by invoking ``typedActorOf(..)`` on that. +Since you can obtain a contextual Typed Actor Extension by passing in an `ActorContext` +you can create child Typed Actors by invoking `typedActorOf(..)` on that. -.. includecode:: code/jdocs/actor/TypedActorDocTest.java - :include: typed-actor-hierarchy +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-hierarchy } -You can also create a child Typed Actor in regular Akka Actors by giving the ``AbstractActor.ActorContext`` +You can also create a child Typed Actor in regular Akka Actors by giving the `AbstractActor.ActorContext` as an input parameter to TypedActor.get(…). -Supervisor Strategy -------------------- +## Supervisor Strategy -By having your Typed Actor implementation class implement ``TypedActor.Supervisor`` +By having your Typed Actor implementation class implement `TypedActor.Supervisor` you can define the strategy to use for supervising child actors, as described in -:ref:`supervision` and :ref:`fault-tolerance-java`. + supervision and @ref:[Fault Tolerance](fault-tolerance.md). -Receive arbitrary messages --------------------------- +## Receive arbitrary messages -If your implementation class of your TypedActor extends ``akka.actor.TypedActor.Receiver``, -all messages that are not ``MethodCall`` instances will be passed into the ``onReceive``-method. +If your implementation class of your TypedActor extends `akka.actor.TypedActor.Receiver`, +all messages that are not `MethodCall` instances will be passed into the `onReceive`-method. -This allows you to react to DeathWatch ``Terminated``-messages and other types of messages, +This allows you to react to DeathWatch `Terminated`-messages and other types of messages, e.g. when interfacing with untyped actors. -Lifecycle callbacks -------------------- +## Lifecycle callbacks By having your Typed Actor implementation class implement any and all of the following: - * ``TypedActor.PreStart`` - * ``TypedActor.PostStop`` - * ``TypedActor.PreRestart`` - * ``TypedActor.PostRestart`` +> + * `TypedActor.PreStart` + * `TypedActor.PostStop` + * `TypedActor.PreRestart` + * `TypedActor.PostRestart` You can hook into the lifecycle of your Typed Actor. -Proxying --------- +## Proxying -You can use the ``typedActorOf`` that takes a TypedProps and an ActorRef to proxy the given ActorRef as a TypedActor. -This is usable if you want to communicate remotely with TypedActors on other machines, just pass the ``ActorRef`` to ``typedActorOf``. +You can use the `typedActorOf` that takes a TypedProps and an ActorRef to proxy the given ActorRef as a TypedActor. +This is usable if you want to communicate remotely with TypedActors on other machines, just pass the `ActorRef` to `typedActorOf`. -Lookup & Remoting ------------------ +## Lookup & Remoting -Since ``TypedActors`` are backed by ``Akka Actors``, you can use ``typedActorOf`` to proxy ``ActorRefs`` potentially residing on remote nodes. +Since `TypedActors` are backed by `Akka Actors`, you can use `typedActorOf` to proxy `ActorRefs` potentially residing on remote nodes. -.. includecode:: code/jdocs/actor/TypedActorDocTest.java#typed-actor-remote +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-actor-remote } -Typed Router pattern --------------------- +## Typed Router pattern -Sometimes you want to spread messages between multiple actors. The easiest way to achieve this in Akka is to use a :ref:`Router `, -which can implement a specific routing logic, such as ``smallest-mailbox`` or ``consistent-hashing`` etc. +Sometimes you want to spread messages between multiple actors. The easiest way to achieve this in Akka is to use a @ref:[Router](routing.md), +which can implement a specific routing logic, such as `smallest-mailbox` or `consistent-hashing` etc. Routers are not provided directly for typed actors, but it is really easy to leverage an untyped router and use a typed proxy in front of it. -To showcase this let's create typed actors that assign themselves some random ``id``, so we know that in fact, the router has sent the message to different actors: +To showcase this let's create typed actors that assign themselves some random `id`, so we know that in fact, the router has sent the message to different actors: -.. includecode:: code/jdocs/actor/TypedActorDocTest.java#typed-router-types +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-router-types } In order to round robin among a few instances of such actors, you can simply create a plain untyped router, -and then facade it with a ``TypedActor`` like shown in the example below. This works because typed actors of course -communicate using the same mechanisms as normal actors, and methods calls on them get transformed into message sends of ``MethodCall`` messages. - -.. includecode:: code/jdocs/actor/TypedActorDocTest.java#typed-router +and then facade it with a `TypedActor` like shown in the example below. This works because typed actors of course +communicate using the same mechanisms as normal actors, and methods calls on them get transformed into message sends of `MethodCall` messages. +@@snip [TypedActorDocTest.java](code/jdocs/actor/TypedActorDocTest.java) { #typed-router } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/index.md b/akka-docs/src/main/paradox/project/index.md index 216735294e..a522d9b047 100644 --- a/akka-docs/src/main/paradox/project/index.md +++ b/akka-docs/src/main/paradox/project/index.md @@ -1,11 +1,13 @@ -Project Information -=================== +# Project Information -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - migration-guides - issue-tracking - licenses - sponsors - links +@@@ index + +* [migration-guides](migration-guides.md) +* [issue-tracking](issue-tracking.md) +* [licenses](licenses.md) +* [sponsors](sponsors.md) +* [links](links.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/issue-tracking.md b/akka-docs/src/main/paradox/project/issue-tracking.md index 98406eaa68..4e53b37da3 100644 --- a/akka-docs/src/main/paradox/project/issue-tracking.md +++ b/akka-docs/src/main/paradox/project/issue-tracking.md @@ -1,44 +1,36 @@ -.. _issue_tracking: - -Issue Tracking -============== +# Issue Tracking Akka is using GitHub Issues as its issue tracking system. -Browsing --------- +## Browsing -Tickets -^^^^^^^ +### Tickets -Before filing a ticket, please check the existing `Akka tickets -`_ for earlier reports of the same +Before filing a ticket, please check the existing [Akka tickets](https://github.com/akka/akka/issues) for earlier reports of the same problem. You are very welcome to comment on existing tickets, especially if you have reproducible test cases that you can share. -Roadmaps -^^^^^^^^ +### Roadmaps -Short and long-term plans are published in the `akka/akka-meta `_ repository. +Short and long-term plans are published in the [akka/akka-meta](https://github.com/akka/akka-meta/issues) repository. -Creating tickets ----------------- +## Creating tickets *Please include the versions of Scala and Akka and relevant configuration files.* -You can create a `new ticket `_ if you +You can create a [new ticket](https://github.com/akka/akka/issues/new) if you have registered a GitHub user account. Thanks a lot for reporting bugs and suggesting features! -Submitting Pull Requests ------------------------- +## Submitting Pull Requests -.. note:: *A pull request is worth a thousand +1's.* -- Old Klangian Proverb +@@@ note + +*A pull request is worth a thousand +1's.* -- Old Klangian Proverb + +@@@ Pull Requests fixing issues or adding functionality are very welcome. -Please read `CONTRIBUTING.md `_ for -more information about contributing to Akka. - - - +Please read [CONTRIBUTING.md](https://github.com/akka/akka/blob/master/CONTRIBUTING.md) for +more information about contributing to Akka. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/licenses.md b/akka-docs/src/main/paradox/project/licenses.md index 73916591c3..0e80c9e88d 100644 --- a/akka-docs/src/main/paradox/project/licenses.md +++ b/akka-docs/src/main/paradox/project/licenses.md @@ -1,40 +1,31 @@ -.. _licenses: +# Licenses -Licenses -======== +## Akka License -.. highlight:: text +``` +This software is licensed under the Apache 2 license, quoted below. -Akka License ------------- +Copyright 2009-2017 Lightbend Inc. -:: +Licensed under the Apache License, Version 2.0 (the "License"); you may not +use this file except in compliance with the License. You may obtain a copy of +the License at - This software is licensed under the Apache 2 license, quoted below. + http://www.apache.org/licenses/LICENSE-2.0 - Copyright 2009-2017 Lightbend Inc. +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +License for the specific language governing permissions and limitations under +the License. +``` - Licensed under the Apache License, Version 2.0 (the "License"); you may not - use this file except in compliance with the License. You may obtain a copy of - the License at +## Akka Committer License Agreement - http://www.apache.org/licenses/LICENSE-2.0 +All committers have signed this [CLA](http://www.lightbend.com/contribute/current-cla). +It can be [signed online](http://www.lightbend.com/contribute/cla). - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, WITHOUT - WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the - License for the specific language governing permissions and limitations under - the License. - -Akka Committer License Agreement --------------------------------- - -All committers have signed this `CLA `_. -It can be `signed online `_. - -Licenses for Dependency Libraries ---------------------------------- +## Licenses for Dependency Libraries Each dependency and its license can be seen in the project build file (the comment on the side of each dependency): -`AkkaBuild.scala <@github@/project/AkkaBuild.scala#L1054>`_ - +[AkkaBuild.scala](@github@/project/AkkaBuild.scala#L1054) \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/links.md b/akka-docs/src/main/paradox/project/links.md index df3ff70f10..285a151c2f 100644 --- a/akka-docs/src/main/paradox/project/links.md +++ b/akka-docs/src/main/paradox/project/links.md @@ -1,92 +1,80 @@ -.. _support: +# Project -######### - Project -######### +## Commercial Support -Commercial Support -^^^^^^^^^^^^^^^^^^ +Commercial support is provided by [Lightbend](http://www.lightbend.com). +Akka is part of the [Lightbend Reactive Platform](http://www.lightbend.com/platform). -Commercial support is provided by `Lightbend `_. -Akka is part of the `Lightbend Reactive Platform `_. +## Mailing List -Mailing List -^^^^^^^^^^^^ +[Akka User Google Group](http://groups.google.com/group/akka-user) -`Akka User Google Group `_ +[Akka Developer Google Group](http://groups.google.com/group/akka-dev) -`Akka Developer Google Group `_ +## Downloads +[http://akka.io/downloads](http://akka.io/downloads) -Downloads -^^^^^^^^^ +## Source Code -``_ +Akka uses Git and is hosted at [Github](http://github.com). + * Akka: clone the Akka repository from [http://github.com/akka/akka](http://github.com/akka/akka) -Source Code -^^^^^^^^^^^ - -Akka uses Git and is hosted at `Github `_. - -* Akka: clone the Akka repository from ``_ - - -Releases Repository -^^^^^^^^^^^^^^^^^^^ +## Releases Repository All Akka releases are published via Sonatype to Maven Central, see -`search.maven.org -`_ +[search.maven.org](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.typesafe.akka%22) or -`search.maven.org (Akka versions before 2.4.3) -`_ +[search.maven.org (Akka versions before 2.4.3)](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.typesafe.akka%22) -Snapshots Repository -^^^^^^^^^^^^^^^^^^^^ +## Snapshots Repository -Nightly builds are available in http://repo.akka.io/snapshots/ as both ``SNAPSHOT`` and +Nightly builds are available in [http://repo.akka.io/snapshots](http://repo.akka.io/snapshots)/ as both `SNAPSHOT` and timestamped versions. For timestamped versions, pick a timestamp from -http://repo.akka.io/snapshots/com/lightbend/akka/akka-actor_@binVersion@/. +[http://repo.akka.io/snapshots/com/lightbend/akka/akka-actor_@binVersion@](http://repo.akka.io/snapshots/com/lightbend/akka/akka-actor_@binVersion@)/. All Akka modules that belong to the same build have the same timestamp. -sbt definition of snapshot repository -------------------------------------- +### sbt definition of snapshot repository -Make sure that you add the repository to the sbt resolvers:: +Make sure that you add the repository to the sbt resolvers: - resolvers += "Lightbend Snapshots" at "http://repo.akka.io/snapshots/" +``` +resolvers += "Lightbend Snapshots" at "http://repo.akka.io/snapshots/" +``` -Define the library dependencies with the timestamp as version. For example:: +Define the library dependencies with the timestamp as version. For example: - libraryDependencies += "com.typesafe.akka" % "akka-remote_@binVersion@" % - "2.1-20121016-001042" +``` +libraryDependencies += "com.typesafe.akka" % "akka-remote_@binVersion@" % + "2.1-20121016-001042" +``` -maven definition of snapshot repository ---------------------------------------- +### maven definition of snapshot repository -Make sure that you add the repository to the maven repositories in pom.xml:: - - - - akka-snapshots - Akka Snapshots - http://repo.akka.io/snapshots/ - default - - - -Define the library dependencies with the timestamp as version. For example:: - - - - com.typesafe.akka - akka-remote_@binVersion@ - 2.1-20121016-001042 - - +Make sure that you add the repository to the maven repositories in pom.xml: +``` + + + akka-snapshots + Akka Snapshots + http://repo.akka.io/snapshots/ + default + + +``` +Define the library dependencies with the timestamp as version. For example: +``` + + + com.typesafe.akka + akka-remote_@binVersion@ + 2.1-20121016-001042 + + +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-1.3.x-2.0.x.md b/akka-docs/src/main/paradox/project/migration-guide-1.3.x-2.0.x.md index 0ffa5e245e..dbc0545138 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-1.3.x-2.0.x.md +++ b/akka-docs/src/main/paradox/project/migration-guide-1.3.x-2.0.x.md @@ -1,10 +1,4 @@ -.. _migration-2.0: - -################################ - Migration Guide 1.3.x to 2.0.x -################################ +# Migration Guide 1.3.x to 2.0.x Migration from 1.3.x to 2.0.x is described in the -`documentation of 2.0 `_. - - +[documentation of 2.0](http://doc.akka.io/docs/akka/2.0.5/project/migration-guide-1.3.x-2.0.x.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-2.0.x-2.1.x.md b/akka-docs/src/main/paradox/project/migration-guide-2.0.x-2.1.x.md index 325aeb8fc0..a1abf1bbb3 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-2.0.x-2.1.x.md +++ b/akka-docs/src/main/paradox/project/migration-guide-2.0.x-2.1.x.md @@ -1,8 +1,4 @@ -.. _migration-2.1: - -################################ - Migration Guide 2.0.x to 2.1.x -################################ +# Migration Guide 2.0.x to 2.1.x Migration from 2.0.x to 2.1.x is described in the -`documentation of 2.1 `_. \ No newline at end of file +[documentation of 2.1](http://doc.akka.io/docs/akka/2.1.4/project/migration-guide-2.0.x-2.1.x.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-2.1.x-2.2.x.md b/akka-docs/src/main/paradox/project/migration-guide-2.1.x-2.2.x.md index 4618faded7..8553d4725d 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-2.1.x-2.2.x.md +++ b/akka-docs/src/main/paradox/project/migration-guide-2.1.x-2.2.x.md @@ -1,8 +1,4 @@ -.. _migration-2.2: - -################################ - Migration Guide 2.1.x to 2.2.x -################################ +# Migration Guide 2.1.x to 2.2.x Migration from 2.1.x to 2.2.x is described in the -`documentation of 2.2 `_. \ No newline at end of file +[documentation of 2.2](http://doc.akka.io/docs/akka/2.2.3/project/migration-guide-2.1.x-2.2.x.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-2.2.x-2.3.x.md b/akka-docs/src/main/paradox/project/migration-guide-2.2.x-2.3.x.md index dfd467e646..59bdc1db42 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-2.2.x-2.3.x.md +++ b/akka-docs/src/main/paradox/project/migration-guide-2.2.x-2.3.x.md @@ -1,8 +1,4 @@ -.. _migration-2.3: - -################################ - Migration Guide 2.2.x to 2.3.x -################################ +# Migration Guide 2.2.x to 2.3.x Migration from 2.2.x to 2.3.x is described in the -`documentation of 2.3 `_. \ No newline at end of file +[documentation of 2.3](http://doc.akka.io/docs/akka/2.3.12/project/migration-guide-2.2.x-2.3.x.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-2.3.x-2.4.x.md b/akka-docs/src/main/paradox/project/migration-guide-2.3.x-2.4.x.md index f2eb1fb44b..607094488a 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-2.3.x-2.4.x.md +++ b/akka-docs/src/main/paradox/project/migration-guide-2.3.x-2.4.x.md @@ -1,9 +1,4 @@ -.. _migration-2.4: - -############################## -Migration Guide 2.3.x to 2.4.x -############################## +# Migration Guide 2.3.x to 2.4.x Migration from 2.3.x to 2.4.x is described in the -`documentation of 2.4 `_. - +[documentation of 2.4](http://doc.akka.io/docs/akka/2.4/project/migration-guide-2.3.x-2.4.x.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-2.4.x-2.5.x.md b/akka-docs/src/main/paradox/project/migration-guide-2.4.x-2.5.x.md index 4d4e24e07f..96289f8dac 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-2.4.x-2.5.x.md +++ b/akka-docs/src/main/paradox/project/migration-guide-2.4.x-2.5.x.md @@ -1,414 +1,409 @@ -.. _migration-guide-2.4.x-2.5.x: +# Migration Guide 2.4.x to 2.5.x -############################## -Migration Guide 2.4.x to 2.5.x -############################## +## Actor (Java) -Actor (Java) -============ +### AbstractActor -AbstractActor -------------- - -``AbstractActor`` has been promoted from its experimental/may change state and while doing this we +`AbstractActor` has been promoted from its experimental/may change state and while doing this we did some small, but important, improvements to the API that will require some mechanical changes of your source code. -Previously the receive behavior was set with the ``receive`` method, but now an actor has -to define its initial receive behavior by implementing the ``createReceive`` method in -the ``AbstractActor``. This has the advantages: +Previously the receive behavior was set with the `receive` method, but now an actor has +to define its initial receive behavior by implementing the `createReceive` method in +the `AbstractActor`. This has the advantages: -* It gives a clear entry point of what to implement. The compiler tells you that the - abstract method must be implemented. -* It's impossible to forget to set the receive behavior. -* It's not possible to define the receive behavior more than once. + * It gives a clear entry point of what to implement. The compiler tells you that the +abstract method must be implemented. + * It's impossible to forget to set the receive behavior. + * It's not possible to define the receive behavior more than once. -The return type of ``createReceive`` is ``AbstractActor.Receive``. It defines which messages +The return type of `createReceive` is `AbstractActor.Receive`. It defines which messages your Actor can handle, along with the implementation of how the messages should be processed. -You can build such behavior with a builder named ``ReceiveBuilder``. +You can build such behavior with a builder named `ReceiveBuilder`. -``AbstractActor.Receive`` can also be used in ``getContext().become``. +`AbstractActor.Receive` can also be used in `getContext().become`. -The old ``receive`` method exposed Scala's ``PartialFunction`` and ``BoxedUnit`` in the signature, -which are unnecessary concepts for newcomers to learn. The new ``createReceive`` requires no +The old `receive` method exposed Scala's `PartialFunction` and `BoxedUnit` in the signature, +which are unnecessary concepts for newcomers to learn. The new `createReceive` requires no additional imports. -Note that The ``Receive`` can still be implemented in other ways than using the ``ReceiveBuilder`` -since it in the end is just a wrapper around a Scala ``PartialFunction``. For example, one could -implement an adapter to `Javaslang Pattern Matching DSL `_. +Note that The `Receive` can still be implemented in other ways than using the `ReceiveBuilder` +since it in the end is just a wrapper around a Scala `PartialFunction`. For example, one could +implement an adapter to [Javaslang Pattern Matching DSL](http://www.javaslang.io/javaslang-docs/#_pattern_matching). -The mechanical source code change for migration to the new ``AbstractActor`` is to implement the -``createReceive`` instead of calling ``receive`` (compiler will tell that this is missing). +The mechanical source code change for migration to the new `AbstractActor` is to implement the +`createReceive` instead of calling `receive` (compiler will tell that this is missing). -Old:: +Old: - import akka.actor.AbstractActor; - import akka.japi.pf.ReceiveBuilder; - import scala.PartialFunction; - import scala.runtime.BoxedUnit; +``` +import akka.actor.AbstractActor; +import akka.japi.pf.ReceiveBuilder; +import scala.PartialFunction; +import scala.runtime.BoxedUnit; - public class SomeActor extends AbstractActor { - public SomeActor() { - receive(ReceiveBuilder - .match(String.class, s -> System.out.println(s.toLowerCase())). - .build()); - } +public class SomeActor extends AbstractActor { + public SomeActor() { + receive(ReceiveBuilder + .match(String.class, s -> System.out.println(s.toLowerCase())). + .build()); } +} +``` -New:: +New: - import akka.actor.AbstractActor; +``` +import akka.actor.AbstractActor; - public class SomeActor extends AbstractActor { - @Override - public Receive createReceive() { - return receiveBuilder() - .match(String.class, s -> System.out.println(s.toLowerCase())) - .build(); - } +public class SomeActor extends AbstractActor { + @Override + public Receive createReceive() { + return receiveBuilder() + .match(String.class, s -> System.out.println(s.toLowerCase())) + .build(); } +} +``` -See :ref:`actors-receive-java` documentation for more advice about how to implement -``createReceive``. +See @ref:[Receive messages](../java/actors.md#actors-receive-java) documentation for more advice about how to implement +`createReceive`. -A few new methods have been added with deprecation of the old. Worth noting is ``preRestart``. +A few new methods have been added with deprecation of the old. Worth noting is `preRestart`. -Old:: +Old: + +``` +@Override +public void preRestart(Throwable reason, scala.Option message) { + super.preRestart(reason, message); +} +``` + +New: + +``` +@Override +public void preRestart(Throwable reason, java.util.Optional message) { + super.preRestart(reason, message); +} +``` + +### AbstractPersistentActor + +Similar change as described above for `AbstractActor` is needed for `AbstractPersistentActor`. Implement `createReceiveRecover` +instead of `receiveRecover`, and `createReceive` instead of `receiveCommand`. + +Old: + +``` +@Override +public PartialFunction receiveCommand() { + return ReceiveBuilder. + match(String.class, cmd -> {/* ... */}).build(); +} + +@Override +public PartialFunction receiveRecover() { + return ReceiveBuilder. + match(String.class, evt -> {/* ... */}).build(); +} +``` + +New: + +``` +@Override +public Receive createReceive() { + return receiveBuilder(). + match(String.class, cmd -> {/* ... */}).build(); +} + +@Override +public Receive createReceiveRecover() { + return receiveBuilder(). + match(String.class, evt -> {/* ... */}).build(); +} +``` + +### UntypedActor + +`UntypedActor` has been deprecated in favor of `AbstractActor`. As a migration path you can extend +`UntypedAbstractActor` instead of `UntypedActor`. + +Old: + +``` +import akka.actor.UntypedActor; + +public class SomeActor extends UntypedActor { + + public static class Msg1 {} @Override - public void preRestart(Throwable reason, scala.Option message) { - super.preRestart(reason, message); + public void onReceive(Object msg) throws Exception { + if (msg instanceof Msg1) { + Msg1 msg1 = (Msg1) msg; + // actual work + } else { + unhandled(msg); + } } +} +``` -New:: +New: + +``` +import akka.actor.UntypedAbstractActor; + +public class SomeActor extends UntypedAbstractActor { + + public static class Msg1 {} @Override - public void preRestart(Throwable reason, java.util.Optional message) { - super.preRestart(reason, message); - } - -AbstractPersistentActor ------------------------ - -Similar change as described above for ``AbstractActor`` is needed for ``AbstractPersistentActor``. Implement ``createReceiveRecover`` -instead of ``receiveRecover``, and ``createReceive`` instead of ``receiveCommand``. - -Old:: - - @Override - public PartialFunction receiveCommand() { - return ReceiveBuilder. - match(String.class, cmd -> {/* ... */}).build(); - } - - @Override - public PartialFunction receiveRecover() { - return ReceiveBuilder. - match(String.class, evt -> {/* ... */}).build(); - } - -New:: - - @Override - public Receive createReceive() { - return receiveBuilder(). - match(String.class, cmd -> {/* ... */}).build(); - } - - @Override - public Receive createReceiveRecover() { - return receiveBuilder(). - match(String.class, evt -> {/* ... */}).build(); - } - -UntypedActor ------------- - -``UntypedActor`` has been deprecated in favor of ``AbstractActor``. As a migration path you can extend -``UntypedAbstractActor`` instead of ``UntypedActor``. - -Old:: - - import akka.actor.UntypedActor; - - public class SomeActor extends UntypedActor { - - public static class Msg1 {} - - @Override - public void onReceive(Object msg) throws Exception { - if (msg instanceof Msg1) { - Msg1 msg1 = (Msg1) msg; - // actual work - } else { - unhandled(msg); - } + public void onReceive(Object msg) throws Exception { + if (msg instanceof Msg1) { + Msg1 msg1 = (Msg1) msg; + // actual work + } else { + unhandled(msg); } } +} +``` +It's recommended to migrate `UntypedActor` to `AbstractActor` by implementing +`createReceive` instead of `onReceive`. -New:: +Old: - import akka.actor.UntypedAbstractActor; +``` +import akka.actor.UntypedActor; - public class SomeActor extends UntypedAbstractActor { +public class SomeActor extends UntypedActor { - public static class Msg1 {} - - @Override - public void onReceive(Object msg) throws Exception { - if (msg instanceof Msg1) { - Msg1 msg1 = (Msg1) msg; - // actual work - } else { - unhandled(msg); - } + @Override + public void onReceive(Object msg) throws Exception { + if (msg instanceof String) { + String s = (String) msg; + System.out.println(s.toLowerCase()); + } else { + unhandled(msg); } } +} +``` -It's recommended to migrate ``UntypedActor`` to ``AbstractActor`` by implementing -``createReceive`` instead of ``onReceive``. +New: -Old:: +``` +import akka.actor.AbstractActor; - import akka.actor.UntypedActor; - - public class SomeActor extends UntypedActor { - - @Override - public void onReceive(Object msg) throws Exception { - if (msg instanceof String) { - String s = (String) msg; +public class SomeActor extends AbstractActor { + @Override + public Receive createReceive() { + return receiveBuilder() + .match(String.class, s -> { System.out.println(s.toLowerCase()); - } else { - unhandled(msg); - } - } + }) + .build(); } +} +``` -New:: +See @ref:[Receive messages](../java/actors.md#actors-receive-java) documentation for more advice about how to implement +`createReceive`. - import akka.actor.AbstractActor; +Similar with `UntypedActorWithStash`, `UntypedPersistentActor`, and +`UntypedPersistentActorWithAtLeastOnceDelivery`. - public class SomeActor extends AbstractActor { - @Override - public Receive createReceive() { - return receiveBuilder() - .match(String.class, s -> { - System.out.println(s.toLowerCase()); - }) - .build(); - } - } +## Actor (Scala) -See :ref:`actors-receive-java` documentation for more advice about how to implement -``createReceive``. - -Similar with ``UntypedActorWithStash``, ``UntypedPersistentActor``, and -``UntypedPersistentActorWithAtLeastOnceDelivery``. - -Actor (Scala) -============= - -Actor DSL deprecation ---------------------- +### Actor DSL deprecation Actor DSL is a rarely used feature and thus will be deprecated and removed. -Use plain ``system.actorOf`` instead of the DSL to create Actors if you have been using it. +Use plain `system.actorOf` instead of the DSL to create Actors if you have been using it. -ExtensionKey Deprecation ------------------------- +### ExtensionKey Deprecation -``ExtensionKey`` is a shortcut for writing :ref:`extending-akka-scala` but extensions created with it -cannot be used from Java and it does in fact not save many lines of code over directly implementing ``ExtensionId``. +`ExtensionKey` is a shortcut for writing @ref:[Akka Extensions](../scala/extending-akka.md) but extensions created with it +cannot be used from Java and it does in fact not save many lines of code over directly implementing `ExtensionId`. +Old: -Old:: +``` +object MyExtension extends ExtensionKey[MyExtension] +``` - object MyExtension extends ExtensionKey[MyExtension] +New: -New:: +``` +object MyExtension extends ExtensionId[MyExtension] with ExtensionIdProvider { - object MyExtension extends ExtensionId[MyExtension] with ExtensionIdProvider { + override def lookup = MyExtension - override def lookup = MyExtension + override def createExtension(system: ExtendedActorSystem): MyExtension = + new MyExtension(system) - override def createExtension(system: ExtendedActorSystem): MyExtension = - new MyExtension(system) + // needed to get the type right when used from Java + override def get(system: ActorSystem): MyExtension = super.get(system) +} +``` - // needed to get the type right when used from Java - override def get(system: ActorSystem): MyExtension = super.get(system) - } - -Actor Mailbox -============= +## Actor Mailbox Scala 2.12 is using the standard JDK8 ForkJoinPool, which may cause performance regression for some Actor messaging scenarios. Therefore we have embedded the ForkJoinPool from Scala 2.11 in -Akka. This breaks binary backwards compatibility for custom ``Mailbox`` implementations that were +Akka. This breaks binary backwards compatibility for custom `Mailbox` implementations that were compiled with Akka 2.4. -This is only a problem if you are using a library that provides a custom ``Mailbox`` implementation. +This is only a problem if you are using a library that provides a custom `Mailbox` implementation. The change is source compatible and such library should be recompiled and released for Akka 2.5. -Streams -======= +## Streams -Removal of StatefulStage, PushPullStage ---------------------------------------- +### Removal of StatefulStage, PushPullStage -``StatefulStage`` and ``PushPullStage`` were first introduced in Akka Streams 1.0, and later deprecated -and replaced by ``GraphStage`` in 2.0-M2. The ``GraphStage`` API has all features (and even more) as the +`StatefulStage` and `PushPullStage` were first introduced in Akka Streams 1.0, and later deprecated +and replaced by `GraphStage` in 2.0-M2. The `GraphStage` API has all features (and even more) as the previous APIs and is even nicer to use. -Please refer to the GraphStage documentation :ref:` for Scala ` or -the documentation :ref:`for Java `, for details on building custom GraphStages. +Please refer to the GraphStage documentation @ref:[ for Scala](../scala/stream/stream-customize.md#graphstage-scala) or +the documentation @ref:[for Java](../scala/stream/stream-customize.md#graphstage-scala), for details on building custom GraphStages. -``StatefulStage`` would be migrated to a simple ``GraphStage`` that contains some mutable state in its ``GraphStageLogic``, -and ``PushPullStage`` directly translate to graph stages. +`StatefulStage` would be migrated to a simple `GraphStage` that contains some mutable state in its `GraphStageLogic`, +and `PushPullStage` directly translate to graph stages. -Removal of ``Source.transform``, replaced by ``via`` ----------------------------------------------------- +### Removal of `Source.transform`, replaced by `via` -Along with the removal of ``Stage`` (as described above), the ``transform`` methods creating Flows/Sources/Sinks -from ``Stage`` have been removed. They are replaced by using ``GraphStage`` instances with ``via``, e.g.:: +Along with the removal of `Stage` (as described above), the `transform` methods creating Flows/Sources/Sinks +from `Stage` have been removed. They are replaced by using `GraphStage` instances with `via`, e.g.: - exampleFlow.transform(() => new MyStage()) +``` +exampleFlow.transform(() => new MyStage()) +``` -would now be:: +would now be: - myFlow.via(new MyGraphStage) +``` +myFlow.via(new MyGraphStage) +``` -as the ``GraphStage`` itself is a factory of logic instances. +as the `GraphStage` itself is a factory of logic instances. -SubFlow.zip and SubSource.zip now emit akka.japi.Pair instead of Scala's Pair ------------------------------------------------------------------------------ +### SubFlow.zip and SubSource.zip now emit akka.japi.Pair instead of Scala's Pair -The the Java API's ``zip`` operator on ``SubFlow`` and ``SubSource`` has been emiting -Scala's ``Pair`` (``Tuple2``) instead of ``akka.japi.Pair``. This is fixed in Akka 2.5 where it emits the proper +The the Java API's `zip` operator on `SubFlow` and `SubSource` has been emiting +Scala's `Pair` (`Tuple2`) instead of `akka.japi.Pair`. This is fixed in Akka 2.5 where it emits the proper Java DSl type. -Please note that the ``zip`` operator on ``Source`` and ``Flow`` has had the correct type, -this change only affects the ``Sub...`` versions of those classes. +Please note that the `zip` operator on `Source` and `Flow` has had the correct type, +this change only affects the `Sub...` versions of those classes. -Deprecation of ActorSubscriber and ActorPublisher -------------------------------------------------- +### Deprecation of ActorSubscriber and ActorPublisher -The classes ``ActorPublisher`` and ``ActorSubscriber`` were the first user-facing Reactive Streams integration +The classes `ActorPublisher` and `ActorSubscriber` were the first user-facing Reactive Streams integration API that we provided for end-users. Akka Streams APIs have evolved and improved a lot since then, and now there is no need to use these low-level abstractions anymore. It is easy to get things wrong when implementing them, and one would have to validate each implementation of such Actor using the Reactive Streams Technology Compatibility Kit. -The replacement API is the powerful ``GraphStage``. It has all features that raw Actors provided for implementing Stream +The replacement API is the powerful `GraphStage`. It has all features that raw Actors provided for implementing Stream stages and adds additional protocol and type-safety. You can learn all about it in the documentation: -:ref:`stream-customize-scala`and :ref:`Custom stream processing in JavaDSL `. +:ref:`stream-customize-scala`and @ref:[Custom stream processing in JavaDSL](../java/stream/stream-customize.md). -You should also read the blog post series on the official team blog, starting with `Mastering GraphStages, part I`_, +You should also read the blog post series on the official team blog, starting with [Mastering GraphStages, part I](http://blog.akka.io/streams/2016/07/30/mastering-graph-stage-part-1), which explains using and implementing GraphStages in more practical terms than the reference documentation. -Order of Attributes List ------------------------- +### Order of Attributes List -Imporant performance improvement could be achieved by reverting the order of the ``attributesList`` in ``Attributes``. +Imporant performance improvement could be achieved by reverting the order of the `attributesList` in `Attributes`. -The ``attributeList`` is ordered with the most specific attribute first, least specific last. +The `attributeList` is ordered with the most specific attribute first, least specific last. Note that the order was the opposite in Akka 2.4.x (but it was not really documented). -The semantics of the convenience methods, such as ``get`` and ``getFirst`` are the same, but if you use the ``attributesList`` -directly or via ``filtered`` or ``getAttributeList`` you need to take the new order into consideration. +The semantics of the convenience methods, such as `get` and `getFirst` are the same, but if you use the `attributesList` +directly or via `filtered` or `getAttributeList` you need to take the new order into consideration. -.. _Mastering GraphStages, part I: http://blog.akka.io/streams/2016/07/30/mastering-graph-stage-part-1 +### Dispatcher attribute -Dispatcher attribute --------------------- - -The ``ActorAttributes.dispatcher`` attribute is adding an async boundary in 2.5, since that is the typical desired behavior. -In 2.4 an explicit `async` marker (``AsyncBoundary`` attribute) had to be added. For example, this means that ``Source`` that -defined ``blocking-io-dispatcher`` as default followed by a ``map`` will now be separated by an async boundary, which was not +The `ActorAttributes.dispatcher` attribute is adding an async boundary in 2.5, since that is the typical desired behavior. +In 2.4 an explicit *async* marker (`AsyncBoundary` attribute) had to be added. For example, this means that `Source` that +defined `blocking-io-dispatcher` as default followed by a `map` will now be separated by an async boundary, which was not the case in 2.4. -Removal of the auto-fuse setting --------------------------------- +### Removal of the auto-fuse setting In 2.4 fusing stages together into the same actor could be completely disabled with the setting -``akka.stream.materializer.auto-fusing``. The new materializer introduced in Akka 2.5 does not support disabling fusing, +`akka.stream.materializer.auto-fusing`. The new materializer introduced in Akka 2.5 does not support disabling fusing, so this setting does not have any effect any more and has been deprecated. Running each stage in a stream on a separate actor can be done by adding explicit async boundaries around every stage. How to add asynchronous boundaries can be seen -in :ref:`operator-fusion-java` (Java) and :ref:`operator-fusion-scala` (Scala). +in @ref:[Operator Fusion](../java/stream/stream-flows-and-basics.md#operator-fusion-java) (Java) and @ref:[Operator Fusion](../scala/stream/stream-flows-and-basics.md#operator-fusion-scala) (Scala). -Remote -====== +## Remote -.. _mig25_mutual: - -Mutual TLS authentication now required by default for netty-based SSL transport -------------------------------------------------------------------------------- + +### Mutual TLS authentication now required by default for netty-based SSL transport Mutual TLS authentication is now required by default for the netty-based SSL transport. -Nodes that are configured with this setting to ``on`` might not be able to receive messages from nodes that run on older +Nodes that are configured with this setting to `on` might not be able to receive messages from nodes that run on older versions of akka-remote. This is because in versions of Akka < 2.4.12 the active side of the remoting connection will not send over certificates even if asked to. It is still possible to make a rolling upgrade from a version < 2.4.12 by doing the upgrade stepwise: - * first, upgrade Akka to the latest version but keep ``akka.remote.netty.ssl.require-mutual-authentication`` at ``off`` - and do a first rolling upgrade - * second, turn the setting to ``on`` and do another rolling upgrade +: + * first, upgrade Akka to the latest version but keep `akka.remote.netty.ssl.require-mutual-authentication` at `off` +and do a first rolling upgrade + * second, turn the setting to `on` and do another rolling upgrade -For more information see the documentation for the ``akka.remote.netty.ssl.require-mutual-authentication`` configuration setting -in :ref:`akka-remote's reference.conf `. -.. _mig25_addser: +For more information see the documentation for the `akka.remote.netty.ssl.require-mutual-authentication` configuration setting +in @ref:[akka-remote's reference.conf](../general/configuration.md#config-akka-remote). -additional-serialization-bindings ---------------------------------- + +### additional-serialization-bindings -From Akka 2.5.0 the ``additional-serialization-bindings`` are enabled by default. That defines +From Akka 2.5.0 the `additional-serialization-bindings` are enabled by default. That defines serializers that are replacing some Java serialization that were used in 2.4. This setting was disabled by default in Akka 2.4.16 but can also be enabled in an Akka 2.4 system. To still be able to support rolling upgrade from a system with this setting disabled, e.g. default for 2.4.16, it is possible to disable the additional serializers and continue using Java serialization for those messages. -.. code-block:: ruby - - akka.actor { - # Set this to off to disable serialization-bindings define in - # additional-serialization-bindings. That should only be needed - # for backwards compatibility reasons. - enable-additional-serialization-bindings = off - } +```ruby +akka.actor { + # Set this to off to disable serialization-bindings define in + # additional-serialization-bindings. That should only be needed + # for backwards compatibility reasons. + enable-additional-serialization-bindings = off +} +``` Please note that this setting must be the same on all nodes participating in a cluster, otherwise the mis-aligned serialization configurations will cause deserialization errors on the receiving nodes. -With serialize-messages the deserialized message is actually sent ------------------------------------------------------------------ +### With serialize-messages the deserialized message is actually sent -The flag ``akka.actor.serialize-message = on`` triggers serialization and deserialization of each message sent in the -``ActorSystem``. With this setting enabled the message actually passed on to the actor previously was the original +The flag `akka.actor.serialize-message = on` triggers serialization and deserialization of each message sent in the +`ActorSystem`. With this setting enabled the message actually passed on to the actor previously was the original message instance, this has now changed to be the deserialized message instance. This may cause tests that rely on messages being the same instance (for example by having mutable messages with attributes that are asserted in the tests) to not work any more with this setting enabled. For such cases the recommendation is to either not rely on messages being the same instance or turn the setting off. - -Wire Protocol Compatibility ---------------------------- +### Wire Protocol Compatibility It is possible to use Akka Remoting between nodes running Akka 2.4.16 and 2.5-M1, but some settings have changed so you might need -to adjust some configuration as described in :ref:`mig25_rolling`. +to adjust some configuration as described in [mig25_rolling](#mig25-rolling). Note however that if using Java serialization it will not be possible to mix nodes using Scala 2.11 and 2.12. -Cluster -======= +## Cluster -.. _mig25_rolling: - -Rolling Update --------------- + +### Rolling Update It is possible to do a rolling update from Akka 2.4.16 to 2.5-M1, i.e. running a cluster of 2.4.16 nodes and join nodes running 2.5-M1 followed by shutting down the old nodes. @@ -422,184 +417,171 @@ Some settings have changed default value in 2.5-M1 and therefore you need to rev before doing a rolling update to 2.5-M1. Such settings are mentioned elsewhere in this migration guide and here is a summary of things to consider. -* :ref:`mig25_addser` -* :ref:`mig25_weaklyup` -* :ref:`mig25_sharding_store` -* :ref:`mig25_mutual` + * [mig25_addser](#mig25-addser) + * [mig25_weaklyup](#mig25-weaklyup) + * [mig25_sharding_store](#mig25-sharding-store) + * [mig25_mutual](#mig25-mutual) -Coordinated Shutdown --------------------- +### Coordinated Shutdown -There is a new extension named ``CoordinatedShutdown`` that will stop certain actors and +There is a new extension named `CoordinatedShutdown` that will stop certain actors and services in a specific order and perform registered tasks during the shutdown process. When using Akka Cluster, tasks for graceful leaving of cluster including graceful shutdown of Cluster Singletons and Cluster Sharding are now performed automatically. -Previously it was documented that things like terminating the ``ActorSystem`` should be +Previously it was documented that things like terminating the `ActorSystem` should be done when the cluster member was removed, but this was very difficult to get right. That is now taken care of automatically. This might result in changed behavior, hopefully to the better. It might also be in conflict with your previous shutdown code so please read the documentation for the Coordinated Shutdown and revisit your own implementations. Most likely your implementation will not be needed any more or it can be simplified. -More information can be found in the :ref:`documentation for Scala ` or -:ref:`documentation for Java ` +More information can be found in the @ref:[documentation for Scala](../scala/actors.md#coordinated-shutdown-scala) or +@ref:[documentation for Java](../java/actors.md#coordinated-shutdown-java) -For some tests it might be undesired to terminate the ``ActorSystem`` via ``CoordinatedShutdown``. -You can disable that by adding the following to the configuration of the ``ActorSystem`` that is -used in the test:: +For some tests it might be undesired to terminate the `ActorSystem` via `CoordinatedShutdown`. +You can disable that by adding the following to the configuration of the `ActorSystem` that is +used in the test: - # Don't terminate ActorSystem via CoordinatedShutdown in tests - akka.coordinated-shutdown.terminate-actor-system = off - akka.coordinated-shutdown.run-by-jvm-shutdown-hook = off - akka.cluster.run-coordinated-shutdown-when-down = off +``` +# Don't terminate ActorSystem via CoordinatedShutdown in tests +akka.coordinated-shutdown.terminate-actor-system = off +akka.coordinated-shutdown.run-by-jvm-shutdown-hook = off +akka.cluster.run-coordinated-shutdown-when-down = off +``` -.. _mig25_weaklyup: + +### WeaklyUp -WeaklyUp --------- +@ref:[weakly_up_scala](../scala/cluster-usage.md#weakly-up-scala) is now enabled by default, but it can be disabled with configuration option: -:ref:`weakly_up_scala` is now enabled by default, but it can be disabled with configuration option:: - - akka.cluster.allow-weakly-up-members = off +``` +akka.cluster.allow-weakly-up-members = off +``` You should not run a cluster with this feature enabled on some nodes and disabled on some. Therefore you might need to enable/disable it in configuration when performing rolling upgrade from 2.4.x to 2.5.0. -.. _mig25_sharding_store: + +### Cluster Sharding state-store-mode -Cluster Sharding state-store-mode ---------------------------------- - -Distributed Data mode is now the default ``state-store-mode`` for Cluster Sharding. The persistence mode -is also supported. Read more in the documentation :ref:`for Scala ` or -the documentation :ref:`for Java `. +Distributed Data mode is now the default `state-store-mode` for Cluster Sharding. The persistence mode +is also supported. Read more in the documentation @ref:[for Scala](../scala/cluster-sharding.md#cluster-sharding-mode-scala) or +the documentation @ref:[for Java](../java/cluster-sharding.md#cluster-sharding-mode-java). It's important to use the same mode on all nodes in the cluster, i.e. if you perform a rolling upgrade -from 2.4.16 you might need to change the ``state-store-mode`` to be the same (``persistence`` is default -in 2.4.x):: +from 2.4.16 you might need to change the `state-store-mode` to be the same (`persistence` is default +in 2.4.x): - akka.cluster.sharding.state-store-mode = persistence +``` +akka.cluster.sharding.state-store-mode = persistence +``` -Note that the stored :ref:`cluster_sharding_remembering_java` data with ``persistence`` mode cannot -be migrated to the ``data`` mode. Such entities must be started again in some other way when using -``ddata`` mode. +Note that the stored @ref:[cluster_sharding_remembering_java](../java/cluster-sharding.md#cluster-sharding-remembering-java) data with `persistence` mode cannot +be migrated to the `data` mode. Such entities must be started again in some other way when using +`ddata` mode. -Cluster Management Command Line Tool ------------------------------------- +### Cluster Management Command Line Tool There is a new cluster management tool with HTTP API that has the same functionality as the command line tool. The HTTP API gives you access to cluster membership information as JSON including full reachability status between the nodes. It supports the ordinary cluster operations such as join, leave, and down. -See documentation of `akka/akka-cluster-management `_. +See documentation of [akka/akka-cluster-management](https://github.com/akka/akka-cluster-management). The command line script for cluster management has been deprecated and is scheduled for removal -in the next major version. Use the HTTP API with `curl `_ or similar instead. +in the next major version. Use the HTTP API with [curl](https://curl.haxx.se/) or similar instead. -Distributed Data -================ +## Distributed Data Distributed Data has been promoted to a stable module. This means that we will keep the API stable from this point. As a result -the module name is changed from `akka-distributed-data-experimental` to `akka-distributed-data` and you need to change that in your +the module name is changed from *akka-distributed-data-experimental* to *akka-distributed-data* and you need to change that in your build tool (sbt/mvn/...). -Map allow generic type for the keys ------------------------------------ +### Map allow generic type for the keys In 2.4 the key of any Distributed Data map always needed to be of type String. In 2.5 you can use any type for the key. This means that every map (ORMap, LWWMap, PNCounterMap, ORMultiMap) now takes an extra type parameter to specify the key type. To migrate -existing code from 2.4 to 2.5 you simple add String as key type, for example: `ORMultiMap[Foo]` becomes `ORMultiMap[String, Foo]`. -`PNCounterMap` didn't take a type parameter in version 2.4, so `PNCounterMap` in 2.4 becomes `PNCounterMap[String]` in 2.5. -Java developers should use `<>` instead of `[]`, e.g: `PNCounterMap`. +existing code from 2.4 to 2.5 you simple add String as key type, for example: *ORMultiMap[Foo]* becomes *ORMultiMap[String, Foo]*. +*PNCounterMap* didn't take a type parameter in version 2.4, so *PNCounterMap* in 2.4 becomes *PNCounterMap[String]* in 2.5. +Java developers should use *<>* instead of *[]*, e.g: *PNCounterMap*. **NOTE: Even though the interface is not compatible between 2.4 and 2.5, the binary protocol over the wire is (as long as you use String as key type). This means that 2.4 nodes can synchronize with 2.5 nodes.** -Subscribers ------------ +### Subscribers -When an entity is removed subscribers will not receive ``Replicator.DataDeleted`` any more. -They will receive ``Replicator.Deleted`` instead. +When an entity is removed subscribers will not receive `Replicator.DataDeleted` any more. +They will receive `Replicator.Deleted` instead. +## Persistence -Persistence -=========== +### Binary incompatibility of PersistentActor and AtLeastOneDelivery -Binary incompatibility of PersistentActor and AtLeastOneDelivery ----------------------------------------------------------------- - -To be able to evolve the Java APIs ``AbstractPersistentActor`` and ``AbstractPersistentActorWithAtLeastOnceDelivery`` +To be able to evolve the Java APIs `AbstractPersistentActor` and `AbstractPersistentActorWithAtLeastOnceDelivery` to work with Scala 2.12 we could find no other solution but to break the binary compatibility of the Scala versions (which the Java ones were based on). This means that the Akka 2.5 artifact cannot be a class path drop in replacement of Akka 2.4 if you use -``PersistentActor`` or ``AtLeastOnceDelivery``, to do this upgrade you _must_ recompile your project with the new +`PersistentActor` or `AtLeastOnceDelivery`, to do this upgrade you _must_ recompile your project with the new version of Akka. +### Removal of PersistentView -Removal of PersistentView -------------------------- +After being deprecated for a long time, and replaced by @ref:[Persistence Query Java](../java/persistence-query.md) +(@ref:[Persistence Query Scala](../scala/persistence-query.md)) `PersistentView` has been removed now removed. -After being deprecated for a long time, and replaced by :ref:`Persistence Query Java ` -(:ref:`Persistence Query Scala `) ``PersistentView`` has been removed now removed. +The corresponding query type is `EventsByPersistenceId`. There are several alternatives for connecting the `Source` +to an actor corresponding to a previous `PersistentView`. There are several alternatives for connecting the `Source` +to an actor corresponding to a previous `PersistentView` actor which are documented in @ref:[Integration](../scala/stream/stream-integrations.md) +for Scala and @ref:[Java](../java/stream/stream-integrations.md). -The corresponding query type is ``EventsByPersistenceId``. There are several alternatives for connecting the ``Source`` -to an actor corresponding to a previous ``PersistentView``. There are several alternatives for connecting the ``Source`` -to an actor corresponding to a previous ``PersistentView`` actor which are documented in :ref:`stream-integrations-scala` -for Scala and :ref:`Java `. +The consuming actor may be a plain `Actor` or an `PersistentActor` if it needs to store its own state (e.g. `fromSequenceNr` offset). -The consuming actor may be a plain ``Actor`` or an ``PersistentActor`` if it needs to store its own state (e.g. ``fromSequenceNr`` offset). +Please note that Persistence Query is not experimental/may-change anymore in Akka `2.5.0`, so you can safely upgrade to it. -Please note that Persistence Query is not experimental/may-change anymore in Akka ``2.5.0``, so you can safely upgrade to it. +### Persistence Plugin Proxy -Persistence Plugin Proxy ------------------------- +A new @ref:[persistence plugin proxy](../scala/persistence.md#persistence-plugin-proxy) was added, that allows sharing of an otherwise +non-sharable journal or snapshot store. The proxy is available by setting `akka.persistence.journal.plugin` or +`akka.persistence.snapshot-store.plugin` to `akka.persistence.journal.proxy` or `akka.persistence.snapshot-store.proxy`, +respectively. The proxy supplants the @ref:[Shared LevelDB journal](../scala/persistence.md#shared-leveldb-journal). -A new :ref:`persistence plugin proxy` was added, that allows sharing of an otherwise -non-sharable journal or snapshot store. The proxy is available by setting ``akka.persistence.journal.plugin`` or -``akka.persistence.snapshot-store.plugin`` to ``akka.persistence.journal.proxy`` or ``akka.persistence.snapshot-store.proxy``, -respectively. The proxy supplants the :ref:`Shared LevelDB journal`. +## Persistence Query -Persistence Query -================= - -Persistence Query has been promoted to a stable module. As a result the module name is changed from `akka-persistence-query-experimental` -to `akka-persistence-query` and you need to change that in your build tool (sbt/mvn/...). +Persistence Query has been promoted to a stable module. As a result the module name is changed from *akka-persistence-query-experimental* +to *akka-persistence-query* and you need to change that in your build tool (sbt/mvn/...). Only slight API changes were made since the module was introduced: -Query naming consistency improved ---------------------------------- +### Query naming consistency improved + Queries always fall into one of the two categories: infinite or finite ("current"). The naming convention for these categories of queries was solidified and is now as follows: -- "infinite" - e.g. ``eventsByTag``, ``persistenceIds`` - which will keep emitting events as they are persisted and match the query. -- "finite", also known as "current" - e.g. ``currentEventsByTag``, ``currentPersistenceIds`` - which will complete the stream once the query completed, - for the journal's definition of "current". For example in an SQL store it would mean it only queries the database once. + * "infinite" - e.g. `eventsByTag`, `persistenceIds` - which will keep emitting events as they are persisted and match the query. + * "finite", also known as "current" - e.g. `currentEventsByTag`, `currentPersistenceIds` - which will complete the stream once the query completed, +for the journal's definition of "current". For example in an SQL store it would mean it only queries the database once. -Only the ``AllPersistenceIdsQuery`` class and method name changed due to this. -The class is now called ``PersistenceIdsQuery``, and the method which used to be ``allPersistenceIds`` is now ``persistenceIds``. +Only the `AllPersistenceIdsQuery` class and method name changed due to this. +The class is now called `PersistenceIdsQuery`, and the method which used to be `allPersistenceIds` is now `persistenceIds`. -Queries now use ``Offset`` instead of ``Long`` for offsets ----------------------------------------------------------- +### Queries now use `Offset` instead of `Long` for offsets This change was made to better accomodate the various types of Journals and their understanding what an offset is. For example, in some journals an offset is always a time, while in others it is a numeric offset (like a sequence id). -Instead of the previous ``Long`` offset you can now use the provided ``Offset`` factories (and types): +Instead of the previous `Long` offset you can now use the provided `Offset` factories (and types): -- ``akka.persistence.query.Offset.sequence(value: Long)``, -- ``akka.persistence.query.Offset.timeBasedUUID(value: UUID)`` -- and finally ``NoOffset`` if not offset should be used. + * `akka.persistence.query.Offset.sequence(value: Long)`, + * `akka.persistence.query.Offset.timeBasedUUID(value: UUID)` + * and finally `NoOffset` if not offset should be used. -Journals are also free to provide their own specific ``Offset`` types. Consult your journal plugin's documentation for details. +Journals are also free to provide their own specific `Offset` types. Consult your journal plugin's documentation for details. -Agents -====== +## Agents -Agents are now deprecated -------------------------- +### Agents are now deprecated Akka Agents are a very simple way of containing mutable state and allowing to access it safely from multiple threads. The abstraction is leaky though, as Agents do not work over the network (unlike Akka Actors). @@ -610,10 +592,9 @@ We also anticipate to replace the uses of Agents by the upcoming Akka Typed, so If you use Agents and would like to take over the maintanance thereof, please contact the team on gitter or github. -Camel -===== +## Camel -``akka-camel`` has been deprecated in favour of `Alpakka `_ , +`akka-camel` has been deprecated in favour of [Alpakka](https://github.com/akka/alpakka) , the Akka Streams based collection of integrations to various endpoints (including Camel) We acknowledge that Akka Camel is a very useful and important module. It will not be removed until @@ -621,142 +602,136 @@ Alpakka has reached the needed production quality to be a full replacement. The Akka Camel should be seen as a signal that new development is to be invested in Alpakka instead of Akka Camel. -Contrib -======= +## Contrib -``akka-contrib`` has been deprecated and is scheduled for removal in the next major version. +`akka-contrib` has been deprecated and is scheduled for removal in the next major version. The reason is to reduce the amount of things to maintain in the core Akka projects. Contributions to the core of Akka or its satellite projects are welcome. Contributions that don't fit into existing modules can be hosted in new Akka Github repositories in the -``akka`` Github organization or outside of it depending on what kind of library it is. +`akka` Github organization or outside of it depending on what kind of library it is. Please ask. -Aggregator ----------- +### Aggregator -``Aggregator`` has been deprecated. Feel free to copy the source into your project or create a +`Aggregator` has been deprecated. Feel free to copy the source into your project or create a separate library outside of Akka. -CircuitBreakerProxy -------------------- +### CircuitBreakerProxy -``CircuitBreakerProxy`` has been deprecated in favor of ``akka.pattern.CircuitBreaker`` with explicit ``ask`` requests. +`CircuitBreakerProxy` has been deprecated in favor of `akka.pattern.CircuitBreaker` with explicit `ask` requests. -JavaLogger ----------- +### JavaLogger -``akka.contrib.jul.JavaLogger`` has been deprecated and included in ``akka-actor`` instead as -``akka.event.jul.JavaLogger``. See :ref:`documentation `. +`akka.contrib.jul.JavaLogger` has been deprecated and included in `akka-actor` instead as +`akka.event.jul.JavaLogger`. See @ref:[documentation](../scala/logging.md#jul-scala). -The ``JavaLoggingAdapter`` has also been deprecated, but not included in ``akka-actor``. +The `JavaLoggingAdapter` has also been deprecated, but not included in `akka-actor`. Feel free to copy the source into your project or create a separate library outside of Akka. -PeekMailbox ------------ +### PeekMailbox -``PeekMailbox`` has been deprecated. Use an explicit supervisor or proxy actor instead. +`PeekMailbox` has been deprecated. Use an explicit supervisor or proxy actor instead. -.. _migration-guide-TimerBasedThrottler: + +### ReceivePipeline -ReceivePipeline ---------------- - -``ReceivePipeline`` has been deprecated. Feel free to copy the source into your project or create +`ReceivePipeline` has been deprecated. Feel free to copy the source into your project or create a separate library outside of Akka. -ReliableProxy -------------- +### ReliableProxy -``ReliableProxy`` has been deprecated. Use :ref:`at-least-once-delivery-scala` instead. ``ReliableProxy`` +`ReliableProxy` has been deprecated. Use @ref:[At-Least-Once Delivery](../scala/persistence.md#at-least-once-delivery-scala) instead. `ReliableProxy` was only intended as an example and doesn't have full production quality. If there is demand for a lightweight (non-durable) at-least once delivery mechanism we are open for a design discussion. -TimerBasedThrottler -------------------- +### TimerBasedThrottler -``TimerBasedThrottler`` has been deprecated. Use the ``throttle`` stage in Akka Streams instead. +`TimerBasedThrottler` has been deprecated. Use the `throttle` stage in Akka Streams instead. -Example in Scala:: +Example in Scala: - import scala.concurrent.duration._ - import akka.NotUsed - import akka.actor.ActorRef - import akka.actor.ActorSystem - import akka.stream.ActorMaterializer - import akka.stream.OverflowStrategy - import akka.stream.ThrottleMode - import akka.stream.scaladsl.Sink - import akka.stream.scaladsl.Source - - val system: ActorSystem = ??? // TODO real ActorSystem here - val target: ActorRef = ??? // TODO real target ActorRef here - implicit val materializer = ActorMaterializer.create(system) - - val throttler: ActorRef = - Source.actorRef(bufferSize = 1000, OverflowStrategy.dropNew) - .throttle(100, 1.second, 10, ThrottleMode.Shaping) - .to(Sink.actorRef(target, NotUsed)) - .run() +``` +import scala.concurrent.duration._ +import akka.NotUsed +import akka.actor.ActorRef +import akka.actor.ActorSystem +import akka.stream.ActorMaterializer +import akka.stream.OverflowStrategy +import akka.stream.ThrottleMode +import akka.stream.scaladsl.Sink +import akka.stream.scaladsl.Source -Example in Java:: +val system: ActorSystem = ??? // TODO real ActorSystem here +val target: ActorRef = ??? // TODO real target ActorRef here +implicit val materializer = ActorMaterializer.create(system) - import java.util.concurrent.TimeUnit; - import scala.concurrent.duration.FiniteDuration; - import akka.NotUsed; - import akka.actor.ActorRef; - import akka.actor.ActorSystem; - import akka.stream.ActorMaterializer; - import akka.stream.Materializer; - import akka.stream.OverflowStrategy; - import akka.stream.ThrottleMode; - import akka.stream.javadsl.Sink; - import akka.stream.javadsl.Source; - - final ActorSystem system = null; // TODO real ActorSystem here - final ActorRef target = null; // TODO real target ActorRef here - final Materializer materializer = ActorMaterializer.create(system); +val throttler: ActorRef = + Source.actorRef(bufferSize = 1000, OverflowStrategy.dropNew) + .throttle(100, 1.second, 10, ThrottleMode.Shaping) + .to(Sink.actorRef(target, NotUsed)) + .run() +``` - final ActorRef throttler = - Source.actorRef(1000, OverflowStrategy.dropNew()) - .throttle(100, FiniteDuration.create(1, TimeUnit.SECONDS), 10, ThrottleMode.shaping()) - .to(Sink.actorRef(target, NotUsed.getInstance())) - .run(materializer); +Example in Java: -Akka Typed -========== +``` +import java.util.concurrent.TimeUnit; +import scala.concurrent.duration.FiniteDuration; +import akka.NotUsed; +import akka.actor.ActorRef; +import akka.actor.ActorSystem; +import akka.stream.ActorMaterializer; +import akka.stream.Materializer; +import akka.stream.OverflowStrategy; +import akka.stream.ThrottleMode; +import akka.stream.javadsl.Sink; +import akka.stream.javadsl.Source; -With the new term :ref:`may change ` we will no longer have a different artifact for modules that are not -stable, and ``akka-typed-experimental`` has therefore been renamed to ``akka-typed``. Note that it is still not +final ActorSystem system = null; // TODO real ActorSystem here +final ActorRef target = null; // TODO real target ActorRef here +final Materializer materializer = ActorMaterializer.create(system); + +final ActorRef throttler = + Source.actorRef(1000, OverflowStrategy.dropNew()) + .throttle(100, FiniteDuration.create(1, TimeUnit.SECONDS), 10, ThrottleMode.shaping()) + .to(Sink.actorRef(target, NotUsed.getInstance())) + .run(materializer); +``` + +## Akka Typed + +With the new term @ref:[may change](../common/may-change.md) we will no longer have a different artifact for modules that are not +stable, and `akka-typed-experimental` has therefore been renamed to `akka-typed`. Note that it is still not promoted to a stable module. -Experimental modules -==================== +## Experimental modules We have previously marked modules that we did not want to freeze the APIs of a **experimental**, such modules will -instead be marked as :ref:`may change ` from now on. +instead be marked as @ref:[may change](../common/may-change.md) from now on. -Testkit -======= +## Testkit -``JavaTestKit`` has been deprecated since it does all kinds of tricks to achieve what Scala testkit does. -Use ``akka.testkit.javadsl.TestKit`` instead which introduces nicer APIs. +`JavaTestKit` has been deprecated since it does all kinds of tricks to achieve what Scala testkit does. +Use `akka.testkit.javadsl.TestKit` instead which introduces nicer APIs. -Old:: +Old: - new JavaTestKit(system) {{ - final JavaTestKit probe = new JavaTestKit(system); - new Within(duration("1 second")) { - public void run() { - probe.expectMsgEquals("hello"); - } - }; - }}; +``` +new JavaTestKit(system) {{ + final JavaTestKit probe = new JavaTestKit(system); + new Within(duration("1 second")) { + public void run() { + probe.expectMsgEquals("hello"); + } + }; +}}; +``` +New: - -New:: - - new TestKit(system) {{ - final TestKit probe = new TestKit(system); - within(duration("1 second"), () -> probe.expectMsgEquals("hello")); - }}; +``` +new TestKit(system) {{ + final TestKit probe = new TestKit(system); + within(duration("1 second"), () -> probe.expectMsgEquals("hello")); +}}; +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-eventsourced-2.3.x.md b/akka-docs/src/main/paradox/project/migration-guide-eventsourced-2.3.x.md index e5a923cdeb..9acd96a3d6 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-eventsourced-2.3.x.md +++ b/akka-docs/src/main/paradox/project/migration-guide-eventsourced-2.3.x.md @@ -1,9 +1,4 @@ -.. _migration-eventsourced-2.3: - -###################################################### -Migration Guide Eventsourced to Akka Persistence 2.3.x -###################################################### +# Migration Guide Eventsourced to Akka Persistence 2.3.x Migration from Eventsourced to Akka Persistence 2.3.x is described in the -`documentation of 2.4 `_. - +[documentation of 2.4](http://doc.akka.io/docs/akka/2.4/project/migration-guide-eventsourced-2.3.x.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-persistence-experimental-2.3.x-2.4.x.md b/akka-docs/src/main/paradox/project/migration-guide-persistence-experimental-2.3.x-2.4.x.md index 45034ec502..f6ccc30aa0 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-persistence-experimental-2.3.x-2.4.x.md +++ b/akka-docs/src/main/paradox/project/migration-guide-persistence-experimental-2.3.x-2.4.x.md @@ -1,8 +1,4 @@ -.. _migration-guide-persistence-experimental-2.3.x-2.4.x: - -########################################################################## -Migration Guide Akka Persistence (experimental) 2.3.3 to 2.3.4 (and 2.4.x) -########################################################################## +# Migration Guide Akka Persistence (experimental) 2.3.3 to 2.3.4 (and 2.4.x) Migration from Akka Persistence 2.3.3 to 2.3.4 (and 2.4.x) is described in the -`documentation of 2.4 `_. +[documentation of 2.4](http://doc.akka.io/docs/akka/2.4/project/migration-guide-persistence-experimental-2.3.x-2.4.x.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-stream-1.0-2.x.md b/akka-docs/src/main/paradox/project/migration-guide-stream-1.0-2.x.md index 47d833b7cb..404da03dae 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-stream-1.0-2.x.md +++ b/akka-docs/src/main/paradox/project/migration-guide-stream-1.0-2.x.md @@ -1,12 +1,4 @@ -.. _migration-2.0-scala: +# Migration Guide Akka Streams 1.0 to 2.x -####################################### -Migration Guide Akka Streams 1.0 to 2.x -####################################### - -For this migration guide see `the Scala documentation for Akka Streams 2.0`_ -or `the Java documentation for Akka Streams 2.0`_. - -.. _`the Scala documentation for Akka Streams 2.0`: http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0.2/scala/migration-guide-1.0-2.x-scala.html - -.. _`the Java documentation for Akka Streams 2.0`: http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0.2/java/migration-guide-1.0-2.x-java.html +For this migration guide see [the Scala documentation for Akka Streams 2.0](http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0.2/scala/migration-guide-1.0-2.x-scala.html) +or [the Java documentation for Akka Streams 2.0](http://doc.akka.io/docs/akka-stream-and-http-experimental/2.0.2/java/migration-guide-1.0-2.x-java.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guide-stream-2.0-2.4.md b/akka-docs/src/main/paradox/project/migration-guide-stream-2.0-2.4.md index bd7029b5ad..b54d2f6fe1 100644 --- a/akka-docs/src/main/paradox/project/migration-guide-stream-2.0-2.4.md +++ b/akka-docs/src/main/paradox/project/migration-guide-stream-2.0-2.4.md @@ -1,9 +1,5 @@ -.. _migration-streams-2.0-2.4-scala: - -########################################### -Migration Guide Akka Streams 2.0.x to 2.4.x -########################################### +# Migration Guide Akka Streams 2.0.x to 2.4.x Migration from Akka Streams 2.0.x to 2.4.x is described in the -`Scala documentation of 2.4 `_ or -`Java documentation of 2.4 `_.. +[Scala documentation of 2.4](http://doc.akka.io/docs/akka/2.4/scala/stream/migration-guide-2.0-2.4-scala.html) or +[Java documentation of 2.4](http://doc.akka.io/docs/akka/2.4/java/stream/migration-guide-2.0-2.4-java.html).. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/migration-guides.md b/akka-docs/src/main/paradox/project/migration-guides.md index 6bd071e562..ee81bc0e39 100644 --- a/akka-docs/src/main/paradox/project/migration-guides.md +++ b/akka-docs/src/main/paradox/project/migration-guides.md @@ -1,18 +1,18 @@ -.. _migration: +# Migration Guides -Migration Guides -================ +@@toc { depth=1 } -.. toctree:: - :maxdepth: 1 +@@@ index - migration-guide-1.3.x-2.0.x - migration-guide-2.0.x-2.1.x - migration-guide-2.1.x-2.2.x - migration-guide-2.2.x-2.3.x - migration-guide-persistence-experimental-2.3.x-2.4.x - migration-guide-eventsourced-2.3.x - migration-guide-2.3.x-2.4.x - migration-guide-stream-1.0-2.x - migration-guide-stream-2.0-2.4 - migration-guide-2.4.x-2.5.x +* [migration-guide-1.3.x-2.0.x](migration-guide-1.3.x-2.0.x.md) +* [migration-guide-2.0.x-2.1.x](migration-guide-2.0.x-2.1.x.md) +* [migration-guide-2.1.x-2.2.x](migration-guide-2.1.x-2.2.x.md) +* [migration-guide-2.2.x-2.3.x](migration-guide-2.2.x-2.3.x.md) +* [migration-guide-persistence-experimental-2.3.x-2.4.x](migration-guide-persistence-experimental-2.3.x-2.4.x.md) +* [migration-guide-eventsourced-2.3.x](migration-guide-eventsourced-2.3.x.md) +* [migration-guide-2.3.x-2.4.x](migration-guide-2.3.x-2.4.x.md) +* [migration-guide-stream-1.0-2.x](migration-guide-stream-1.0-2.x.md) +* [migration-guide-stream-2.0-2.4](migration-guide-stream-2.0-2.4.md) +* [migration-guide-2.4.x-2.5.x](migration-guide-2.4.x-2.5.x.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/project/sponsors.md b/akka-docs/src/main/paradox/project/sponsors.md index 17c03925aa..c3480c13d7 100644 --- a/akka-docs/src/main/paradox/project/sponsors.md +++ b/akka-docs/src/main/paradox/project/sponsors.md @@ -1,13 +1,9 @@ -.. _sponsors: +# Sponsors -Sponsors -======== - -Lightbend ---------- +## Lightbend Lightbend is the company behind the Akka Project, Scala Programming Language, Play Web Framework, Scala IDE, sbt and many other open source projects. It also provides the Lightbend Stack, a full-featured development stack consisting of Akka, Play and Scala. Learn more at -`lightbend.com `_. +[lightbend.com](http://www.lightbend.com). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/reference/index.md b/akka-docs/src/main/paradox/reference/index.md deleted file mode 100644 index 59dc0e41cd..0000000000 --- a/akka-docs/src/main/paradox/reference/index.md +++ /dev/null @@ -1,5 +0,0 @@ -# Reference - -@@@ index - -@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala.md b/akka-docs/src/main/paradox/scala.md index 1366d91fa4..919a7d3e34 100644 --- a/akka-docs/src/main/paradox/scala.md +++ b/akka-docs/src/main/paradox/scala.md @@ -1,22 +1,21 @@ -.. _scala-api: +# Scala Documentation -Scala Documentation -=================== +@@toc { depth=2 } -.. toctree:: - :maxdepth: 2 +@@@ index - security/index - intro/index-scala - general/index - scala/index-actors - scala/index-futures - scala/index-network - scala/index-utilities - scala/stream/index - scala/http/index - scala/howto - dev/index - project/index - additional/index +* [security/index](security/index.md) +* [intro/index-scala](intro/index-scala.md) +* [general/index](general/index.md) +* [scala/index-actors](scala/index-actors.md) +* [scala/index-futures](scala/index-futures.md) +* [scala/index-network](scala/index-network.md) +* [scala/index-utilities](scala/index-utilities.md) +* [scala/stream/index](scala/stream/index.md) +* [scala/http/index](scala/http/index.md) +* [scala/howto](scala/howto.md) +* [dev/index](dev/index.md) +* [project/index](project/index.md) +* [additional/index](additional/index.md) +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/actordsl.md b/akka-docs/src/main/paradox/scala/actordsl.md index 4fd277afe2..10cdb83461 100644 --- a/akka-docs/src/main/paradox/scala/actordsl.md +++ b/akka-docs/src/main/paradox/scala/actordsl.md @@ -1,83 +1,82 @@ -.. _actordsl-scala: +# Actor DSL -Actor DSL -######### +@@@ warning -.. warning:: - Actor DSL is deprecated and will be removed in the near future. - Use plain ``system.actorOf`` or ``context.actorOf`` instead. +Actor DSL is deprecated and will be removed in the near future. +Use plain `system.actorOf` or `context.actorOf` instead. -The Actor DSL -============= +@@@ + +## The Actor DSL Simple actors—for example one-off workers or even when trying things out in the -REPL—can be created more concisely using the :class:`Act` trait. The supporting +REPL—can be created more concisely using the `Act` trait. The supporting infrastructure is bundled in the following import: -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#import +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #import } This import is assumed for all code samples throughout this section. The -implicit actor system serves as :class:`ActorRefFactory` for all examples +implicit actor system serves as `ActorRefFactory` for all examples below. To define a simple actor, the following is sufficient: -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#simple-actor +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #simple-actor } -Here, :meth:`actor` takes the role of either ``system.actorOf`` or -``context.actorOf``, depending on which context it is called in: it takes an -implicit :class:`ActorRefFactory`, which within an actor is available in the -form of the ``implicit val context: ActorContext``. Outside of an actor, you’ll -have to either declare an implicit :class:`ActorSystem`, or you can give the +Here, `actor` takes the role of either `system.actorOf` or +`context.actorOf`, depending on which context it is called in: it takes an +implicit `ActorRefFactory`, which within an actor is available in the +form of the `implicit val context: ActorContext`. Outside of an actor, you’ll +have to either declare an implicit `ActorSystem`, or you can give the factory explicitly (see further below). -The two possible ways of issuing a ``context.become`` (replacing or adding the +The two possible ways of issuing a `context.become` (replacing or adding the new behavior) are offered separately to enable a clutter-free notation of nested receives: -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#becomeStacked +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #becomeStacked } -Please note that calling ``unbecome`` more often than ``becomeStacked`` results -in the original behavior being installed, which in case of the :class:`Act` -trait is the empty behavior (the outer ``become`` just replaces it during +Please note that calling `unbecome` more often than `becomeStacked` results +in the original behavior being installed, which in case of the `Act` +trait is the empty behavior (the outer `become` just replaces it during construction). -Life-cycle management ---------------------- +### Life-cycle management -Life-cycle hooks are also exposed as DSL elements (see :ref:`start-hook-scala` and :ref:`stop-hook-scala`), where later invocations of the methods shown below will replace the contents of the respective hooks: +Life-cycle hooks are also exposed as DSL elements (see @ref:[Start Hook](actors.md#start-hook-scala) and @ref:[Stop Hook](actors.md#stop-hook-scala)), where later invocations of the methods shown below will replace the contents of the respective hooks: -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#simple-start-stop +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #simple-start-stop } The above is enough if the logical life-cycle of the actor matches the restart -cycles (i.e. ``whenStopping`` is executed before a restart and ``whenStarting`` -afterwards). If that is not desired, use the following two hooks (see :ref:`restart-hook-scala`): +cycles (i.e. `whenStopping` is executed before a restart and `whenStarting` +afterwards). If that is not desired, use the following two hooks (see @ref:[Restart Hooks](actors.md#restart-hook-scala)): -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#failing-actor +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #failing-actor } It is also possible to create nested actors, i.e. grand-children, like this: -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#nested-actor +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #nested-actor } -.. note:: +@@@ note - In some cases it will be necessary to explicitly pass the - :class:`ActorRefFactory` to the :meth:`actor()` method (you will notice when - the compiler tells you about ambiguous implicits). +In some cases it will be necessary to explicitly pass the +`ActorRefFactory` to the `actor()` method (you will notice when +the compiler tells you about ambiguous implicits). + +@@@ The grand-child will be supervised by the child; the supervisor strategy for this relationship can also be configured using a DSL element (supervision -directives are part of the :class:`Act` trait): +directives are part of the `Act` trait): -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#supervise-with +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #supervise-with } -Actor with :class:`Stash` -------------------------- +### Actor with `Stash` Last but not least there is a little bit of convenience magic built-in, which detects if the runtime class of the statically given actor subtype extends the -:class:`RequiresMessageQueue` trait via the :class:`Stash` trait (this is a -complicated way of saying that ``new Act with Stash`` would not work because its -runtime erased type is just an anonymous subtype of ``Act``). The purpose is to -automatically use the appropriate deque-based mailbox type required by :class:`Stash`. -If you want to use this magic, simply extend :class:`ActWithStash`: +`RequiresMessageQueue` trait via the `Stash` trait (this is a +complicated way of saying that `new Act with Stash` would not work because its +runtime erased type is just an anonymous subtype of `Act`). The purpose is to +automatically use the appropriate deque-based mailbox type required by `Stash`. +If you want to use this magic, simply extend `ActWithStash`: -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#act-with-stash +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #act-with-stash } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/actors.md b/akka-docs/src/main/paradox/scala/actors.md index 876f8af5f9..115d87a654 100644 --- a/akka-docs/src/main/paradox/scala/actors.md +++ b/akka-docs/src/main/paradox/scala/actors.md @@ -1,11 +1,6 @@ -.. _actors-scala: +# Actors -################ - Actors -################ - - -The `Actor Model`_ provides a higher level of abstraction for writing concurrent +The [Actor Model](http://en.wikipedia.org/wiki/Actor_model) provides a higher level of abstraction for writing concurrent and distributed systems. It alleviates the developer from having to deal with explicit locking and thread management, making it easier to write correct concurrent and parallel systems. Actors were defined in the 1973 paper by Carl @@ -16,176 +11,171 @@ systems. The API of Akka’s Actors is similar to Scala Actors which has borrowed some of its syntax from Erlang. -.. _Actor Model: http://en.wikipedia.org/wiki/Actor_model +## Creating Actors +@@@ note -Creating Actors -=============== +Since Akka enforces parental supervision every actor is supervised and +(potentially) the supervisor of its children, it is advisable that you +familiarize yourself with @ref:[Actor Systems](../general/actor-systems.md) and supervision and it +may also help to read @ref:[Actor References, Paths and Addresses](../general/addressing.md). -.. note:: +@@@ - Since Akka enforces parental supervision every actor is supervised and - (potentially) the supervisor of its children, it is advisable that you - familiarize yourself with :ref:`actor-systems` and :ref:`supervision` and it - may also help to read :ref:`addressing`. +### Defining an Actor class -Defining an Actor class ------------------------ - -Actors are implemented by extending the :class:`Actor` base trait and implementing the -:meth:`receive` method. The :meth:`receive` method should define a series of case -statements (which has the type ``PartialFunction[Any, Unit]``) that defines +Actors are implemented by extending the `Actor` base trait and implementing the +`receive` method. The `receive` method should define a series of case +statements (which has the type `PartialFunction[Any, Unit]`) that defines which messages your Actor can handle, using standard Scala pattern matching, along with the implementation of how the messages should be processed. Here is an example: -.. includecode:: code/docs/actor/ActorDocSpec.scala - :include: imports1,my-actor +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #imports1 #my-actor } -Please note that the Akka Actor ``receive`` message loop is exhaustive, which +Please note that the Akka Actor `receive` message loop is exhaustive, which is different compared to Erlang and the late Scala Actors. This means that you need to provide a pattern match for all messages that it can accept and if you want to be able to handle unknown messages then you need to have a default case -as in the example above. Otherwise an ``akka.actor.UnhandledMessage(message, -sender, recipient)`` will be published to the ``ActorSystem``'s -``EventStream``. +as in the example above. Otherwise an `akka.actor.UnhandledMessage(message, +sender, recipient)` will be published to the `ActorSystem`'s +`EventStream`. -Note further that the return type of the behavior defined above is ``Unit``; if +Note further that the return type of the behavior defined above is `Unit`; if the actor shall reply to the received message then this must be done explicitly as explained below. -The result of the :meth:`receive` method is a partial function object, which is -stored within the actor as its “initial behavior”, see `Become/Unbecome`_ for +The result of the `receive` method is a partial function object, which is +stored within the actor as its “initial behavior”, see [Become/Unbecome](#become-unbecome) for further information on changing the behavior of an actor after its construction. -Props ------ +### Props -:class:`Props` is a configuration class to specify options for the creation +`Props` is a configuration class to specify options for the creation of actors, think of it as an immutable and thus freely shareable recipe for creating an actor including associated deployment information (e.g. which dispatcher to use, see more below). Here are some examples of how to create a -:class:`Props` instance. +`Props` instance. -.. includecode:: code/docs/actor/ActorDocSpec.scala#creating-props +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #creating-props } The second variant shows how to pass constructor arguments to the -:class:`Actor` being created, but it should only be used outside of actors as +`Actor` being created, but it should only be used outside of actors as explained below. The last line shows a possibility to pass constructor arguments regardless of the context it is being used in. The presence of a matching constructor is -verified during construction of the :class:`Props` object, resulting in an -:class:`IllegalArgumentException` if no or multiple matching constructors are +verified during construction of the `Props` object, resulting in an +`IllegalArgumentException` if no or multiple matching constructors are found. -.. note:: +@@@ note - The recommended approach to create the actor :class:`Props` is not supported - for cases when the actor constructor takes value classes as arguments. +The recommended approach to create the actor `Props` is not supported +for cases when the actor constructor takes value classes as arguments. -Dangerous Variants -^^^^^^^^^^^^^^^^^^ +@@@ -.. includecode:: code/docs/actor/ActorDocSpec.scala#creating-props-deprecated +#### Dangerous Variants + +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #creating-props-deprecated } This method is not recommended to be used within another actor because it encourages to close over the enclosing scope, resulting in non-serializable -:class:`Props` and possibly race conditions (breaking the actor encapsulation). -On the other hand using this variant in a :class:`Props` factory in +`Props` and possibly race conditions (breaking the actor encapsulation). +On the other hand using this variant in a `Props` factory in the actor’s companion object as documented under “Recommended Practices” below is completely fine. There were two use-cases for these methods: passing constructor arguments to the actor—which is solved by the newly introduced -:meth:`Props.apply(clazz, args)` method above or the recommended practice +`Props.apply(clazz, args)` method above or the recommended practice below—and creating actors “on the spot” as anonymous classes. The latter should be solved by making these actors named classes instead (if they are not -declared within a top-level ``object`` then the enclosing instance’s ``this`` +declared within a top-level `object` then the enclosing instance’s `this` reference needs to be passed as the first argument). -.. warning:: +@@@ warning - Declaring one actor within another is very dangerous and breaks actor - encapsulation. Never pass an actor’s ``this`` reference into :class:`Props`! +Declaring one actor within another is very dangerous and breaks actor +encapsulation. Never pass an actor’s `this` reference into `Props`! -Edge cases -^^^^^^^^^^ -There are two edge cases in actor creation with :class:`Props`: +@@@ -* An actor with :class:`AnyVal` arguments. +#### Edge cases -.. includecode:: code/docs/actor/PropsEdgeCaseSpec.scala#props-edge-cases-value-class -.. includecode:: code/docs/actor/PropsEdgeCaseSpec.scala#props-edge-cases-value-class-example +There are two edge cases in actor creation with `Props`: -* An actor with default constructor values. + * An actor with `AnyVal` arguments. -.. includecode:: code/docs/actor/PropsEdgeCaseSpec.scala#props-edge-cases-default-values +@@snip [PropsEdgeCaseSpec.scala](code/docs/actor/PropsEdgeCaseSpec.scala) { #props-edge-cases-value-class } -In both cases an :class:`IllegalArgumentException` will be thrown stating +@@snip [PropsEdgeCaseSpec.scala](code/docs/actor/PropsEdgeCaseSpec.scala) { #props-edge-cases-value-class-example } + + * An actor with default constructor values. + +@@snip [PropsEdgeCaseSpec.scala](code/docs/actor/PropsEdgeCaseSpec.scala) { #props-edge-cases-default-values } + +In both cases an `IllegalArgumentException` will be thrown stating no matching constructor could be found. -The next section explains the recommended ways to create :class:`Actor` props in a way, +The next section explains the recommended ways to create `Actor` props in a way, which simultaneously safe-guards against these edge cases. -Recommended Practices -^^^^^^^^^^^^^^^^^^^^^ +#### Recommended Practices It is a good idea to provide factory methods on the companion object of each -:class:`Actor` which help keeping the creation of suitable :class:`Props` as +`Actor` which help keeping the creation of suitable `Props` as close to the actor definition as possible. This also avoids the pitfalls -associated with using the ``Props.apply(...)`` method which takes a by-name +associated with using the `Props.apply(...)` method which takes a by-name argument, since within a companion object the given code block will not retain a reference to its enclosing scope: -.. includecode:: code/docs/actor/ActorDocSpec.scala#props-factory +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #props-factory } Another good practice is to declare what messages an Actor can receive in the companion object of the Actor, which makes easier to know what it can receive: -.. includecode:: code/docs/actor/ActorDocSpec.scala#messages-in-companion +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #messages-in-companion } -Creating Actors with Props --------------------------- +### Creating Actors with Props -Actors are created by passing a :class:`Props` instance into the -:meth:`actorOf` factory method which is available on :class:`ActorSystem` and -:class:`ActorContext`. +Actors are created by passing a `Props` instance into the +`actorOf` factory method which is available on `ActorSystem` and +`ActorContext`. -.. includecode:: code/docs/actor/ActorDocSpec.scala#system-actorOf +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #system-actorOf } -Using the :class:`ActorSystem` will create top-level actors, supervised by the +Using the `ActorSystem` will create top-level actors, supervised by the actor system’s provided guardian actor, while using an actor’s context will create a child actor. -.. includecode:: code/docs/actor/ActorDocSpec.scala#context-actorOf - :exclude: plus-some-behavior +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #context-actorOf } It is recommended to create a hierarchy of children, grand-children and so on such that it fits the logical failure-handling structure of the application, -see :ref:`actor-systems`. +see @ref:[Actor Systems](../general/actor-systems.md). -The call to :meth:`actorOf` returns an instance of :class:`ActorRef`. This is a +The call to `actorOf` returns an instance of `ActorRef`. This is a handle to the actor instance and the only way to interact with it. The -:class:`ActorRef` is immutable and has a one to one relationship with the Actor -it represents. The :class:`ActorRef` is also serializable and network-aware. +`ActorRef` is immutable and has a one to one relationship with the Actor +it represents. The `ActorRef` is also serializable and network-aware. This means that you can serialize it, send it over the wire and use it on a remote host and it will still be representing the same Actor on the original node, across the network. The name parameter is optional, but you should preferably name your actors, since that is used in log messages and for identifying actors. The name must -not be empty or start with ``$``, but it may contain URL encoded characters -(eg. ``%20`` for a blank space). If the given name is already in use by -another child to the same parent an :class:`InvalidActorNameException` is thrown. +not be empty or start with `$`, but it may contain URL encoded characters +(eg. `%20` for a blank space). If the given name is already in use by +another child to the same parent an `InvalidActorNameException` is thrown. Actors are automatically started asynchronously when created. -Value classes as constructor arguments -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Value classes as constructor arguments The recommended way to instantiate actor props uses reflection at runtime to determine the correct actor constructor to be invoked and due to technical @@ -194,171 +184,159 @@ value classes. In these cases you should either unpack the arguments or create the props by calling the constructor manually: -.. includecode:: code/docs/actor/ActorDocSpec.scala#actor-with-value-class-argument +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #actor-with-value-class-argument } -Dependency Injection --------------------- +### Dependency Injection If your Actor has a constructor that takes parameters then those need to -be part of the :class:`Props` as well, as described `above`__. But there +be part of the `Props` as well, as described [above](Props_). But there are cases when a factory method must be used, for example when the actual constructor arguments are determined by a dependency injection framework. -__ Props_ +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #creating-indirectly } -.. includecode:: code/docs/actor/ActorDocSpec.scala - :include: creating-indirectly - :exclude: obtain-fresh-Actor-instance-from-DI-framework +@@@ warning -.. warning:: +You might be tempted at times to offer an `IndirectActorProducer` +which always returns the same instance, e.g. by using a `lazy val`. This is +not supported, as it goes against the meaning of an actor restart, which is +described here: @ref:[What Restarting Means](../general/supervision.md#supervision-restart). - You might be tempted at times to offer an :class:`IndirectActorProducer` - which always returns the same instance, e.g. by using a ``lazy val``. This is - not supported, as it goes against the meaning of an actor restart, which is - described here: :ref:`supervision-restart`. +When using a dependency injection framework, actor beans *MUST NOT* have +singleton scope. - When using a dependency injection framework, actor beans *MUST NOT* have - singleton scope. +@@@ Techniques for dependency injection and integration with dependency injection frameworks are described in more depth in the -`Using Akka with Dependency Injection `_ -guideline and the `Akka Java Spring `_ tutorial. +[Using Akka with Dependency Injection](http://letitcrash.com/post/55958814293/akka-dependency-injection) +guideline and the [Akka Java Spring](https://github.com/typesafehub/activator-akka-java-spring) tutorial. -The Inbox ---------- +### The Inbox When writing code outside of actors which shall communicate with actors, the -``ask`` pattern can be a solution (see below), but there are two things it -cannot do: receiving multiple replies (e.g. by subscribing an :class:`ActorRef` +`ask` pattern can be a solution (see below), but there are two things it +cannot do: receiving multiple replies (e.g. by subscribing an `ActorRef` to a notification service) and watching other actors’ lifecycle. For these -purposes there is the :class:`Inbox` class: +purposes there is the `Inbox` class: -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#inbox +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #inbox } There is an implicit conversion from inbox to actor reference which means that in this example the sender reference will be that of the actor hidden away within the inbox. This allows the reply to be received on the last line. Watching an actor is quite simple as well: -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala#watch +@@snip [ActorDSLSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/actor/ActorDSLSpec.scala) { #watch } -Actor API -========= +## Actor API -The :class:`Actor` trait defines only one abstract method, the above mentioned -:meth:`receive`, which implements the behavior of the actor. +The `Actor` trait defines only one abstract method, the above mentioned +`receive`, which implements the behavior of the actor. If the current actor behavior does not match a received message, -:meth:`unhandled` is called, which by default publishes an -``akka.actor.UnhandledMessage(message, sender, recipient)`` on the actor +`unhandled` is called, which by default publishes an +`akka.actor.UnhandledMessage(message, sender, recipient)` on the actor system’s event stream (set configuration item -``akka.actor.debug.unhandled`` to ``on`` to have them converted into +`akka.actor.debug.unhandled` to `on` to have them converted into actual Debug messages). In addition, it offers: -* :obj:`self` reference to the :class:`ActorRef` of the actor + * `self` reference to the `ActorRef` of the actor + * `sender` reference sender Actor of the last received message, typically used as described in [Actor.Reply](#actor-reply) + * + `supervisorStrategy` user overridable definition the strategy to use for supervising child actors + This strategy is typically declared inside the actor in order to have access +to the actor’s internal state within the decider function: since failure is +communicated as a message sent to the supervisor and processed like other +messages (albeit outside of the normal behavior), all values and variables +within the actor are available, as is the `sender` reference (which will +be the immediate child reporting the failure; if the original failure +occurred within a distant descendant it is still reported one level up at a +time). + * + `context` exposes contextual information for the actor and the current message, such as: + * factory methods to create child actors (`actorOf`) + * system that the actor belongs to + * parent supervisor + * supervised children + * lifecycle monitoring + * hotswap behavior stack as described in [Actor.HotSwap](#actor-hotswap) -* :obj:`sender` reference sender Actor of the last received message, typically used as described in :ref:`Actor.Reply` +You can import the members in the `context` to avoid prefixing access with `context.` -* :obj:`supervisorStrategy` user overridable definition the strategy to use for supervising child actors - - This strategy is typically declared inside the actor in order to have access - to the actor’s internal state within the decider function: since failure is - communicated as a message sent to the supervisor and processed like other - messages (albeit outside of the normal behavior), all values and variables - within the actor are available, as is the ``sender`` reference (which will - be the immediate child reporting the failure; if the original failure - occurred within a distant descendant it is still reported one level up at a - time). - -* :obj:`context` exposes contextual information for the actor and the current message, such as: - - * factory methods to create child actors (:meth:`actorOf`) - * system that the actor belongs to - * parent supervisor - * supervised children - * lifecycle monitoring - * hotswap behavior stack as described in :ref:`Actor.HotSwap` - -You can import the members in the :obj:`context` to avoid prefixing access with ``context.`` - -.. includecode:: code/docs/actor/ActorDocSpec.scala#import-context +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #import-context } The remaining visible methods are user-overridable life-cycle hooks which are described in the following: -.. includecode:: ../../../akka-actor/src/main/scala/akka/actor/Actor.scala#lifecycle-hooks +@@snip [Actor.scala]../../../../../akka-actor/src/main/scala/akka/actor/Actor.scala) { #lifecycle-hooks } -The implementations shown above are the defaults provided by the :class:`Actor` +The implementations shown above are the defaults provided by the `Actor` trait. -.. _actor-lifecycle-scala: + +### Actor Lifecycle -Actor Lifecycle ---------------- - -.. image:: ../images/actor_lifecycle.png - :align: center - :width: 680 +![actor_lifecycle.png](../images/actor_lifecycle.png) A path in an actor system represents a "place" which might be occupied by a living actor. Initially (apart from system initialized actors) a path is -empty. When ``actorOf()`` is called it assigns an *incarnation* of the actor -described by the passed ``Props`` to the given path. An actor incarnation is -identified by the path *and a UID*. A restart only swaps the ``Actor`` -instance defined by the ``Props`` but the incarnation and hence the UID remains +empty. When `actorOf()` is called it assigns an *incarnation* of the actor +described by the passed `Props` to the given path. An actor incarnation is +identified by the path *and a UID*. A restart only swaps the `Actor` +instance defined by the `Props` but the incarnation and hence the UID remains the same. The lifecycle of an incarnation ends when the actor is stopped. At that point the appropriate lifecycle events are called and watching actors are notified of the termination. After the incarnation is stopped, the path can -be reused again by creating an actor with ``actorOf()``. In this case the +be reused again by creating an actor with `actorOf()`. In this case the name of the new incarnation will be the same as the previous one but the UIDs will differ. An actor can be stopped by the actor itself, another actor -or the ``ActorSystem`` (see :ref:`stopping-actors-scala`). +or the `ActorSystem` (see [Stopping actors](#stopping-actors-scala)). -.. note:: +@@@ note - It is important to note that Actors do not stop automatically when no longer - referenced, every Actor that is created must also explicitly be destroyed. - The only simplification is that stopping a parent Actor will also recursively - stop all the child Actors that this parent has created. +It is important to note that Actors do not stop automatically when no longer +referenced, every Actor that is created must also explicitly be destroyed. +The only simplification is that stopping a parent Actor will also recursively +stop all the child Actors that this parent has created. -An ``ActorRef`` always represents an incarnation (path and UID) not just a +@@@ + +An `ActorRef` always represents an incarnation (path and UID) not just a given path. Therefore if an actor is stopped and a new one with the same -name is created an ``ActorRef`` of the old incarnation will not point +name is created an `ActorRef` of the old incarnation will not point to the new one. -``ActorSelection`` on the other hand points to the path (or multiple paths +`ActorSelection` on the other hand points to the path (or multiple paths if wildcards are used) and is completely oblivious to which incarnation is currently -occupying it. ``ActorSelection`` cannot be watched for this reason. It is -possible to resolve the current incarnation's ``ActorRef`` living under the -path by sending an ``Identify`` message to the ``ActorSelection`` which -will be replied to with an ``ActorIdentity`` containing the correct reference -(see :ref:`actorSelection-scala`). This can also be done with the ``resolveOne`` -method of the :class:`ActorSelection`, which returns a ``Future`` of the matching -:class:`ActorRef`. +occupying it. `ActorSelection` cannot be watched for this reason. It is +possible to resolve the current incarnation's `ActorRef` living under the +path by sending an `Identify` message to the `ActorSelection` which +will be replied to with an `ActorIdentity` containing the correct reference +(see [actorSelection-scala](#actorselection-scala)). This can also be done with the `resolveOne` +method of the `ActorSelection`, which returns a `Future` of the matching +`ActorRef`. -.. _deathwatch-scala: - -Lifecycle Monitoring aka DeathWatch ------------------------------------ + +### Lifecycle Monitoring aka DeathWatch In order to be notified when another actor terminates (i.e. stops permanently, not temporary failure and restart), an actor may register itself for reception -of the :class:`Terminated` message dispatched by the other actor upon -termination (see `Stopping Actors`_). This service is provided by the -:class:`DeathWatch` component of the actor system. +of the `Terminated` message dispatched by the other actor upon +termination (see [Stopping Actors](#stopping-actors)). This service is provided by the +`DeathWatch` component of the actor system. Registering a monitor is easy: -.. includecode:: code/docs/actor/ActorDocSpec.scala#watch +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #watch } -It should be noted that the :class:`Terminated` message is generated +It should be noted that the `Terminated` message is generated independent of the order in which registration and termination occur. -In particular, the watching actor will receive a :class:`Terminated` message even if the +In particular, the watching actor will receive a `Terminated` message even if the watched actor has already been terminated at the time of registration. Registering multiple times does not necessarily lead to multiple messages being @@ -367,157 +345,151 @@ received: if termination of the watched actor has generated and queued the message, and another registration is done before this message has been processed, then a second message will be queued, because registering for monitoring of an already terminated actor leads to the immediate generation of -the :class:`Terminated` message. +the `Terminated` message. It is also possible to deregister from watching another actor’s liveliness -using ``context.unwatch(target)``. This works even if the :class:`Terminated` -message has already been enqueued in the mailbox; after calling :meth:`unwatch` -no :class:`Terminated` message for that actor will be processed anymore. +using `context.unwatch(target)`. This works even if the `Terminated` +message has already been enqueued in the mailbox; after calling `unwatch` +no `Terminated` message for that actor will be processed anymore. -.. _start-hook-scala: + +### Start Hook -Start Hook ----------- +Right after starting the actor, its `preStart` method is invoked. -Right after starting the actor, its :meth:`preStart` method is invoked. - -.. includecode:: code/docs/actor/ActorDocSpec.scala#preStart +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #preStart } This method is called when the actor is first created. During restarts it is -called by the default implementation of :meth:`postRestart`, which means that +called by the default implementation of `postRestart`, which means that by overriding that method you can choose whether the initialization code in this method is called only exactly once for this actor or for every restart. Initialization code which is part of the actor’s constructor will always be called when an instance of the actor class is created, which happens at every restart. -.. _restart-hook-scala: - -Restart Hooks -------------- + +### Restart Hooks All actors are supervised, i.e. linked to another actor with a fault handling strategy. Actors may be restarted in case an exception is thrown while -processing a message (see :ref:`supervision`). This restart involves the hooks +processing a message (see supervision). This restart involves the hooks mentioned above: -1. The old actor is informed by calling :meth:`preRestart` with the exception - which caused the restart and the message which triggered that exception; the - latter may be ``None`` if the restart was not caused by processing a - message, e.g. when a supervisor does not trap the exception and is restarted - in turn by its supervisor, or if an actor is restarted due to a sibling’s - failure. If the message is available, then that message’s sender is also - accessible in the usual way (i.e. by calling ``sender``). - - This method is the best place for cleaning up, preparing hand-over to the - fresh actor instance, etc. By default it stops all children and calls - :meth:`postStop`. - -2. The initial factory from the ``actorOf`` call is used - to produce the fresh instance. - -3. The new actor’s :meth:`postRestart` method is invoked with the exception - which caused the restart. By default the :meth:`preStart` - is called, just as in the normal start-up case. + 1. + The old actor is informed by calling `preRestart` with the exception +which caused the restart and the message which triggered that exception; the +latter may be `None` if the restart was not caused by processing a +message, e.g. when a supervisor does not trap the exception and is restarted +in turn by its supervisor, or if an actor is restarted due to a sibling’s +failure. If the message is available, then that message’s sender is also +accessible in the usual way (i.e. by calling `sender`). + This method is the best place for cleaning up, preparing hand-over to the +fresh actor instance, etc. By default it stops all children and calls +`postStop`. + 2. The initial factory from the `actorOf` call is used +to produce the fresh instance. + 3. The new actor’s `postRestart` method is invoked with the exception +which caused the restart. By default the `preStart` +is called, just as in the normal start-up case. An actor restart replaces only the actual actor object; the contents of the mailbox is unaffected by the restart, so processing of messages will resume -after the :meth:`postRestart` hook returns. The message +after the `postRestart` hook returns. The message that triggered the exception will not be received again. Any message sent to an actor while it is being restarted will be queued to its mailbox as usual. -.. warning:: +@@@ warning - Be aware that the ordering of failure notifications relative to user messages - is not deterministic. In particular, a parent might restart its child before - it has processed the last messages sent by the child before the failure. - See :ref:`message-ordering` for details. +Be aware that the ordering of failure notifications relative to user messages +is not deterministic. In particular, a parent might restart its child before +it has processed the last messages sent by the child before the failure. +See @ref:[Discussion: Message Ordering](../general/message-delivery-reliability.md#message-ordering) for details. -.. _stop-hook-scala: +@@@ -Stop Hook ---------- + +### Stop Hook -After stopping an actor, its :meth:`postStop` hook is called, which may be used +After stopping an actor, its `postStop` hook is called, which may be used e.g. for deregistering this actor from other services. This hook is guaranteed to run after message queuing has been disabled for this actor, i.e. messages -sent to a stopped actor will be redirected to the :obj:`deadLetters` of the -:obj:`ActorSystem`. +sent to a stopped actor will be redirected to the `deadLetters` of the +`ActorSystem`. -.. _actorSelection-scala: + +## Identifying Actors via Actor Selection -Identifying Actors via Actor Selection -====================================== - -As described in :ref:`addressing`, each actor has a unique logical path, which +As described in @ref:[Actor References, Paths and Addresses](../general/addressing.md), each actor has a unique logical path, which is obtained by following the chain of actors from child to parent until reaching the root of the actor system, and it has a physical path, which may differ if the supervision chain includes any remote supervisors. These paths are used by the system to look up actors, e.g. when a remote message is received and the recipient is searched, but they are also useful more directly: actors may look up other actors by specifying absolute or relative -paths—logical or physical—and receive back an :class:`ActorSelection` with the +paths—logical or physical—and receive back an `ActorSelection` with the result: -.. includecode:: code/docs/actor/ActorDocSpec.scala#selection-local +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #selection-local } -.. note:: +@@@ note - It is always preferable to communicate with other Actors using their ActorRef - instead of relying upon ActorSelection. Exceptions are +It is always preferable to communicate with other Actors using their ActorRef +instead of relying upon ActorSelection. Exceptions are - * sending messages using the :ref:`at-least-once-delivery-scala` facility - * initiating first contact with a remote system +> + * sending messages using the @ref:[At-Least-Once Delivery](persistence.md#at-least-once-delivery-scala) facility + * initiating first contact with a remote system - In all other cases ActorRefs can be provided during Actor creation or - initialization, passing them from parent to child or introducing Actors by - sending their ActorRefs to other Actors within messages. +In all other cases ActorRefs can be provided during Actor creation or +initialization, passing them from parent to child or introducing Actors by +sending their ActorRefs to other Actors within messages. -The supplied path is parsed as a :class:`java.net.URI`, which basically means -that it is split on ``/`` into path elements. If the path starts with ``/``, it +@@@ + +The supplied path is parsed as a `java.net.URI`, which basically means +that it is split on `/` into path elements. If the path starts with `/`, it is absolute and the look-up starts at the root guardian (which is the parent of -``"/user"``); otherwise it starts at the current actor. If a path element equals -``..``, the look-up will take a step “up” towards the supervisor of the +`"/user"`); otherwise it starts at the current actor. If a path element equals +`..`, the look-up will take a step “up” towards the supervisor of the currently traversed actor, otherwise it will step “down” to the named child. -It should be noted that the ``..`` in actor paths here always means the logical +It should be noted that the `..` in actor paths here always means the logical structure, i.e. the supervisor. The path elements of an actor selection may contain wildcard patterns allowing for broadcasting of messages to that section: -.. includecode:: code/docs/actor/ActorDocSpec.scala#selection-wildcard +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #selection-wildcard } -Messages can be sent via the :class:`ActorSelection` and the path of the -:class:`ActorSelection` is looked up when delivering each message. If the selection +Messages can be sent via the `ActorSelection` and the path of the +`ActorSelection` is looked up when delivering each message. If the selection does not match any actors the message will be dropped. -To acquire an :class:`ActorRef` for an :class:`ActorSelection` you need to send -a message to the selection and use the ``sender()`` reference of the reply from -the actor. There is a built-in ``Identify`` message that all Actors will -understand and automatically reply to with a ``ActorIdentity`` message -containing the :class:`ActorRef`. This message is handled specially by the +To acquire an `ActorRef` for an `ActorSelection` you need to send +a message to the selection and use the `sender()` reference of the reply from +the actor. There is a built-in `Identify` message that all Actors will +understand and automatically reply to with a `ActorIdentity` message +containing the `ActorRef`. This message is handled specially by the actors which are traversed in the sense that if a concrete name lookup fails (i.e. a non-wildcard path element does not correspond to a live actor) then a negative result is generated. Please note that this does not mean that delivery of that reply is guaranteed, it still is a normal message. -.. includecode:: code/docs/actor/ActorDocSpec.scala#identify +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #identify } -You can also acquire an :class:`ActorRef` for an :class:`ActorSelection` with -the ``resolveOne`` method of the :class:`ActorSelection`. It returns a ``Future`` -of the matching :class:`ActorRef` if such an actor exists. It is completed with +You can also acquire an `ActorRef` for an `ActorSelection` with +the `resolveOne` method of the `ActorSelection`. It returns a `Future` +of the matching `ActorRef` if such an actor exists. It is completed with failure [[akka.actor.ActorNotFound]] if no such actor exists or the identification -didn't complete within the supplied `timeout`. +didn't complete within the supplied *timeout*. -Remote actor addresses may also be looked up, if :ref:`remoting ` is enabled: +Remote actor addresses may also be looked up, if @ref:[remoting](remoting.md) is enabled: -.. includecode:: code/docs/actor/ActorDocSpec.scala#selection-remote +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #selection-remote } -An example demonstrating actor look-up is given in :ref:`remote-sample-scala`. +An example demonstrating actor look-up is given in @ref:[Remoting Sample](remoting.md#remote-sample-scala). -Messages and immutability -========================= +## Messages and immutability **IMPORTANT**: Messages can be any kind of object but have to be immutable. Scala can’t enforce immutability (yet) so this has to be by @@ -528,199 +500,191 @@ pattern matching at the receiver side. Here is an example: -.. code-block:: scala +```scala +// define the case class +case class Register(user: User) - // define the case class - case class Register(user: User) +// create a new case class message +val message = Register(user) +``` - // create a new case class message - val message = Register(user) - -Send messages -============= +## Send messages Messages are sent to an Actor through one of the following methods. -* ``!`` means “fire-and-forget”, e.g. send a message asynchronously and return - immediately. Also known as ``tell``. -* ``?`` sends a message asynchronously and returns a :class:`Future` - representing a possible reply. Also known as ``ask``. + * `!` means “fire-and-forget”, e.g. send a message asynchronously and return +immediately. Also known as `tell`. + * `?` sends a message asynchronously and returns a `Future` +representing a possible reply. Also known as `ask`. Message ordering is guaranteed on a per-sender basis. -.. note:: +@@@ note - There are performance implications of using ``ask`` since something needs to - keep track of when it times out, there needs to be something that bridges - a ``Promise`` into an ``ActorRef`` and it also needs to be reachable through - remoting. So always prefer ``tell`` for performance, and only ``ask`` if you must. +There are performance implications of using `ask` since something needs to +keep track of when it times out, there needs to be something that bridges +a `Promise` into an `ActorRef` and it also needs to be reachable through +remoting. So always prefer `tell` for performance, and only `ask` if you must. -.. _actors-tell-sender-scala: +@@@ -Tell: Fire-forget ------------------ + +### Tell: Fire-forget This is the preferred way of sending messages. No blocking waiting for a message. This gives the best concurrency and scalability characteristics. -.. includecode:: code/docs/actor/ActorDocSpec.scala#tell +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #tell } If invoked from within an Actor, then the sending actor reference will be implicitly passed along with the message and available to the receiving Actor -in its ``sender(): ActorRef`` member method. The target actor can use this -to reply to the original sender, by using ``sender() ! replyMsg``. +in its `sender(): ActorRef` member method. The target actor can use this +to reply to the original sender, by using `sender() ! replyMsg`. If invoked from an instance that is **not** an Actor the sender will be -:obj:`deadLetters` actor reference by default. +`deadLetters` actor reference by default. -.. _actors-ask-scala: + +### Ask: Send-And-Receive-Future -Ask: Send-And-Receive-Future ----------------------------- +The `ask` pattern involves actors as well as futures, hence it is offered as +a use pattern rather than a method on `ActorRef`: -The ``ask`` pattern involves actors as well as futures, hence it is offered as -a use pattern rather than a method on :class:`ActorRef`: +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #ask-pipeTo } -.. includecode:: code/docs/actor/ActorDocSpec.scala#ask-pipeTo - -This example demonstrates ``ask`` together with the ``pipeTo`` pattern on +This example demonstrates `ask` together with the `pipeTo` pattern on futures, because this is likely to be a common combination. Please note that -all of the above is completely non-blocking and asynchronous: ``ask`` produces -a :class:`Future`, three of which are composed into a new future using the -for-comprehension and then ``pipeTo`` installs an ``onComplete``-handler on the -future to affect the submission of the aggregated :class:`Result` to another +all of the above is completely non-blocking and asynchronous: `ask` produces +a `Future`, three of which are composed into a new future using the +for-comprehension and then `pipeTo` installs an `onComplete`-handler on the +future to affect the submission of the aggregated `Result` to another actor. -Using ``ask`` will send a message to the receiving Actor as with ``tell``, and -the receiving actor must reply with ``sender() ! reply`` in order to complete the -returned :class:`Future` with a value. The ``ask`` operation involves creating +Using `ask` will send a message to the receiving Actor as with `tell`, and +the receiving actor must reply with `sender() ! reply` in order to complete the +returned `Future` with a value. The `ask` operation involves creating an internal actor for handling this reply, which needs to have a timeout after which it is destroyed in order not to leak resources; see more below. -.. warning:: +@@@ warning - To complete the future with an exception you need send a Failure message to the sender. - This is *not done automatically* when an actor throws an exception while processing a message. +To complete the future with an exception you need send a Failure message to the sender. +This is *not done automatically* when an actor throws an exception while processing a message. -.. includecode:: code/docs/actor/ActorDocSpec.scala#reply-exception +@@@ + +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #reply-exception } If the actor does not complete the future, it will expire after the timeout -period, completing it with an :class:`AskTimeoutException`. The timeout is +period, completing it with an `AskTimeoutException`. The timeout is taken from one of the following locations in order of precedence: -1. explicitly given timeout as in: + 1. explicitly given timeout as in: -.. includecode:: code/docs/actor/ActorDocSpec.scala#using-explicit-timeout +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #using-explicit-timeout } -2. implicit argument of type :class:`akka.util.Timeout`, e.g. + 2. implicit argument of type `akka.util.Timeout`, e.g. -.. includecode:: code/docs/actor/ActorDocSpec.scala#using-implicit-timeout +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #using-implicit-timeout } -See :ref:`futures-scala` for more information on how to await or query a +See @ref:[Futures](futures.md) for more information on how to await or query a future. -The ``onComplete``, ``onSuccess``, or ``onFailure`` methods of the ``Future`` can be +The `onComplete`, `onSuccess`, or `onFailure` methods of the `Future` can be used to register a callback to get a notification when the Future completes, giving you a way to avoid blocking. -.. warning:: +@@@ warning - When using future callbacks, such as ``onComplete``, ``onSuccess``, and ``onFailure``, - inside actors you need to carefully avoid closing over - the containing actor’s reference, i.e. do not call methods or access mutable state - on the enclosing actor from within the callback. This would break the actor - encapsulation and may introduce synchronization bugs and race conditions because - the callback will be scheduled concurrently to the enclosing actor. Unfortunately - there is not yet a way to detect these illegal accesses at compile time. - See also: :ref:`jmm-shared-state` +When using future callbacks, such as `onComplete`, `onSuccess`, and `onFailure`, +inside actors you need to carefully avoid closing over +the containing actor’s reference, i.e. do not call methods or access mutable state +on the enclosing actor from within the callback. This would break the actor +encapsulation and may introduce synchronization bugs and race conditions because +the callback will be scheduled concurrently to the enclosing actor. Unfortunately +there is not yet a way to detect these illegal accesses at compile time. +See also: @ref:[Actors and shared mutable state](../general/jmm.md#jmm-shared-state) -Forward message ---------------- +@@@ + +### Forward message You can forward a message from one actor to another. This means that the original sender address/reference is maintained even though the message is going through a 'mediator'. This can be useful when writing actors that work as routers, load-balancers, replicators etc. -.. includecode:: code/docs/actor/ActorDocSpec.scala#forward +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #forward } -Receive messages -================ +## Receive messages -An Actor has to implement the ``receive`` method to receive messages: +An Actor has to implement the `receive` method to receive messages: -.. includecode:: ../../../akka-actor/src/main/scala/akka/actor/Actor.scala#receive +@@snip [Actor.scala]../../../../../akka-actor/src/main/scala/akka/actor/Actor.scala) { #receive } -This method returns a ``PartialFunction``, e.g. a ‘match/case’ clause in +This method returns a `PartialFunction`, e.g. a ‘match/case’ clause in which the message can be matched against the different case clauses using Scala pattern matching. Here is an example: -.. includecode:: code/docs/actor/ActorDocSpec.scala - :include: imports1,my-actor +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #imports1 #my-actor } - -.. _Actor.Reply: - -Reply to messages -================= + +## Reply to messages If you want to have a handle for replying to a message, you can use -``sender()``, which gives you an ActorRef. You can reply by sending to -that ActorRef with ``sender() ! replyMsg``. You can also store the ActorRef +`sender()`, which gives you an ActorRef. You can reply by sending to +that ActorRef with `sender() ! replyMsg`. You can also store the ActorRef for replying later, or passing on to other actors. If there is no sender (a message was sent without an actor or future context) then the sender defaults to a 'dead-letter' actor ref. -.. code-block:: scala +```scala +case request => + val result = process(request) + sender() ! result // will have dead-letter actor as default +``` - case request => - val result = process(request) - sender() ! result // will have dead-letter actor as default +## Receive timeout -Receive timeout -=============== - -The `ActorContext` :meth:`setReceiveTimeout` defines the inactivity timeout after which -the sending of a `ReceiveTimeout` message is triggered. -When specified, the receive function should be able to handle an `akka.actor.ReceiveTimeout` message. +The *ActorContext* `setReceiveTimeout` defines the inactivity timeout after which +the sending of a *ReceiveTimeout* message is triggered. +When specified, the receive function should be able to handle an *akka.actor.ReceiveTimeout* message. 1 millisecond is the minimum supported timeout. -Please note that the receive timeout might fire and enqueue the `ReceiveTimeout` message right after +Please note that the receive timeout might fire and enqueue the *ReceiveTimeout* message right after another message was enqueued; hence it is **not guaranteed** that upon reception of the receive timeout there must have been an idle period beforehand as configured via this method. Once set, the receive timeout stays in effect (i.e. continues firing repeatedly after inactivity -periods). Pass in `Duration.Undefined` to switch off this feature. +periods). Pass in *Duration.Undefined* to switch off this feature. -.. includecode:: code/docs/actor/ActorDocSpec.scala#receive-timeout +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #receive-timeout } -Messages marked with ``NotInfluenceReceiveTimeout`` will not reset the timer. This can be useful when -``ReceiveTimeout`` should be fired by external inactivity but not influenced by internal activity, +Messages marked with `NotInfluenceReceiveTimeout` will not reset the timer. This can be useful when +`ReceiveTimeout` should be fired by external inactivity but not influenced by internal activity, e.g. scheduled tick messages. -.. _stopping-actors-scala: + +## Stopping actors -Stopping actors -=============== - -Actors are stopped by invoking the :meth:`stop` method of a ``ActorRefFactory``, -i.e. ``ActorContext`` or ``ActorSystem``. Typically the context is used for stopping +Actors are stopped by invoking the `stop` method of a `ActorRefFactory`, +i.e. `ActorContext` or `ActorSystem`. Typically the context is used for stopping the actor itself or child actors and the system for stopping top level actors. The actual -termination of the actor is performed asynchronously, i.e. :meth:`stop` may return before +termination of the actor is performed asynchronously, i.e. `stop` may return before the actor is stopped. -.. includecode:: code/docs/actor/ActorDocSpec.scala#stoppingActors-actor +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #stoppingActors-actor } Processing of the current message, if any, will continue before the actor is stopped, but additional messages in the mailbox will not be processed. By default these -messages are sent to the :obj:`deadLetters` of the :obj:`ActorSystem`, but that +messages are sent to the `deadLetters` of the `ActorSystem`, but that depends on the mailbox implementation. Termination of an actor proceeds in two steps: first the actor suspends its mailbox processing and sends a stop command to all its children, then it keeps processing the internal termination notifications from its children until the last one is -gone, finally terminating itself (invoking :meth:`postStop`, dumping mailbox, -publishing :class:`Terminated` on the :ref:`DeathWatch `, telling +gone, finally terminating itself (invoking `postStop`, dumping mailbox, +publishing `Terminated` on the [DeathWatch](#deathwatch-scala), telling its supervisor). This procedure ensures that actor system sub-trees terminate in an orderly fashion, propagating the stop command to the leaves and collecting their confirmation back to the stopped supervisor. If one of the @@ -728,276 +692,274 @@ actors does not respond (i.e. processing a message for extended periods of time and therefore not receiving the stop command), this whole process will be stuck. -Upon :meth:`ActorSystem.terminate()`, the system guardian actors will be +Upon `ActorSystem.terminate()`, the system guardian actors will be stopped, and the aforementioned process will ensure proper termination of the whole system. -The :meth:`postStop()` hook is invoked after an actor is fully stopped. This +The `postStop()` hook is invoked after an actor is fully stopped. This enables cleaning up of resources: -.. includecode:: code/docs/actor/ActorDocSpec.scala#postStop - :exclude: clean-up-some-resources +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #postStop } -.. note:: +@@@ note - Since stopping an actor is asynchronous, you cannot immediately reuse the - name of the child you just stopped; this will result in an - :class:`InvalidActorNameException`. Instead, :meth:`watch()` the terminating - actor and create its replacement in response to the :class:`Terminated` - message which will eventually arrive. +Since stopping an actor is asynchronous, you cannot immediately reuse the +name of the child you just stopped; this will result in an +`InvalidActorNameException`. Instead, `watch()` the terminating +actor and create its replacement in response to the `Terminated` +message which will eventually arrive. -.. _poison-pill-scala: +@@@ -PoisonPill ----------- + +### PoisonPill -You can also send an actor the ``akka.actor.PoisonPill`` message, which will -stop the actor when the message is processed. ``PoisonPill`` is enqueued as +You can also send an actor the `akka.actor.PoisonPill` message, which will +stop the actor when the message is processed. `PoisonPill` is enqueued as ordinary messages and will be handled after messages that were already queued in the mailbox. -Graceful Stop -------------- +### Graceful Stop -:meth:`gracefulStop` is useful if you need to wait for termination or compose ordered +`gracefulStop` is useful if you need to wait for termination or compose ordered termination of several actors: -.. includecode:: code/docs/actor/ActorDocSpec.scala#gracefulStop +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #gracefulStop } -.. includecode:: code/docs/actor/ActorDocSpec.scala#gracefulStop-actor +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #gracefulStop-actor } -When ``gracefulStop()`` returns successfully, the actor’s ``postStop()`` hook +When `gracefulStop()` returns successfully, the actor’s `postStop()` hook will have been executed: there exists a happens-before edge between the end of -``postStop()`` and the return of ``gracefulStop()``. +`postStop()` and the return of `gracefulStop()`. -In the above example a custom ``Manager.Shutdown`` message is sent to the target -actor to initiate the process of stopping the actor. You can use ``PoisonPill`` for +In the above example a custom `Manager.Shutdown` message is sent to the target +actor to initiate the process of stopping the actor. You can use `PoisonPill` for this, but then you have limited possibilities to perform interactions with other actors -before stopping the target actor. Simple cleanup tasks can be handled in ``postStop``. +before stopping the target actor. Simple cleanup tasks can be handled in `postStop`. -.. warning:: +@@@ warning - Keep in mind that an actor stopping and its name being deregistered are - separate events which happen asynchronously from each other. Therefore it may - be that you will find the name still in use after ``gracefulStop()`` - returned. In order to guarantee proper deregistration, only reuse names from - within a supervisor you control and only in response to a :class:`Terminated` - message, i.e. not for top-level actors. +Keep in mind that an actor stopping and its name being deregistered are +separate events which happen asynchronously from each other. Therefore it may +be that you will find the name still in use after `gracefulStop()` +returned. In order to guarantee proper deregistration, only reuse names from +within a supervisor you control and only in response to a `Terminated` +message, i.e. not for top-level actors. -.. _coordinated-shutdown-scala: +@@@ -Coordinated Shutdown --------------------- + +### Coordinated Shutdown -There is an extension named ``CoordinatedShutdown`` that will stop certain actors and +There is an extension named `CoordinatedShutdown` that will stop certain actors and services in a specific order and perform registered tasks during the shutdown process. -The order of the shutdown phases is defined in configuration ``akka.coordinated-shutdown.phases``. +The order of the shutdown phases is defined in configuration `akka.coordinated-shutdown.phases`. The default phases are defined as: -.. includecode:: ../../../akka-actor/src/main/resources/reference.conf#coordinated-shutdown-phases +@@snip [reference.conf]../../../../../akka-actor/src/main/resources/reference.conf) { #coordinated-shutdown-phases } More phases can be be added in the application's configuration if needed by overriding a phase with an -additional ``depends-on``. Especially the phases ``before-service-unbind``, ``before-cluster-shutdown`` and -``before-actor-system-terminate`` are intended for application specific phases or tasks. +additional `depends-on`. Especially the phases `before-service-unbind`, `before-cluster-shutdown` and +`before-actor-system-terminate` are intended for application specific phases or tasks. The default phases are defined in a single linear order, but the phases can be ordered as a directed acyclic graph (DAG) by defining the dependencies between the phases. -The phases are ordered with `topological `_ sort of the DAG. +The phases are ordered with [topological](https://en.wikipedia.org/wiki/Topological_sorting) sort of the DAG. Tasks can be added to a phase with: -.. includecode:: code/docs/actor/ActorDocSpec.scala#coordinated-shutdown-addTask +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #coordinated-shutdown-addTask } -The returned ``Future[Done]`` should be completed when the task is completed. The task name parameter +The returned `Future[Done]` should be completed when the task is completed. The task name parameter is only used for debugging/logging. Tasks added to the same phase are executed in parallel without any ordering assumptions. Next phase will not start until all tasks of previous phase have been completed. -If tasks are not completed within a configured timeout (see :ref:`reference.conf `) -the next phase will be started anyway. It is possible to configure ``recover=off`` for a phase +If tasks are not completed within a configured timeout (see @ref:[reference.conf](../general/configuration.md#config-akka-actor)) +the next phase will be started anyway. It is possible to configure `recover=off` for a phase to abort the rest of the shutdown process if a task fails or is not completed within the timeout. Tasks should typically be registered as early as possible after system startup. When running the coordinated shutdown tasks that have been registered will be performed but tasks that are added too late will not be run. -To start the coordinated shutdown process you can invoke ``run`` on the ``CoordinatedShutdown`` +To start the coordinated shutdown process you can invoke `run` on the `CoordinatedShutdown` extension: -.. includecode:: code/docs/actor/ActorDocSpec.scala#coordinated-shutdown-run +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #coordinated-shutdown-run } -It's safe to call the ``run`` method multiple times. It will only run once. +It's safe to call the `run` method multiple times. It will only run once. -That also means that the ``ActorSystem`` will be terminated in the last phase. By default, the +That also means that the `ActorSystem` will be terminated in the last phase. By default, the JVM is not forcefully stopped (it will be stopped if all non-daemon threads have been terminated). -To enable a hard ``System.exit`` as a final action you can configure:: +To enable a hard `System.exit` as a final action you can configure: - akka.coordinated-shutdown.exit-jvm = on +``` +akka.coordinated-shutdown.exit-jvm = on +``` -When using :ref:`Akka Cluster ` the ``CoordinatedShutdown`` will automatically run -when the cluster node sees itself as ``Exiting``, i.e. leaving from another node will trigger +When using @ref:[Akka Cluster](cluster-usage.md) the `CoordinatedShutdown` will automatically run +when the cluster node sees itself as `Exiting`, i.e. leaving from another node will trigger the shutdown process on the leaving node. Tasks for graceful leaving of cluster including graceful shutdown of Cluster Singletons and Cluster Sharding are added automatically when Akka Cluster is used, i.e. running the shutdown process will also trigger the graceful leaving if it's not already in progress. -By default, the ``CoordinatedShutdown`` will be run when the JVM process exits, e.g. -via ``kill SIGTERM`` signal (``SIGINT`` ctrl-c doesn't work). This behavior can be disabled with:: +By default, the `CoordinatedShutdown` will be run when the JVM process exits, e.g. +via `kill SIGTERM` signal (`SIGINT` ctrl-c doesn't work). This behavior can be disabled with: - akka.coordinated-shutdown.run-by-jvm-shutdown-hook=off +``` +akka.coordinated-shutdown.run-by-jvm-shutdown-hook=off +``` If you have application specific JVM shutdown hooks it's recommended that you register them via the -``CoordinatedShutdown`` so that they are running before Akka internal shutdown hooks, e.g. +`CoordinatedShutdown` so that they are running before Akka internal shutdown hooks, e.g. those shutting down Akka Remoting (Artery). -.. includecode:: code/docs/actor/ActorDocSpec.scala#coordinated-shutdown-jvm-hook +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #coordinated-shutdown-jvm-hook } -For some tests it might be undesired to terminate the ``ActorSystem`` via ``CoordinatedShutdown``. -You can disable that by adding the following to the configuration of the ``ActorSystem`` that is -used in the test:: +For some tests it might be undesired to terminate the `ActorSystem` via `CoordinatedShutdown`. +You can disable that by adding the following to the configuration of the `ActorSystem` that is +used in the test: - # Don't terminate ActorSystem via CoordinatedShutdown in tests - akka.coordinated-shutdown.terminate-actor-system = off - akka.coordinated-shutdown.run-by-jvm-shutdown-hook = off - akka.cluster.run-coordinated-shutdown-when-down = off +``` +# Don't terminate ActorSystem via CoordinatedShutdown in tests +akka.coordinated-shutdown.terminate-actor-system = off +akka.coordinated-shutdown.run-by-jvm-shutdown-hook = off +akka.cluster.run-coordinated-shutdown-when-down = off +``` -.. _Actor.HotSwap: + +## Become/Unbecome -Become/Unbecome -=============== - -Upgrade -------- +### Upgrade Akka supports hotswapping the Actor’s message loop (e.g. its implementation) at -runtime: invoke the ``context.become`` method from within the Actor. -:meth:`become` takes a ``PartialFunction[Any, Unit]`` that implements the new +runtime: invoke the `context.become` method from within the Actor. +`become` takes a `PartialFunction[Any, Unit]` that implements the new message handler. The hotswapped code is kept in a Stack which can be pushed and popped. -.. warning:: +@@@ warning - Please note that the actor will revert to its original behavior when restarted by its Supervisor. +Please note that the actor will revert to its original behavior when restarted by its Supervisor. -To hotswap the Actor behavior using ``become``: +@@@ -.. includecode:: code/docs/actor/ActorDocSpec.scala#hot-swap-actor +To hotswap the Actor behavior using `become`: -This variant of the :meth:`become` method is useful for many different things, -such as to implement a Finite State Machine (FSM, for an example see `Dining -Hakkers`_). It will replace the current behavior (i.e. the top of the behavior -stack), which means that you do not use :meth:`unbecome`, instead always the +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #hot-swap-actor } + +This variant of the `become` method is useful for many different things, +such as to implement a Finite State Machine (FSM, for an example see [Dining +Hakkers](http://www.lightbend.com/activator/template/akka-sample-fsm-scala)). It will replace the current behavior (i.e. the top of the behavior +stack), which means that you do not use `unbecome`, instead always the next behavior is explicitly installed. -.. _Dining Hakkers: http://www.lightbend.com/activator/template/akka-sample-fsm-scala - -The other way of using :meth:`become` does not replace but add to the top of +The other way of using `become` does not replace but add to the top of the behavior stack. In this case care must be taken to ensure that the number -of “pop” operations (i.e. :meth:`unbecome`) matches the number of “push” ones +of “pop” operations (i.e. `unbecome`) matches the number of “push” ones in the long run, otherwise this amounts to a memory leak (which is why this behavior is not the default). -.. includecode:: code/docs/actor/ActorDocSpec.scala#swapper +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #swapper } -Encoding Scala Actors nested receives without accidentally leaking memory -------------------------------------------------------------------------- +### Encoding Scala Actors nested receives without accidentally leaking memory -See this `Unnested receive example <@github@/akka-docs/rst/scala/code/docs/actor/UnnestedReceives.scala>`_. +See this [Unnested receive example](@github@/akka-docs/rst/scala/code/docs/actor/UnnestedReceives.scala). -.. _stash-scala: + +## Stash -Stash -===== - -The `Stash` trait enables an actor to temporarily stash away messages +The *Stash* trait enables an actor to temporarily stash away messages that can not or should not be handled using the actor's current behavior. Upon changing the actor's message handler, i.e., right -before invoking ``context.become`` or ``context.unbecome``, all +before invoking `context.become` or `context.unbecome`, all stashed messages can be "unstashed", thereby prepending them to the actor's mailbox. This way, the stashed messages can be processed in the same order as they have been received originally. -.. note:: +@@@ note - The trait ``Stash`` extends the marker trait - ``RequiresMessageQueue[DequeBasedMessageQueueSemantics]`` which - requests the system to automatically choose a deque based - mailbox implementation for the actor. If you want more control over the - mailbox, see the documentation on mailboxes: :ref:`mailboxes-scala`. +The trait `Stash` extends the marker trait +`RequiresMessageQueue[DequeBasedMessageQueueSemantics]` which +requests the system to automatically choose a deque based +mailbox implementation for the actor. If you want more control over the +mailbox, see the documentation on mailboxes: @ref:[Mailboxes](mailboxes.md). -Here is an example of the ``Stash`` in action: +@@@ -.. includecode:: code/docs/actor/ActorDocSpec.scala#stash +Here is an example of the `Stash` in action: -Invoking ``stash()`` adds the current message (the message that the +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #stash } + +Invoking `stash()` adds the current message (the message that the actor received last) to the actor's stash. It is typically invoked when handling the default case in the actor's message handler to stash messages that aren't handled by the other cases. It is illegal to stash the same message twice; to do so results in an -``IllegalStateException`` being thrown. The stash may also be bounded -in which case invoking ``stash()`` may lead to a capacity violation, -which results in a ``StashOverflowException``. The capacity of the -stash can be configured using the ``stash-capacity`` setting (an ``Int``) of the +`IllegalStateException` being thrown. The stash may also be bounded +in which case invoking `stash()` may lead to a capacity violation, +which results in a `StashOverflowException`. The capacity of the +stash can be configured using the `stash-capacity` setting (an `Int`) of the mailbox's configuration. -Invoking ``unstashAll()`` enqueues messages from the stash to the +Invoking `unstashAll()` enqueues messages from the stash to the actor's mailbox until the capacity of the mailbox (if any) has been reached (note that messages from the stash are prepended to the mailbox). In case a bounded mailbox overflows, a -``MessageQueueAppendFailedException`` is thrown. -The stash is guaranteed to be empty after calling ``unstashAll()``. +`MessageQueueAppendFailedException` is thrown. +The stash is guaranteed to be empty after calling `unstashAll()`. -The stash is backed by a ``scala.collection.immutable.Vector``. As a +The stash is backed by a `scala.collection.immutable.Vector`. As a result, even a very large number of messages may be stashed without a major impact on performance. -.. warning:: +@@@ warning - Note that the ``Stash`` trait must be mixed into (a subclass of) the - ``Actor`` trait before any trait/class that overrides the ``preRestart`` - callback. This means it's not possible to write - ``Actor with MyActor with Stash`` if ``MyActor`` overrides ``preRestart``. +Note that the `Stash` trait must be mixed into (a subclass of) the +`Actor` trait before any trait/class that overrides the `preRestart` +callback. This means it's not possible to write +`Actor with MyActor with Stash` if `MyActor` overrides `preRestart`. + +@@@ Note that the stash is part of the ephemeral actor state, unlike the mailbox. Therefore, it should be managed like other parts of the -actor's state which have the same property. The :class:`Stash` trait’s -implementation of :meth:`preRestart` will call ``unstashAll()``, which is +actor's state which have the same property. The `Stash` trait’s +implementation of `preRestart` will call `unstashAll()`, which is usually the desired behavior. -.. note:: +@@@ note - If you want to enforce that your actor can only work with an unbounded stash, - then you should use the ``UnboundedStash`` trait instead. +If you want to enforce that your actor can only work with an unbounded stash, +then you should use the `UnboundedStash` trait instead. +@@@ -.. _killing-actors-scala: + +## Killing an Actor -Killing an Actor -================ - -You can kill an actor by sending a ``Kill`` message. This will cause the actor -to throw a :class:`ActorKilledException`, triggering a failure. The actor will +You can kill an actor by sending a `Kill` message. This will cause the actor +to throw a `ActorKilledException`, triggering a failure. The actor will suspend operation and its supervisor will be asked how to handle the failure, which may mean resuming the actor, restarting it or terminating it completely. -See :ref:`supervision-directives` for more information. +See @ref:[What Supervision Means](../general/supervision.md#supervision-directives) for more information. -Use ``Kill`` like this: +Use `Kill` like this: -.. code-block:: scala +```scala +// kill the 'victim' actor +victim ! Kill +``` - // kill the 'victim' actor - victim ! Kill - - -Actors and exceptions -===================== +## Actors and exceptions It can happen that while a message is being processed by an actor, that some kind of exception is thrown, e.g. a database exception. -What happens to the Message ---------------------------- +### What happens to the Message If an exception is thrown while a message is being processed (i.e. taken out of its mailbox and handed over to the current behavior), then this message will be @@ -1006,96 +968,90 @@ if you want to retry processing of a message, you need to deal with it yourself by catching the exception and retry your flow. Make sure that you put a bound on the number of retries since you don't want a system to livelock (so consuming a lot of cpu cycles without making progress). Another possibility -would be to have a look at the :ref:`PeekMailbox pattern `. +would be to have a look at the mailbox-acking. -What happens to the mailbox ---------------------------- +### What happens to the mailbox If an exception is thrown while a message is being processed, nothing happens to the mailbox. If the actor is restarted, the same mailbox will be there. So all messages on that mailbox will be there as well. -What happens to the actor -------------------------- +### What happens to the actor If code within an actor throws an exception, that actor is suspended and the -supervision process is started (see :ref:`supervision`). Depending on the +supervision process is started (see supervision). Depending on the supervisor’s decision the actor is resumed (as if nothing happened), restarted (wiping out its internal state and starting from scratch) or terminated. - -Extending Actors using PartialFunction chaining -=============================================== +## Extending Actors using PartialFunction chaining Sometimes it can be useful to share common behavior among a few actors, or compose one actor's behavior from multiple smaller functions. -This is possible because an actor's :meth:`receive` method returns an ``Actor.Receive``, which is a type alias for ``PartialFunction[Any,Unit]``, -and partial functions can be chained together using the ``PartialFunction#orElse`` method. You can chain as many functions as you need, +This is possible because an actor's `receive` method returns an `Actor.Receive`, which is a type alias for `PartialFunction[Any,Unit]`, +and partial functions can be chained together using the `PartialFunction#orElse` method. You can chain as many functions as you need, however you should keep in mind that "first match" wins - which may be important when combining functions that both can handle the same type of message. -For example, imagine you have a set of actors which are either ``Producers`` or ``Consumers``, yet sometimes it makes sense to +For example, imagine you have a set of actors which are either `Producers` or `Consumers`, yet sometimes it makes sense to have an actor share both behaviors. This can be easily achieved without having to duplicate code by extracting the behaviors to -traits and implementing the actor's :meth:`receive` as combination of these partial functions. +traits and implementing the actor's `receive` as combination of these partial functions. -.. includecode:: code/docs/actor/ActorDocSpec.scala#receive-orElse +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #receive-orElse } Instead of inheritance the same pattern can be applied via composition - one would simply compose the receive method using partial functions from delegates. -Initialization patterns -======================= +## Initialization patterns The rich lifecycle hooks of Actors provide a useful toolkit to implement various initialization patterns. During the -lifetime of an ``ActorRef``, an actor can potentially go through several restarts, where the old instance is replaced by -a fresh one, invisibly to the outside observer who only sees the ``ActorRef``. +lifetime of an `ActorRef`, an actor can potentially go through several restarts, where the old instance is replaced by +a fresh one, invisibly to the outside observer who only sees the `ActorRef`. One may think about the new instances as "incarnations". Initialization might be necessary for every incarnation of an actor, but sometimes one needs initialization to happen only at the birth of the first instance when the -``ActorRef`` is created. The following sections provide patterns for different initialization needs. +`ActorRef` is created. The following sections provide patterns for different initialization needs. -Initialization via constructor ------------------------------- +### Initialization via constructor -Using the constructor for initialization has various benefits. First of all, it makes it possible to use ``val`` fields to store +Using the constructor for initialization has various benefits. First of all, it makes it possible to use `val` fields to store any state that does not change during the life of the actor instance, making the implementation of the actor more robust. The constructor is invoked for every incarnation of the actor, therefore the internals of the actor can always assume that proper initialization happened. This is also the drawback of this approach, as there are cases when one would like to avoid reinitializing internals on restart. For example, it is often useful to preserve child actors across restarts. The following section provides a pattern for this case. -Initialization via preStart ---------------------------- +### Initialization via preStart -The method ``preStart()`` of an actor is only called once directly during the initialization of the first instance, that -is, at creation of its ``ActorRef``. In the case of restarts, ``preStart()`` is called from ``postRestart()``, therefore -if not overridden, ``preStart()`` is called on every incarnation. However, by overriding ``postRestart()`` one can disable -this behavior, and ensure that there is only one call to ``preStart()``. +The method `preStart()` of an actor is only called once directly during the initialization of the first instance, that +is, at creation of its `ActorRef`. In the case of restarts, `preStart()` is called from `postRestart()`, therefore +if not overridden, `preStart()` is called on every incarnation. However, by overriding `postRestart()` one can disable +this behavior, and ensure that there is only one call to `preStart()`. -One useful usage of this pattern is to disable creation of new ``ActorRefs`` for children during restarts. This can be -achieved by overriding ``preRestart()``: +One useful usage of this pattern is to disable creation of new `ActorRefs` for children during restarts. This can be +achieved by overriding `preRestart()`: -.. includecode:: code/docs/actor/InitializationDocSpec.scala#preStartInit +@@snip [InitializationDocSpec.scala](code/docs/actor/InitializationDocSpec.scala) { #preStartInit } -Please note, that the child actors are *still restarted*, but no new ``ActorRef`` is created. One can recursively apply -the same principles for the children, ensuring that their ``preStart()`` method is called only at the creation of their +Please note, that the child actors are *still restarted*, but no new `ActorRef` is created. One can recursively apply +the same principles for the children, ensuring that their `preStart()` method is called only at the creation of their refs. -For more information see :ref:`supervision-restart`. +For more information see @ref:[What Restarting Means](../general/supervision.md#supervision-restart). -Initialization via message passing ----------------------------------- +### Initialization via message passing There are cases when it is impossible to pass all the information needed for actor initialization in the constructor, for example in the presence of circular dependencies. In this case the actor should listen for an initialization message, -and use ``become()`` or a finite state-machine state transition to encode the initialized and uninitialized states +and use `become()` or a finite state-machine state transition to encode the initialized and uninitialized states of the actor. -.. includecode:: code/docs/actor/InitializationDocSpec.scala#messageInit +@@snip [InitializationDocSpec.scala](code/docs/actor/InitializationDocSpec.scala) { #messageInit } -If the actor may receive messages before it has been initialized, a useful tool can be the ``Stash`` to save messages +If the actor may receive messages before it has been initialized, a useful tool can be the `Stash` to save messages until the initialization finishes, and replaying them after the actor became initialized. -.. warning:: +@@@ warning - This pattern should be used with care, and applied only when none of the patterns above are applicable. One of - the potential issues is that messages might be lost when sent to remote actors. Also, publishing an ``ActorRef`` in - an uninitialized state might lead to the condition that it receives a user message before the initialization has been - done. +This pattern should be used with care, and applied only when none of the patterns above are applicable. One of +the potential issues is that messages might be lost when sent to remote actors. Also, publishing an `ActorRef` in +an uninitialized state might lead to the condition that it receives a user message before the initialization has been +done. + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/agents.md b/akka-docs/src/main/paradox/scala/agents.md index 472494057a..5158766235 100644 --- a/akka-docs/src/main/paradox/scala/agents.md +++ b/akka-docs/src/main/paradox/scala/agents.md @@ -1,17 +1,15 @@ -.. _agents-scala: +# Agents -Agents -###### +Agents in Akka are inspired by [agents in Clojure](http://clojure.org/agents). -Agents in Akka are inspired by `agents in Clojure`_. +@@@ warning -.. warning:: - **Deprecation warning** - Agents have been deprecated and are scheduled for removal - in the next major version. We have found that their leaky abstraction (they do not - work over the network) make them inferior to pure Actors, and in face of the soon - inclusion of Akka Typed we see little value in maintaining the current Agents. +**Deprecation warning** - Agents have been deprecated and are scheduled for removal +in the next major version. We have found that their leaky abstraction (they do not +work over the network) make them inferior to pure Actors, and in face of the soon +inclusion of Akka Typed we see little value in maintaining the current Agents. -.. _agents in Clojure: http://clojure.org/agents +@@@ Agents provide asynchronous change of individual locations. Agents are bound to a single storage location for their lifetime, and only allow mutation of that @@ -20,87 +18,84 @@ functions that are asynchronously applied to the Agent's state and whose return value becomes the Agent's new state. The state of an Agent should be immutable. While updates to Agents are asynchronous, the state of an Agent is always -immediately available for reading by any thread (using ``get`` or ``apply``) +immediately available for reading by any thread (using `get` or `apply`) without any messages. Agents are reactive. The update actions of all Agents get interleaved amongst -threads in an ``ExecutionContext``. At any point in time, at most one ``send`` action for +threads in an `ExecutionContext`. At any point in time, at most one `send` action for each Agent is being executed. Actions dispatched to an agent from another thread will occur in the order they were sent, potentially interleaved with actions dispatched to the same agent from other threads. -.. note:: +@@@ note - Agents are local to the node on which they are created. This implies that you - should generally not include them in messages that may be passed to remote Actors - or as constructor parameters for remote Actors; those remote Actors will not be able to - read or update the Agent. - -Creating Agents -=============== +Agents are local to the node on which they are created. This implies that you +should generally not include them in messages that may be passed to remote Actors +or as constructor parameters for remote Actors; those remote Actors will not be able to +read or update the Agent. -Agents are created by invoking ``Agent(value)`` passing in the Agent's initial -value and providing an implicit ``ExecutionContext`` to be used for it, for these +@@@ + +## Creating Agents + +Agents are created by invoking `Agent(value)` passing in the Agent's initial +value and providing an implicit `ExecutionContext` to be used for it, for these examples we're going to use the default global one, but YMMV: -.. includecode:: code/docs/agent/AgentDocSpec.scala#create +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #create } -Reading an Agent's value -======================== +## Reading an Agent's value Agents can be dereferenced (you can get an Agent's value) by invoking the Agent with parentheses like this: -.. includecode:: code/docs/agent/AgentDocSpec.scala#read-apply +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #read-apply } Or by using the get method: -.. includecode:: code/docs/agent/AgentDocSpec.scala#read-get +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #read-get } Reading an Agent's current value does not involve any message passing and happens immediately. So while updates to an Agent are asynchronous, reading the state of an Agent is synchronous. -Updating Agents (send & alter) -============================== +## Updating Agents (send & alter) You update an Agent by sending a function that transforms the current value or by sending just a new value. The Agent will apply the new value or function atomically and asynchronously. The update is done in a fire-forget manner and you are only guaranteed that it will be applied. There is no guarantee of when the update will be applied but dispatches to an Agent from a single thread will -occur in order. You apply a value or a function by invoking the ``send`` +occur in order. You apply a value or a function by invoking the `send` function. -.. includecode:: code/docs/agent/AgentDocSpec.scala#send +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #send } You can also dispatch a function to update the internal state but on its own thread. This does not use the reactive thread pool and can be used for -long-running or blocking operations. You do this with the ``sendOff`` -method. Dispatches using either ``sendOff`` or ``send`` will still be executed +long-running or blocking operations. You do this with the `sendOff` +method. Dispatches using either `sendOff` or `send` will still be executed in order. -.. includecode:: code/docs/agent/AgentDocSpec.scala#send-off +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #send-off } -All ``send`` methods also have a corresponding ``alter`` method that returns a ``Future``. -See :ref:`futures-scala` for more information on ``Futures``. +All `send` methods also have a corresponding `alter` method that returns a `Future`. +See @ref:[Futures](futures.md) for more information on `Futures`. -.. includecode:: code/docs/agent/AgentDocSpec.scala#alter +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #alter } -.. includecode:: code/docs/agent/AgentDocSpec.scala#alter-off +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #alter-off } -Awaiting an Agent's value -========================= +## Awaiting an Agent's value -You can also get a ``Future`` to the Agents value, that will be completed after the +You can also get a `Future` to the Agents value, that will be completed after the currently queued updates have completed: -.. includecode:: code/docs/agent/AgentDocSpec.scala#read-future +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #read-future } -See :ref:`futures-scala` for more information on ``Futures``. +See @ref:[Futures](futures.md) for more information on `Futures`. -Monadic usage -============= +## Monadic usage Agents are also monadic, allowing you to compose operations using for-comprehensions. In monadic usage, new Agents are created leaving the @@ -109,16 +104,14 @@ as-is. They are so-called 'persistent'. Example of monadic usage: -.. includecode:: code/docs/agent/AgentDocSpec.scala#monadic-example +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #monadic-example } -Configuration -============= +## Configuration There are several configuration properties for the agents module, please refer -to the :ref:`reference configuration `. +to the @ref:[reference configuration](../general/configuration.md#config-akka-agent). -Deprecated Transactional Agents -=============================== +## Deprecated Transactional Agents Agents participating in enclosing STM transaction is a deprecated feature in 2.3. @@ -127,4 +120,4 @@ that transaction. If you send to an Agent within a transaction then the dispatch to the Agent will be held until that transaction commits, and discarded if the transaction is aborted. Here's an example: -.. includecode:: code/docs/agent/AgentDocSpec.scala#transfer-example +@@snip [AgentDocSpec.scala](code/docs/agent/AgentDocSpec.scala) { #transfer-example } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/camel.md b/akka-docs/src/main/paradox/scala/camel.md index a85ce820bd..1a98a5f099 100644 --- a/akka-docs/src/main/paradox/scala/camel.md +++ b/akka-docs/src/main/paradox/scala/camel.md @@ -1,16 +1,12 @@ +# Camel -.. _camel-scala: +@@@ warning -Camel -##### +Akka Camel is deprecated in favour of [Alpakka](https://github.com/akka/alpakka) , the Akka Streams based collection of integrations to various endpoints (including Camel). -.. warning:: - Akka Camel is deprecated in favour of `Alpakka`_ , the Akka Streams based collection of integrations to various endpoints (including Camel). +@@@ -.. _Alpakka: https://github.com/akka/alpakka - -Introduction -============ +## Introduction The akka-camel module allows Untyped Actors to receive and send messages over a great variety of protocols and APIs. @@ -18,58 +14,49 @@ In addition to the native Scala and Java actor API, actors can now exchange mess of protocols and APIs such as HTTP, SOAP, TCP, FTP, SMTP or JMS, to mention a few. At the moment, approximately 80 protocols and APIs are supported. -Apache Camel ------------- -The akka-camel module is based on `Apache Camel`_, a powerful and light-weight +### Apache Camel + +The akka-camel module is based on [Apache Camel](http://camel.apache.org/), a powerful and light-weight integration framework for the JVM. For an introduction to Apache Camel you may -want to read this `Apache Camel article`_. Camel comes with a -large number of `components`_ that provide bindings to different protocols and -APIs. The `camel-extra`_ project provides further components. +want to read this [Apache Camel article](http://architects.dzone.com/articles/apache-camel-integration). Camel comes with a +large number of [components](http://camel.apache.org/components.html) that provide bindings to different protocols and +APIs. The [camel-extra](http://code.google.com/p/camel-extra/) project provides further components. -.. _Apache Camel: http://camel.apache.org/ -.. _Apache Camel article: http://architects.dzone.com/articles/apache-camel-integration -.. _components: http://camel.apache.org/components.html -.. _camel-extra: http://code.google.com/p/camel-extra/ +### Consumer -Consumer --------- Usage of Camel's integration components in Akka is essentially a one-liner. Here's an example. -.. includecode:: code/docs/camel/Introduction.scala#Consumer-mina +@@snip [Introduction.scala](code/docs/camel/Introduction.scala) { #Consumer-mina } The above example exposes an actor over a TCP endpoint via Apache -Camel's `Mina component`_. The actor implements the endpointUri method to define +Camel's [Mina component](http://camel.apache.org/mina2.html). The actor implements the endpointUri method to define an endpoint from which it can receive messages. After starting the actor, TCP clients can immediately send messages to and receive responses from that -actor. If the message exchange should go over HTTP (via Camel's `Jetty +actor. If the message exchange should go over HTTP (via Camel's `Jetty component`_), only the actor's endpointUri method must be changed. -.. _Mina component: http://camel.apache.org/mina2.html -.. _Jetty component: http://camel.apache.org/jetty.html +@@snip [Introduction.scala](code/docs/camel/Introduction.scala) { #Consumer } -.. includecode:: code/docs/camel/Introduction.scala#Consumer +### Producer -Producer --------- Actors can also trigger message exchanges with external systems i.e. produce to Camel endpoints. -.. includecode:: code/docs/camel/Introduction.scala - :include: imports,Producer +@@snip [Introduction.scala](code/docs/camel/Introduction.scala) { #imports #Producer } In the above example, any message sent to this actor will be sent to -the JMS queue ``orders``. Producer actors may choose from the same set of Camel +the JMS queue `orders`. Producer actors may choose from the same set of Camel components as Consumer actors do. -CamelMessage ------------- +### CamelMessage + The number of Camel components is constantly increasing. The akka-camel module can support these in a plug-and-play manner. Just add them to your application's classpath, define a component-specific endpoint URI and use it to exchange messages over the component-specific protocols or APIs. This is possible because Camel components bind protocol-specific message formats to a Camel-specific -`normalized message format`__. The normalized message format hides +[normalized message format](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Message.java). The normalized message format hides protocol-specific details from Akka and makes it therefore very easy to support a large number of protocols through a uniform Camel component interface. The akka-camel module further converts mutable Camel messages into immutable @@ -78,101 +65,77 @@ matching, transformation, serialization or storage. In the above example of the the XML message is put in the body of a newly created Camel Message with an empty set of headers. You can also create a CamelMessage yourself with the appropriate body and headers as you see fit. -__ https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Message.java +### CamelExtension -CamelExtension --------------- -The akka-camel module is implemented as an Akka Extension, the ``CamelExtension`` object. -Extensions will only be loaded once per ``ActorSystem``, which will be managed by Akka. -The ``CamelExtension`` object provides access to the `Camel`_ trait. -The `Camel`_ trait in turn provides access to two important Apache Camel objects, the `CamelContext`_ and the `ProducerTemplate`_. +The akka-camel module is implemented as an Akka Extension, the `CamelExtension` object. +Extensions will only be loaded once per `ActorSystem`, which will be managed by Akka. +The `CamelExtension` object provides access to the [Camel](@github@/akka-camel/src/main/scala/akka/camel/Camel.scala) trait. +The [Camel](@github@/akka-camel/src/main/scala/akka/camel/Camel.scala) trait in turn provides access to two important Apache Camel objects, the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) and the `ProducerTemplate`_. Below you can see how you can get access to these Apache Camel objects. -.. includecode:: code/docs/camel/Introduction.scala#CamelExtension +@@snip [Introduction.scala](code/docs/camel/Introduction.scala) { #CamelExtension } -One ``CamelExtension`` is only loaded once for every one ``ActorSystem``, which makes it safe to call the ``CamelExtension`` at any point in your code to get to the -Apache Camel objects associated with it. There is one `CamelContext`_ and one `ProducerTemplate`_ for every one ``ActorSystem`` that uses a ``CamelExtension``. -By Default, a new `CamelContext`_ is created when the ``CamelExtension`` starts. If you want to inject your own context instead, -you can extend the `ContextProvider`_ trait and add the FQCN of your implementation in the config, as the value of the "akka.camel.context-provider". -This interface define a single method ``getContext`` used to load the `CamelContext`_. +One `CamelExtension` is only loaded once for every one `ActorSystem`, which makes it safe to call the `CamelExtension` at any point in your code to get to the +Apache Camel objects associated with it. There is one [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) and one `ProducerTemplate`_ for every one `ActorSystem` that uses a `CamelExtension`. +By Default, a new [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) is created when the `CamelExtension` starts. If you want to inject your own context instead, +you can extend the [ContextProvider](@github@/akka-camel/src/main/scala/akka/camel/ContextProvider.scala) trait and add the FQCN of your implementation in the config, as the value of the "akka.camel.context-provider". +This interface define a single method `getContext` used to load the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java). -.. _ContextProvider: @github@/akka-camel/src/main/scala/akka/camel/ContextProvider.scala +Below an example on how to add the ActiveMQ component to the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java), which is required when you would like to use the ActiveMQ component. -Below an example on how to add the ActiveMQ component to the `CamelContext`_, which is required when you would like to use the ActiveMQ component. +@@snip [Introduction.scala](code/docs/camel/Introduction.scala) { #CamelExtensionAddComponent } -.. includecode:: code/docs/camel/Introduction.scala#CamelExtensionAddComponent +The [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) joins the lifecycle of the `ActorSystem` and `CamelExtension` it is associated with; the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) is started when +the `CamelExtension` is created, and it is shut down when the associated `ActorSystem` is shut down. The same is true for the `ProducerTemplate`_. -The `CamelContext`_ joins the lifecycle of the ``ActorSystem`` and ``CamelExtension`` it is associated with; the `CamelContext`_ is started when -the ``CamelExtension`` is created, and it is shut down when the associated ``ActorSystem`` is shut down. The same is true for the `ProducerTemplate`_. - -The ``CamelExtension`` is used by both `Producer` and `Consumer` actors to interact with Apache Camel internally. -You can access the ``CamelExtension`` inside a `Producer` or a `Consumer` using the ``camel`` definition, or get straight at the `CamelContext` using the ``camelContext`` definition. -Actors are created and started asynchronously. When a `Consumer` actor is created, the `Consumer` is published at its Camel endpoint (more precisely, the route is added to the `CamelContext`_ from the `Endpoint`_ to the actor). -When a `Producer` actor is created, a `SendProcessor`_ and `Endpoint`_ are created so that the Producer can send messages to it. +The `CamelExtension` is used by both *Producer* and *Consumer* actors to interact with Apache Camel internally. +You can access the `CamelExtension` inside a *Producer* or a *Consumer* using the `camel` definition, or get straight at the *CamelContext* using the `camelContext` definition. +Actors are created and started asynchronously. When a *Consumer* actor is created, the *Consumer* is published at its Camel endpoint (more precisely, the route is added to the [CamelContext](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java) from the [Endpoint](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Endpoint.java) to the actor). +When a *Producer* actor is created, a [SendProcessor](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/processor/SendProcessor.java) and [Endpoint](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Endpoint.java) are created so that the Producer can send messages to it. Publication is done asynchronously; setting up an endpoint may still be in progress after you have requested the actor to be created. Some Camel components can take a while to startup, and in some cases you might want to know when the endpoints are activated and ready to be used. -The `Camel`_ trait allows you to find out when the endpoint is activated or deactivated. +The [Camel](@github@/akka-camel/src/main/scala/akka/camel/Camel.scala) trait allows you to find out when the endpoint is activated or deactivated. -.. includecode:: code/docs/camel/Introduction.scala#CamelActivation +@@snip [Introduction.scala](code/docs/camel/Introduction.scala) { #CamelActivation } -The above code shows that you can get a ``Future`` to the activation of the route from the endpoint to the actor, or you can wait in a blocking fashion on the activation of the route. -An ``ActivationTimeoutException`` is thrown if the endpoint could not be activated within the specified timeout. Deactivation works in a similar fashion: +The above code shows that you can get a `Future` to the activation of the route from the endpoint to the actor, or you can wait in a blocking fashion on the activation of the route. +An `ActivationTimeoutException` is thrown if the endpoint could not be activated within the specified timeout. Deactivation works in a similar fashion: -.. includecode:: code/docs/camel/Introduction.scala#CamelDeactivation +@@snip [Introduction.scala](code/docs/camel/Introduction.scala) { #CamelDeactivation } -Deactivation of a Consumer or a Producer actor happens when the actor is terminated. For a Consumer, the route to the actor is stopped. For a Producer, the `SendProcessor`_ is stopped. -A ``DeActivationTimeoutException`` is thrown if the associated camel objects could not be deactivated within the specified timeout. +Deactivation of a Consumer or a Producer actor happens when the actor is terminated. For a Consumer, the route to the actor is stopped. For a Producer, the [SendProcessor](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/processor/SendProcessor.java) is stopped. +A `DeActivationTimeoutException` is thrown if the associated camel objects could not be deactivated within the specified timeout. -.. _Camel: @github@/akka-camel/src/main/scala/akka/camel/Camel.scala -.. _CamelContext: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/CamelContext.java -.. _ProducerTemplate: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/ProducerTemplate.java -.. _SendProcessor: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/processor/SendProcessor.java -.. _Endpoint: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Endpoint.java +## Consumer Actors -Consumer Actors -================ - -For objects to receive messages, they must mixin the `Consumer`_ +For objects to receive messages, they must mixin the [Consumer](@github@/akka-camel/src/main/scala/akka/camel/Consumer.scala) trait. For example, the following actor class (Consumer1) implements the endpointUri method, which is declared in the Consumer trait, in order to receive -messages from the ``file:data/input/actor`` Camel endpoint. +messages from the `file:data/input/actor` Camel endpoint. -.. _Consumer: @github@/akka-camel/src/main/scala/akka/camel/Consumer.scala - -.. includecode:: code/docs/camel/Consumers.scala#Consumer1 +@@snip [Consumers.scala](code/docs/camel/Consumers.scala) { #Consumer1 } Whenever a file is put into the data/input/actor directory, its content is -picked up by the Camel `file component`_ and sent as message to the +picked up by the Camel [file component](http://camel.apache.org/file2.html) and sent as message to the actor. Messages consumed by actors from Camel endpoints are of type -`CamelMessage`_. These are immutable representations of Camel messages. - -.. _file component: http://camel.apache.org/file2.html -.. _Message: @github@/akka-camel/src/main/scala/akka/camel/CamelMessage.scala - +[CamelMessage](@github@/akka-camel/src/main/scala/akka/camel/CamelMessage.scala). These are immutable representations of Camel messages. Here's another example that sets the endpointUri to -``jetty:http://localhost:8877/camel/default``. It causes Camel's `Jetty -component`_ to start an embedded `Jetty`_ server, accepting HTTP connections +`jetty:http://localhost:8877/camel/default`. It causes Camel's `Jetty +component`_ to start an embedded [Jetty](http://www.eclipse.org/jetty/) server, accepting HTTP connections from localhost on port 8877. -.. _Jetty component: http://camel.apache.org/jetty.html -.. _Jetty: http://www.eclipse.org/jetty/ - -.. includecode:: code/docs/camel/Consumers.scala#Consumer2 +@@snip [Consumers.scala](code/docs/camel/Consumers.scala) { #Consumer2 } After starting the actor, clients can send messages to that actor by POSTing to -``http://localhost:8877/camel/default``. The actor sends a response by using the -sender `!` method. For returning a message body and headers to the HTTP -client the response type should be `CamelMessage`_. For any other response type, a +`http://localhost:8877/camel/default`. The actor sends a response by using the +sender *!* method. For returning a message body and headers to the HTTP +client the response type should be [CamelMessage](@github@/akka-camel/src/main/scala/akka/camel/CamelMessage.scala). For any other response type, a new CamelMessage object is created by akka-camel with the actor response as message body. -.. _CamelMessage: @github@/akka-camel/src/main/scala/akka/camel/CamelMessage.scala - -.. _camel-acknowledgements: - -Delivery acknowledgements -------------------------- + +### Delivery acknowledgements With in-out message exchanges, clients usually know that a message exchange is done when they receive a reply from a consumer actor. The reply message can be a @@ -184,162 +147,141 @@ is added to the consumer actor's mailbox. Any failure or exception that occurs during processing of that message by the consumer actor cannot be reported back to the endpoint in this case. To allow consumer actors to positively or negatively acknowledge the receipt of a message from an in-only message -exchange, they need to override the ``autoAck`` method to return false. +exchange, they need to override the `autoAck` method to return false. In this case, consumer actors must reply either with a special akka.camel.Ack message (positive acknowledgement) or a akka.actor.Status.Failure (negative acknowledgement). -.. includecode:: code/docs/camel/Consumers.scala#Consumer3 +@@snip [Consumers.scala](code/docs/camel/Consumers.scala) { #Consumer3 } -.. _camel-timeout: - -Consumer timeout ----------------- + +### Consumer timeout Camel Exchanges (and their corresponding endpoints) that support two-way communications need to wait for a response from an actor before returning it to the initiating client. For some endpoint types, timeout values can be defined in an endpoint-specific -way which is described in the documentation of the individual `Camel +way which is described in the documentation of the individual `Camel components`_. Another option is to configure timeouts on the level of consumer actors. -.. _Camel components: http://camel.apache.org/components.html - Two-way communications between a Camel endpoint and an actor are -initiated by sending the request message to the actor with the `ask`_ pattern +initiated by sending the request message to the actor with the [ask](@github@/akka-actor/src/main/scala/akka/pattern/AskSupport.scala) pattern and the actor replies to the endpoint when the response is ready. The ask request to the actor can timeout, which will -result in the `Exchange`_ failing with a TimeoutException set on the failure of the `Exchange`_. -The timeout on the consumer actor can be overridden with the ``replyTimeout``, as shown below. +result in the [Exchange](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Exchange.java) failing with a TimeoutException set on the failure of the [Exchange](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Exchange.java). +The timeout on the consumer actor can be overridden with the `replyTimeout`, as shown below. -.. includecode:: code/docs/camel/Consumers.scala#Consumer4 -.. _Exchange: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/Exchange.java -.. _ask: @github@/akka-actor/src/main/scala/akka/pattern/AskSupport.scala +@@snip [Consumers.scala](code/docs/camel/Consumers.scala) { #Consumer4 } -Producer Actors -=============== +## Producer Actors -For sending messages to Camel endpoints, actors need to mixin the `Producer`_ trait and implement the endpointUri method. +For sending messages to Camel endpoints, actors need to mixin the [Producer](@github@/akka-camel/src/main/scala/akka/camel/Producer.scala) trait and implement the endpointUri method. -.. includecode:: code/docs/camel/Producers.scala#Producer1 +@@snip [Producers.scala](code/docs/camel/Producers.scala) { #Producer1 } Producer1 inherits a default implementation of the receive method from the -Producer trait. To customize a producer actor's default behavior you must override the `Producer`_.transformResponse and -`Producer`_.transformOutgoingMessage methods. This is explained later in more detail. -Producer Actors cannot override the default `Producer`_.receive method. +Producer trait. To customize a producer actor's default behavior you must override the [Producer](@github@/akka-camel/src/main/scala/akka/camel/Producer.scala).transformResponse and +[Producer](@github@/akka-camel/src/main/scala/akka/camel/Producer.scala).transformOutgoingMessage methods. This is explained later in more detail. +Producer Actors cannot override the default [Producer](@github@/akka-camel/src/main/scala/akka/camel/Producer.scala).receive method. -Any message sent to a `Producer`_ actor will be sent to +Any message sent to a [Producer](@github@/akka-camel/src/main/scala/akka/camel/Producer.scala) actor will be sent to the associated Camel endpoint, in the above example to -``http://localhost:8080/news``. The `Producer`_ always sends messages asynchronously. Response messages (if supported by the +`http://localhost:8080/news`. The [Producer](@github@/akka-camel/src/main/scala/akka/camel/Producer.scala) always sends messages asynchronously. Response messages (if supported by the configured endpoint) will, by default, be returned to the original sender. The following example uses the ask pattern to send a message to a Producer actor and waits for a response. -.. includecode:: code/docs/camel/Producers.scala#AskProducer +@@snip [Producers.scala](code/docs/camel/Producers.scala) { #AskProducer } -The future contains the response CamelMessage, or an ``AkkaCamelException`` when an error occurred, which contains the headers of the response. +The future contains the response CamelMessage, or an `AkkaCamelException` when an error occurred, which contains the headers of the response. -.. _camel-custom-processing-scala: - -Custom Processing ------------------ + +### Custom Processing Instead of replying to the initial sender, producer actors can implement custom response processing by overriding the routeResponse method. In the following example, the response message is forwarded to a target actor instead of being replied to the original sender. -.. includecode:: code/docs/camel/Producers.scala#RouteResponse +@@snip [Producers.scala](code/docs/camel/Producers.scala) { #RouteResponse } Before producing messages to endpoints, producer actors can pre-process them by -overriding the `Producer`_.transformOutgoingMessage method. +overriding the [Producer](@github@/akka-camel/src/main/scala/akka/camel/Producer.scala).transformOutgoingMessage method. -.. includecode:: code/docs/camel/Producers.scala#TransformOutgoingMessage +@@snip [Producers.scala](code/docs/camel/Producers.scala) { #TransformOutgoingMessage } -Producer configuration options ------------------------------- +### Producer configuration options The interaction of producer actors with Camel endpoints can be configured to be one-way or two-way (by initiating in-only or in-out message exchanges, respectively). By default, the producer initiates an in-out message exchange with the endpoint. For initiating an in-only exchange, producer actors have to override the oneway method to return true. -.. includecode:: code/docs/camel/Producers.scala#Oneway +@@snip [Producers.scala](code/docs/camel/Producers.scala) { #Oneway } -Message correlation -------------------- +### Message correlation To correlate request with response messages, applications can set the -`Message.MessageExchangeId` message header. +*Message.MessageExchangeId* message header. -.. includecode:: code/docs/camel/Producers.scala#Correlate +@@snip [Producers.scala](code/docs/camel/Producers.scala) { #Correlate } -ProducerTemplate ----------------- +### ProducerTemplate -The `Producer`_ trait is a very -convenient way for actors to produce messages to Camel endpoints. Actors may also use a Camel `ProducerTemplate`_ for producing +The [Producer](@github@/akka-camel/src/main/scala/akka/camel/Producer.scala) trait is a very +convenient way for actors to produce messages to Camel endpoints. Actors may also use a Camel `ProducerTemplate`_ for producing messages to endpoints. -.. includecode:: code/docs/camel/Producers.scala#ProducerTemplate +@@snip [Producers.scala](code/docs/camel/Producers.scala) { #ProducerTemplate } For initiating a two-way message exchange, one of the -``ProducerTemplate.request*`` methods must be used. +`ProducerTemplate.request*` methods must be used. -.. includecode:: code/docs/camel/Producers.scala#RequestProducerTemplate +@@snip [Producers.scala](code/docs/camel/Producers.scala) { #RequestProducerTemplate } -.. _Producer: @github@/akka-camel/src/main/scala/akka/camel/Producer.scala -.. _ProducerTemplate: https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/ProducerTemplate.java - -.. _camel-asynchronous-routing: - -Asynchronous routing -==================== + +## Asynchronous routing In-out message exchanges between endpoints and actors are designed to be asynchronous. This is the case for both, consumer and producer actors. -* A consumer endpoint sends request messages to its consumer actor using the ``!`` - (tell) operator and the actor returns responses with ``sender !`` once they are - ready. - -* A producer actor sends request messages to its endpoint using Camel's - asynchronous routing engine. Asynchronous responses are wrapped and added to the - producer actor's mailbox for later processing. By default, response messages are - returned to the initial sender but this can be overridden by Producer - implementations (see also description of the ``routeResponse`` method - in :ref:`camel-custom-processing-scala`). + * A consumer endpoint sends request messages to its consumer actor using the `!` +(tell) operator and the actor returns responses with `sender !` once they are +ready. + * A producer actor sends request messages to its endpoint using Camel's +asynchronous routing engine. Asynchronous responses are wrapped and added to the +producer actor's mailbox for later processing. By default, response messages are +returned to the initial sender but this can be overridden by Producer +implementations (see also description of the `routeResponse` method +in [Custom Processing](#camel-custom-processing-scala)). However, asynchronous two-way message exchanges, without allocating a thread for the full duration of exchange, cannot be generically supported by Camel's asynchronous routing engine alone. This must be supported by the individual -`Camel components`_ (from which endpoints are created) as well. They must be + `Camel components`_ (from which endpoints are created) as well. They must be able to suspend any work started for request processing (thereby freeing threads to do other work) and resume processing when the response is ready. This is -currently the case for a `subset of components`_ such as the `Jetty component`_. +currently the case for a [subset of components](http://camel.apache.org/asynchronous-routing-engine.html) such as the `Jetty component`_. All other Camel components can still be used, of course, but they will cause allocation of a thread for the duration of an in-out message exchange. There's -also :ref:`camel-examples` that implements both, an asynchronous +also [Examples](#camel-examples) that implements both, an asynchronous consumer and an asynchronous producer, with the jetty component. If the used Camel component is blocking it might be necessary to use a separate -:ref:`dispatcher ` for the producer. The Camel processor is +@ref:[dispatcher](dispatchers.md) for the producer. The Camel processor is invoked by a child actor of the producer and the dispatcher can be defined in the deployment section of the configuration. For example, if your producer actor -has path ``/user/integration/output`` the dispatcher of the child actor can be -defined with:: +has path `/user/integration/output` the dispatcher of the child actor can be +defined with: - akka.actor.deployment { - /integration/output/* { - dispatcher = my-dispatcher - } +``` +akka.actor.deployment { + /integration/output/* { + dispatcher = my-dispatcher } +} +``` -.. _Camel components: http://camel.apache.org/components.html -.. _subset of components: http://camel.apache.org/asynchronous-routing-engine.html -.. _Jetty component: http://camel.apache.org/jetty.html - -Custom Camel routes -=================== +## Custom Camel routes In all the examples so far, routes to consumer actors have been automatically constructed by akka-camel, when the actor was started. Although the default @@ -348,105 +290,83 @@ most use cases, some applications may require more specialized routes to actors. The akka-camel module provides two mechanisms for customizing routes to actors, which will be explained in this section. These are: -* Usage of :ref:`camel-components` to access actors. - Any Camel route can use these components to access Akka actors. + * Usage of [camel-components](#camel-components-2) to access actors. +Any Camel route can use these components to access Akka actors. + * [Intercepting route construction](#camel-intercepting-route-construction) to actors. +This option gives you the ability to change routes that have already been added to Camel. +Consumer actors have a hook into the route definition process which can be used to change the route. -* :ref:`camel-intercepting-route-construction` to actors. - This option gives you the ability to change routes that have already been added to Camel. - Consumer actors have a hook into the route definition process which can be used to change the route. + +### Akka Camel components - -.. _camel-components: - -Akka Camel components ---------------------- - -Akka actors can be accessed from Camel routes using the `actor`_ Camel component. This component can be used to +Akka actors can be accessed from Camel routes using the `actor`_ Camel component. This component can be used to access any Akka actor (not only consumer actors) from Camel routes, as described in the following sections. -.. _actor: @github@/akka-camel/src/main/scala/akka/camel/internal/component/ActorComponent.scala + +### Access to actors -.. _access-to-actors: - -Access to actors ----------------- - -To access actors from custom Camel routes, the `actor`_ Camel -component should be used. It fully supports Camel's `asynchronous routing -engine`_. - -.. _actor: @github@/akka-camel/src/main/scala/akka/camel/internal/component/ActorComponent.scala -.. _asynchronous routing engine: http://camel.apache.org/asynchronous-routing-engine.html +To access actors from custom Camel routes, the `actor`_ Camel +component should be used. It fully supports Camel's [asynchronous routing +engine](http://camel.apache.org/asynchronous-routing-engine.html). This component accepts the following endpoint URI format: -* ``[]?`` + * `[]?` -where ```` is the ``ActorPath`` to the actor. The ```` are -name-value pairs separated by ``&`` (i.e. ``name1=value1&name2=value2&...``). +where `` is the `ActorPath` to the actor. The `` are +name-value pairs separated by `&` (i.e. `name1=value1&name2=value2&...`). - -URI options -^^^^^^^^^^^ +#### URI options The following URI options are supported: -.. tabularcolumns:: |l|l|l|L| +|Name | Type | Default | Description | +|-------------|----------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|replyTimeout | Duration | false | +The reply timeout, specified in the same +way that you use the duration in akka, +for instance `10 seconds` except that +in the url it is handy to use a + +between the amount and the unit, like +for example `200+millis` +See also [Consumer timeout](#camel-timeout).| +|autoAck | Boolean | true | +If set to true, in-only message exchanges +are auto-acknowledged when the message is +added to the actor's mailbox. If set to +false, actors must acknowledge the +receipt of the message. +See also [Delivery acknowledgements](#camel-acknowledgements). | -+--------------+----------+---------+-------------------------------------------+ -| Name | Type | Default | Description | -+==============+==========+=========+===========================================+ -| replyTimeout | Duration | false | The reply timeout, specified in the same | -| | | | way that you use the duration in akka, | -| | | | for instance ``10 seconds`` except that | -| | | | in the url it is handy to use a + | -| | | | between the amount and the unit, like | -| | | | for example ``200+millis`` | -| | | | | -| | | | See also :ref:`camel-timeout`. | -+--------------+----------+---------+-------------------------------------------+ -| autoAck | Boolean | true | If set to true, in-only message exchanges | -| | | | are auto-acknowledged when the message is | -| | | | added to the actor's mailbox. If set to | -| | | | false, actors must acknowledge the | -| | | | receipt of the message. | -| | | | | -| | | | See also :ref:`camel-acknowledgements`. | -+--------------+----------+---------+-------------------------------------------+ +Here's an actor endpoint URI example containing an actor path: -Here's an actor endpoint URI example containing an actor path:: - - akka://some-system/user/myconsumer?autoAck=false&replyTimeout=100+millis +``` +akka://some-system/user/myconsumer?autoAck=false&replyTimeout=100+millis +``` In the following example, a custom route to an actor is created, using the -actor's path. the Akka camel package contains an implicit ``toActorRouteDefinition`` that allows for a route to -reference an ``ActorRef`` directly as shown in the below example, The route starts from a `Jetty`_ endpoint and +actor's path. the Akka camel package contains an implicit `toActorRouteDefinition` that allows for a route to +reference an `ActorRef` directly as shown in the below example, The route starts from a [Jetty](http://www.eclipse.org/jetty/) endpoint and ends at the target actor. -.. includecode:: code/docs/camel/CustomRoute.scala#CustomRoute +@@snip [CustomRoute.scala](code/docs/camel/CustomRoute.scala) { #CustomRoute } When a message is received on the jetty endpoint, it is routed to the Responder actor, which in return replies back to the client of the HTTP request. + +### Intercepting route construction -.. _camel-intercepting-route-construction: - -Intercepting route construction -------------------------------- - -The previous section, :ref:`camel-components`, explained how to setup a route to an actor manually. +The previous section, [camel-components](#camel-components-2), explained how to setup a route to an actor manually. It was the application's responsibility to define the route and add it to the current CamelContext. This section explains a more convenient way to define custom routes: akka-camel is still setting up the routes to consumer actors (and adds these routes to the current CamelContext) but applications can define extensions to these routes. -Extensions can be defined with Camel's `Java DSL`_ or `Scala DSL`_. +Extensions can be defined with Camel's [Java DSL](http://camel.apache.org/dsl.html) or [Scala DSL](http://camel.apache.org/scala-dsl.html). For example, an extension could be a custom error handler that redelivers messages from an endpoint to an actor's bounded mailbox when the mailbox was full. -.. _Java DSL: http://camel.apache.org/dsl.html -.. _Scala DSL: http://camel.apache.org/scala-dsl.html - The following examples demonstrate how to extend a route to a consumer actor for handling exceptions thrown by that actor. -.. includecode:: code/docs/camel/CustomRoute.scala#ErrorThrowingConsumer +@@snip [CustomRoute.scala](code/docs/camel/CustomRoute.scala) { #ErrorThrowingConsumer } The above ErrorThrowingConsumer sends the Failure back to the sender in preRestart because the Exception that is thrown in the actor would @@ -455,54 +375,44 @@ otherwise just crash the actor, by default the actor would be restarted, and the The akka-camel module creates a RouteDefinition instance by calling from(endpointUri) on a Camel RouteBuilder (where endpointUri is the endpoint URI of the consumer actor) and passes that instance as argument to the route -definition handler \*). The route definition handler then extends the route and +definition handler *). The route definition handler then extends the route and returns a ProcessorDefinition (in the above example, the ProcessorDefinition -returned by the end method. See the `org.apache.camel.model`__ package for +returned by the end method. See the [org.apache.camel.model](https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/model/) package for details). After executing the route definition handler, akka-camel finally calls a to(targetActorUri) on the returned ProcessorDefinition to complete the -route to the consumer actor (where targetActorUri is the actor component URI as described in :ref:`access-to-actors`). -If the actor cannot be found, a `ActorNotRegisteredException` is thrown. +route to the consumer actor (where targetActorUri is the actor component URI as described in [Access to actors](#access-to-actors)). +If the actor cannot be found, a *ActorNotRegisteredException* is thrown. -\*) Before passing the RouteDefinition instance to the route definition handler, +*) Before passing the RouteDefinition instance to the route definition handler, akka-camel may make some further modifications to it. -__ https://svn.apache.org/repos/asf/camel/tags/camel-2.8.0/camel-core/src/main/java/org/apache/camel/model/ + +## Examples -.. _camel-examples: - -Examples -======== - -The sample named `Akka Camel Samples with Scala <@exampleCodeService@/akka-samples-camel-scala>`_ (`source code <@samples@/akka-sample-camel-scala>`_) +The sample named [Akka Camel Samples with Scala](@exampleCodeService@/akka-samples-camel-scala) ([source code](@samples@/akka-sample-camel-scala)) contains 3 samples: +> * Asynchronous routing and transformation - This example demonstrates how to implement consumer and - producer actors that support :ref:`camel-asynchronous-routing` with their Camel endpoints. - - * Custom Camel route - Demonstrates the combined usage of a ``Producer`` and a - ``Consumer`` actor as well as the inclusion of a custom Camel route. - +producer actors that support [Asynchronous routing](#camel-asynchronous-routing) with their Camel endpoints. + * Custom Camel route - Demonstrates the combined usage of a `Producer` and a +`Consumer` actor as well as the inclusion of a custom Camel route. * Quartz Scheduler Example - Showing how simple is to implement a cron-style scheduler by - using the Camel Quartz component +using the Camel Quartz component -Configuration -============= +## Configuration There are several configuration properties for the Camel module, please refer -to the :ref:`reference configuration `. +to the @ref:[reference configuration](../general/configuration.md#config-akka-camel). -Additional Resources -==================== -For an introduction to akka-camel 2, see also the Peter Gabryanczyk's talk `Migrating akka-camel module to Akka 2.x`_. +## Additional Resources -For an introduction to akka-camel 1, see also the `Appendix E - Akka and Camel`_ -(pdf) of the book `Camel in Action`_. +For an introduction to akka-camel 2, see also the Peter Gabryanczyk's talk [Migrating akka-camel module to Akka 2.x](http://skillsmatter.com/podcast/scala/akka-2-x). -.. _Appendix E - Akka and Camel: http://www.manning.com/ibsen/appEsample.pdf -.. _Camel in Action: http://www.manning.com/ibsen/ -.. _Migrating akka-camel module to Akka 2.x: http://skillsmatter.com/podcast/scala/akka-2-x +For an introduction to akka-camel 1, see also the [Appendix E - Akka and Camel](http://www.manning.com/ibsen/appEsample.pdf) +(pdf) of the book [Camel in Action](http://www.manning.com/ibsen/). Other, more advanced external articles (for version 1) are: -* `Akka Consumer Actors: New Features and Best Practices `_ -* `Akka Producer Actors: New Features and Best Practices `_ + * [Akka Consumer Actors: New Features and Best Practices](http://krasserm.blogspot.com/2011/02/akka-consumer-actors-new-features-and.html) + * [Akka Producer Actors: New Features and Best Practices](http://krasserm.blogspot.com/2011/02/akka-producer-actor-new-features-and.html) \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/cluster-client.md b/akka-docs/src/main/paradox/scala/cluster-client.md index e0d18a84f1..ac17fbcd33 100644 --- a/akka-docs/src/main/paradox/scala/cluster-client.md +++ b/akka-docs/src/main/paradox/scala/cluster-client.md @@ -1,46 +1,45 @@ -.. _cluster-client-scala: - -Cluster Client -============== +# Cluster Client An actor system that is not part of the cluster can communicate with actors -somewhere in the cluster via this ``ClusterClient``. The client can of course be part of +somewhere in the cluster via this `ClusterClient`. The client can of course be part of another cluster. It only needs to know the location of one (or more) nodes to use as initial -contact points. It will establish a connection to a ``ClusterReceptionist`` somewhere in +contact points. It will establish a connection to a `ClusterReceptionist` somewhere in the cluster. It will monitor the connection to the receptionist and establish a new connection if the link goes down. When looking for a new receptionist it uses fresh contact points retrieved from previous establishment, or periodically refreshed contacts, i.e. not necessarily the initial contact points. -.. note:: +@@@ note - ``ClusterClient`` should not be used when sending messages to actors that run - within the same cluster. Similar functionality as the ``ClusterClient`` is - provided in a more efficient way by :ref:`distributed-pub-sub-scala` for actors that - belong to the same cluster. +`ClusterClient` should not be used when sending messages to actors that run +within the same cluster. Similar functionality as the `ClusterClient` is +provided in a more efficient way by @ref:[Distributed Publish Subscribe in Cluster](distributed-pub-sub.md) for actors that +belong to the same cluster. -Also, note it's necessary to change ``akka.actor.provider`` from ``local`` -to ``remote`` or ``cluster`` when using +@@@ + +Also, note it's necessary to change `akka.actor.provider` from `local` +to `remote` or `cluster` when using the cluster client. The receptionist is supposed to be started on all nodes, or all nodes with specified role, -in the cluster. The receptionist can be started with the ``ClusterClientReceptionist`` extension +in the cluster. The receptionist can be started with the `ClusterClientReceptionist` extension or as an ordinary actor. -You can send messages via the ``ClusterClient`` to any actor in the cluster that is registered -in the ``DistributedPubSubMediator`` used by the ``ClusterReceptionist``. -The ``ClusterClientReceptionist`` provides methods for registration of actors that -should be reachable from the client. Messages are wrapped in ``ClusterClient.Send``, -``ClusterClient.SendToAll`` or ``ClusterClient.Publish``. +You can send messages via the `ClusterClient` to any actor in the cluster that is registered +in the `DistributedPubSubMediator` used by the `ClusterReceptionist`. +The `ClusterClientReceptionist` provides methods for registration of actors that +should be reachable from the client. Messages are wrapped in `ClusterClient.Send`, +`ClusterClient.SendToAll` or `ClusterClient.Publish`. -Both the ``ClusterClient`` and the ``ClusterClientReceptionist`` emit events that can be subscribed to. -The ``ClusterClient`` sends out notifications in relation to having received a list of contact points -from the ``ClusterClientReceptionist``. One use of this list might be for the client to record its +Both the `ClusterClient` and the `ClusterClientReceptionist` emit events that can be subscribed to. +The `ClusterClient` sends out notifications in relation to having received a list of contact points +from the `ClusterClientReceptionist`. One use of this list might be for the client to record its contact points. A client that is restarted could then use this information to supersede any previously configured contact points. -The ``ClusterClientReceptionist`` sends out notifications in relation to having received contact -from a ``ClusterClient``. This notification enables the server containing the receptionist to become aware of +The `ClusterClientReceptionist` sends out notifications in relation to having received contact +from a `ClusterClient`. This notification enables the server containing the receptionist to become aware of what clients are connected. **1. ClusterClient.Send** @@ -63,130 +62,133 @@ to the named topic. Response messages from the destination actor are tunneled via the receptionist to avoid inbound connections from other cluster nodes to the client, i.e. -the ``sender()``, as seen by the destination actor, is not the client itself. -The ``sender()`` of the response messages, as seen by the client, is ``deadLetters`` -since the client should normally send subsequent messages via the ``ClusterClient``. +the `sender()`, as seen by the destination actor, is not the client itself. +The `sender()` of the response messages, as seen by the client, is `deadLetters` +since the client should normally send subsequent messages via the `ClusterClient`. It is possible to pass the original sender inside the reply messages if the client is supposed to communicate directly to the actor in the cluster. -While establishing a connection to a receptionist the ``ClusterClient`` will buffer +While establishing a connection to a receptionist the `ClusterClient` will buffer messages and send them when the connection is established. If the buffer is full -the ``ClusterClient`` will drop old messages when new messages are sent via the client. +the `ClusterClient` will drop old messages when new messages are sent via the client. The size of the buffer is configurable and it can be disabled by using a buffer size of 0. It's worth noting that messages can always be lost because of the distributed nature of these actors. As always, additional logic should be implemented in the destination (acknowledgement) and in the client (retry) actors to ensure at-least-once message delivery. -An Example ----------- +## An Example On the cluster nodes first start the receptionist. Note, it is recommended to load the extension -when the actor system is started by defining it in the ``akka.extensions`` configuration property:: +when the actor system is started by defining it in the `akka.extensions` configuration property: - akka.extensions = ["akka.cluster.client.ClusterClientReceptionist"] +``` +akka.extensions = ["akka.cluster.client.ClusterClientReceptionist"] +``` Next, register the actors that should be available for the client. -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala#server +@@snip [ClusterClientSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala) { #server } -On the client you create the ``ClusterClient`` actor and use it as a gateway for sending +On the client you create the `ClusterClient` actor and use it as a gateway for sending messages to the actors identified by their path (without address information) somewhere in the cluster. -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala#client +@@snip [ClusterClientSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala) { #client } -The ``initialContacts`` parameter is a ``Set[ActorPath]``, which can be created like this: +The `initialContacts` parameter is a `Set[ActorPath]`, which can be created like this: -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala#initialContacts +@@snip [ClusterClientSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala) { #initialContacts } You will probably define the address information of the initial contact points in configuration or system property. -See also :ref:`cluster-client-config-scala`. +See also [Configuration](#cluster-client-config-scala). -A more comprehensive sample is available in the tutorial named `Distributed workers with Akka and Scala! `_. +A more comprehensive sample is available in the tutorial named [Distributed workers with Akka and Scala!](https://github.com/typesafehub/activator-akka-distributed-workers-scala). -ClusterClientReceptionist Extension ------------------------------------ +## ClusterClientReceptionist Extension -In the example above the receptionist is started and accessed with the ``akka.cluster.client.ClusterClientReceptionist`` extension. +In the example above the receptionist is started and accessed with the `akka.cluster.client.ClusterClientReceptionist` extension. That is convenient and perfectly fine in most cases, but it can be good to know that it is possible to -start the ``akka.cluster.client.ClusterReceptionist`` actor as an ordinary actor and you can have several +start the `akka.cluster.client.ClusterReceptionist` actor as an ordinary actor and you can have several different receptionists at the same time, serving different types of clients. -Note that the ``ClusterClientReceptionist`` uses the ``DistributedPubSub`` extension, which is described -in :ref:`distributed-pub-sub-scala`. +Note that the `ClusterClientReceptionist` uses the `DistributedPubSub` extension, which is described +in @ref:[Distributed Publish Subscribe in Cluster](distributed-pub-sub.md). It is recommended to load the extension when the actor system is started by defining it in the -``akka.extensions`` configuration property:: +`akka.extensions` configuration property: - akka.extensions = ["akka.cluster.client.ClusterClientReceptionist"] +``` +akka.extensions = ["akka.cluster.client.ClusterClientReceptionist"] +``` -Events ------- -As mentioned earlier, both the ``ClusterClient`` and ``ClusterClientReceptionist`` emit events that can be subscribed to. +## Events + +As mentioned earlier, both the `ClusterClient` and `ClusterClientReceptionist` emit events that can be subscribed to. The following code snippet declares an actor that will receive notifications on contact points (addresses to the available -receptionists), as they become available. The code illustrates subscribing to the events and receiving the ``ClusterClient`` +receptionists), as they become available. The code illustrates subscribing to the events and receiving the `ClusterClient` initial state. -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala#clientEventsListener +@@snip [ClusterClientSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala) { #clientEventsListener } -Similarly we can have an actor that behaves in a similar fashion for learning what cluster clients contact a ``ClusterClientReceptionist``: +Similarly we can have an actor that behaves in a similar fashion for learning what cluster clients contact a `ClusterClientReceptionist`: -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala#receptionistEventsListener +@@snip [ClusterClientSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/client/ClusterClientSpec.scala) { #receptionistEventsListener } -Dependencies ------------- +## Dependencies To use the Cluster Client you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` -maven:: +maven: - - com.typesafe.akka - akka-cluster-tools_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-cluster-tools_@binVersion@ + @version@ + +``` -.. _cluster-client-config-scala: + +## Configuration -Configuration -------------- - -The ``ClusterClientReceptionist`` extension (or ``ClusterReceptionistSettings``) can be configured +The `ClusterClientReceptionist` extension (or `ClusterReceptionistSettings`) can be configured with the following properties: -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#receptionist-ext-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #receptionist-ext-config } -The following configuration properties are read by the ``ClusterClientSettings`` -when created with a ``ActorSystem`` parameter. It is also possible to amend the ``ClusterClientSettings`` -or create it from another config section with the same layout as below. ``ClusterClientSettings`` is -a parameter to the ``ClusterClient.props`` factory method, i.e. each client can be configured +The following configuration properties are read by the `ClusterClientSettings` +when created with a `ActorSystem` parameter. It is also possible to amend the `ClusterClientSettings` +or create it from another config section with the same layout as below. `ClusterClientSettings` is +a parameter to the `ClusterClient.props` factory method, i.e. each client can be configured with different settings if needed. -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#cluster-client-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #cluster-client-config } + +## Failure handling -Failure handling ----------------- When the cluster client is started it must be provided with a list of initial contacts which are cluster nodes where receptionists are running. It will then repeatedly (with an interval configurable -by ``establishing-get-contacts-interval``) try to contact those until it gets in contact with one of them. +by `establishing-get-contacts-interval`) try to contact those until it gets in contact with one of them. While running, the list of contacts are continuously updated with data from the receptionists (again, with an -interval configurable with ``refresh-contacts-interval``), so that if there are more receptionists in the cluster +interval configurable with `refresh-contacts-interval`), so that if there are more receptionists in the cluster than the initial contacts provided to the client the client will learn about them. While the client is running it will detect failures in its connection to the receptionist by heartbeats if more than a configurable amount of heartbeats are missed the client will try to reconnect to its known set of contacts to find a receptionist it can access. -When the cluster cannot be reached at all ------------------------------------------ +## When the cluster cannot be reached at all + It is possible to make the cluster client stop entirely if it cannot find a receptionist it can talk to -within a configurable interval. This is configured with the ``reconnect-timeout``, which defaults to ``off``. +within a configurable interval. This is configured with the `reconnect-timeout`, which defaults to `off`. This can be useful when initial contacts are provided from some kind of service registry, cluster node addresses are entirely dynamic and the entire cluster might shut down or crash, be restarted on new addresses. Since the -client will be stopped in that case a monitoring actor can watch it and upon ``Terminate`` a new set of initial -contacts can be fetched and a new cluster client started. +client will be stopped in that case a monitoring actor can watch it and upon `Terminate` a new set of initial +contacts can be fetched and a new cluster client started. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/cluster-metrics.md b/akka-docs/src/main/paradox/scala/cluster-metrics.md index fff16824d0..f48f6059be 100644 --- a/akka-docs/src/main/paradox/scala/cluster-metrics.md +++ b/akka-docs/src/main/paradox/scala/cluster-metrics.md @@ -1,11 +1,6 @@ +# Cluster Metrics Extension -.. _cluster_metrics_scala: - -Cluster Metrics Extension -========================= - -Introduction ------------- +## Introduction The member nodes of the cluster can collect system health metrics and publish that to other cluster nodes and to the registered subscribers on the system event bus with the help of Cluster Metrics Extension. @@ -14,62 +9,65 @@ Cluster metrics information is primarily used for load-balancing routers, and can also be used to implement advanced metrics-based node life cycles, such as "Node Let-it-crash" when CPU steal time becomes excessive. -Cluster Metrics Extension is a separate Akka module delivered in ``akka-cluster-metrics`` jar. +Cluster Metrics Extension is a separate Akka module delivered in `akka-cluster-metrics` jar. To enable usage of the extension you need to add the following dependency to your project: -:: +: - "com.typesafe.akka" % "akka-cluster-metrics_@binVersion@" % "@version@" +``` +"com.typesafe.akka" % "akka-cluster-metrics_@binVersion@" % "@version@" +``` -and add the following configuration stanza to your ``application.conf`` -:: +and add the following configuration stanza to your `application.conf` +: - akka.extensions = [ "akka.cluster.metrics.ClusterMetricsExtension" ] +``` +akka.extensions = [ "akka.cluster.metrics.ClusterMetricsExtension" ] +``` -Cluster members with status :ref:`WeaklyUp `, if that feature is enabled, +Cluster members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-scala), if that feature is enabled, will participate in Cluster Metrics collection and dissemination. -Metrics Collector ------------------ +## Metrics Collector -Metrics collection is delegated to an implementation of ``akka.cluster.metrics.MetricsCollector``. +Metrics collection is delegated to an implementation of `akka.cluster.metrics.MetricsCollector`. Different collector implementations provide different subsets of metrics published to the cluster. Certain message routing and let-it-crash functions may not work when Sigar is not provisioned. Cluster metrics extension comes with two built-in collector implementations: -#. ``akka.cluster.metrics.SigarMetricsCollector``, which requires Sigar provisioning, and is more rich/precise -#. ``akka.cluster.metrics.JmxMetricsCollector``, which is used as fall back, and is less rich/precise + 1. `akka.cluster.metrics.SigarMetricsCollector`, which requires Sigar provisioning, and is more rich/precise + 2. `akka.cluster.metrics.JmxMetricsCollector`, which is used as fall back, and is less rich/precise You can also plug-in your own metrics collector implementation. By default, metrics extension will use collector provider fall back and will try to load them in this order: -#. configured user-provided collector -#. built-in ``akka.cluster.metrics.SigarMetricsCollector`` -#. and finally ``akka.cluster.metrics.JmxMetricsCollector`` + 1. configured user-provided collector + 2. built-in `akka.cluster.metrics.SigarMetricsCollector` + 3. and finally `akka.cluster.metrics.JmxMetricsCollector` -Metrics Events --------------- +## Metrics Events Metrics extension periodically publishes current snapshot of the cluster metrics to the node system event bus. -The publication interval is controlled by the ``akka.cluster.metrics.collector.sample-interval`` setting. +The publication interval is controlled by the `akka.cluster.metrics.collector.sample-interval` setting. -The payload of the ``akka.cluster.metrics.ClusterMetricsChanged`` event will contain +The payload of the `akka.cluster.metrics.ClusterMetricsChanged` event will contain latest metrics of the node as well as other cluster member nodes metrics gossip which was received during the collector sample interval. You can subscribe your metrics listener actors to these events in order to implement custom node lifecycle -:: +: - ClusterMetricsExtension(system).subscribe(metricsListenerActor) +``` +ClusterMetricsExtension(system).subscribe(metricsListenerActor) +``` -Hyperic Sigar Provisioning --------------------------- +## Hyperic Sigar Provisioning -Both user-provided and built-in metrics collectors can optionally use `Hyperic Sigar `_ +Both user-provided and built-in metrics collectors can optionally use [Hyperic Sigar](http://www.hyperic.com/products/sigar) for a wider and more accurate range of metrics compared to what can be retrieved from ordinary JMX MBeans. Sigar is using a native o/s library, and requires library provisioning, i.e. @@ -77,114 +75,112 @@ deployment, extraction and loading of the o/s native library into JVM at runtime User can provision Sigar classes and native library in one of the following ways: -#. Use `Kamon sigar-loader `_ as a project dependency for the user project. - Metrics extension will extract and load sigar library on demand with help of Kamon sigar provisioner. -#. Use `Kamon sigar-loader `_ as java agent: ``java -javaagent:/path/to/sigar-loader.jar``. - Kamon sigar loader agent will extract and load sigar library during JVM start. -#. Place ``sigar.jar`` on the ``classpath`` and Sigar native library for the o/s on the ``java.library.path``. - User is required to manage both project dependency and library deployment manually. + 1. Use [Kamon sigar-loader](https://github.com/kamon-io/sigar-loader) as a project dependency for the user project. +Metrics extension will extract and load sigar library on demand with help of Kamon sigar provisioner. + 2. Use [Kamon sigar-loader](https://github.com/kamon-io/sigar-loader) as java agent: `java -javaagent:/path/to/sigar-loader.jar`. +Kamon sigar loader agent will extract and load sigar library during JVM start. + 3. Place `sigar.jar` on the `classpath` and Sigar native library for the o/s on the `java.library.path`. +User is required to manage both project dependency and library deployment manually. -.. warning:: +@@@ warning - When using `Kamon sigar-loader `_ and running multiple - instances of the same application on the same host, you have to make sure that sigar library is extracted to a - unique per instance directory. You can control the extract directory with the - ``akka.cluster.metrics.native-library-extract-folder`` configuration setting. +When using [Kamon sigar-loader](https://github.com/kamon-io/sigar-loader) and running multiple +instances of the same application on the same host, you have to make sure that sigar library is extracted to a +unique per instance directory. You can control the extract directory with the +`akka.cluster.metrics.native-library-extract-folder` configuration setting. + +@@@ To enable usage of Sigar you can add the following dependency to the user project -:: +: - "io.kamon" % "sigar-loader" % "@sigarLoaderVersion@" +``` +"io.kamon" % "sigar-loader" % "@sigarLoaderVersion@" +``` -You can download Kamon sigar-loader from `Maven Central `_ +You can download Kamon sigar-loader from [Maven Central](http://search.maven.org/#search%7Cga%7C1%7Csigar-loader) +## Adaptive Load Balancing -Adaptive Load Balancing ------------------------ - -The ``AdaptiveLoadBalancingPool`` / ``AdaptiveLoadBalancingGroup`` performs load balancing of messages to cluster nodes based on the cluster metrics data. +The `AdaptiveLoadBalancingPool` / `AdaptiveLoadBalancingGroup` performs load balancing of messages to cluster nodes based on the cluster metrics data. It uses random selection of routees with probabilities derived from the remaining capacity of the corresponding node. It can be configured to use a specific MetricsSelector to produce the probabilities, a.k.a. weights: -* ``heap`` / ``HeapMetricsSelector`` - Used and max JVM heap memory. Weights based on remaining heap capacity; (max - used) / max -* ``load`` / ``SystemLoadAverageMetricsSelector`` - System load average for the past 1 minute, corresponding value can be found in ``top`` of Linux systems. The system is possibly nearing a bottleneck if the system load average is nearing number of cpus/cores. Weights based on remaining load capacity; 1 - (load / processors) -* ``cpu`` / ``CpuMetricsSelector`` - CPU utilization in percentage, sum of User + Sys + Nice + Wait. Weights based on remaining cpu capacity; 1 - utilization -* ``mix`` / ``MixMetricsSelector`` - Combines heap, cpu and load. Weights based on mean of remaining capacity of the combined selectors. -* Any custom implementation of ``akka.cluster.metrics.MetricsSelector`` + * `heap` / `HeapMetricsSelector` - Used and max JVM heap memory. Weights based on remaining heap capacity; (max - used) / max + * `load` / `SystemLoadAverageMetricsSelector` - System load average for the past 1 minute, corresponding value can be found in `top` of Linux systems. The system is possibly nearing a bottleneck if the system load average is nearing number of cpus/cores. Weights based on remaining load capacity; 1 - (load / processors) + * `cpu` / `CpuMetricsSelector` - CPU utilization in percentage, sum of User + Sys + Nice + Wait. Weights based on remaining cpu capacity; 1 - utilization + * `mix` / `MixMetricsSelector` - Combines heap, cpu and load. Weights based on mean of remaining capacity of the combined selectors. + * Any custom implementation of `akka.cluster.metrics.MetricsSelector` -The collected metrics values are smoothed with `exponential weighted moving average `_. In the :ref:`cluster_configuration_scala` you can adjust how quickly past data is decayed compared to new data. +The collected metrics values are smoothed with [exponential weighted moving average](http://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average). In the @ref:[cluster_configuration_scala](cluster-usage.md#cluster-configuration-scala) you can adjust how quickly past data is decayed compared to new data. Let's take a look at this router in action. What can be more demanding than calculating factorials? The backend worker that performs the factorial calculation: -.. includecode:: code/docs/cluster/FactorialBackend.scala#backend +@@snip [FactorialBackend.scala](code/docs/cluster/FactorialBackend.scala) { #backend } The frontend that receives user jobs and delegates to the backends via the router: -.. includecode:: code/docs/cluster/FactorialFrontend.scala#frontend - +@@snip [FactorialFrontend.scala](code/docs/cluster/FactorialFrontend.scala) { #frontend } As you can see, the router is defined in the same way as other routers, and in this case it is configured as follows: -:: - - akka.actor.deployment { - /factorialFrontend/factorialBackendRouter = { - # Router type provided by metrics extension. - router = cluster-metrics-adaptive-group - # Router parameter specific for metrics extension. - # metrics-selector = heap - # metrics-selector = load - # metrics-selector = cpu - metrics-selector = mix - # - routees.paths = ["/user/factorialBackend"] - cluster { - enabled = on - use-role = backend - allow-local-routees = off - } +``` +akka.actor.deployment { + /factorialFrontend/factorialBackendRouter = { + # Router type provided by metrics extension. + router = cluster-metrics-adaptive-group + # Router parameter specific for metrics extension. + # metrics-selector = heap + # metrics-selector = load + # metrics-selector = cpu + metrics-selector = mix + # + routees.paths = ["/user/factorialBackend"] + cluster { + enabled = on + use-role = backend + allow-local-routees = off } } +} +``` -It is only ``router`` type and the ``metrics-selector`` parameter that is specific to this router, +It is only `router` type and the `metrics-selector` parameter that is specific to this router, other things work in the same way as other routers. The same type of router could also have been defined in code: -.. includecode:: code/docs/cluster/FactorialFrontend.scala#router-lookup-in-code +@@snip [FactorialFrontend.scala](code/docs/cluster/FactorialFrontend.scala) { #router-lookup-in-code } -.. includecode:: code/docs/cluster/FactorialFrontend.scala#router-deploy-in-code +@@snip [FactorialFrontend.scala](code/docs/cluster/FactorialFrontend.scala) { #router-deploy-in-code } The easiest way to run **Adaptive Load Balancing** example yourself is to download the ready to run -`Akka Cluster Sample with Scala <@exampleCodeService@/akka-samples-cluster-scala>`_ +[Akka Cluster Sample with Scala](@exampleCodeService@/akka-samples-cluster-scala) together with the tutorial. It contains instructions on how to run the **Adaptive Load Balancing** sample. -The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-cluster-scala>`_. +The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-cluster-scala). -Subscribe to Metrics Events ---------------------------- +## Subscribe to Metrics Events It is possible to subscribe to the metrics events directly to implement other functionality. -.. includecode:: code/docs/cluster/MetricsListener.scala#metrics-listener +@@snip [MetricsListener.scala](code/docs/cluster/MetricsListener.scala) { #metrics-listener } -Custom Metrics Collector ------------------------- +## Custom Metrics Collector -Metrics collection is delegated to the implementation of ``akka.cluster.metrics.MetricsCollector`` +Metrics collection is delegated to the implementation of `akka.cluster.metrics.MetricsCollector` You can plug-in your own metrics collector instead of built-in -``akka.cluster.metrics.SigarMetricsCollector`` or ``akka.cluster.metrics.JmxMetricsCollector``. +`akka.cluster.metrics.SigarMetricsCollector` or `akka.cluster.metrics.JmxMetricsCollector`. Look at those two implementations for inspiration. Custom metrics collector implementation class must be specified in the -``akka.cluster.metrics.collector.provider`` configuration property. +`akka.cluster.metrics.collector.provider` configuration property. -Configuration -------------- +## Configuration The Cluster metrics extension can be configured with the following properties: -.. includecode:: ../../../akka-cluster-metrics/src/main/resources/reference.conf +@@snip [reference.conf]../../../../../akka-cluster-metrics/src/main/resources/reference.conf) { # } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/cluster-sharding.md b/akka-docs/src/main/paradox/scala/cluster-sharding.md index 31ed128bc9..f34d46507f 100644 --- a/akka-docs/src/main/paradox/scala/cluster-sharding.md +++ b/akka-docs/src/main/paradox/scala/cluster-sharding.md @@ -1,7 +1,4 @@ -.. _cluster_sharding_scala: - -Cluster Sharding -================ +# Cluster Sharding Cluster sharding is useful when you need to distribute actors across several nodes in the cluster and want to be able to interact with them using their logical identifier, but without having to care about @@ -13,61 +10,64 @@ but this feature is not limited to actors with persistent state. Cluster sharding is typically used when you have many stateful actors that together consume more resources (e.g. memory) than fit on one machine. If you only have a few stateful actors -it might be easier to run them on a :ref:`cluster-singleton-scala` node. +it might be easier to run them on a @ref:[Cluster Singleton](cluster-singleton.md) node. In this context sharding means that actors with an identifier, so called entities, can be automatically distributed across multiple nodes in the cluster. Each entity actor runs only at one place, and messages can be sent to the entity without requiring the sender to know the location of the destination actor. This is achieved by sending -the messages via a ``ShardRegion`` actor provided by this extension, which knows how +the messages via a `ShardRegion` actor provided by this extension, which knows how to route the message with the entity id to the final destination. -Cluster sharding will not be active on members with status :ref:`WeaklyUp ` +Cluster sharding will not be active on members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-scala) if that feature is enabled. -.. warning:: - **Don't use Cluster Sharding together with Automatic Downing**, - since it allows the cluster to split up into two separate clusters, which in turn will result - in *multiple shards and entities* being started, one in each separate cluster! - See :ref:`automatic-vs-manual-downing-java`. +@@@ warning -An Example ----------- +**Don't use Cluster Sharding together with Automatic Downing**, +since it allows the cluster to split up into two separate clusters, which in turn will result +in *multiple shards and entities* being started, one in each separate cluster! +See @ref:[Downing](../java/cluster-usage.md#automatic-vs-manual-downing-java). + +@@@ + +## An Example This is how an entity actor may look like: -.. includecode:: ../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala#counter-actor +@@snip [ClusterShardingSpec.scala]../../../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala) { #counter-actor } -The above actor uses event sourcing and the support provided in ``PersistentActor`` to store its state. +The above actor uses event sourcing and the support provided in `PersistentActor` to store its state. It does not have to be a persistent actor, but in case of failure or migration of entities between nodes it must be able to recover its state if it is valuable. -Note how the ``persistenceId`` is defined. The name of the actor is the entity identifier (utf-8 URL-encoded). +Note how the `persistenceId` is defined. The name of the actor is the entity identifier (utf-8 URL-encoded). You may define it another way, but it must be unique. When using the sharding extension you are first, typically at system startup on each node -in the cluster, supposed to register the supported entity types with the ``ClusterSharding.start`` -method. ``ClusterSharding.start`` gives you the reference which you can pass along. +in the cluster, supposed to register the supported entity types with the `ClusterSharding.start` +method. `ClusterSharding.start` gives you the reference which you can pass along. -.. includecode:: ../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala#counter-start +@@snip [ClusterShardingSpec.scala]../../../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala) { #counter-start } -The ``extractEntityId`` and ``extractShardId`` are two application specific functions to extract the entity +The `extractEntityId` and `extractShardId` are two application specific functions to extract the entity identifier and the shard identifier from incoming messages. -.. includecode:: ../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala#counter-extractor +@@snip [ClusterShardingSpec.scala]../../../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala) { #counter-extractor } This example illustrates two different ways to define the entity identifier in the messages: - * The ``Get`` message includes the identifier itself. - * The ``EntityEnvelope`` holds the identifier, and the actual message that is - sent to the entity actor is wrapped in the envelope. +> + * The `Get` message includes the identifier itself. + * The `EntityEnvelope` holds the identifier, and the actual message that is +sent to the entity actor is wrapped in the envelope. -Note how these two messages types are handled in the ``extractEntityId`` function shown above. -The message sent to the entity actor is the second part of the tuple return by the ``extractEntityId`` and that makes it +Note how these two messages types are handled in the `extractEntityId` function shown above. +The message sent to the entity actor is the second part of the tuple return by the `extractEntityId` and that makes it possible to unwrap envelopes if needed. A shard is a group of entities that will be managed together. The grouping is defined by the -``extractShardId`` function shown above. For a specific entity identifier the shard identifier must always +`extractShardId` function shown above. For a specific entity identifier the shard identifier must always be the same. Creating a good sharding algorithm is an interesting challenge in itself. Try to produce a uniform distribution, @@ -78,122 +78,119 @@ overhead, and increased latency because the coordinator is involved in the routi shard. The sharding algorithm must be the same on all nodes in a running cluster. It can be changed after stopping all nodes in the cluster. -A simple sharding algorithm that works fine in most cases is to take the absolute value of the ``hashCode`` of +A simple sharding algorithm that works fine in most cases is to take the absolute value of the `hashCode` of the entity identifier modulo number of shards. As a convenience this is provided by the -``ShardRegion.HashCodeMessageExtractor``. +`ShardRegion.HashCodeMessageExtractor`. -Messages to the entities are always sent via the local ``ShardRegion``. The ``ShardRegion`` actor reference for a -named entity type is returned by ``ClusterSharding.start`` and it can also be retrieved with ``ClusterSharding.shardRegion``. -The ``ShardRegion`` will lookup the location of the shard for the entity if it does not already know its location. It will +Messages to the entities are always sent via the local `ShardRegion`. The `ShardRegion` actor reference for a +named entity type is returned by `ClusterSharding.start` and it can also be retrieved with `ClusterSharding.shardRegion`. +The `ShardRegion` will lookup the location of the shard for the entity if it does not already know its location. It will delegate the message to the right node and it will create the entity actor on demand, i.e. when the first message for a specific entity is delivered. -.. includecode:: ../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala#counter-usage +@@snip [ClusterShardingSpec.scala]../../../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala) { #counter-usage } A more comprehensive sample is available in the -tutorial named `Akka Cluster Sharding with Scala! `_. +tutorial named [Akka Cluster Sharding with Scala!](https://github.com/typesafehub/activator-akka-cluster-sharding-scala). -How it works ------------- +## How it works -The ``ShardRegion`` actor is started on each node in the cluster, or group of nodes -tagged with a specific role. The ``ShardRegion`` is created with two application specific +The `ShardRegion` actor is started on each node in the cluster, or group of nodes +tagged with a specific role. The `ShardRegion` is created with two application specific functions to extract the entity identifier and the shard identifier from incoming messages. A shard is a group of entities that will be managed together. For the first message in a -specific shard the ``ShardRegion`` request the location of the shard from a central coordinator, -the ``ShardCoordinator``. +specific shard the `ShardRegion` request the location of the shard from a central coordinator, +the `ShardCoordinator`. -The ``ShardCoordinator`` decides which ``ShardRegion`` shall own the ``Shard`` and informs -that ``ShardRegion``. The region will confirm this request and create the ``Shard`` supervisor -as a child actor. The individual ``Entities`` will then be created when needed by the ``Shard`` -actor. Incoming messages thus travel via the ``ShardRegion`` and the ``Shard`` to the target -``Entity``. +The `ShardCoordinator` decides which `ShardRegion` shall own the `Shard` and informs +that `ShardRegion`. The region will confirm this request and create the `Shard` supervisor +as a child actor. The individual `Entities` will then be created when needed by the `Shard` +actor. Incoming messages thus travel via the `ShardRegion` and the `Shard` to the target +`Entity`. -If the shard home is another ``ShardRegion`` instance messages will be forwarded -to that ``ShardRegion`` instance instead. While resolving the location of a +If the shard home is another `ShardRegion` instance messages will be forwarded +to that `ShardRegion` instance instead. While resolving the location of a shard incoming messages for that shard are buffered and later delivered when the shard home is known. Subsequent messages to the resolved shard can be delivered -to the target destination immediately without involving the ``ShardCoordinator``. +to the target destination immediately without involving the `ShardCoordinator`. Scenario 1: -#. Incoming message M1 to ``ShardRegion`` instance R1. -#. M1 is mapped to shard S1. R1 doesn't know about S1, so it asks the coordinator C for the location of S1. -#. C answers that the home of S1 is R1. -#. R1 creates child actor for the entity E1 and sends buffered messages for S1 to E1 child -#. All incoming messages for S1 which arrive at R1 can be handled by R1 without C. It creates entity children as needed, and forwards messages to them. + 1. Incoming message M1 to `ShardRegion` instance R1. + 2. M1 is mapped to shard S1. R1 doesn't know about S1, so it asks the coordinator C for the location of S1. + 3. C answers that the home of S1 is R1. + 4. R1 creates child actor for the entity E1 and sends buffered messages for S1 to E1 child + 5. All incoming messages for S1 which arrive at R1 can be handled by R1 without C. It creates entity children as needed, and forwards messages to them. Scenario 2: -#. Incoming message M2 to R1. -#. M2 is mapped to S2. R1 doesn't know about S2, so it asks C for the location of S2. -#. C answers that the home of S2 is R2. -#. R1 sends buffered messages for S2 to R2 -#. All incoming messages for S2 which arrive at R1 can be handled by R1 without C. It forwards messages to R2. -#. R2 receives message for S2, ask C, which answers that the home of S2 is R2, and we are in Scenario 1 (but for R2). + 1. Incoming message M2 to R1. + 2. M2 is mapped to S2. R1 doesn't know about S2, so it asks C for the location of S2. + 3. C answers that the home of S2 is R2. + 4. R1 sends buffered messages for S2 to R2 + 5. All incoming messages for S2 which arrive at R1 can be handled by R1 without C. It forwards messages to R2. + 6. R2 receives message for S2, ask C, which answers that the home of S2 is R2, and we are in Scenario 1 (but for R2). To make sure that at most one instance of a specific entity actor is running somewhere in the cluster it is important that all nodes have the same view of where the shards are located. Therefore the shard allocation decisions are taken by the central -``ShardCoordinator``, which is running as a cluster singleton, i.e. one instance on +`ShardCoordinator`, which is running as a cluster singleton, i.e. one instance on the oldest member among all cluster nodes or a group of nodes tagged with a specific role. The logic that decides where a shard is to be located is defined in a pluggable shard -allocation strategy. The default implementation ``ShardCoordinator.LeastShardAllocationStrategy`` -allocates new shards to the ``ShardRegion`` with least number of previously allocated shards. +allocation strategy. The default implementation `ShardCoordinator.LeastShardAllocationStrategy` +allocates new shards to the `ShardRegion` with least number of previously allocated shards. This strategy can be replaced by an application specific implementation. To be able to use newly added members in the cluster the coordinator facilitates rebalancing of shards, i.e. migrate entities from one node to another. In the rebalance process the -coordinator first notifies all ``ShardRegion`` actors that a handoff for a shard has started. +coordinator first notifies all `ShardRegion` actors that a handoff for a shard has started. That means they will start buffering incoming messages for that shard, in the same way as if the shard location is unknown. During the rebalance process the coordinator will not answer any requests for the location of shards that are being rebalanced, i.e. local buffering will -continue until the handoff is completed. The ``ShardRegion`` responsible for the rebalanced shard -will stop all entities in that shard by sending the specified ``handOffStopMessage`` -(default ``PoisonPill``) to them. When all entities have been terminated the ``ShardRegion`` +continue until the handoff is completed. The `ShardRegion` responsible for the rebalanced shard +will stop all entities in that shard by sending the specified `handOffStopMessage` +(default `PoisonPill`) to them. When all entities have been terminated the `ShardRegion` owning the entities will acknowledge the handoff as completed to the coordinator. Thereafter the coordinator will reply to requests for the location of the shard and thereby allocate a new home for the shard and then buffered messages in the -``ShardRegion`` actors are delivered to the new location. This means that the state of the entities +`ShardRegion` actors are delivered to the new location. This means that the state of the entities are not transferred or migrated. If the state of the entities are of importance it should be -persistent (durable), e.g. with :ref:`persistence-scala`, so that it can be recovered at the new +persistent (durable), e.g. with @ref:[Persistence](persistence.md), so that it can be recovered at the new location. The logic that decides which shards to rebalance is defined in a pluggable shard -allocation strategy. The default implementation ``ShardCoordinator.LeastShardAllocationStrategy`` -picks shards for handoff from the ``ShardRegion`` with most number of previously allocated shards. -They will then be allocated to the ``ShardRegion`` with least number of previously allocated shards, +allocation strategy. The default implementation `ShardCoordinator.LeastShardAllocationStrategy` +picks shards for handoff from the `ShardRegion` with most number of previously allocated shards. +They will then be allocated to the `ShardRegion` with least number of previously allocated shards, i.e. new members in the cluster. There is a configurable threshold of how large the difference must be to begin the rebalancing. This strategy can be replaced by an application specific implementation. -The state of shard locations in the ``ShardCoordinator`` is persistent (durable) with -:ref:`distributed_data_scala` or :ref:`persistence-scala` to survive failures. When a crashed or -unreachable coordinator node has been removed (via down) from the cluster a new ``ShardCoordinator`` singleton +The state of shard locations in the `ShardCoordinator` is persistent (durable) with +@ref:[distributed_data_scala](distributed-data.md) or @ref:[Persistence](persistence.md) to survive failures. When a crashed or +unreachable coordinator node has been removed (via down) from the cluster a new `ShardCoordinator` singleton actor will take over and the state is recovered. During such a failure period shards with known location are still available, while messages for new (unknown) shards -are buffered until the new ``ShardCoordinator`` becomes available. +are buffered until the new `ShardCoordinator` becomes available. -As long as a sender uses the same ``ShardRegion`` actor to deliver messages to an entity +As long as a sender uses the same `ShardRegion` actor to deliver messages to an entity actor the order of the messages is preserved. As long as the buffer limit is not reached messages are delivered on a best effort basis, with at-most once delivery semantics, in the same way as ordinary message sending. Reliable end-to-end messaging, with -at-least-once semantics can be added by using ``AtLeastOnceDelivery`` in :ref:`persistence-scala`. +at-least-once semantics can be added by using `AtLeastOnceDelivery` in @ref:[Persistence](persistence.md). Some additional latency is introduced for messages targeted to new or previously unused shards due to the round-trip to the coordinator. Rebalancing of shards may also add latency. This should be considered when designing the application specific shard resolution, e.g. to avoid too fine grained shards. -.. _cluster_sharding_mode_scala: + +## Distributed Data vs. Persistence Mode -Distributed Data vs. Persistence Mode -------------------------------------- - -The state of the coordinator and the state of :ref:`cluster_sharding_remembering_scala` of the shards -are persistent (durable) to survive failures. :ref:`distributed_data_scala` or :ref:`persistence-scala` +The state of the coordinator and the state of [cluster_sharding_remembering_scala](#cluster-sharding-remembering-scala) of the shards +are persistent (durable) to survive failures. @ref:[distributed_data_scala](distributed-data.md) or @ref:[Persistence](persistence.md) can be used for the storage. Distributed Data is used by default. The functionality when using the two modes is the same. If your sharded entities are not using Akka Persistence @@ -204,221 +201,221 @@ no major reasons for using one mode over the the other. It's important to use the same mode on all nodes in the cluster, i.e. it's not possible to perform a rolling upgrade to change this setting. -Distributed Data Mode -^^^^^^^^^^^^^^^^^^^^^ +### Distributed Data Mode -This mode is enabled with configuration (enabled by default):: +This mode is enabled with configuration (enabled by default): - akka.cluster.sharding.state-store-mode = ddata +``` +akka.cluster.sharding.state-store-mode = ddata +``` -The state of the ``ShardCoordinator`` will be replicated inside a cluster by the -:ref:`distributed_data_scala` module with ``WriteMajority``/``ReadMajority`` consistency. +The state of the `ShardCoordinator` will be replicated inside a cluster by the +@ref:[distributed_data_scala](distributed-data.md) module with `WriteMajority`/`ReadMajority` consistency. The state of the coordinator is not durable, it's not stored to disk. When all nodes in the cluster have been stopped the state is lost and not needed any more. -The state of :ref:`cluster_sharding_remembering_scala` is also durable, i.e. it is stored to +The state of [cluster_sharding_remembering_scala](#cluster-sharding-remembering-scala) is also durable, i.e. it is stored to disk. The stored entities are started also after a complete cluster restart. -Cluster Sharding is using its own Distributed Data ``Replicator`` per node role. In this way you can use a subset of +Cluster Sharding is using its own Distributed Data `Replicator` per node role. In this way you can use a subset of all nodes for some entity types and another subset for other entity types. Each such replicator has a name that contains the node role and therefore the role configuration must be the same on all nodes in the cluster, i.e. you can't change the roles when performing a rolling upgrade. The settings for Distributed Data is configured in the the section -``akka.cluster.sharding.distributed-data``. It's not possible to have different -``distributed-data`` settings for different sharding entity types. +`akka.cluster.sharding.distributed-data`. It's not possible to have different +`distributed-data` settings for different sharding entity types. -Persistence Mode -^^^^^^^^^^^^^^^^ +### Persistence Mode -This mode is enabled with configuration:: +This mode is enabled with configuration: - akka.cluster.sharding.state-store-mode = persistence +``` +akka.cluster.sharding.state-store-mode = persistence +``` -Since it is running in a cluster :ref:`persistence-scala` must be configured with a distributed journal. +Since it is running in a cluster @ref:[Persistence](persistence.md) must be configured with a distributed journal. +## Startup after minimum number of members -Startup after minimum number of members ---------------------------------------- - -It's good to use Cluster Sharding with the Cluster setting ``akka.cluster.min-nr-of-members`` or -``akka.cluster.role..min-nr-of-members``. That will defer the allocation of the shards +It's good to use Cluster Sharding with the Cluster setting `akka.cluster.min-nr-of-members` or +`akka.cluster.role..min-nr-of-members`. That will defer the allocation of the shards until at least that number of regions have been started and registered to the coordinator. This avoids that many shards are allocated to the first region that registers and only later are rebalanced to other nodes. -See :ref:`min-members_scala` for more information about ``min-nr-of-members``. +See @ref:[min-members_scala](cluster-usage.md#min-members-scala) for more information about `min-nr-of-members`. -Proxy Only Mode ---------------- +## Proxy Only Mode -The ``ShardRegion`` actor can also be started in proxy only mode, i.e. it will not +The `ShardRegion` actor can also be started in proxy only mode, i.e. it will not host any entities itself, but knows how to delegate messages to the right location. -A ``ShardRegion`` is started in proxy only mode with the method ``ClusterSharding.startProxy`` +A `ShardRegion` is started in proxy only mode with the method `ClusterSharding.startProxy` method. -Passivation ------------ +## Passivation If the state of the entities are persistent you may stop entities that are not used to reduce memory consumption. This is done by the application specific implementation of -the entity actors for example by defining receive timeout (``context.setReceiveTimeout``). +the entity actors for example by defining receive timeout (`context.setReceiveTimeout`). If a message is already enqueued to the entity when it stops itself the enqueued message in the mailbox will be dropped. To support graceful passivation without losing such -messages the entity actor can send ``ShardRegion.Passivate`` to its parent ``Shard``. -The specified wrapped message in ``Passivate`` will be sent back to the entity, which is -then supposed to stop itself. Incoming messages will be buffered by the ``Shard`` -between reception of ``Passivate`` and termination of the entity. Such buffered messages +messages the entity actor can send `ShardRegion.Passivate` to its parent `Shard`. +The specified wrapped message in `Passivate` will be sent back to the entity, which is +then supposed to stop itself. Incoming messages will be buffered by the `Shard` +between reception of `Passivate` and termination of the entity. Such buffered messages are thereafter delivered to a new incarnation of the entity. -.. _cluster_sharding_remembering_scala: + +## Remembering Entities -Remembering Entities --------------------- - -The list of entities in each ``Shard`` can be made persistent (durable) by setting -the ``rememberEntities`` flag to true in ``ClusterShardingSettings`` when calling -``ClusterSharding.start``. When configured to remember entities, whenever a ``Shard`` +The list of entities in each `Shard` can be made persistent (durable) by setting +the `rememberEntities` flag to true in `ClusterShardingSettings` when calling +`ClusterSharding.start`. When configured to remember entities, whenever a `Shard` is rebalanced onto another node or recovers after a crash it will recreate all the -entities which were previously running in that ``Shard``. To permanently stop entities, -a ``Passivate`` message must be sent to the parent of the entity actor, otherwise the +entities which were previously running in that `Shard`. To permanently stop entities, +a `Passivate` message must be sent to the parent of the entity actor, otherwise the entity will be automatically restarted after the entity restart backoff specified in the configuration. -When :ref:`Distributed Data mode ` is used the identifiers of the entities are -stored in :ref:`ddata_durable_scala` of Distributed Data. You may want to change the +When [Distributed Data mode](#cluster-sharding-mode-scala) is used the identifiers of the entities are +stored in @ref:[ddata_durable_scala](distributed-data.md#ddata-durable-scala) of Distributed Data. You may want to change the configuration of the akka.cluster.sharding.distributed-data.durable.lmdb.dir`, since the default directory contains the remote port of the actor system. If using a dynamically assigned port (0) it will be different each time and the previously stored data will not be loaded. -When ``rememberEntities`` is set to false, a ``Shard`` will not automatically restart any entities +When `rememberEntities` is set to false, a `Shard` will not automatically restart any entities after a rebalance or recovering from a crash. Entities will only be started once the first message -for that entity has been received in the ``Shard``. Entities will not be restarted if they stop without -using a ``Passivate``. +for that entity has been received in the `Shard`. Entities will not be restarted if they stop without +using a `Passivate`. Note that the state of the entities themselves will not be restored unless they have been made persistent, -e.g. with :ref:`persistence-scala`. +e.g. with @ref:[Persistence](persistence.md). -The performance cost of ``rememberEntities`` is rather high when starting/stopping entities and when +The performance cost of `rememberEntities` is rather high when starting/stopping entities and when shards are rebalanced. This cost increases with number of entities per shard and we currently don't recommend using it with more than 10000 entities per shard. -Supervision ------------ +## Supervision -If you need to use another ``supervisorStrategy`` for the entity actors than the default (restarting) strategy -you need to create an intermediate parent actor that defines the ``supervisorStrategy`` to the +If you need to use another `supervisorStrategy` for the entity actors than the default (restarting) strategy +you need to create an intermediate parent actor that defines the `supervisorStrategy` to the child entity actor. -.. includecode:: ../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala#supervisor +@@snip [ClusterShardingSpec.scala]../../../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala) { #supervisor } You start such a supervisor in the same way as if it was the entity actor. -.. includecode:: ../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala#counter-supervisor-start +@@snip [ClusterShardingSpec.scala]../../../../../akka-cluster-sharding/src/multi-jvm/scala/akka/cluster/sharding/ClusterShardingSpec.scala) { #counter-supervisor-start } Note that stopped entities will be started again when a new message is targeted to the entity. -Graceful Shutdown ------------------ +## Graceful Shutdown -You can send the message ``ShardRegion.GracefulShutdown`` message to the ``ShardRegion`` actor to handoff all shards that are hosted by that ``ShardRegion`` and then the -``ShardRegion`` actor will be stopped. You can ``watch`` the ``ShardRegion`` actor to know when it is completed. +You can send the message `ShardRegion.GracefulShutdown` message to the `ShardRegion` actor to handoff all shards that are hosted by that `ShardRegion` and then the +`ShardRegion` actor will be stopped. You can `watch` the `ShardRegion` actor to know when it is completed. During this period other regions will buffer messages for those shards in the same way as when a rebalance is triggered by the coordinator. When the shards have been stopped the coordinator will allocate these shards elsewhere. -This is performed automatically by the :ref:`coordinated-shutdown-scala` and is therefore part of the +This is performed automatically by the @ref:[Coordinated Shutdown](actors.md#coordinated-shutdown-scala) and is therefore part of the graceful leaving process of a cluster member. -.. _RemoveInternalClusterShardingData-scala: - -Removal of Internal Cluster Sharding Data ------------------------------------------ + +## Removal of Internal Cluster Sharding Data The Cluster Sharding coordinator stores the locations of the shards using Akka Persistence. This data can safely be removed when restarting the whole Akka Cluster. Note that this is not application data. -There is a utility program ``akka.cluster.sharding.RemoveInternalClusterShardingData`` +There is a utility program `akka.cluster.sharding.RemoveInternalClusterShardingData` that removes this data. -.. warning:: +@@@ warning - Never use this program while there are running Akka Cluster nodes that are - using Cluster Sharding. Stop all Cluster nodes before using this program. +Never use this program while there are running Akka Cluster nodes that are +using Cluster Sharding. Stop all Cluster nodes before using this program. + +@@@ It can be needed to remove the data if the Cluster Sharding coordinator cannot startup because of corrupt data, which may happen if accidentally two clusters were running at the same time, e.g. caused by using auto-down and there was a network partition. -.. warning:: - **Don't use Cluster Sharding together with Automatic Downing**, - since it allows the cluster to split up into two separate clusters, which in turn will result - in *multiple shards and entities* being started, one in each separate cluster! - See :ref:`automatic-vs-manual-downing-scala`. +@@@ warning -Use this program as a standalone Java main program:: +**Don't use Cluster Sharding together with Automatic Downing**, +since it allows the cluster to split up into two separate clusters, which in turn will result +in *multiple shards and entities* being started, one in each separate cluster! +See @ref:[Downing](cluster-usage.md#automatic-vs-manual-downing-scala). - java -classpath - akka.cluster.sharding.RemoveInternalClusterShardingData - -2.3 entityType1 entityType2 entityType3 +@@@ -The program is included in the ``akka-cluster-sharding`` jar file. It +Use this program as a standalone Java main program: + +``` +java -classpath + akka.cluster.sharding.RemoveInternalClusterShardingData + -2.3 entityType1 entityType2 entityType3 +``` + +The program is included in the `akka-cluster-sharding` jar file. It is easiest to run it with same classpath and configuration as your ordinary application. It can be run from sbt or maven in similar way. -Specify the entity type names (same as you use in the ``start`` method -of ``ClusterSharding``) as program arguments. +Specify the entity type names (same as you use in the `start` method +of `ClusterSharding`) as program arguments. -If you specify ``-2.3`` as the first program argument it will also try +If you specify `-2.3` as the first program argument it will also try to remove data that was stored by Cluster Sharding in Akka 2.3.x using different persistenceId. -Dependencies ------------- +## Dependencies To use the Cluster Sharding you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-cluster-sharding" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-cluster-sharding" % "@version@" @crossString@ +``` -maven:: +maven: - - com.typesafe.akka - akka-cluster-sharding_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-cluster-sharding_@binVersion@ + @version@ + +``` -Configuration -------------- +## Configuration -The ``ClusterSharding`` extension can be configured with the following properties. These configuration -properties are read by the ``ClusterShardingSettings`` when created with a ``ActorSystem`` parameter. -It is also possible to amend the ``ClusterShardingSettings`` or create it from another config section -with the same layout as below. ``ClusterShardingSettings`` is a parameter to the ``start`` method of -the ``ClusterSharding`` extension, i.e. each each entity type can be configured with different settings +The `ClusterSharding` extension can be configured with the following properties. These configuration +properties are read by the `ClusterShardingSettings` when created with a `ActorSystem` parameter. +It is also possible to amend the `ClusterShardingSettings` or create it from another config section +with the same layout as below. `ClusterShardingSettings` is a parameter to the `start` method of +the `ClusterSharding` extension, i.e. each each entity type can be configured with different settings if needed. -.. includecode:: ../../../akka-cluster-sharding/src/main/resources/reference.conf#sharding-ext-config +@@snip [reference.conf]../../../../../akka-cluster-sharding/src/main/resources/reference.conf) { #sharding-ext-config } Custom shard allocation strategy can be defined in an optional parameter to -``ClusterSharding.start``. See the API documentation of ``ShardAllocationStrategy`` for details of +`ClusterSharding.start`. See the API documentation of `ShardAllocationStrategy` for details of how to implement a custom shard allocation strategy. +## Inspecting cluster sharding state -Inspecting cluster sharding state ---------------------------------- Two requests to inspect the cluster state are available: -``ShardRegion.GetShardRegionState`` which will return a ``ShardRegion.CurrentShardRegionState`` that contains +`ShardRegion.GetShardRegionState` which will return a `ShardRegion.CurrentShardRegionState` that contains the identifiers of the shards running in a Region and what entities are alive for each of them. -``ShardRegion.GetClusterShardingStats`` which will query all the regions in the cluster and return -a ``ShardRegion.ClusterShardingStats`` containing the identifiers of the shards running in each region and a count +`ShardRegion.GetClusterShardingStats` which will query all the regions in the cluster and return +a `ShardRegion.ClusterShardingStats` containing the identifiers of the shards running in each region and a count of entities that are alive in each shard. The purpose of these messages is testing and monitoring, they are not provided to give access to -directly sending messages to the individual entities. +directly sending messages to the individual entities. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/cluster-singleton.md b/akka-docs/src/main/paradox/scala/cluster-singleton.md index 134dd8a1f9..825b9bc55a 100644 --- a/akka-docs/src/main/paradox/scala/cluster-singleton.md +++ b/akka-docs/src/main/paradox/scala/cluster-singleton.md @@ -1,34 +1,31 @@ -.. _cluster-singleton-scala: - -Cluster Singleton -================= +# Cluster Singleton For some use cases it is convenient and sometimes also mandatory to ensure that you have exactly one actor of a certain type running somewhere in the cluster. Some examples: -* single point of responsibility for certain cluster-wide consistent decisions, or - coordination of actions across the cluster system -* single entry point to an external system -* single master, many workers -* centralized naming service, or routing logic + * single point of responsibility for certain cluster-wide consistent decisions, or +coordination of actions across the cluster system + * single entry point to an external system + * single master, many workers + * centralized naming service, or routing logic Using a singleton should not be the first design choice. It has several drawbacks, such as single-point of bottleneck. Single-point of failure is also a relevant concern, but for some cases this feature takes care of that by making sure that another singleton instance will eventually be started. -The cluster singleton pattern is implemented by ``akka.cluster.singleton.ClusterSingletonManager``. +The cluster singleton pattern is implemented by `akka.cluster.singleton.ClusterSingletonManager`. It manages one singleton actor instance among all cluster nodes or a group of nodes tagged with -a specific role. ``ClusterSingletonManager`` is an actor that is supposed to be started on +a specific role. `ClusterSingletonManager` is an actor that is supposed to be started on all nodes, or all nodes with specified role, in the cluster. The actual singleton actor is -started by the ``ClusterSingletonManager`` on the oldest node by creating a child actor from -supplied ``Props``. ``ClusterSingletonManager`` makes sure that at most one singleton instance +started by the `ClusterSingletonManager` on the oldest node by creating a child actor from +supplied `Props`. `ClusterSingletonManager` makes sure that at most one singleton instance is running at any point in time. The singleton actor is always running on the oldest member with specified role. -The oldest member is determined by ``akka.cluster.Member#isOlderThan``. +The oldest member is determined by `akka.cluster.Member#isOlderThan`. This can change when removing that member from the cluster. Be aware that there is a short time period when there is no active singleton during the hand-over process. @@ -38,46 +35,47 @@ take over and a new singleton actor is created. For these failure scenarios ther not be a graceful hand-over, but more than one active singletons is prevented by all reasonable means. Some corner cases are eventually resolved by configurable timeouts. -You can access the singleton actor by using the provided ``akka.cluster.singleton.ClusterSingletonProxy``, +You can access the singleton actor by using the provided `akka.cluster.singleton.ClusterSingletonProxy`, which will route all messages to the current instance of the singleton. The proxy will keep track of -the oldest node in the cluster and resolve the singleton's ``ActorRef`` by explicitly sending the -singleton's ``actorSelection`` the ``akka.actor.Identify`` message and waiting for it to reply. +the oldest node in the cluster and resolve the singleton's `ActorRef` by explicitly sending the +singleton's `actorSelection` the `akka.actor.Identify` message and waiting for it to reply. This is performed periodically if the singleton doesn't reply within a certain (configurable) time. -Given the implementation, there might be periods of time during which the ``ActorRef`` is unavailable, +Given the implementation, there might be periods of time during which the `ActorRef` is unavailable, e.g., when a node leaves the cluster. In these cases, the proxy will buffer the messages sent to the singleton and then deliver them when the singleton is finally available. If the buffer is full -the ``ClusterSingletonProxy`` will drop old messages when new messages are sent via the proxy. +the `ClusterSingletonProxy` will drop old messages when new messages are sent via the proxy. The size of the buffer is configurable and it can be disabled by using a buffer size of 0. It's worth noting that messages can always be lost because of the distributed nature of these actors. As always, additional logic should be implemented in the singleton (acknowledgement) and in the client (retry) actors to ensure at-least-once message delivery. -The singleton instance will not run on members with status :ref:`WeaklyUp `. +The singleton instance will not run on members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-scala). -Potential problems to be aware of ---------------------------------- +## Potential problems to be aware of This pattern may seem to be very tempting to use at first, but it has several drawbacks, some of them are listed below: -* the cluster singleton may quickly become a *performance bottleneck*, -* you can not rely on the cluster singleton to be *non-stop* available — e.g. when the node on which the singleton has - been running dies, it will take a few seconds for this to be noticed and the singleton be migrated to another node, -* in the case of a *network partition* appearing in a Cluster that is using Automatic Downing (see Auto Downing docs for - :ref:`automatic-vs-manual-downing-scala`), - it may happen that the isolated clusters each decide to spin up their own singleton, meaning that there might be multiple - singletons running in the system, yet the Clusters have no way of finding out about them (because of the partition). + * the cluster singleton may quickly become a *performance bottleneck*, + * you can not rely on the cluster singleton to be *non-stop* available — e.g. when the node on which the singleton has +been running dies, it will take a few seconds for this to be noticed and the singleton be migrated to another node, + * in the case of a *network partition* appearing in a Cluster that is using Automatic Downing (see Auto Downing docs for +@ref:[Downing](cluster-usage.md#automatic-vs-manual-downing-scala)), +it may happen that the isolated clusters each decide to spin up their own singleton, meaning that there might be multiple +singletons running in the system, yet the Clusters have no way of finding out about them (because of the partition). Especially the last point is something you should be aware of — in general when using the Cluster Singleton pattern you should take care of downing nodes yourself and not rely on the timing based auto-down feature. -.. warning:: - **Don't use Cluster Singleton together with Automatic Downing**, - since it allows the cluster to split up into two separate clusters, which in turn will result - in *multiple Singletons* being started, one in each separate cluster! +@@@ warning -An Example ----------- +**Don't use Cluster Singleton together with Automatic Downing**, +since it allows the cluster to split up into two separate clusters, which in turn will result +in *multiple Singletons* being started, one in each separate cluster! + +@@@ + +## An Example Assume that we need one single entry point to an external system. An actor that receives messages from a JMS queue with the strict requirement that only one @@ -85,62 +83,63 @@ JMS consumer must exist to be make sure that the messages are processed in order That is perhaps not how one would like to design things, but a typical real-world scenario when integrating with external systems. -On each node in the cluster you need to start the ``ClusterSingletonManager`` and -supply the ``Props`` of the singleton actor, in this case the JMS queue consumer. +On each node in the cluster you need to start the `ClusterSingletonManager` and +supply the `Props` of the singleton actor, in this case the JMS queue consumer. -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/singleton/ClusterSingletonManagerSpec.scala#create-singleton-manager +@@snip [ClusterSingletonManagerSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/singleton/ClusterSingletonManagerSpec.scala) { #create-singleton-manager } -Here we limit the singleton to nodes tagged with the ``"worker"`` role, but all nodes, independent of -role, can be used by not specifying ``withRole``. +Here we limit the singleton to nodes tagged with the `"worker"` role, but all nodes, independent of +role, can be used by not specifying `withRole`. -Here we use an application specific ``terminationMessage`` to be able to close the -resources before actually stopping the singleton actor. Note that ``PoisonPill`` is a -perfectly fine ``terminationMessage`` if you only need to stop the actor. +Here we use an application specific `terminationMessage` to be able to close the +resources before actually stopping the singleton actor. Note that `PoisonPill` is a +perfectly fine `terminationMessage` if you only need to stop the actor. -Here is how the singleton actor handles the ``terminationMessage`` in this example. +Here is how the singleton actor handles the `terminationMessage` in this example. -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/singleton/ClusterSingletonManagerSpec.scala#consumer-end +@@snip [ClusterSingletonManagerSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/singleton/ClusterSingletonManagerSpec.scala) { #consumer-end } With the names given above, access to the singleton can be obtained from any cluster node using a properly configured proxy. -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/singleton/ClusterSingletonManagerSpec.scala#create-singleton-proxy +@@snip [ClusterSingletonManagerSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/singleton/ClusterSingletonManagerSpec.scala) { #create-singleton-proxy } -A more comprehensive sample is available in the tutorial named `Distributed workers with Akka and Scala! `_. +A more comprehensive sample is available in the tutorial named [Distributed workers with Akka and Scala!](https://github.com/typesafehub/activator-akka-distributed-workers). -Dependencies ------------- +## Dependencies To use the Cluster Singleton you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` -maven:: +maven: - - com.typesafe.akka - akka-cluster-tools_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-cluster-tools_@binVersion@ + @version@ + +``` +## Configuration -Configuration -------------- - -The following configuration properties are read by the ``ClusterSingletonManagerSettings`` -when created with a ``ActorSystem`` parameter. It is also possible to amend the ``ClusterSingletonManagerSettings`` -or create it from another config section with the same layout as below. ``ClusterSingletonManagerSettings`` is -a parameter to the ``ClusterSingletonManager.props`` factory method, i.e. each singleton can be configured +The following configuration properties are read by the `ClusterSingletonManagerSettings` +when created with a `ActorSystem` parameter. It is also possible to amend the `ClusterSingletonManagerSettings` +or create it from another config section with the same layout as below. `ClusterSingletonManagerSettings` is +a parameter to the `ClusterSingletonManager.props` factory method, i.e. each singleton can be configured with different settings if needed. -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#singleton-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #singleton-config } -The following configuration properties are read by the ``ClusterSingletonProxySettings`` -when created with a ``ActorSystem`` parameter. It is also possible to amend the ``ClusterSingletonProxySettings`` -or create it from another config section with the same layout as below. ``ClusterSingletonProxySettings`` is -a parameter to the ``ClusterSingletonProxy.props`` factory method, i.e. each singleton proxy can be configured +The following configuration properties are read by the `ClusterSingletonProxySettings` +when created with a `ActorSystem` parameter. It is also possible to amend the `ClusterSingletonProxySettings` +or create it from another config section with the same layout as below. `ClusterSingletonProxySettings` is +a parameter to the `ClusterSingletonProxy.props` factory method, i.e. each singleton proxy can be configured with different settings if needed. -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#singleton-proxy-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #singleton-proxy-config } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/cluster-usage.md b/akka-docs/src/main/paradox/scala/cluster-usage.md index 41b53acf54..4f504ca7c2 100644 --- a/akka-docs/src/main/paradox/scala/cluster-usage.md +++ b/akka-docs/src/main/paradox/scala/cluster-usage.md @@ -1,91 +1,87 @@ +# Cluster Usage -.. _cluster_usage_scala: +For introduction to the Akka Cluster concepts please see cluster. -############# -Cluster Usage -############# +## Preparing Your Project for Clustering -For introduction to the Akka Cluster concepts please see :ref:`cluster`. +The Akka cluster is a separate jar file. Make sure that you have the following dependency in your project: -Preparing Your Project for Clustering -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``` +"com.typesafe.akka" %% "akka-cluster" % "@version@" @crossString@ +``` -The Akka cluster is a separate jar file. Make sure that you have the following dependency in your project:: +## A Simple Cluster Example - "com.typesafe.akka" %% "akka-cluster" % "@version@" @crossString@ - -A Simple Cluster Example -^^^^^^^^^^^^^^^^^^^^^^^^ - -The following configuration enables the ``Cluster`` extension to be used. +The following configuration enables the `Cluster` extension to be used. It joins the cluster and an actor subscribes to cluster membership events and logs them. -The ``application.conf`` configuration looks like this: +The `application.conf` configuration looks like this: -:: - - akka { - actor { - provider = "cluster" - } - remote { - log-remote-lifecycle-events = off - netty.tcp { - hostname = "127.0.0.1" - port = 0 - } - } - - cluster { - seed-nodes = [ - "akka.tcp://ClusterSystem@127.0.0.1:2551", - "akka.tcp://ClusterSystem@127.0.0.1:2552"] - - # auto downing is NOT safe for production deployments. - # you may want to use it during development, read more about it in the docs. - # - # auto-down-unreachable-after = 10s - } +``` +akka { + actor { + provider = "cluster" + } + remote { + log-remote-lifecycle-events = off + netty.tcp { + hostname = "127.0.0.1" + port = 0 } + } - # Disable legacy metrics in akka-cluster. - akka.cluster.metrics.enabled=off + cluster { + seed-nodes = [ + "akka.tcp://ClusterSystem@127.0.0.1:2551", + "akka.tcp://ClusterSystem@127.0.0.1:2552"] - # Enable metrics extension in akka-cluster-metrics. - akka.extensions=["akka.cluster.metrics.ClusterMetricsExtension"] + # auto downing is NOT safe for production deployments. + # you may want to use it during development, read more about it in the docs. + # + # auto-down-unreachable-after = 10s + } +} - # Sigar native library extract location during tests. - # Note: use per-jvm-instance folder when running multiple jvm on one host. - akka.cluster.metrics.native-library-extract-folder=${user.dir}/target/native +# Disable legacy metrics in akka-cluster. +akka.cluster.metrics.enabled=off -To enable cluster capabilities in your Akka project you should, at a minimum, add the :ref:`remoting-scala` -settings, but with ``cluster``. -The ``akka.cluster.seed-nodes`` should normally also be added to your ``application.conf`` file. +# Enable metrics extension in akka-cluster-metrics. +akka.extensions=["akka.cluster.metrics.ClusterMetricsExtension"] -.. note:: - If you are running Akka in a Docker container or the nodes for some other reason have separate internal and - external ip addresses you must configure remoting according to :ref:`remote-configuration-nat` +# Sigar native library extract location during tests. +# Note: use per-jvm-instance folder when running multiple jvm on one host. +akka.cluster.metrics.native-library-extract-folder=${user.dir}/target/native +``` + +To enable cluster capabilities in your Akka project you should, at a minimum, add the @ref:[Remoting](remoting.md) +settings, but with `cluster`. +The `akka.cluster.seed-nodes` should normally also be added to your `application.conf` file. + +@@@ note + +If you are running Akka in a Docker container or the nodes for some other reason have separate internal and +external ip addresses you must configure remoting according to @ref:[Akka behind NAT or in a Docker container](remoting.md#remote-configuration-nat) + +@@@ The seed nodes are configured contact points for initial, automatic, join of the cluster. Note that if you are going to start the nodes on different machines you need to specify the -ip-addresses or host names of the machines in ``application.conf`` instead of ``127.0.0.1`` +ip-addresses or host names of the machines in `application.conf` instead of `127.0.0.1` An actor that uses the cluster extension may look like this: -.. literalinclude:: code/docs/cluster/SimpleClusterListener.scala - :language: scala +@@snip [SimpleClusterListener.scala](code/docs/cluster/SimpleClusterListener.scala) { type=scala } The actor registers itself as subscriber of certain cluster events. It receives events corresponding to the current state of the cluster when the subscription starts and then it receives events for changes that happen in the cluster. The easiest way to run this example yourself is to download the ready to run -`Akka Cluster Sample with Scala <@exampleCodeService@/akka-samples-cluster-scala>`_ -together with the tutorial. It contains instructions on how to run the ``SimpleClusterApp``. -The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-cluster-scala>`_. +[Akka Cluster Sample with Scala](@exampleCodeService@/akka-samples-cluster-scala) +together with the tutorial. It contains instructions on how to run the `SimpleClusterApp`. +The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-cluster-scala). -Joining to Seed Nodes -^^^^^^^^^^^^^^^^^^^^^ +## Joining to Seed Nodes You may decide if joining to the cluster should be done manually or automatically to configured initial contact points, so-called seed nodes. When a new node is started @@ -93,25 +89,29 @@ it sends a message to all seed nodes and then sends join command to the one that answers first. If no one of the seed nodes replied (might not be started yet) it retries this procedure until successful or shutdown. -You define the seed nodes in the :ref:`cluster_configuration_scala` file (application.conf):: +You define the seed nodes in the [cluster_configuration_scala](#cluster-configuration-scala) file (application.conf): - akka.cluster.seed-nodes = [ - "akka.tcp://ClusterSystem@host1:2552", - "akka.tcp://ClusterSystem@host2:2552"] +``` +akka.cluster.seed-nodes = [ + "akka.tcp://ClusterSystem@host1:2552", + "akka.tcp://ClusterSystem@host2:2552"] +``` -This can also be defined as Java system properties when starting the JVM using the following syntax:: +This can also be defined as Java system properties when starting the JVM using the following syntax: - -Dakka.cluster.seed-nodes.0=akka.tcp://ClusterSystem@host1:2552 - -Dakka.cluster.seed-nodes.1=akka.tcp://ClusterSystem@host2:2552 +``` +-Dakka.cluster.seed-nodes.0=akka.tcp://ClusterSystem@host1:2552 +-Dakka.cluster.seed-nodes.1=akka.tcp://ClusterSystem@host2:2552 +``` The seed nodes can be started in any order and it is not necessary to have all -seed nodes running, but the node configured as the first element in the ``seed-nodes`` +seed nodes running, but the node configured as the first element in the `seed-nodes` configuration list must be started when initially starting a cluster, otherwise the other seed-nodes will not become initialized and no other node can join the cluster. The reason for the special first seed node is to avoid forming separated islands when starting from an empty cluster. It is quickest to start all configured seed nodes at the same time (order doesn't matter), -otherwise it can take up to the configured ``seed-node-timeout`` until the nodes +otherwise it can take up to the configured `seed-node-timeout` until the nodes can join. Once more than two seed nodes have been started it is no problem to shut down the first @@ -120,27 +120,27 @@ seed nodes in the existing cluster. If you don't configure seed nodes you need to join the cluster programmatically or manually. -Manual joining can be performed by using :ref:`cluster_jmx_scala` or :ref:`cluster_http_scala`. -Joining programmatically can be performed with ``Cluster(system).join``. Unsuccessful join attempts are -automatically retried after the time period defined in configuration property ``retry-unsuccessful-join-after``. -Retries can be disabled by setting the property to ``off``. +Manual joining can be performed by using [cluster_jmx_scala](#cluster-jmx-scala) or [cluster_http_scala](#cluster-http-scala). +Joining programmatically can be performed with `Cluster(system).join`. Unsuccessful join attempts are +automatically retried after the time period defined in configuration property `retry-unsuccessful-join-after`. +Retries can be disabled by setting the property to `off`. You can join to any node in the cluster. It does not have to be configured as a seed node. Note that you can only join to an existing cluster member, which means that for bootstrapping some node must join itself,and then the following nodes could join them to make up a cluster. -You may also use ``Cluster(system).joinSeedNodes`` to join programmatically, +You may also use `Cluster(system).joinSeedNodes` to join programmatically, which is attractive when dynamically discovering other nodes at startup by using some external tool or API. -When using ``joinSeedNodes`` you should not include the node itself except for the node that is +When using `joinSeedNodes` you should not include the node itself except for the node that is supposed to be the first seed node, and that should be placed first in parameter to -``joinSeedNodes``. +`joinSeedNodes`. Unsuccessful attempts to contact seed nodes are automatically retried after the time period defined in -configuration property ``seed-node-timeout``. Unsuccessful attempt to join a specific seed node is -automatically retried after the configured ``retry-unsuccessful-join-after``. Retrying means that it +configuration property `seed-node-timeout`. Unsuccessful attempt to join a specific seed node is +automatically retried after the configured `retry-unsuccessful-join-after`. Retrying means that it tries to contact all seed nodes and then joins the node that answers first. The first node in the list of seed nodes will join itself if it cannot contact any of the other seed nodes within the -configured ``seed-node-timeout``. +configured `seed-node-timeout`. An actor system can only join a cluster once. Additional attempts will be ignored. When it has successfully joined it must be restarted to be able to join another @@ -149,41 +149,42 @@ after the restart, when it come up as new incarnation of existing member in the trying to join in, then the existing one will be removed from the cluster and then it will be allowed to join. -.. note:: +@@@ note - The name of the ``ActorSystem`` must be the same for all members of a cluster. The name is given - when you start the ``ActorSystem``. +The name of the `ActorSystem` must be the same for all members of a cluster. The name is given +when you start the `ActorSystem`. -.. _automatic-vs-manual-downing-scala: +@@@ -Downing -^^^^^^^ + +## Downing When a member is considered by the failure detector to be unreachable the leader is not allowed to perform its duties, such as changing status of new joining members to 'Up'. The node must first become reachable again, or the status of the unreachable member must be changed to 'Down'. Changing status to 'Down' can be performed automatically or manually. By default it must be done manually, using -:ref:`cluster_jmx_scala` or :ref:`cluster_http_scala`. +[cluster_jmx_scala](#cluster-jmx-scala) or [cluster_http_scala](#cluster-http-scala). -It can also be performed programmatically with ``Cluster(system).down(address)``. +It can also be performed programmatically with `Cluster(system).down(address)`. A pre-packaged solution for the downing problem is provided by -`Split Brain Resolver `_, -which is part of the `Lightbend Reactive Platform `_. -If you don’t use RP, you should anyway carefully read the `documentation `_ +[Split Brain Resolver](http://developer.lightbend.com/docs/akka-commercial-addons/current/split-brain-resolver.html), +which is part of the [Lightbend Reactive Platform](http://www.lightbend.com/platform). +If you don’t use RP, you should anyway carefully read the [documentation](http://developer.lightbend.com/docs/akka-commercial-addons/current/split-brain-resolver.html) of the Split Brain Resolver and make sure that the solution you are using handles the concerns described there. -Auto-downing (DO NOT USE) -------------------------- +### Auto-downing (DO NOT USE) -There is an automatic downing feature that you should not use in production. For testing purpose you can enable it with configuration:: +There is an automatic downing feature that you should not use in production. For testing purpose you can enable it with configuration: - akka.cluster.auto-down-unreachable-after = 120s +``` +akka.cluster.auto-down-unreachable-after = 120s +``` -This means that the cluster leader member will change the ``unreachable`` node -status to ``down`` automatically after the configured time of unreachability. +This means that the cluster leader member will change the `unreachable` node +status to `down` automatically after the configured time of unreachability. This is a naïve approach to remove unreachable nodes from the cluster membership. It works great for crashes and short transient network partitions, but not for long network @@ -192,17 +193,17 @@ and after a while remove it from its cluster membership. Since this happens on b sides the result is that two separate disconnected clusters have been created. This can also happen because of long GC pauses or system overload. -.. warning:: +@@@ warning - We recommend against using the auto-down feature of Akka Cluster in production. - This is crucial for correct behavior if you use :ref:`cluster-singleton-scala` or - :ref:`cluster_sharding_scala`, especially together with Akka :ref:`persistence-scala`. - For Akka Persistence with Cluster Sharding it can result in corrupt data in case - of network partitions. +We recommend against using the auto-down feature of Akka Cluster in production. +This is crucial for correct behavior if you use @ref:[Cluster Singleton](cluster-singleton.md) or +@ref:[cluster_sharding_scala](cluster-sharding.md), especially together with Akka @ref:[Persistence](persistence.md). +For Akka Persistence with Cluster Sharding it can result in corrupt data in case +of network partitions. +@@@ -Leaving -^^^^^^^ +## Leaving There are two ways to remove a member from the cluster. @@ -211,95 +212,91 @@ as unreachable and removed after the automatic or manual downing as described above. A more graceful exit can be performed if you tell the cluster that a node shall leave. -This can be performed using :ref:`cluster_jmx_scala` or :ref:`cluster_http_scala`. +This can be performed using [cluster_jmx_scala](#cluster-jmx-scala) or [cluster_http_scala](#cluster-http-scala). It can also be performed programmatically with: -.. includecode:: code/docs/cluster/ClusterDocSpec.scala#leave +@@snip [ClusterDocSpec.scala](code/docs/cluster/ClusterDocSpec.scala) { #leave } Note that this command can be issued to any member in the cluster, not necessarily the one that is leaving. -The :ref:`coordinated-shutdown-scala` will automatically run when the cluster node sees itself as -``Exiting``, i.e. leaving from another node will trigger the shutdown process on the leaving node. +The @ref:[Coordinated Shutdown](actors.md#coordinated-shutdown-scala) will automatically run when the cluster node sees itself as +`Exiting`, i.e. leaving from another node will trigger the shutdown process on the leaving node. Tasks for graceful leaving of cluster including graceful shutdown of Cluster Singletons and Cluster Sharding are added automatically when Akka Cluster is used, i.e. running the shutdown process will also trigger the graceful leaving if it's not already in progress. Normally this is handled automatically, but in case of network failures during this process it might still -be necessary to set the node’s status to ``Down`` in order to complete the removal. +be necessary to set the node’s status to `Down` in order to complete the removal. -.. _weakly_up_scala: + +## WeaklyUp Members -WeaklyUp Members -^^^^^^^^^^^^^^^^ - -If a node is ``unreachable`` then gossip convergence is not possible and therefore any -``leader`` actions are also not possible. However, we still might want new nodes to join +If a node is `unreachable` then gossip convergence is not possible and therefore any +`leader` actions are also not possible. However, we still might want new nodes to join the cluster in this scenario. -``Joining`` members will be promoted to ``WeaklyUp`` and become part of the cluster if -convergence can't be reached. Once gossip convergence is reached, the leader will move ``WeaklyUp`` -members to ``Up``. +`Joining` members will be promoted to `WeaklyUp` and become part of the cluster if +convergence can't be reached. Once gossip convergence is reached, the leader will move `WeaklyUp` +members to `Up`. -This feature is enabled by default, but it can be disabled with configuration option:: +This feature is enabled by default, but it can be disabled with configuration option: - akka.cluster.allow-weakly-up-members = off +``` +akka.cluster.allow-weakly-up-members = off +``` -You can subscribe to the ``WeaklyUp`` membership event to make use of the members that are +You can subscribe to the `WeaklyUp` membership event to make use of the members that are in this state, but you should be aware of that members on the other side of a network partition have no knowledge about the existence of the new members. You should for example not count -``WeaklyUp`` members in quorum decisions. +`WeaklyUp` members in quorum decisions. - -.. _cluster_subscriber_scala: - -Subscribe to Cluster Events -^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## Subscribe to Cluster Events You can subscribe to change notifications of the cluster membership by using -``Cluster(system).subscribe``. +`Cluster(system).subscribe`. -.. includecode:: code/docs/cluster/SimpleClusterListener2.scala +@@snip [SimpleClusterListener2.scala](code/docs/cluster/SimpleClusterListener2.scala) { # } -A snapshot of the full state, ``akka.cluster.ClusterEvent.CurrentClusterState``, is sent to the subscriber +A snapshot of the full state, `akka.cluster.ClusterEvent.CurrentClusterState`, is sent to the subscriber as the first message, followed by events for incremental updates. -Note that you may receive an empty ``CurrentClusterState``, containing no members, +Note that you may receive an empty `CurrentClusterState`, containing no members, if you start the subscription before the initial join procedure has completed. This is expected behavior. When the node has been accepted in the cluster you will -receive ``MemberUp`` for that node, and other nodes. +receive `MemberUp` for that node, and other nodes. -If you find it inconvenient to handle the ``CurrentClusterState`` you can use -``ClusterEvent.InitialStateAsEvents`` as parameter to ``subscribe``. -That means that instead of receiving ``CurrentClusterState`` as the first message you will receive +If you find it inconvenient to handle the `CurrentClusterState` you can use +`ClusterEvent.InitialStateAsEvents` as parameter to `subscribe`. +That means that instead of receiving `CurrentClusterState` as the first message you will receive the events corresponding to the current state to mimic what you would have seen if you were listening to the events when they occurred in the past. Note that those initial events only correspond to the current state and it is not the full history of all changes that actually has occurred in the cluster. -.. includecode:: code/docs/cluster/SimpleClusterListener.scala#subscribe +@@snip [SimpleClusterListener.scala](code/docs/cluster/SimpleClusterListener.scala) { #subscribe } The events to track the life-cycle of members are: -* ``ClusterEvent.MemberJoined`` - A new member has joined the cluster and its status has been changed to ``Joining``. -* ``ClusterEvent.MemberUp`` - A new member has joined the cluster and its status has been changed to ``Up``. -* ``ClusterEvent.MemberExited`` - A member is leaving the cluster and its status has been changed to ``Exiting``. - Note that the node might already have been shutdown when this event is published on another node. -* ``ClusterEvent.MemberRemoved`` - Member completely removed from the cluster. -* ``ClusterEvent.UnreachableMember`` - A member is considered as unreachable, detected by the failure detector - of at least one other node. -* ``ClusterEvent.ReachableMember`` - A member is considered as reachable again, after having been unreachable. - All nodes that previously detected it as unreachable has detected it as reachable again. + * `ClusterEvent.MemberJoined` - A new member has joined the cluster and its status has been changed to `Joining`. + * `ClusterEvent.MemberUp` - A new member has joined the cluster and its status has been changed to `Up`. + * `ClusterEvent.MemberExited` - A member is leaving the cluster and its status has been changed to `Exiting`. +Note that the node might already have been shutdown when this event is published on another node. + * `ClusterEvent.MemberRemoved` - Member completely removed from the cluster. + * `ClusterEvent.UnreachableMember` - A member is considered as unreachable, detected by the failure detector +of at least one other node. + * `ClusterEvent.ReachableMember` - A member is considered as reachable again, after having been unreachable. +All nodes that previously detected it as unreachable has detected it as reachable again. There are more types of change events, consult the API documentation -of classes that extends ``akka.cluster.ClusterEvent.ClusterDomainEvent`` +of classes that extends `akka.cluster.ClusterEvent.ClusterDomainEvent` for details about the events. Instead of subscribing to cluster events it can sometimes be convenient to only get the full membership state with -``Cluster(system).state``. Note that this state is not necessarily in sync with the events published to a +`Cluster(system).state`. Note that this state is not necessarily in sync with the events published to a cluster subscription. -Worker Dial-in Example ----------------------- +### Worker Dial-in Example Let's take a look at an example that illustrates how workers, here named *backend*, can detect and register to new master nodes, here named *frontend*. @@ -312,181 +309,180 @@ added or removed to the cluster dynamically. Messages: -.. includecode:: code/docs/cluster/TransformationMessages.scala#messages +@@snip [TransformationMessages.scala](code/docs/cluster/TransformationMessages.scala) { #messages } The backend worker that performs the transformation job: -.. includecode:: code/docs/cluster/TransformationBackend.scala#backend +@@snip [TransformationBackend.scala](code/docs/cluster/TransformationBackend.scala) { #backend } -Note that the ``TransformationBackend`` actor subscribes to cluster events to detect new, +Note that the `TransformationBackend` actor subscribes to cluster events to detect new, potential, frontend nodes, and send them a registration message so that they know that they can use the backend worker. The frontend that receives user jobs and delegates to one of the registered backend workers: -.. includecode:: code/docs/cluster/TransformationFrontend.scala#frontend +@@snip [TransformationFrontend.scala](code/docs/cluster/TransformationFrontend.scala) { #frontend } -Note that the ``TransformationFrontend`` actor watch the registered backend +Note that the `TransformationFrontend` actor watch the registered backend to be able to remove it from its list of available backend workers. Death watch uses the cluster failure detector for nodes in the cluster, i.e. it detects network failures and JVM crashes, in addition to graceful termination of watched -actor. Death watch generates the ``Terminated`` message to the watching actor when the +actor. Death watch generates the `Terminated` message to the watching actor when the unreachable cluster node has been downed and removed. The easiest way to run **Worker Dial-in Example** example yourself is to download the ready to run -`Akka Cluster Sample with Scala <@exampleCodeService@/akka-samples-cluster-scala>`_ +[Akka Cluster Sample with Scala](@exampleCodeService@/akka-samples-cluster-scala) together with the tutorial. It contains instructions on how to run the **Worker Dial-in Example** sample. -The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-cluster-scala>`_. +The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-cluster-scala). -Node Roles -^^^^^^^^^^ +## Node Roles Not all nodes of a cluster need to perform the same function: there might be one sub-set which runs the web front-end, one which runs the data access layer and one for the number-crunching. Deployment of actors—for example by cluster-aware routers—can take node roles into account to achieve this distribution of responsibilities. -The roles of a node is defined in the configuration property named ``akka.cluster.roles`` +The roles of a node is defined in the configuration property named `akka.cluster.roles` and it is typically defined in the start script as a system property or environment variable. -The roles of the nodes is part of the membership information in ``MemberEvent`` that you can subscribe to. +The roles of the nodes is part of the membership information in `MemberEvent` that you can subscribe to. -.. _min-members_scala: - -How To Startup when Cluster Size Reached -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## How To Startup when Cluster Size Reached A common use case is to start actors after the cluster has been initialized, members have joined, and the cluster has reached a certain size. With a configuration option you can define required number of members -before the leader changes member status of 'Joining' members to 'Up'.:: +before the leader changes member status of 'Joining' members to 'Up'.: - akka.cluster.min-nr-of-members = 3 +``` +akka.cluster.min-nr-of-members = 3 +``` In a similar way you can define required number of members of a certain role -before the leader changes member status of 'Joining' members to 'Up'.:: +before the leader changes member status of 'Joining' members to 'Up'.: - akka.cluster.role { - frontend.min-nr-of-members = 1 - backend.min-nr-of-members = 2 - } +``` +akka.cluster.role { + frontend.min-nr-of-members = 1 + backend.min-nr-of-members = 2 +} +``` -You can start the actors in a ``registerOnMemberUp`` callback, which will +You can start the actors in a `registerOnMemberUp` callback, which will be invoked when the current member status is changed to 'Up', i.e. the cluster has at least the defined number of members. -.. includecode:: code/docs/cluster/FactorialFrontend.scala#registerOnUp +@@snip [FactorialFrontend.scala](code/docs/cluster/FactorialFrontend.scala) { #registerOnUp } This callback can be used for other things than starting actors. -How To Cleanup when Member is Removed -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## How To Cleanup when Member is Removed -You can do some clean up in a ``registerOnMemberRemoved`` callback, which will +You can do some clean up in a `registerOnMemberRemoved` callback, which will be invoked when the current member status is changed to 'Removed' or the cluster have been shutdown. -An alternative is to register tasks to the :ref:`coordinated-shutdown-scala`. +An alternative is to register tasks to the @ref:[Coordinated Shutdown](actors.md#coordinated-shutdown-scala). -.. note:: - Register a OnMemberRemoved callback on a cluster that have been shutdown, the callback will be invoked immediately on - the caller thread, otherwise it will be invoked later when the current member status changed to 'Removed'. You may - want to install some cleanup handling after the cluster was started up, but the cluster might already be shutting - down when you installing, and depending on the race is not healthy. +@@@ note -Cluster Singleton -^^^^^^^^^^^^^^^^^ +Register a OnMemberRemoved callback on a cluster that have been shutdown, the callback will be invoked immediately on +the caller thread, otherwise it will be invoked later when the current member status changed to 'Removed'. You may +want to install some cleanup handling after the cluster was started up, but the cluster might already be shutting +down when you installing, and depending on the race is not healthy. + +@@@ + +## Cluster Singleton For some use cases it is convenient and sometimes also mandatory to ensure that you have exactly one actor of a certain type running somewhere in the cluster. This can be implemented by subscribing to member events, but there are several corner cases to consider. Therefore, this specific use case is made easily accessible by the -:ref:`cluster-singleton-scala`. +@ref:[Cluster Singleton](cluster-singleton.md). -Cluster Sharding -^^^^^^^^^^^^^^^^ +## Cluster Sharding Distributes actors across several nodes in the cluster and supports interaction with the actors using their logical identifier, but without having to care about their physical location in the cluster. -See :ref:`cluster_sharding_scala` +See @ref:[cluster_sharding_scala](cluster-sharding.md) -Distributed Publish Subscribe -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## Distributed Publish Subscribe Publish-subscribe messaging between actors in the cluster, and point-to-point messaging using the logical path of the actors, i.e. the sender does not have to know on which node the destination actor is running. -See :ref:`distributed-pub-sub-scala`. +See @ref:[Distributed Publish Subscribe in Cluster](distributed-pub-sub.md). -Cluster Client -^^^^^^^^^^^^^^ +## Cluster Client Communication from an actor system that is not part of the cluster to actors running somewhere in the cluster. The client does not have to know on which node the destination actor is running. -See :ref:`cluster-client-scala`. +See @ref:[Cluster Client](cluster-client.md). -Distributed Data -^^^^^^^^^^^^^^^^ +## Distributed Data *Akka Distributed Data* is useful when you need to share data between nodes in an Akka Cluster. The data is accessed with an actor providing a key-value store like API. -See :ref:`distributed_data_scala`. +See @ref:[distributed_data_scala](distributed-data.md). -Failure Detector -^^^^^^^^^^^^^^^^ +## Failure Detector In a cluster each node is monitored by a few (default maximum 5) other nodes, and when -any of these detects the node as ``unreachable`` that information will spread to +any of these detects the node as `unreachable` that information will spread to the rest of the cluster through the gossip. In other words, only one node needs to -mark a node ``unreachable`` to have the rest of the cluster mark that node ``unreachable``. +mark a node `unreachable` to have the rest of the cluster mark that node `unreachable`. -The failure detector will also detect if the node becomes ``reachable`` again. When -all nodes that monitored the ``unreachable`` node detects it as ``reachable`` again -the cluster, after gossip dissemination, will consider it as ``reachable``. +The failure detector will also detect if the node becomes `reachable` again. When +all nodes that monitored the `unreachable` node detects it as `reachable` again +the cluster, after gossip dissemination, will consider it as `reachable`. If system messages cannot be delivered to a node it will be quarantined and then it -cannot come back from ``unreachable``. This can happen if the there are too many +cannot come back from `unreachable`. This can happen if the there are too many unacknowledged system messages (e.g. watch, Terminated, remote actor deployment, failures of actors supervised by remote parent). Then the node needs to be moved -to the ``down`` or ``removed`` states and the actor system of the quarantined node +to the `down` or `removed` states and the actor system of the quarantined node must be restarted before it can join the cluster again. The nodes in the cluster monitor each other by sending heartbeats to detect if a node is unreachable from the rest of the cluster. The heartbeat arrival times is interpreted by an implementation of -`The Phi Accrual Failure Detector `_. +[The Phi Accrual Failure Detector](http://www.jaist.ac.jp/~defago/files/pdf/IS_RR_2004_010.pdf). The suspicion level of failure is given by a value called *phi*. The basic idea of the phi failure detector is to express the value of *phi* on a scale that is dynamically adjusted to reflect current network conditions. -The value of *phi* is calculated as:: +The value of *phi* is calculated as: - phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` +phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` where F is the cumulative distribution function of a normal distribution with mean and standard deviation estimated from historical heartbeat inter-arrival times. -In the :ref:`cluster_configuration_scala` you can adjust the ``akka.cluster.failure-detector.threshold`` +In the [cluster_configuration_scala](#cluster-configuration-scala) you can adjust the `akka.cluster.failure-detector.threshold` to define when a *phi* value is considered to be a failure. -A low ``threshold`` is prone to generate many false positives but ensures -a quick detection in the event of a real crash. Conversely, a high ``threshold`` +A low `threshold` is prone to generate many false positives but ensures +a quick detection in the event of a real crash. Conversely, a high `threshold` generates fewer mistakes but needs more time to detect actual crashes. The -default ``threshold`` is 8 and is appropriate for most situations. However in +default `threshold` is 8 and is appropriate for most situations. However in cloud environments, such as Amazon EC2, the value could be increased to 12 in order to account for network issues that sometimes occur on such platforms. The following chart illustrates how *phi* increase with increasing time since the previous heartbeat. -.. image:: ../images/phi1.png +![phi1.png](../images/phi1.png) Phi is calculated from the mean and standard deviation of historical inter arrival times. The previous chart is an example for standard deviation @@ -494,95 +490,94 @@ of 200 ms. If the heartbeats arrive with less deviation the curve becomes steepe i.e. it is possible to determine failure more quickly. The curve looks like this for a standard deviation of 100 ms. -.. image:: ../images/phi2.png +![phi2.png](../images/phi2.png) To be able to survive sudden abnormalities, such as garbage collection pauses and transient network failures the failure detector is configured with a margin, -``akka.cluster.failure-detector.acceptable-heartbeat-pause``. You may want to -adjust the :ref:`cluster_configuration_scala` of this depending on you environment. -This is how the curve looks like for ``acceptable-heartbeat-pause`` configured to +`akka.cluster.failure-detector.acceptable-heartbeat-pause`. You may want to +adjust the [cluster_configuration_scala](#cluster-configuration-scala) of this depending on you environment. +This is how the curve looks like for `acceptable-heartbeat-pause` configured to 3 seconds. -.. image:: ../images/phi3.png - +![phi3.png](../images/phi3.png) Death watch uses the cluster failure detector for nodes in the cluster, i.e. it detects network failures and JVM crashes, in addition to graceful termination of watched -actor. Death watch generates the ``Terminated`` message to the watching actor when the +actor. Death watch generates the `Terminated` message to the watching actor when the unreachable cluster node has been downed and removed. If you encounter suspicious false positives when the system is under load you should -define a separate dispatcher for the cluster actors as described in :ref:`cluster_dispatcher_scala`. +define a separate dispatcher for the cluster actors as described in [cluster_dispatcher_scala](#cluster-dispatcher-scala). -.. _cluster_aware_routers_scala: + +## Cluster Aware Routers -Cluster Aware Routers -^^^^^^^^^^^^^^^^^^^^^ - -All :ref:`routers ` can be made aware of member nodes in the cluster, i.e. +All @ref:[routers](routing.md) can be made aware of member nodes in the cluster, i.e. deploying new routees or looking up routees on nodes in the cluster. When a node becomes unreachable or leaves the cluster the routees of that node are automatically unregistered from the router. When new nodes join the cluster, additional routees are added to the router, according to the configuration. Routees are also added when a node becomes reachable again, after having been unreachable. -Cluster aware routers make use of members with status :ref:`WeaklyUp ` if that feature +Cluster aware routers make use of members with status [WeaklyUp](#weakly-up-scala) if that feature is enabled. There are two distinct types of routers. -* **Group - router that sends messages to the specified path using actor selection** - The routees can be shared among routers running on different nodes in the cluster. - One example of a use case for this type of router is a service running on some backend - nodes in the cluster and used by routers running on front-end nodes in the cluster. + * **Group - router that sends messages to the specified path using actor selection** +The routees can be shared among routers running on different nodes in the cluster. +One example of a use case for this type of router is a service running on some backend +nodes in the cluster and used by routers running on front-end nodes in the cluster. + * **Pool - router that creates routees as child actors and deploys them on remote nodes.** +Each router will have its own routee instances. For example, if you start a router +on 3 nodes in a 10-node cluster, you will have 30 routees in total if the router is +configured to use one instance per node. The routees created by the different routers +will not be shared among the routers. One example of a use case for this type of router +is a single master that coordinates jobs and delegates the actual work to routees running +on other nodes in the cluster. -* **Pool - router that creates routees as child actors and deploys them on remote nodes.** - Each router will have its own routee instances. For example, if you start a router - on 3 nodes in a 10-node cluster, you will have 30 routees in total if the router is - configured to use one instance per node. The routees created by the different routers - will not be shared among the routers. One example of a use case for this type of router - is a single master that coordinates jobs and delegates the actual work to routees running - on other nodes in the cluster. +### Router with Group of Routees -Router with Group of Routees ----------------------------- +When using a `Group` you must start the routee actors on the cluster member nodes. +That is not done by the router. The configuration for a group looks like this:: -When using a ``Group`` you must start the routee actors on the cluster member nodes. -That is not done by the router. The configuration for a group looks like this::: - - akka.actor.deployment { - /statsService/workerRouter { - router = consistent-hashing-group - routees.paths = ["/user/statsWorker"] - cluster { - enabled = on - allow-local-routees = on - use-role = compute - } - } +``` +akka.actor.deployment { + /statsService/workerRouter { + router = consistent-hashing-group + routees.paths = ["/user/statsWorker"] + cluster { + enabled = on + allow-local-routees = on + use-role = compute + } } +} +``` -.. note:: - The routee actors should be started as early as possible when starting the actor system, because - the router will try to use them as soon as the member status is changed to 'Up'. +@@@ note -The actor paths without address information that are defined in ``routees.paths`` are used for selecting the +The routee actors should be started as early as possible when starting the actor system, because +the router will try to use them as soon as the member status is changed to 'Up'. + +@@@ + +The actor paths without address information that are defined in `routees.paths` are used for selecting the actors to which the messages will be forwarded to by the router. -Messages will be forwarded to the routees using :ref:`ActorSelection `, so the same delivery semantics should be expected. -It is possible to limit the lookup of routees to member nodes tagged with a certain role by specifying ``use-role``. +Messages will be forwarded to the routees using @ref:[ActorSelection](actors.md#actorselection-scala), so the same delivery semantics should be expected. +It is possible to limit the lookup of routees to member nodes tagged with a certain role by specifying `use-role`. -``max-total-nr-of-instances`` defines total number of routees in the cluster. By default ``max-total-nr-of-instances`` +`max-total-nr-of-instances` defines total number of routees in the cluster. By default `max-total-nr-of-instances` is set to a high value (10000) that will result in new routees added to the router when nodes join the cluster. Set it to a lower value if you want to limit total number of routees. The same type of router could also have been defined in code: -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsService.scala#router-lookup-in-code +@@snip [StatsService.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsService.scala) { #router-lookup-in-code } -See :ref:`cluster_configuration_scala` section for further descriptions of the settings. +See [cluster_configuration_scala](#cluster-configuration-scala) section for further descriptions of the settings. -Router Example with Group of Routees ------------------------------------- +### Router Example with Group of Routees Let's take a look at how to use a cluster aware router with a group of routees, i.e. router sending to the paths of the routees. @@ -595,296 +590,296 @@ the average number of characters per word when all results have been collected. Messages: -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsMessages.scala#messages +@@snip [StatsMessages.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsMessages.scala) { #messages } The worker that counts number of characters in each word: -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsWorker.scala#worker +@@snip [StatsWorker.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsWorker.scala) { #worker } The service that receives text from users and splits it up into words, delegates to workers and aggregates: -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsService.scala#service - +@@snip [StatsService.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsService.scala) { #service } Note, nothing cluster specific so far, just plain actors. -All nodes start ``StatsService`` and ``StatsWorker`` actors. Remember, routees are the workers in this case. -The router is configured with ``routees.paths``::: +All nodes start `StatsService` and `StatsWorker` actors. Remember, routees are the workers in this case. +The router is configured with `routees.paths`:: - akka.actor.deployment { - /statsService/workerRouter { - router = consistent-hashing-group - routees.paths = ["/user/statsWorker"] - cluster { - enabled = on - allow-local-routees = on - use-role = compute - } - } +``` +akka.actor.deployment { + /statsService/workerRouter { + router = consistent-hashing-group + routees.paths = ["/user/statsWorker"] + cluster { + enabled = on + allow-local-routees = on + use-role = compute } + } +} +``` -This means that user requests can be sent to ``StatsService`` on any node and it will use -``StatsWorker`` on all nodes. +This means that user requests can be sent to `StatsService` on any node and it will use +`StatsWorker` on all nodes. The easiest way to run **Router Example with Group of Routees** example yourself is to download the ready to run -`Akka Cluster Sample with Scala <@exampleCodeService@/akka-samples-cluster-scala>`_ +[Akka Cluster Sample with Scala](@exampleCodeService@/akka-samples-cluster-scala) together with the tutorial. It contains instructions on how to run the **Router Example with Group of Routees** sample. -The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-cluster-scala>`_. +The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-cluster-scala). -Router with Pool of Remote Deployed Routees -------------------------------------------- +### Router with Pool of Remote Deployed Routees -When using a ``Pool`` with routees created and deployed on the cluster member nodes -the configuration for a router looks like this::: +When using a `Pool` with routees created and deployed on the cluster member nodes +the configuration for a router looks like this:: - akka.actor.deployment { - /statsService/singleton/workerRouter { - router = consistent-hashing-pool - cluster { - enabled = on - max-nr-of-instances-per-node = 3 - allow-local-routees = on - use-role = compute - } - } +``` +akka.actor.deployment { + /statsService/singleton/workerRouter { + router = consistent-hashing-pool + cluster { + enabled = on + max-nr-of-instances-per-node = 3 + allow-local-routees = on + use-role = compute + } } +} +``` It is possible to limit the deployment of routees to member nodes tagged with a certain role by -specifying ``use-role``. +specifying `use-role`. -``max-total-nr-of-instances`` defines total number of routees in the cluster, but the number of routees -per node, ``max-nr-of-instances-per-node``, will not be exceeded. By default ``max-total-nr-of-instances`` +`max-total-nr-of-instances` defines total number of routees in the cluster, but the number of routees +per node, `max-nr-of-instances-per-node`, will not be exceeded. By default `max-total-nr-of-instances` is set to a high value (10000) that will result in new routees added to the router when nodes join the cluster. Set it to a lower value if you want to limit total number of routees. The same type of router could also have been defined in code: -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsService.scala#router-deploy-in-code +@@snip [StatsService.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsService.scala) { #router-deploy-in-code } -See :ref:`cluster_configuration_scala` section for further descriptions of the settings. +See [cluster_configuration_scala](#cluster-configuration-scala) section for further descriptions of the settings. -Router Example with Pool of Remote Deployed Routees ---------------------------------------------------- +### Router Example with Pool of Remote Deployed Routees Let's take a look at how to use a cluster aware router on single master node that creates -and deploys workers. To keep track of a single master we use the :ref:`cluster-singleton-scala` -in the cluster-tools module. The ``ClusterSingletonManager`` is started on each node.:: +and deploys workers. To keep track of a single master we use the @ref:[Cluster Singleton](cluster-singleton.md) +in the cluster-tools module. The `ClusterSingletonManager` is started on each node.: - system.actorOf( - ClusterSingletonManager.props( - singletonProps = Props[StatsService], - terminationMessage = PoisonPill, - settings = ClusterSingletonManagerSettings(system).withRole("compute")), - name = "statsService") +``` +system.actorOf( + ClusterSingletonManager.props( + singletonProps = Props[StatsService], + terminationMessage = PoisonPill, + settings = ClusterSingletonManagerSettings(system).withRole("compute")), + name = "statsService") +``` We also need an actor on each node that keeps track of where current single master exists and -delegates jobs to the ``StatsService``. That is provided by the ``ClusterSingletonProxy``.:: +delegates jobs to the `StatsService`. That is provided by the `ClusterSingletonProxy`.: - system.actorOf( - ClusterSingletonProxy.props( - singletonManagerPath = "/user/statsService", - settings = ClusterSingletonProxySettings(system).withRole("compute")), - name = "statsServiceProxy") +``` +system.actorOf( + ClusterSingletonProxy.props( + singletonManagerPath = "/user/statsService", + settings = ClusterSingletonProxySettings(system).withRole("compute")), + name = "statsServiceProxy") +``` -The ``ClusterSingletonProxy`` receives text from users and delegates to the current ``StatsService``, the single -master. It listens to cluster events to lookup the ``StatsService`` on the oldest node. +The `ClusterSingletonProxy` receives text from users and delegates to the current `StatsService`, the single +master. It listens to cluster events to lookup the `StatsService` on the oldest node. -All nodes start ``ClusterSingletonProxy`` and the ``ClusterSingletonManager``. The router is now configured like this::: +All nodes start `ClusterSingletonProxy` and the `ClusterSingletonManager`. The router is now configured like this:: - akka.actor.deployment { - /statsService/singleton/workerRouter { - router = consistent-hashing-pool - cluster { - enabled = on - max-nr-of-instances-per-node = 3 - allow-local-routees = on - use-role = compute - } - } +``` +akka.actor.deployment { + /statsService/singleton/workerRouter { + router = consistent-hashing-pool + cluster { + enabled = on + max-nr-of-instances-per-node = 3 + allow-local-routees = on + use-role = compute } + } +} +``` The easiest way to run **Router Example with Pool of Remote Deployed Routees** example yourself is to download the ready to run -`Akka Cluster Sample with Scala <@exampleCodeService@/akka-samples-cluster-scala>`_ +[Akka Cluster Sample with Scala](@exampleCodeService@/akka-samples-cluster-scala) together with the tutorial. It contains instructions on how to run the **Router Example with Pool of Remote Deployed Routees** sample. -The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-cluster-scala>`_. +The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-cluster-scala). -Cluster Metrics -^^^^^^^^^^^^^^^ +## Cluster Metrics The member nodes of the cluster can collect system health metrics and publish that to other cluster nodes -and to the registered subscribers on the system event bus with the help of :doc:`cluster-metrics`. +and to the registered subscribers on the system event bus with the help of `cluster-metrics`. +## How to Test -How to Test -^^^^^^^^^^^ +@ref:[Multi Node Testing](../dev/multi-node-testing.md) is useful for testing cluster applications. -:ref:`multi-node-testing` is useful for testing cluster applications. +Set up your project according to the instructions in @ref:[Multi Node Testing](../dev/multi-node-testing.md) and @ref:[Multi JVM Testing](../dev/multi-jvm-testing.md), i.e. +add the `sbt-multi-jvm` plugin and the dependency to `akka-multi-node-testkit`. -Set up your project according to the instructions in :ref:`multi-node-testing` and :ref:`multi-jvm-testing`, i.e. -add the ``sbt-multi-jvm`` plugin and the dependency to ``akka-multi-node-testkit``. +First, as described in @ref:[Multi Node Testing](../dev/multi-node-testing.md), we need some scaffolding to configure the `MultiNodeSpec`. +Define the participating roles and their [cluster_configuration_scala](#cluster-configuration-scala) in an object extending `MultiNodeConfig`: -First, as described in :ref:`multi-node-testing`, we need some scaffolding to configure the ``MultiNodeSpec``. -Define the participating roles and their :ref:`cluster_configuration_scala` in an object extending ``MultiNodeConfig``: - -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala - :include: MultiNodeConfig - :exclude: router-lookup-config +@@snip [StatsSampleSpec.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala) { #MultiNodeConfig } Define one concrete test class for each role/node. These will be instantiated on the different nodes (JVMs). They can be implemented differently, but often they are the same and extend an abstract test class, as illustrated here. -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala#concrete-tests +@@snip [StatsSampleSpec.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala) { #concrete-tests } -Note the naming convention of these classes. The name of the classes must end with ``MultiJvmNode1``, ``MultiJvmNode2`` -and so on. It is possible to define another suffix to be used by the ``sbt-multi-jvm``, but the default should be +Note the naming convention of these classes. The name of the classes must end with `MultiJvmNode1`, `MultiJvmNode2` +and so on. It is possible to define another suffix to be used by the `sbt-multi-jvm`, but the default should be fine in most cases. -Then the abstract ``MultiNodeSpec``, which takes the ``MultiNodeConfig`` as constructor parameter. +Then the abstract `MultiNodeSpec`, which takes the `MultiNodeConfig` as constructor parameter. -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala#abstract-test +@@snip [StatsSampleSpec.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala) { #abstract-test } Most of this can of course be extracted to a separate trait to avoid repeating this in all your tests. Typically you begin your test by starting up the cluster and let the members join, and create some actors. That can be done like this: -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala#startup-cluster +@@snip [StatsSampleSpec.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala) { #startup-cluster } -From the test you interact with the cluster using the ``Cluster`` extension, e.g. ``join``. +From the test you interact with the cluster using the `Cluster` extension, e.g. `join`. -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala#join +@@snip [StatsSampleSpec.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala) { #join } -Notice how the `testActor` from :ref:`testkit ` is added as :ref:`subscriber ` +Notice how the *testActor* from @ref:[testkit](testing.md) is added as [subscriber](#cluster-subscriber-scala) to cluster changes and then waiting for certain events, such as in this case all members becoming 'Up'. -The above code was running for all roles (JVMs). ``runOn`` is a convenient utility to declare that a certain block +The above code was running for all roles (JVMs). `runOn` is a convenient utility to declare that a certain block of code should only run for a specific role. -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala#test-statsService +@@snip [StatsSampleSpec.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala) { #test-statsService } -Once again we take advantage of the facilities in :ref:`testkit ` to verify expected behavior. -Here using ``testActor`` as sender (via ``ImplicitSender``) and verifying the reply with ``expectMsgPF``. +Once again we take advantage of the facilities in @ref:[testkit](testing.md) to verify expected behavior. +Here using `testActor` as sender (via `ImplicitSender`) and verifying the reply with `expectMsgPF`. -In the above code you can see ``node(third)``, which is useful facility to get the root actor reference of -the actor system for a specific role. This can also be used to grab the ``akka.actor.Address`` of that node. +In the above code you can see `node(third)`, which is useful facility to get the root actor reference of +the actor system for a specific role. This can also be used to grab the `akka.actor.Address` of that node. -.. includecode:: ../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala#addresses +@@snip [StatsSampleSpec.scala]../../../../../akka-cluster-metrics/src/multi-jvm/scala/akka/cluster/metrics/sample/StatsSampleSpec.scala) { #addresses } +## Management -Management -^^^^^^^^^^ - -.. _cluster_http_scala: - -HTTP ----- + +### HTTP Information and management of the cluster is available with a HTTP API. -See documentation of `akka/akka-cluster-management `_. +See documentation of [akka/akka-cluster-management](https://github.com/akka/akka-cluster-management). -.. _cluster_jmx_scala: + +### JMX -JMX ---- - -Information and management of the cluster is available as JMX MBeans with the root name ``akka.Cluster``. +Information and management of the cluster is available as JMX MBeans with the root name `akka.Cluster`. The JMX information can be displayed with an ordinary JMX console such as JConsole or JVisualVM. From JMX you can: -* see what members that are part of the cluster -* see status of this node -* see roles of each member -* join this node to another node in cluster -* mark any node in the cluster as down -* tell any node in the cluster to leave + * see what members that are part of the cluster + * see status of this node + * see roles of each member + * join this node to another node in cluster + * mark any node in the cluster as down + * tell any node in the cluster to leave -Member nodes are identified by their address, in format `akka.://@:`. +Member nodes are identified by their address, in format *akka.://@:*. -.. _cluster_command_line_scala: + +### Command Line -Command Line ------------- +@@@ warning -.. warning:: - **Deprecation warning** - The command line script has been deprecated and is scheduled for removal - in the next major version. Use the :ref:`cluster_http_scala` API with `curl `_ - or similar instead. +**Deprecation warning** - The command line script has been deprecated and is scheduled for removal +in the next major version. Use the [cluster_http_scala](#cluster-http-scala) API with [curl](https://curl.haxx.se/) +or similar instead. -The cluster can be managed with the script ``akka-cluster`` provided in the Akka github repository here: @github@/akka-cluster/jmx-client. Place the script and the ``jmxsh-R5.jar`` library in the same directory. +@@@ -Run it without parameters to see instructions about how to use the script:: +The cluster can be managed with the script `akka-cluster` provided in the Akka github repository here: @[github@/akka-cluster/jmx-client](mailto:github@/akka-cluster/jmx-client). Place the script and the `jmxsh-R5.jar` library in the same directory. - Usage: ./akka-cluster ... +Run it without parameters to see instructions about how to use the script: - Supported commands are: - join - Sends request a JOIN node with the specified URL - leave - Sends a request for node with URL to LEAVE the cluster - down - Sends a request for marking node with URL as DOWN - member-status - Asks the member node for its current status - members - Asks the cluster for addresses of current members - unreachable - Asks the cluster for addresses of unreachable members - cluster-status - Asks the cluster for its current status (member ring, - unavailable nodes, meta data etc.) - leader - Asks the cluster who the current leader is - is-singleton - Checks if the cluster is a singleton cluster (single - node cluster) - is-available - Checks if the member node is available - Where the should be on the format of - 'akka.://@:' +``` +Usage: ./akka-cluster ... - Examples: ./akka-cluster localhost 9999 is-available - ./akka-cluster localhost 9999 join akka.tcp://MySystem@darkstar:2552 - ./akka-cluster localhost 9999 cluster-status +Supported commands are: + join - Sends request a JOIN node with the specified URL + leave - Sends a request for node with URL to LEAVE the cluster + down - Sends a request for marking node with URL as DOWN + member-status - Asks the member node for its current status + members - Asks the cluster for addresses of current members + unreachable - Asks the cluster for addresses of unreachable members + cluster-status - Asks the cluster for its current status (member ring, + unavailable nodes, meta data etc.) + leader - Asks the cluster who the current leader is + is-singleton - Checks if the cluster is a singleton cluster (single + node cluster) + is-available - Checks if the member node is available +Where the should be on the format of + 'akka.://@:' +Examples: ./akka-cluster localhost 9999 is-available + ./akka-cluster localhost 9999 join akka.tcp://MySystem@darkstar:2552 + ./akka-cluster localhost 9999 cluster-status +``` To be able to use the script you must enable remote monitoring and management when starting the JVMs of the cluster nodes, -as described in `Monitoring and Management Using JMX Technology `_. +as described in [Monitoring and Management Using JMX Technology](http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html). Make sure you understand the security implications of enabling remote monitoring and management. -.. _cluster_configuration_scala: - -Configuration -^^^^^^^^^^^^^ + +## Configuration There are several configuration properties for the cluster. We refer to the -:ref:`reference configuration ` for more information. +@ref:[reference configuration](../general/configuration.md#config-akka-cluster) for more information. -Cluster Info Logging --------------------- +### Cluster Info Logging -You can silence the logging of cluster events at info level with configuration property:: +You can silence the logging of cluster events at info level with configuration property: - akka.cluster.log-info = off +``` +akka.cluster.log-info = off +``` -.. _cluster_dispatcher_scala: - -Cluster Dispatcher ------------------- + +### Cluster Dispatcher Under the hood the cluster extension is implemented with actors and it can be necessary to create a bulkhead for those actors to avoid disturbance from other actors. Especially the heartbeating actors that is used for failure detection can generate false positives if they are not given a chance to run at regular intervals. -For this purpose you can define a separate dispatcher to be used for the cluster actors:: +For this purpose you can define a separate dispatcher to be used for the cluster actors: - akka.cluster.use-dispatcher = cluster-dispatcher +``` +akka.cluster.use-dispatcher = cluster-dispatcher - cluster-dispatcher { - type = "Dispatcher" - executor = "fork-join-executor" - fork-join-executor { - parallelism-min = 2 - parallelism-max = 4 - } +cluster-dispatcher { + type = "Dispatcher" + executor = "fork-join-executor" + fork-join-executor { + parallelism-min = 2 + parallelism-max = 4 } +} +``` -.. note:: - Normally it should not be necessary to configure a separate dispatcher for the Cluster. - The default-dispatcher should be sufficient for performing the Cluster tasks, i.e. ``akka.cluster.use-dispatcher`` - should not be changed. If you have Cluster related problems when using the default-dispatcher that is typically an - indication that you are running blocking or CPU intensive actors/tasks on the default-dispatcher. - Use dedicated dispatchers for such actors/tasks instead of running them on the default-dispatcher, - because that may starve system internal tasks. - Related config properties: ``akka.cluster.use-dispatcher = akka.cluster.cluster-dispatcher``. - Corresponding default values: ``akka.cluster.use-dispatcher =``. +@@@ note + +Normally it should not be necessary to configure a separate dispatcher for the Cluster. +The default-dispatcher should be sufficient for performing the Cluster tasks, i.e. `akka.cluster.use-dispatcher` +should not be changed. If you have Cluster related problems when using the default-dispatcher that is typically an +indication that you are running blocking or CPU intensive actors/tasks on the default-dispatcher. +Use dedicated dispatchers for such actors/tasks instead of running them on the default-dispatcher, +because that may starve system internal tasks. +Related config properties: `akka.cluster.use-dispatcher = akka.cluster.cluster-dispatcher`. +Corresponding default values: `akka.cluster.use-dispatcher =`. + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/code/docs/camel/Introduction.scala b/akka-docs/src/main/paradox/scala/code/docs/camel/Introduction.scala index ed5e6f7383..f79c7e16e4 100644 --- a/akka-docs/src/main/paradox/scala/code/docs/camel/Introduction.scala +++ b/akka-docs/src/main/paradox/scala/code/docs/camel/Introduction.scala @@ -1,10 +1,12 @@ package docs.camel +//#imports import akka.actor.{ Props, ActorSystem } import akka.camel.CamelExtension import language.postfixOps import akka.util.Timeout +//#imports object Introduction { def foo(): Unit = { @@ -106,4 +108,4 @@ object Introduction { //#CamelDeactivation } -} \ No newline at end of file +} diff --git a/akka-docs/src/main/paradox/scala/dispatchers.md b/akka-docs/src/main/paradox/scala/dispatchers.md index c31edf3e39..15e3c8f7ff 100644 --- a/akka-docs/src/main/paradox/scala/dispatchers.md +++ b/akka-docs/src/main/paradox/scala/dispatchers.md @@ -1,152 +1,139 @@ -.. _dispatchers-scala: +# Dispatchers -Dispatchers -=========== +An Akka `MessageDispatcher` is what makes Akka Actors "tick", it is the engine of the machine so to speak. +All `MessageDispatcher` implementations are also an `ExecutionContext`, which means that they can be used +to execute arbitrary code, for instance @ref:[Futures](futures.md). -An Akka ``MessageDispatcher`` is what makes Akka Actors "tick", it is the engine of the machine so to speak. -All ``MessageDispatcher`` implementations are also an ``ExecutionContext``, which means that they can be used -to execute arbitrary code, for instance :ref:`futures-scala`. +## Default dispatcher -Default dispatcher ------------------- - -Every ``ActorSystem`` will have a default dispatcher that will be used in case nothing else is configured for an ``Actor``. -The default dispatcher can be configured, and is by default a ``Dispatcher`` with the specified ``default-executor``. +Every `ActorSystem` will have a default dispatcher that will be used in case nothing else is configured for an `Actor`. +The default dispatcher can be configured, and is by default a `Dispatcher` with the specified `default-executor`. If an ActorSystem is created with an ExecutionContext passed in, this ExecutionContext will be used as the default executor for all dispatchers in this ActorSystem. If no ExecutionContext is given, it will fallback to the executor specified in -``akka.actor.default-dispatcher.default-executor.fallback``. By default this is a "fork-join-executor", which +`akka.actor.default-dispatcher.default-executor.fallback`. By default this is a "fork-join-executor", which gives excellent performance in most cases. -.. _dispatcher-lookup-scala: + +## Looking up a Dispatcher -Looking up a Dispatcher ------------------------ +Dispatchers implement the `ExecutionContext` interface and can thus be used to run `Future` invocations etc. -Dispatchers implement the :class:`ExecutionContext` interface and can thus be used to run :class:`Future` invocations etc. +@@snip [DispatcherDocSpec.scala](code/docs/dispatcher/DispatcherDocSpec.scala) { #lookup } -.. includecode:: code/docs/dispatcher/DispatcherDocSpec.scala#lookup +## Setting the dispatcher for an Actor -Setting the dispatcher for an Actor ------------------------------------ - -So in case you want to give your ``Actor`` a different dispatcher than the default, you need to do two things, of which the first +So in case you want to give your `Actor` a different dispatcher than the default, you need to do two things, of which the first is to configure the dispatcher: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#my-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #my-dispatcher-config } -.. note:: - Note that the ``parallelism-max`` does not set the upper bound on the total number of threads - allocated by the ForkJoinPool. It is a setting specifically talking about the number of *hot* - threads the pool keep running in order to reduce the latency of handling a new incoming task. - You can read more about parallelism in the JDK's `ForkJoinPool documentation`_. +@@@ note + +Note that the `parallelism-max` does not set the upper bound on the total number of threads +allocated by the ForkJoinPool. It is a setting specifically talking about the number of *hot* +threads the pool keep running in order to reduce the latency of handling a new incoming task. +You can read more about parallelism in the JDK's [ForkJoinPool documentation](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html). + +@@@ Another example that uses the "thread-pool-executor": - - .. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#fixed-pool-size-dispatcher-config -.. note:: - The thread pool executor dispatcher is implemented using by a ``java.util.concurrent.ThreadPoolExecutor``. - You can read more about it in the JDK's `ThreadPoolExecutor documentation`_. +> +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #fixed-pool-size-dispatcher-config } -For more options, see the default-dispatcher section of the :ref:`configuration`. +@@@ note + +The thread pool executor dispatcher is implemented using by a `java.util.concurrent.ThreadPoolExecutor`. +You can read more about it in the JDK's [ThreadPoolExecutor documentation](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html). + +@@@ + +For more options, see the default-dispatcher section of the configuration. Then you create the actor as usual and define the dispatcher in the deployment configuration. -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#defining-dispatcher-in-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #defining-dispatcher-in-config } -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#dispatcher-deployment-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #dispatcher-deployment-config } An alternative to the deployment configuration is to define the dispatcher in code. -If you define the ``dispatcher`` in the deployment configuration then this value will be used instead +If you define the `dispatcher` in the deployment configuration then this value will be used instead of programmatically provided parameter. -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#defining-dispatcher-in-code +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #defining-dispatcher-in-code } -.. note:: - The dispatcher you specify in ``withDispatcher`` and the ``dispatcher`` property in the deployment - configuration is in fact a path into your configuration. - So in this example it's a top-level section, but you could for instance put it as a sub-section, - where you'd use periods to denote sub-sections, like this: ``"foo.bar.my-dispatcher"`` +@@@ note -.. _ForkJoinPool documentation: https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html -.. _ThreadPoolExecutor documentation: https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html +The dispatcher you specify in `withDispatcher` and the `dispatcher` property in the deployment +configuration is in fact a path into your configuration. +So in this example it's a top-level section, but you could for instance put it as a sub-section, +where you'd use periods to denote sub-sections, like this: `"foo.bar.my-dispatcher"` -Types of dispatchers --------------------- +@@@ + +## Types of dispatchers There are 3 different types of message dispatchers: -* Dispatcher + * Dispatcher + * This is an event-based dispatcher that binds a set of Actors to a thread pool. It is the default dispatcher +used if one is not specified. + * Sharability: Unlimited + * Mailboxes: Any, creates one per Actor + * Use cases: Default dispatcher, Bulkheading + * + Driven by: + `java.util.concurrent.ExecutorService` + : specify using "executor" using "fork-join-executor", +"thread-pool-executor" or the FQCN of +an `akka.dispatcher.ExecutorServiceConfigurator` + + * PinnedDispatcher + * This dispatcher dedicates a unique thread for each actor using it; i.e. each actor will have its own thread pool with only one thread in the pool. + * Sharability: None + * Mailboxes: Any, creates one per Actor + * Use cases: Bulkheading + * + Driven by: Any + `akka.dispatch.ThreadPoolExecutorConfigurator` + : by default a "thread-pool-executor" + + * CallingThreadDispatcher + * This dispatcher runs invocations on the current thread only. This dispatcher does not create any new threads, +but it can be used from different threads concurrently for the same actor. See @ref:[Scala-CallingThreadDispatcher](testing.md#scala-callingthreaddispatcher) +for details and restrictions. + * Sharability: Unlimited + * Mailboxes: Any, creates one per Actor per Thread (on demand) + * Use cases: Testing + * Driven by: The calling thread (duh) - - This is an event-based dispatcher that binds a set of Actors to a thread pool. It is the default dispatcher - used if one is not specified. - - - Sharability: Unlimited - - - Mailboxes: Any, creates one per Actor - - - Use cases: Default dispatcher, Bulkheading - - - Driven by: ``java.util.concurrent.ExecutorService`` - specify using "executor" using "fork-join-executor", - "thread-pool-executor" or the FQCN of - an ``akka.dispatcher.ExecutorServiceConfigurator`` - -* PinnedDispatcher - - - This dispatcher dedicates a unique thread for each actor using it; i.e. each actor will have its own thread pool with only one thread in the pool. - - - Sharability: None - - - Mailboxes: Any, creates one per Actor - - - Use cases: Bulkheading - - - Driven by: Any ``akka.dispatch.ThreadPoolExecutorConfigurator`` - by default a "thread-pool-executor" - -* CallingThreadDispatcher - - - This dispatcher runs invocations on the current thread only. This dispatcher does not create any new threads, - but it can be used from different threads concurrently for the same actor. See :ref:`Scala-CallingThreadDispatcher` - for details and restrictions. - - - Sharability: Unlimited - - - Mailboxes: Any, creates one per Actor per Thread (on demand) - - - Use cases: Testing - - - Driven by: The calling thread (duh) - -More dispatcher configuration examples -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### More dispatcher configuration examples Configuring a dispatcher with fixed thread pool size, e.g. for actors that perform blocking IO: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#fixed-pool-size-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #fixed-pool-size-dispatcher-config } And then using it: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#defining-fixed-pool-size-dispatcher +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #defining-fixed-pool-size-dispatcher } Another example that uses the thread pool based on the number of cores (e.g. for CPU bound tasks) -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#my-thread-pool-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #my-thread-pool-dispatcher-config } -Configuring a ``PinnedDispatcher``: +Configuring a `PinnedDispatcher`: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#my-pinned-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #my-pinned-dispatcher-config } And then using it: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#defining-pinned-dispatcher +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #defining-pinned-dispatcher } -Note that ``thread-pool-executor`` configuration as per the above ``my-thread-pool-dispatcher`` example is -NOT applicable. This is because every actor will have its own thread pool when using ``PinnedDispatcher``, +Note that `thread-pool-executor` configuration as per the above `my-thread-pool-dispatcher` example is +NOT applicable. This is because every actor will have its own thread pool when using `PinnedDispatcher`, and that pool will have only one thread. Note that it's not guaranteed that the *same* thread is used over time, since the core pool timeout -is used for ``PinnedDispatcher`` to keep resource usage down in case of idle actors. To use the same -thread all the time you need to add ``thread-pool-executor.allow-core-timeout=off`` to the -configuration of the ``PinnedDispatcher``. +is used for `PinnedDispatcher` to keep resource usage down in case of idle actors. To use the same +thread all the time you need to add `thread-pool-executor.allow-core-timeout=off` to the +configuration of the `PinnedDispatcher`. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/distributed-data.md b/akka-docs/src/main/paradox/scala/distributed-data.md index f08f8096cc..559d49de52 100644 --- a/akka-docs/src/main/paradox/scala/distributed-data.md +++ b/akka-docs/src/main/paradox/scala/distributed-data.md @@ -1,9 +1,4 @@ - -.. _distributed_data_scala: - -################## - Distributed Data -################## +# Distributed Data *Akka Distributed Data* is useful when you need to share data between nodes in an Akka Cluster. The data is accessed with an actor providing a key-value store like API. @@ -24,492 +19,491 @@ It is eventually consistent and geared toward providing high read and write avai (partition tolerance), with low latency. Note that in an eventually consistent system a read may return an out-of-date value. +## Using the Replicator -Using the Replicator -==================== - -The ``akka.cluster.ddata.Replicator`` actor provides the API for interacting with the data. -The ``Replicator`` actor must be started on each node in the cluster, or group of nodes tagged -with a specific role. It communicates with other ``Replicator`` instances with the same path +The `akka.cluster.ddata.Replicator` actor provides the API for interacting with the data. +The `Replicator` actor must be started on each node in the cluster, or group of nodes tagged +with a specific role. It communicates with other `Replicator` instances with the same path (without address) that are running on other nodes . For convenience it can be used with the -``akka.cluster.ddata.DistributedData`` extension but it can also be started as an ordinary -actor using the ``Replicator.props``. If it is started as an ordinary actor it is important +`akka.cluster.ddata.DistributedData` extension but it can also be started as an ordinary +actor using the `Replicator.props`. If it is started as an ordinary actor it is important that it is given the same name, started on same path, on all nodes. -Cluster members with status :ref:`WeaklyUp `, +Cluster members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-scala), will participate in Distributed Data. This means that the data will be replicated to the -:ref:`WeaklyUp ` nodes with the background gossip protocol. Note that it +@ref:[WeaklyUp](cluster-usage.md#weakly-up-scala) nodes with the background gossip protocol. Note that it will not participate in any actions where the consistency mode is to read/write from all -nodes or the majority of nodes. The :ref:`WeaklyUp ` node is not counted -as part of the cluster. So 3 nodes + 5 :ref:`WeaklyUp ` is essentially a +nodes or the majority of nodes. The @ref:[WeaklyUp](cluster-usage.md#weakly-up-scala) node is not counted +as part of the cluster. So 3 nodes + 5 @ref:[WeaklyUp](cluster-usage.md#weakly-up-scala) is essentially a 3 node cluster as far as consistent actions are concerned. Below is an example of an actor that schedules tick messages to itself and for each tick -adds or removes elements from a ``ORSet`` (observed-remove set). It also subscribes to +adds or removes elements from a `ORSet` (observed-remove set). It also subscribes to changes of this. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#data-bot +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #data-bot } -.. _replicator_update_scala: + +### Update -Update ------- +To modify and replicate a data value you send a `Replicator.Update` message to the local +`Replicator`. -To modify and replicate a data value you send a ``Replicator.Update`` message to the local -``Replicator``. - -The current data value for the ``key`` of the ``Update`` is passed as parameter to the ``modify`` -function of the ``Update``. The function is supposed to return the new value of the data, which +The current data value for the `key` of the `Update` is passed as parameter to the `modify` +function of the `Update`. The function is supposed to return the new value of the data, which will then be replicated according to the given consistency level. -The ``modify`` function is called by the ``Replicator`` actor and must therefore be a pure +The `modify` function is called by the `Replicator` actor and must therefore be a pure function that only uses the data parameter and stable fields from enclosing scope. It must -for example not access ``sender()`` reference of an enclosing actor. +for example not access `sender()` reference of an enclosing actor. + +`Update` + is intended to only be sent from an actor running in same local +`ActorSystem` + as +: the `Replicator`, because the `modify` function is typically not serializable. -``Update`` is intended to only be sent from an actor running in same local ``ActorSystem`` as - the ``Replicator``, because the ``modify`` function is typically not serializable. You supply a write consistency level which has the following meaning: -* ``WriteLocal`` the value will immediately only be written to the local replica, - and later disseminated with gossip -* ``WriteTo(n)`` the value will immediately be written to at least ``n`` replicas, - including the local replica -* ``WriteMajority`` the value will immediately be written to a majority of replicas, i.e. - at least **N/2 + 1** replicas, where N is the number of nodes in the cluster - (or cluster role group) -* ``WriteAll`` the value will immediately be written to all nodes in the cluster - (or all nodes in the cluster role group) + * `WriteLocal` the value will immediately only be written to the local replica, +and later disseminated with gossip + * `WriteTo(n)` the value will immediately be written to at least `n` replicas, +including the local replica + * `WriteMajority` the value will immediately be written to a majority of replicas, i.e. +at least **N/2 + 1** replicas, where N is the number of nodes in the cluster +(or cluster role group) + * `WriteAll` the value will immediately be written to all nodes in the cluster +(or all nodes in the cluster role group) -When you specify to write to ``n`` out of ``x`` nodes, the update will first replicate to ``n`` nodes. If there are not - enough Acks after 1/5th of the timeout, the update will be replicated to ``n`` other nodes. If there are less than n nodes - left all of the remaining nodes are used. Reachable nodes are prefered over unreachable nodes. +When you specify to write to +`n` + out of +`x` + nodes, the update will first replicate to +`n` + nodes. If there are not +: enough Acks after 1/5th of the timeout, the update will be replicated to `n` other nodes. If there are less than n nodes +left all of the remaining nodes are used. Reachable nodes are prefered over unreachable nodes. -Note that ``WriteMajority`` has a ``minCap`` parameter that is useful to specify to achieve better safety for small clusters. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#update +Note that `WriteMajority` has a `minCap` parameter that is useful to specify to achieve better safety for small clusters. -As reply of the ``Update`` a ``Replicator.UpdateSuccess`` is sent to the sender of the -``Update`` if the value was successfully replicated according to the supplied consistency -level within the supplied timeout. Otherwise a ``Replicator.UpdateFailure`` subclass is -sent back. Note that a ``Replicator.UpdateTimeout`` reply does not mean that the update completely failed +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #update } + +As reply of the `Update` a `Replicator.UpdateSuccess` is sent to the sender of the +`Update` if the value was successfully replicated according to the supplied consistency +level within the supplied timeout. Otherwise a `Replicator.UpdateFailure` subclass is +sent back. Note that a `Replicator.UpdateTimeout` reply does not mean that the update completely failed or was rolled back. It may still have been replicated to some nodes, and will eventually be replicated to all nodes with the gossip protocol. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#update-response1 +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #update-response1 } -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#update-response2 +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #update-response2 } -You will always see your own writes. For example if you send two ``Update`` messages -changing the value of the same ``key``, the ``modify`` function of the second message will -see the change that was performed by the first ``Update`` message. +You will always see your own writes. For example if you send two `Update` messages +changing the value of the same `key`, the `modify` function of the second message will +see the change that was performed by the first `Update` message. -In the ``Update`` message you can pass an optional request context, which the ``Replicator`` +In the `Update` message you can pass an optional request context, which the `Replicator` does not care about, but is included in the reply messages. This is a convenient -way to pass contextual information (e.g. original sender) without having to use ``ask`` +way to pass contextual information (e.g. original sender) without having to use `ask` or maintain local correlation data structures. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#update-request-context +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #update-request-context } -.. _replicator_get_scala: + +### Get -Get ---- +To retrieve the current value of a data you send `Replicator.Get` message to the +`Replicator`. You supply a consistency level which has the following meaning: -To retrieve the current value of a data you send ``Replicator.Get`` message to the -``Replicator``. You supply a consistency level which has the following meaning: + * `ReadLocal` the value will only be read from the local replica + * `ReadFrom(n)` the value will be read and merged from `n` replicas, +including the local replica + * `ReadMajority` the value will be read and merged from a majority of replicas, i.e. +at least **N/2 + 1** replicas, where N is the number of nodes in the cluster +(or cluster role group) + * `ReadAll` the value will be read and merged from all nodes in the cluster +(or all nodes in the cluster role group) -* ``ReadLocal`` the value will only be read from the local replica -* ``ReadFrom(n)`` the value will be read and merged from ``n`` replicas, - including the local replica -* ``ReadMajority`` the value will be read and merged from a majority of replicas, i.e. - at least **N/2 + 1** replicas, where N is the number of nodes in the cluster - (or cluster role group) -* ``ReadAll`` the value will be read and merged from all nodes in the cluster - (or all nodes in the cluster role group) +Note that `ReadMajority` has a `minCap` parameter that is useful to specify to achieve better safety for small clusters. -Note that ``ReadMajority`` has a ``minCap`` parameter that is useful to specify to achieve better safety for small clusters. +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #get } -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#get +As reply of the `Get` a `Replicator.GetSuccess` is sent to the sender of the +`Get` if the value was successfully retrieved according to the supplied consistency +level within the supplied timeout. Otherwise a `Replicator.GetFailure` is sent. +If the key does not exist the reply will be `Replicator.NotFound`. -As reply of the ``Get`` a ``Replicator.GetSuccess`` is sent to the sender of the -``Get`` if the value was successfully retrieved according to the supplied consistency -level within the supplied timeout. Otherwise a ``Replicator.GetFailure`` is sent. -If the key does not exist the reply will be ``Replicator.NotFound``. +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #get-response1 } -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#get-response1 +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #get-response2 } -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#get-response2 +You will always read your own writes. For example if you send a `Update` message +followed by a `Get` of the same `key` the `Get` will retrieve the change that was +performed by the preceding `Update` message. However, the order of the reply messages are +not defined, i.e. in the previous example you may receive the `GetSuccess` before +the `UpdateSuccess`. -You will always read your own writes. For example if you send a ``Update`` message -followed by a ``Get`` of the same ``key`` the ``Get`` will retrieve the change that was -performed by the preceding ``Update`` message. However, the order of the reply messages are -not defined, i.e. in the previous example you may receive the ``GetSuccess`` before -the ``UpdateSuccess``. +In the `Get` message you can pass an optional request context in the same way as for the +`Update` message, described above. For example the original sender can be passed and replied +to after receiving and transforming `GetSuccess`. -In the ``Get`` message you can pass an optional request context in the same way as for the -``Update`` message, described above. For example the original sender can be passed and replied -to after receiving and transforming ``GetSuccess``. +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #get-request-context } -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#get-request-context +### Consistency -Consistency ------------ - -The consistency level that is supplied in the :ref:`replicator_update_scala` and :ref:`replicator_get_scala` +The consistency level that is supplied in the [replicator_update_scala](#replicator-update-scala) and [replicator_get_scala](#replicator-get-scala) specifies per request how many replicas that must respond successfully to a write and read request. -For low latency reads you use ``ReadLocal`` with the risk of retrieving stale data, i.e. updates +For low latency reads you use `ReadLocal` with the risk of retrieving stale data, i.e. updates from other nodes might not be visible yet. -When using ``WriteLocal`` the update is only written to the local replica and then disseminated +When using `WriteLocal` the update is only written to the local replica and then disseminated in the background with the gossip protocol, which can take few seconds to spread to all nodes. -``WriteAll`` and ``ReadAll`` is the strongest consistency level, but also the slowest and with -lowest availability. For example, it is enough that one node is unavailable for a ``Get`` request +`WriteAll` and `ReadAll` is the strongest consistency level, but also the slowest and with +lowest availability. For example, it is enough that one node is unavailable for a `Get` request and you will not receive the value. If consistency is important, you can ensure that a read always reflects the most recent -write by using the following formula:: +write by using the following formula: - (nodes_written + nodes_read) > N +``` +(nodes_written + nodes_read) > N +``` where N is the total number of nodes in the cluster, or the number of nodes with the role that is -used for the ``Replicator``. +used for the `Replicator`. For example, in a 7 node cluster this these consistency properties are achieved by writing to 4 nodes and reading from 4 nodes, or writing to 5 nodes and reading from 3 nodes. -By combining ``WriteMajority`` and ``ReadMajority`` levels a read always reflects the most recent write. -The ``Replicator`` writes and reads to a majority of replicas, i.e. **N / 2 + 1**. For example, +By combining `WriteMajority` and `ReadMajority` levels a read always reflects the most recent write. +The `Replicator` writes and reads to a majority of replicas, i.e. **N / 2 + 1**. For example, in a 5 node cluster it writes to 3 nodes and reads from 3 nodes. In a 6 node cluster it writes to 4 nodes and reads from 4 nodes. -You can define a minimum number of nodes for ``WriteMajority`` and ``ReadMajority``, +You can define a minimum number of nodes for `WriteMajority` and `ReadMajority`, this will minimize the risk of reading steal data. Minimum cap is -provided by minCap property of ``WriteMajority`` and ``ReadMajority`` and defines the required majority. +provided by minCap property of `WriteMajority` and `ReadMajority` and defines the required majority. If the minCap is higher then **N / 2 + 1** the minCap will be used. -For example if the minCap is 5 the ``WriteMajority`` and ``ReadMajority`` for cluster of 3 nodes will be 3, for +For example if the minCap is 5 the `WriteMajority` and `ReadMajority` for cluster of 3 nodes will be 3, for cluster of 6 nodes will be 5 and for cluster of 12 nodes will be 7(**N / 2 + 1**). For small clusters (<7) the risk of membership changes between a WriteMajority and ReadMajority is rather high and then the nice properties of combining majority write and reads are not -guaranteed. Therefore the ``ReadMajority`` and ``WriteMajority`` have a ``minCap`` parameter that +guaranteed. Therefore the `ReadMajority` and `WriteMajority` have a `minCap` parameter that is useful to specify to achieve better safety for small clusters. It means that if the cluster -size is smaller than the majority size it will use the ``minCap`` number of nodes but at most +size is smaller than the majority size it will use the `minCap` number of nodes but at most the total size of the cluster. -Here is an example of using ``WriteMajority`` and ``ReadMajority``: +Here is an example of using `WriteMajority` and `ReadMajority`: -.. includecode:: code/docs/ddata/ShoppingCart.scala#read-write-majority +@@snip [ShoppingCart.scala](code/docs/ddata/ShoppingCart.scala) { #read-write-majority } -.. includecode:: code/docs/ddata/ShoppingCart.scala#get-cart +@@snip [ShoppingCart.scala](code/docs/ddata/ShoppingCart.scala) { #get-cart } -.. includecode:: code/docs/ddata/ShoppingCart.scala#add-item +@@snip [ShoppingCart.scala](code/docs/ddata/ShoppingCart.scala) { #add-item } -In some rare cases, when performing an ``Update`` it is needed to first try to fetch latest data from -other nodes. That can be done by first sending a ``Get`` with ``ReadMajority`` and then continue with -the ``Update`` when the ``GetSuccess``, ``GetFailure`` or ``NotFound`` reply is received. This might be -needed when you need to base a decision on latest information or when removing entries from ``ORSet`` -or ``ORMap``. If an entry is added to an ``ORSet`` or ``ORMap`` from one node and removed from another +In some rare cases, when performing an `Update` it is needed to first try to fetch latest data from +other nodes. That can be done by first sending a `Get` with `ReadMajority` and then continue with +the `Update` when the `GetSuccess`, `GetFailure` or `NotFound` reply is received. This might be +needed when you need to base a decision on latest information or when removing entries from `ORSet` +or `ORMap`. If an entry is added to an `ORSet` or `ORMap` from one node and removed from another node the entry will only be removed if the added entry is visible on the node where the removal is performed (hence the name observed-removed set). The following example illustrates how to do that: -.. includecode:: code/docs/ddata/ShoppingCart.scala#remove-item +@@snip [ShoppingCart.scala](code/docs/ddata/ShoppingCart.scala) { #remove-item } -.. warning:: +@@@ warning - *Caveat:* Even if you use ``WriteMajority`` and ``ReadMajority`` there is small risk that you may - read stale data if the cluster membership has changed between the ``Update`` and the ``Get``. - For example, in cluster of 5 nodes when you ``Update`` and that change is written to 3 nodes: - n1, n2, n3. Then 2 more nodes are added and a ``Get`` request is reading from 4 nodes, which - happens to be n4, n5, n6, n7, i.e. the value on n1, n2, n3 is not seen in the response of the - ``Get`` request. +*Caveat:* Even if you use `WriteMajority` and `ReadMajority` there is small risk that you may +read stale data if the cluster membership has changed between the `Update` and the `Get`. +For example, in cluster of 5 nodes when you `Update` and that change is written to 3 nodes: +n1, n2, n3. Then 2 more nodes are added and a `Get` request is reading from 4 nodes, which +happens to be n4, n5, n6, n7, i.e. the value on n1, n2, n3 is not seen in the response of the +`Get` request. -Subscribe ---------- +@@@ -You may also register interest in change notifications by sending ``Replicator.Subscribe`` -message to the ``Replicator``. It will send ``Replicator.Changed`` messages to the registered +### Subscribe + +You may also register interest in change notifications by sending `Replicator.Subscribe` +message to the `Replicator`. It will send `Replicator.Changed` messages to the registered subscriber when the data for the subscribed key is updated. Subscribers will be notified -periodically with the configured ``notify-subscribers-interval``, and it is also possible to -send an explicit ``Replicator.FlushChanges`` message to the ``Replicator`` to notify the subscribers +periodically with the configured `notify-subscribers-interval`, and it is also possible to +send an explicit `Replicator.FlushChanges` message to the `Replicator` to notify the subscribers immediately. The subscriber is automatically removed if the subscriber is terminated. A subscriber can -also be deregistered with the ``Replicator.Unsubscribe`` message. +also be deregistered with the `Replicator.Unsubscribe` message. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#subscribe +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #subscribe } -Delete ------- +### Delete -A data entry can be deleted by sending a ``Replicator.Delete`` message to the local -local ``Replicator``. As reply of the ``Delete`` a ``Replicator.DeleteSuccess`` is sent to -the sender of the ``Delete`` if the value was successfully deleted according to the supplied -consistency level within the supplied timeout. Otherwise a ``Replicator.ReplicationDeleteFailure`` -is sent. Note that ``ReplicationDeleteFailure`` does not mean that the delete completely failed or +A data entry can be deleted by sending a `Replicator.Delete` message to the local +local `Replicator`. As reply of the `Delete` a `Replicator.DeleteSuccess` is sent to +the sender of the `Delete` if the value was successfully deleted according to the supplied +consistency level within the supplied timeout. Otherwise a `Replicator.ReplicationDeleteFailure` +is sent. Note that `ReplicationDeleteFailure` does not mean that the delete completely failed or was rolled back. It may still have been replicated to some nodes, and may eventually be replicated to all nodes. A deleted key cannot be reused again, but it is still recommended to delete unused data entries because that reduces the replication overhead when new nodes join the cluster. -Subsequent ``Delete``, ``Update`` and ``Get`` requests will be replied with ``Replicator.DataDeleted``. -Subscribers will receive ``Replicator.Deleted``. +Subsequent `Delete`, `Update` and `Get` requests will be replied with `Replicator.DataDeleted`. +Subscribers will receive `Replicator.Deleted`. -In the `Delete` message you can pass an optional request context in the same way as for the -`Update` message, described above. For example the original sender can be passed and replied -to after receiving and transforming `DeleteSuccess`. +In the *Delete* message you can pass an optional request context in the same way as for the +*Update* message, described above. For example the original sender can be passed and replied +to after receiving and transforming *DeleteSuccess*. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#delete +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #delete } -.. warning:: +@@@ warning - As deleted keys continue to be included in the stored data on each node as well as in gossip - messages, a continuous series of updates and deletes of top-level entities will result in - growing memory usage until an ActorSystem runs out of memory. To use Akka Distributed Data - where frequent adds and removes are required, you should use a fixed number of top-level data - types that support both updates and removals, for example ``ORMap`` or ``ORSet``. +As deleted keys continue to be included in the stored data on each node as well as in gossip +messages, a continuous series of updates and deletes of top-level entities will result in +growing memory usage until an ActorSystem runs out of memory. To use Akka Distributed Data +where frequent adds and removes are required, you should use a fixed number of top-level data +types that support both updates and removals, for example `ORMap` or `ORSet`. -.. _delta_crdt_scala: +@@@ -delta-CRDT ----------- + +### delta-CRDT -`Delta State Replicated Data Types `_ +[Delta State Replicated Data Types](http://arxiv.org/abs/1603.01529) are supported. delta-CRDT is a way to reduce the need for sending the full state -for updates. For example adding element ``'c'`` and ``'d'`` to set ``{'a', 'b'}`` would -result in sending the delta ``{'c', 'd'}`` and merge that with the state on the -receiving side, resulting in set ``{'a', 'b', 'c', 'd'}``. +for updates. For example adding element `'c'` and `'d'` to set `{'a', 'b'}` would +result in sending the delta `{'c', 'd'}` and merge that with the state on the +receiving side, resulting in set `{'a', 'b', 'c', 'd'}`. The protocol for replicating the deltas supports causal consistency if the data type -is marked with ``RequiresCausalDeliveryOfDeltas``. Otherwise it is only eventually -consistent. Without causal consistency it means that if elements ``'c'`` and ``'d'`` are -added in two separate `Update` operations these deltas may occasionally be propagated +is marked with `RequiresCausalDeliveryOfDeltas`. Otherwise it is only eventually +consistent. Without causal consistency it means that if elements `'c'` and `'d'` are +added in two separate *Update* operations these deltas may occasionally be propagated to nodes in different order than the causal order of the updates. For this example it -can result in that set ``{'a', 'b', 'd'}`` can be seen before element 'c' is seen. Eventually -it will be ``{'a', 'b', 'c', 'd'}``. +can result in that set `{'a', 'b', 'd'}` can be seen before element 'c' is seen. Eventually +it will be `{'a', 'b', 'c', 'd'}`. Note that the full state is occasionally also replicated for delta-CRDTs, for example when new nodes are added to the cluster or when deltas could not be propagated because of network partitions or similar problems. -The the delta propagation can be disabled with configuration property:: +The the delta propagation can be disabled with configuration property: - akka.cluster.distributed-data.delta-crdt.enabled=off +``` +akka.cluster.distributed-data.delta-crdt.enabled=off +``` -Data Types -========== +## Data Types -The data types must be convergent (stateful) CRDTs and implement the ``ReplicatedData`` trait, +The data types must be convergent (stateful) CRDTs and implement the `ReplicatedData` trait, i.e. they provide a monotonic merge function and the state changes always converge. -You can use your own custom ``ReplicatedData`` or ``DeltaReplicatedData`` types, and several types are provided +You can use your own custom `ReplicatedData` or `DeltaReplicatedData` types, and several types are provided by this package, such as: -* Counters: ``GCounter``, ``PNCounter`` -* Sets: ``GSet``, ``ORSet`` -* Maps: ``ORMap``, ``ORMultiMap``, ``LWWMap``, ``PNCounterMap`` -* Registers: ``LWWRegister``, ``Flag`` + * Counters: `GCounter`, `PNCounter` + * Sets: `GSet`, `ORSet` + * Maps: `ORMap`, `ORMultiMap`, `LWWMap`, `PNCounterMap` + * Registers: `LWWRegister`, `Flag` -Counters --------- +### Counters -``GCounter`` is a "grow only counter". It only supports increments, no decrements. +`GCounter` is a "grow only counter". It only supports increments, no decrements. It works in a similar way as a vector clock. It keeps track of one counter per node and the total -value is the sum of these counters. The ``merge`` is implemented by taking the maximum count for +value is the sum of these counters. The `merge` is implemented by taking the maximum count for each node. -If you need both increments and decrements you can use the ``PNCounter`` (positive/negative counter). +If you need both increments and decrements you can use the `PNCounter` (positive/negative counter). It is tracking the increments (P) separate from the decrements (N). Both P and N are represented -as two internal ``GCounter``. Merge is handled by merging the internal P and N counters. +as two internal `GCounter`. Merge is handled by merging the internal P and N counters. The value of the counter is the value of the P counter minus the value of the N counter. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#pncounter +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #pncounter } -``GCounter`` and ``PNCounter`` have support for :ref:`delta_crdt_scala` and don't need causal +`GCounter` and `PNCounter` have support for [delta_crdt_scala](#delta-crdt-scala) and don't need causal delivery of deltas. -Several related counters can be managed in a map with the ``PNCounterMap`` data type. -When the counters are placed in a ``PNCounterMap`` as opposed to placing them as separate top level +Several related counters can be managed in a map with the `PNCounterMap` data type. +When the counters are placed in a `PNCounterMap` as opposed to placing them as separate top level values they are guaranteed to be replicated together as one unit, which is sometimes necessary for related data. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#pncountermap +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #pncountermap } -Sets ----- +### Sets -If you only need to add elements to a set and not remove elements the ``GSet`` (grow-only set) is +If you only need to add elements to a set and not remove elements the `GSet` (grow-only set) is the data type to use. The elements can be any type of values that can be serialized. Merge is simply the union of the two sets. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#gset +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #gset } -``GSet`` has support for :ref:`delta_crdt_scala` and it doesn't require causal delivery of deltas. +`GSet` has support for [delta_crdt_scala](#delta-crdt-scala) and it doesn't require causal delivery of deltas. -If you need add and remove operations you should use the ``ORSet`` (observed-remove set). +If you need add and remove operations you should use the `ORSet` (observed-remove set). Elements can be added and removed any number of times. If an element is concurrently added and removed, the add will win. You cannot remove an element that you have not seen. -The ``ORSet`` has a version vector that is incremented when an element is added to the set. +The `ORSet` has a version vector that is incremented when an element is added to the set. The version for the node that added the element is also tracked for each element in a so -called "birth dot". The version vector and the dots are used by the ``merge`` function to +called "birth dot". The version vector and the dots are used by the `merge` function to track causality of the operations and resolve concurrent updates. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#orset +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #orset } -``ORSet`` has support for :ref:`delta_crdt_scala` and it requires causal delivery of deltas. +`ORSet` has support for [delta_crdt_scala](#delta-crdt-scala) and it requires causal delivery of deltas. -Maps ----- +### Maps -``ORMap`` (observed-remove map) is a map with keys of ``Any`` type and the values are ``ReplicatedData`` +`ORMap` (observed-remove map) is a map with keys of `Any` type and the values are `ReplicatedData` types themselves. It supports add, remove and delete any number of times for a map entry. If an entry is concurrently added and removed, the add will win. You cannot remove an entry that -you have not seen. This is the same semantics as for the ``ORSet``. +you have not seen. This is the same semantics as for the `ORSet`. If an entry is concurrently updated to different values the values will be merged, hence the -requirement that the values must be ``ReplicatedData`` types. +requirement that the values must be `ReplicatedData` types. -It is rather inconvenient to use the ``ORMap`` directly since it does not expose specific types -of the values. The ``ORMap`` is intended as a low level tool for building more specific maps, +It is rather inconvenient to use the `ORMap` directly since it does not expose specific types +of the values. The `ORMap` is intended as a low level tool for building more specific maps, such as the following specialized maps. -``ORMultiMap`` (observed-remove multi-map) is a multi-map implementation that wraps an -``ORMap`` with an ``ORSet`` for the map's value. +`ORMultiMap` (observed-remove multi-map) is a multi-map implementation that wraps an +`ORMap` with an `ORSet` for the map's value. -``PNCounterMap`` (positive negative counter map) is a map of named counters (where the name can be of any type). -It is a specialized ``ORMap`` with ``PNCounter`` values. +`PNCounterMap` (positive negative counter map) is a map of named counters (where the name can be of any type). +It is a specialized `ORMap` with `PNCounter` values. -``LWWMap`` (last writer wins map) is a specialized ``ORMap`` with ``LWWRegister`` (last writer wins register) +`LWWMap` (last writer wins map) is a specialized `ORMap` with `LWWRegister` (last writer wins register) values. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#ormultimap +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #ormultimap } When a data entry is changed the full state of that entry is replicated to other nodes, i.e. -when you update a map the whole map is replicated. Therefore, instead of using one ``ORMap`` -with 1000 elements it is more efficient to split that up in 10 top level ``ORMap`` entries +when you update a map the whole map is replicated. Therefore, instead of using one `ORMap` +with 1000 elements it is more efficient to split that up in 10 top level `ORMap` entries with 100 elements each. Top level entries are replicated individually, which has the trade-off that different entries may not be replicated at the same time and you may see inconsistencies between related entries. Separate top level entries cannot be updated atomically together. -Note that ``LWWRegister`` and therefore ``LWWMap`` relies on synchronized clocks and should only be used +Note that `LWWRegister` and therefore `LWWMap` relies on synchronized clocks and should only be used when the choice of value is not important for concurrent updates occurring within the clock skew. Read more -in the below section about ``LWWRegister``. +in the below section about `LWWRegister`. -Flags and Registers -------------------- +### Flags and Registers -``Flag`` is a data type for a boolean value that is initialized to ``false`` and can be switched -to ``true``. Thereafter it cannot be changed. ``true`` wins over ``false`` in merge. +`Flag` is a data type for a boolean value that is initialized to `false` and can be switched +to `true`. Thereafter it cannot be changed. `true` wins over `false` in merge. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#flag +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #flag } -``LWWRegister`` (last writer wins register) can hold any (serializable) value. +`LWWRegister` (last writer wins register) can hold any (serializable) value. -Merge of a ``LWWRegister`` takes the register with highest timestamp. Note that this -relies on synchronized clocks. `LWWRegister` should only be used when the choice of +Merge of a `LWWRegister` takes the register with highest timestamp. Note that this +relies on synchronized clocks. *LWWRegister* should only be used when the choice of value is not important for concurrent updates occurring within the clock skew. -Merge takes the register updated by the node with lowest address (``UniqueAddress`` is ordered) +Merge takes the register updated by the node with lowest address (`UniqueAddress` is ordered) if the timestamps are exactly the same. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#lwwregister +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #lwwregister } -Instead of using timestamps based on ``System.currentTimeMillis()`` time it is possible to +Instead of using timestamps based on `System.currentTimeMillis()` time it is possible to use a timestamp value based on something else, for example an increasing version number from a database record that is used for optimistic concurrency control. -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#lwwregister-custom-clock +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #lwwregister-custom-clock } -For first-write-wins semantics you can use the ``LWWRegister#reverseClock`` instead of the -``LWWRegister#defaultClock``. +For first-write-wins semantics you can use the `LWWRegister#reverseClock` instead of the +`LWWRegister#defaultClock`. -The ``defaultClock`` is using max value of ``System.currentTimeMillis()`` and ``currentTimestamp + 1``. +The `defaultClock` is using max value of `System.currentTimeMillis()` and `currentTimestamp + 1`. This means that the timestamp is increased for changes on the same node that occurs within -the same millisecond. It also means that it is safe to use the ``LWWRegister`` without +the same millisecond. It also means that it is safe to use the `LWWRegister` without synchronized clocks when there is only one active writer, e.g. a Cluster Singleton. Such a -single writer should then first read current value with ``ReadMajority`` (or more) before -changing and writing the value with ``WriteMajority`` (or more). +single writer should then first read current value with `ReadMajority` (or more) before +changing and writing the value with `WriteMajority` (or more). -Custom Data Type ----------------- +### Custom Data Type You can rather easily implement your own data types. The only requirement is that it implements -the ``merge`` function of the ``ReplicatedData`` trait. +the `merge` function of the `ReplicatedData` trait. A nice property of stateful CRDTs is that they typically compose nicely, i.e. you can combine several -smaller data types to build richer data structures. For example, the ``PNCounter`` is composed of -two internal ``GCounter`` instances to keep track of increments and decrements separately. +smaller data types to build richer data structures. For example, the `PNCounter` is composed of +two internal `GCounter` instances to keep track of increments and decrements separately. -Here is s simple implementation of a custom ``TwoPhaseSet`` that is using two internal ``GSet`` types -to keep track of addition and removals. A ``TwoPhaseSet`` is a set where an element may be added and +Here is s simple implementation of a custom `TwoPhaseSet` that is using two internal `GSet` types +to keep track of addition and removals. A `TwoPhaseSet` is a set where an element may be added and removed, but never added again thereafter. -.. includecode:: code/docs/ddata/TwoPhaseSet.scala#twophaseset +@@snip [TwoPhaseSet.scala](code/docs/ddata/TwoPhaseSet.scala) { #twophaseset } Data types should be immutable, i.e. "modifying" methods should return a new instance. -Implement the additional methods of ``DeltaReplicatedData`` if it has support for delta-CRDT replication. +Implement the additional methods of `DeltaReplicatedData` if it has support for delta-CRDT replication. -Serialization -^^^^^^^^^^^^^ +#### Serialization -The data types must be serializable with an :ref:`Akka Serializer `. +The data types must be serializable with an @ref:[Akka Serializer](serialization.md). It is highly recommended that you implement efficient serialization with Protobuf or similar -for your custom data types. The built in data types are marked with ``ReplicatedDataSerialization`` -and serialized with ``akka.cluster.ddata.protobuf.ReplicatedDataSerializer``. +for your custom data types. The built in data types are marked with `ReplicatedDataSerialization` +and serialized with `akka.cluster.ddata.protobuf.ReplicatedDataSerializer`. Serialization of the data types are used in remote messages and also for creating message digests (SHA-1) to detect changes. Therefore it is important that the serialization is efficient and produce the same bytes for the same content. For example sets and maps should be sorted deterministically in the serialization. -This is a protobuf representation of the above ``TwoPhaseSet``: +This is a protobuf representation of the above `TwoPhaseSet`: -.. includecode:: ../../src/main/protobuf/TwoPhaseSetMessages.proto#twophaseset +@@snip [TwoPhaseSetMessages.proto]../../protobuf/TwoPhaseSetMessages.proto) { #twophaseset } -The serializer for the ``TwoPhaseSet``: +The serializer for the `TwoPhaseSet`: -.. includecode:: code/docs/ddata/protobuf/TwoPhaseSetSerializer.scala#serializer +@@snip [TwoPhaseSetSerializer.scala](code/docs/ddata/protobuf/TwoPhaseSetSerializer.scala) { #serializer } Note that the elements of the sets are sorted so the SHA-1 digests are the same for the same elements. You register the serializer in configuration: -.. includecode:: code/docs/ddata/DistributedDataDocSpec.scala#serializer-config +@@snip [DistributedDataDocSpec.scala](code/docs/ddata/DistributedDataDocSpec.scala) { #serializer-config } Using compression can sometimes be a good idea to reduce the data size. Gzip compression is -provided by the ``akka.cluster.ddata.protobuf.SerializationSupport`` trait: +provided by the `akka.cluster.ddata.protobuf.SerializationSupport` trait: -.. includecode:: code/docs/ddata/protobuf/TwoPhaseSetSerializer.scala#compression +@@snip [TwoPhaseSetSerializer.scala](code/docs/ddata/protobuf/TwoPhaseSetSerializer.scala) { #compression } -The two embedded ``GSet`` can be serialized as illustrated above, but in general when composing +The two embedded `GSet` can be serialized as illustrated above, but in general when composing new data types from the existing built in types it is better to make use of the existing serializer for those types. This can be done by declaring those as bytes fields in protobuf: -.. includecode:: ../../src/main/protobuf/TwoPhaseSetMessages.proto#twophaseset2 +@@snip [TwoPhaseSetMessages.proto]../../protobuf/TwoPhaseSetMessages.proto) { #twophaseset2 } -and use the methods ``otherMessageToProto`` and ``otherMessageFromBinary`` that are provided -by the ``SerializationSupport`` trait to serialize and deserialize the ``GSet`` instances. This +and use the methods `otherMessageToProto` and `otherMessageFromBinary` that are provided +by the `SerializationSupport` trait to serialize and deserialize the `GSet` instances. This works with any type that has a registered Akka serializer. This is how such an serializer would -look like for the ``TwoPhaseSet``: +look like for the `TwoPhaseSet`: -.. includecode:: code/docs/ddata/protobuf/TwoPhaseSetSerializer2.scala#serializer +@@snip [TwoPhaseSetSerializer2.scala](code/docs/ddata/protobuf/TwoPhaseSetSerializer2.scala) { #serializer } -.. _ddata_durable_scala: - -Durable Storage ---------------- + +### Durable Storage By default the data is only kept in memory. It is redundant since it is replicated to other nodes in the cluster, but if you stop all nodes the data is lost, unless you have saved it @@ -518,30 +512,36 @@ elsewhere. Entries can be configured to be durable, i.e. stored on local disk on each node. The stored data will be loaded next time the replicator is started, i.e. when actor system is restarted. This means data will survive as long as at least one node from the old cluster takes part in a new cluster. The keys of the durable entries -are configured with:: +are configured with: - akka.cluster.distributed-data.durable.keys = ["a", "b", "durable*"] +``` +akka.cluster.distributed-data.durable.keys = ["a", "b", "durable*"] +``` -Prefix matching is supported by using ``*`` at the end of a key. +Prefix matching is supported by using `*` at the end of a key. -All entries can be made durable by specifying:: +All entries can be made durable by specifying: - akka.cluster.distributed-data.durable.keys = ["*"] +``` +akka.cluster.distributed-data.durable.keys = ["*"] +``` -`LMDB `_ is the default storage implementation. It is +[LMDB](https://symas.com/products/lightning-memory-mapped-database/) is the default storage implementation. It is possible to replace that with another implementation by implementing the actor protocol described in -``akka.cluster.ddata.DurableStore`` and defining the ``akka.cluster.distributed-data.durable.store-actor-class`` +`akka.cluster.ddata.DurableStore` and defining the `akka.cluster.distributed-data.durable.store-actor-class` property for the new implementation. -The location of the files for the data is configured with:: +The location of the files for the data is configured with: - # Directory of LMDB file. There are two options: - # 1. A relative or absolute path to a directory that ends with 'ddata' - # the full name of the directory will contain name of the ActorSystem - # and its remote port. - # 2. Otherwise the path is used as is, as a relative or absolute path to - # a directory. - akka.cluster.distributed-data.durable.lmdb.dir = "ddata" +``` +# Directory of LMDB file. There are two options: +# 1. A relative or absolute path to a directory that ends with 'ddata' +# the full name of the directory will contain name of the ActorSystem +# and its remote port. +# 2. Otherwise the path is used as is, as a relative or absolute path to +# a directory. +akka.cluster.distributed-data.durable.lmdb.dir = "ddata" +``` When running in production you may want to configure the directory to a specific path (alt 2), since the default directory contains the remote port of the @@ -550,58 +550,54 @@ port (0) it will be different each time and the previously stored data will not be loaded. Making the data durable has of course a performance cost. By default, each update is flushed -to disk before the ``UpdateSuccess`` reply is sent. For better performance, but with the risk of losing +to disk before the `UpdateSuccess` reply is sent. For better performance, but with the risk of losing the last writes if the JVM crashes, you can enable write behind mode. Changes are then accumulated during a time period before it is written to LMDB and flushed to disk. Enabling write behind is especially efficient when performing many writes to the same key, because it is only the last value for each key that will be serialized and stored. The risk of losing writes if the JVM crashes is small since the -data is typically replicated to other nodes immediately according to the given ``WriteConsistency``. +data is typically replicated to other nodes immediately according to the given `WriteConsistency`. -:: +``` +akka.cluster.distributed-data.lmdb.write-behind-interval = 200 ms +``` - akka.cluster.distributed-data.lmdb.write-behind-interval = 200 ms +Note that you should be prepared to receive `WriteFailure` as reply to an `Update` of a +durable entry if the data could not be stored for some reason. When enabling `write-behind-interval` +such errors will only be logged and `UpdateSuccess` will still be the reply to the `Update`. -Note that you should be prepared to receive ``WriteFailure`` as reply to an ``Update`` of a -durable entry if the data could not be stored for some reason. When enabling ``write-behind-interval`` -such errors will only be logged and ``UpdateSuccess`` will still be the reply to the ``Update``. - -There is one important caveat when it comes pruning of :ref:`crdt_garbage_scala` for durable data. +There is one important caveat when it comes pruning of [crdt_garbage_scala](#crdt-garbage-scala) for durable data. If and old data entry that was never pruned is injected and merged with existing data after that the pruning markers have been removed the value will not be correct. The time-to-live of the markers is defined by configuration -``akka.cluster.distributed-data.durable.remove-pruning-marker-after`` and is in the magnitude of days. +`akka.cluster.distributed-data.durable.remove-pruning-marker-after` and is in the magnitude of days. This would be possible if a node with durable data didn't participate in the pruning (e.g. it was shutdown) and later started after this time. A node with durable data should not be stopped for longer time than this duration and if it is joining again after this duration its data should first be manually removed (from the lmdb directory). -.. _crdt_garbage_scala: - -CRDT Garbage ------------- + +### CRDT Garbage One thing that can be problematic with CRDTs is that some data types accumulate history (garbage). -For example a ``GCounter`` keeps track of one counter per node. If a ``GCounter`` has been updated +For example a `GCounter` keeps track of one counter per node. If a `GCounter` has been updated from one node it will associate the identifier of that node forever. That can become a problem for long running systems with many cluster nodes being added and removed. To solve this problem -the ``Replicator`` performs pruning of data associated with nodes that have been removed from the -cluster. Data types that need pruning have to implement the ``RemovedNodePruning`` trait. See the -API documentation of the ``Replicator`` for details. +the `Replicator` performs pruning of data associated with nodes that have been removed from the +cluster. Data types that need pruning have to implement the `RemovedNodePruning` trait. See the +API documentation of the `Replicator` for details. -Samples -======= +## Samples Several interesting samples are included and described in the -tutorial named `Akka Distributed Data Samples with Scala <@exampleCodeService@/akka-samples-distributed-data-scala>`_ (`source code <@samples@/akka-sample-distributed-data-scala>`_) +tutorial named [Akka Distributed Data Samples with Scala](@exampleCodeService@/akka-samples-distributed-data-scala) ([source code](@samples@/akka-sample-distributed-data-scala)) -* Low Latency Voting Service -* Highly Available Shopping Cart -* Distributed Service Registry -* Replicated Cache -* Replicated Metrics + * Low Latency Voting Service + * Highly Available Shopping Cart + * Distributed Service Registry + * Replicated Cache + * Replicated Metrics -Limitations -=========== +## Limitations There are some limitations that you should be aware of. @@ -618,44 +614,44 @@ be able to improve this if needed, but the design is still not intended for bill All data is held in memory, which is another reason why it is not intended for *Big Data*. When a data entry is changed the full state of that entry may be replicated to other nodes -if it doesn't support :ref:`delta_crdt_scala`. The full state is also replicated for delta-CRDTs, +if it doesn't support [delta_crdt_scala](#delta-crdt-scala). The full state is also replicated for delta-CRDTs, for example when new nodes are added to the cluster or when deltas could not be propagated because of network partitions or similar problems. This means that you cannot have too large data entries, because then the remote message size will be too large. -Learn More about CRDTs -====================== +## Learn More about CRDTs -* `The Final Causal Frontier `_ - talk by Sean Cribbs -* `Eventually Consistent Data Structures `_ - talk by Sean Cribbs -* `Strong Eventual Consistency and Conflict-free Replicated Data Types `_ - talk by Mark Shapiro -* `A comprehensive study of Convergent and Commutative Replicated Data Types `_ - paper by Mark Shapiro et. al. + * [The Final Causal Frontier](http://www.ustream.tv/recorded/61448875) +talk by Sean Cribbs + * [Eventually Consistent Data Structures](https://vimeo.com/43903960) +talk by Sean Cribbs + * [Strong Eventual Consistency and Conflict-free Replicated Data Types](http://research.microsoft.com/apps/video/default.aspx?id=153540&r=1) +talk by Mark Shapiro + * [A comprehensive study of Convergent and Commutative Replicated Data Types](http://hal.upmc.fr/file/index/docid/555588/filename/techreport.pdf) +paper by Mark Shapiro et. al. -Dependencies -============ +## Dependencies To use Distributed Data you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-distributed-data" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-distributed-data" % "@version@" @crossString@ +``` -maven:: +maven: - - com.typesafe.akka - akka-distributed-data_@binVersion@ - @version@ - +``` + + com.typesafe.akka + akka-distributed-data_@binVersion@ + @version@ + +``` -Configuration -============= +## Configuration -The ``DistributedData`` extension can be configured with the following properties: - -.. includecode:: ../../../akka-distributed-data/src/main/resources/reference.conf#distributed-data +The `DistributedData` extension can be configured with the following properties: +@@snip [reference.conf]../../../../../akka-distributed-data/src/main/resources/reference.conf) { #distributed-data } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/distributed-pub-sub.md b/akka-docs/src/main/paradox/scala/distributed-pub-sub.md index f5fd9ffecd..a11f7ac193 100644 --- a/akka-docs/src/main/paradox/scala/distributed-pub-sub.md +++ b/akka-docs/src/main/paradox/scala/distributed-pub-sub.md @@ -1,20 +1,17 @@ -.. _distributed-pub-sub-scala: - -Distributed Publish Subscribe in Cluster -======================================== +# Distributed Publish Subscribe in Cluster How do I send a message to an actor without knowing which node it is running on? How do I send messages to all actors in the cluster that have registered interest in a named topic? -This pattern provides a mediator actor, ``akka.cluster.pubsub.DistributedPubSubMediator``, +This pattern provides a mediator actor, `akka.cluster.pubsub.DistributedPubSubMediator`, that manages a registry of actor references and replicates the entries to peer actors among all cluster nodes or a group of nodes tagged with a specific role. -The ``DistributedPubSubMediator`` actor is supposed to be started on all nodes, +The `DistributedPubSubMediator` actor is supposed to be started on all nodes, or all nodes with specified role, in the cluster. The mediator can be -started with the ``DistributedPubSub`` extension or as an ordinary actor. +started with the `DistributedPubSub` extension or as an ordinary actor. The registry is eventually consistent, i.e. changes are not immediately visible at other nodes, but typically they will be fully replicated to all other nodes after @@ -22,24 +19,22 @@ a few seconds. Changes are only performed in the own part of the registry and th changes are versioned. Deltas are disseminated in a scalable way to other nodes with a gossip protocol. -Cluster members with status :ref:`WeaklyUp `, +Cluster members with status @ref:[WeaklyUp](cluster-usage.md#weakly-up-scala), will participate in Distributed Publish Subscribe, i.e. subscribers on nodes with -``WeaklyUp`` status will receive published messages if the publisher and subscriber are on +`WeaklyUp` status will receive published messages if the publisher and subscriber are on same side of a network partition. You can send messages via the mediator on any node to registered actors on any other node. There a two different modes of message delivery, explained in the sections -:ref:`distributed-pub-sub-publish-scala` and :ref:`distributed-pub-sub-send-scala` below. +[Publish](#distributed-pub-sub-publish-scala) and [Send](#distributed-pub-sub-send-scala) below. A more comprehensive sample is available in the -tutorial named `Akka Clustered PubSub with Scala! `_. +tutorial named [Akka Clustered PubSub with Scala!](https://github.com/typesafehub/activator-akka-clustering). -.. _distributed-pub-sub-publish-scala: - -Publish -------- + +## Publish This is the true pub/sub mode. A typical usage of this mode is a chat room in an instant messaging application. @@ -50,61 +45,60 @@ The message will be delivered to all subscribers of the topic. For efficiency the message is sent over the wire only once per node (that has a matching topic), and then delivered to all subscribers of the local topic representation. (See more in ) -You register actors to the local mediator with ``DistributedPubSubMediator.Subscribe``. -Successful ``Subscribe`` and ``Unsubscribe`` is acknowledged with -``DistributedPubSubMediator.SubscribeAck`` and ``DistributedPubSubMediator.UnsubscribeAck`` +You register actors to the local mediator with `DistributedPubSubMediator.Subscribe`. +Successful `Subscribe` and `Unsubscribe` is acknowledged with +`DistributedPubSubMediator.SubscribeAck` and `DistributedPubSubMediator.UnsubscribeAck` replies. The acknowledgment means that the subscription is registered, but it can still take some time until it is replicated to other nodes. -You publish messages by sending ``DistributedPubSubMediator.Publish`` message to the +You publish messages by sending `DistributedPubSubMediator.Publish` message to the local mediator. Actors are automatically removed from the registry when they are terminated, or you -can explicitly remove entries with ``DistributedPubSubMediator.Unsubscribe``. +can explicitly remove entries with `DistributedPubSubMediator.Unsubscribe`. An example of a subscriber actor: -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala#subscriber +@@snip [DistributedPubSubMediatorSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala) { #subscriber } Subscriber actors can be started on several nodes in the cluster, and all will receive messages published to the "content" topic. -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala#start-subscribers +@@snip [DistributedPubSubMediatorSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala) { #start-subscribers } A simple actor that publishes to this "content" topic: -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala#publisher +@@snip [DistributedPubSubMediatorSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala) { #publisher } It can publish messages to the topic from anywhere in the cluster: -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala#publish-message +@@snip [DistributedPubSubMediatorSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala) { #publish-message } -Topic Groups -^^^^^^^^^^^^ +### Topic Groups -Actors may also be subscribed to a named topic with a ``group`` id. +Actors may also be subscribed to a named topic with a `group` id. If subscribing with a group id, each message published to a topic with the -``sendOneMessageToEachGroup`` flag set to ``true`` is delivered via the supplied ``RoutingLogic`` +`sendOneMessageToEachGroup` flag set to `true` is delivered via the supplied `RoutingLogic` (default random) to one actor within each subscribing group. If all the subscribed actors have the same group id, then this works just like -``Send`` and each message is only delivered to one subscriber. +`Send` and each message is only delivered to one subscriber. If all the subscribed actors have different group names, then this works like -normal ``Publish`` and each message is broadcasted to all subscribers. +normal `Publish` and each message is broadcasted to all subscribers. -.. note:: +@@@ note - Note that if the group id is used it is part of the topic identifier. - Messages published with ``sendOneMessageToEachGroup=false`` will not be delivered - to subscribers that subscribed with a group id. - Messages published with ``sendOneMessageToEachGroup=true`` will not be delivered - to subscribers that subscribed without a group id. +Note that if the group id is used it is part of the topic identifier. +Messages published with `sendOneMessageToEachGroup=false` will not be delivered +to subscribers that subscribed with a group id. +Messages published with `sendOneMessageToEachGroup=true` will not be delivered +to subscribers that subscribed without a group id. -.. _distributed-pub-sub-send-scala: +@@@ -Send ----- + +## Send This is a point-to-point mode where each message is delivered to one destination, but you still do not have to know where the destination is located. @@ -114,43 +108,43 @@ cluster aware router where the routees dynamically can register themselves. The message will be delivered to one recipient with a matching path, if any such exists in the registry. If several entries match the path because it has been registered -on several nodes the message will be sent via the supplied ``RoutingLogic`` (default random) +on several nodes the message will be sent via the supplied `RoutingLogic` (default random) to one destination. The sender() of the message can specify that local affinity is preferred, i.e. the message is sent to an actor in the same local actor system as the used mediator actor, if any such exists, otherwise route to any other matching entry. -You register actors to the local mediator with ``DistributedPubSubMediator.Put``. -The ``ActorRef`` in ``Put`` must belong to the same local actor system as the mediator. +You register actors to the local mediator with `DistributedPubSubMediator.Put`. +The `ActorRef` in `Put` must belong to the same local actor system as the mediator. The path without address information is the key to which you send messages. On each node there can only be one actor for a given path, since the path is unique within one local actor system. -You send messages by sending ``DistributedPubSubMediator.Send`` message to the +You send messages by sending `DistributedPubSubMediator.Send` message to the local mediator with the path (without address information) of the destination actors. Actors are automatically removed from the registry when they are terminated, or you -can explicitly remove entries with ``DistributedPubSubMediator.Remove``. +can explicitly remove entries with `DistributedPubSubMediator.Remove`. An example of a destination actor: -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala#send-destination +@@snip [DistributedPubSubMediatorSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala) { #send-destination } Destination actors can be started on several nodes in the cluster, and all will receive messages sent to the path (without address information). -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala#start-send-destinations +@@snip [DistributedPubSubMediatorSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala) { #start-send-destinations } A simple actor that sends to the path: -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala#sender +@@snip [DistributedPubSubMediatorSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala) { #sender } It can send messages to the path from anywhere in the cluster: -.. includecode:: ../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala#send-message +@@snip [DistributedPubSubMediatorSpec.scala]../../../../../akka-cluster-tools/src/multi-jvm/scala/akka/cluster/pubsub/DistributedPubSubMediatorSpec.scala) { #send-message } It is also possible to broadcast messages to the actors that have been registered with -``Put``. Send ``DistributedPubSubMediator.SendToAll`` message to the local mediator and the wrapped message +`Put`. Send `DistributedPubSubMediator.SendToAll` message to the local mediator and the wrapped message will then be delivered to all recipients with a matching path. Actors with the same path, without address information, can be registered on different nodes. On each node there can only be one such actor, since the path is unique within one @@ -158,53 +152,52 @@ local actor system. Typical usage of this mode is to broadcast messages to all replicas with the same path, e.g. 3 actors on different nodes that all perform the same actions, -for redundancy. You can also optionally specify a property (``allButSelf``) deciding +for redundancy. You can also optionally specify a property (`allButSelf`) deciding if the message should be sent to a matching path on the self node or not. -DistributedPubSub Extension ---------------------------- +## DistributedPubSub Extension -In the example above the mediator is started and accessed with the ``akka.cluster.pubsub.DistributedPubSub`` extension. +In the example above the mediator is started and accessed with the `akka.cluster.pubsub.DistributedPubSub` extension. That is convenient and perfectly fine in most cases, but it can be good to know that it is possible to start the mediator actor as an ordinary actor and you can have several different mediators at the same time to be able to divide a large number of actors/topics to different mediators. For example you might want to use different cluster roles for different mediators. -The ``DistributedPubSub`` extension can be configured with the following properties: +The `DistributedPubSub` extension can be configured with the following properties: -.. includecode:: ../../../akka-cluster-tools/src/main/resources/reference.conf#pub-sub-ext-config +@@snip [reference.conf]../../../../../akka-cluster-tools/src/main/resources/reference.conf) { #pub-sub-ext-config } It is recommended to load the extension when the actor system is started by defining it in -``akka.extensions`` configuration property. Otherwise it will be activated when first used +`akka.extensions` configuration property. Otherwise it will be activated when first used and then it takes a while for it to be populated. -:: +``` +akka.extensions = ["akka.cluster.pubsub.DistributedPubSub"] +``` - akka.extensions = ["akka.cluster.pubsub.DistributedPubSub"] +## Delivery Guarantee - -Delivery Guarantee ------------------- - -As in :ref:`message-delivery-reliability` of Akka, message delivery guarantee in distributed pub sub modes is **at-most-once delivery**. +As in @ref:[Message Delivery Reliability](../general/message-delivery-reliability.md) of Akka, message delivery guarantee in distributed pub sub modes is **at-most-once delivery**. In other words, messages can be lost over the wire. -If you are looking for at-least-once delivery guarantee, we recommend `Kafka Akka Streams integration `_. +If you are looking for at-least-once delivery guarantee, we recommend [Kafka Akka Streams integration](https://github.com/akka/reactive-kafka). -Dependencies ------------- +## Dependencies To use Distributed Publish Subscribe you must add the following dependency in your project. -sbt:: +sbt: - "com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-cluster-tools" % "@version@" @crossString@ +``` -maven:: - - - com.typesafe.akka - akka-cluster-tools_@binVersion@ - @version@ - +maven: +``` + + com.typesafe.akka + akka-cluster-tools_@binVersion@ + @version@ + +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/event-bus.md b/akka-docs/src/main/paradox/scala/event-bus.md index 780504f681..a36226b683 100644 --- a/akka-docs/src/main/paradox/scala/event-bus.md +++ b/akka-docs/src/main/paradox/scala/event-bus.md @@ -1,66 +1,58 @@ -.. _event-bus-scala: - -########### - Event Bus -########### - +# Event Bus Originally conceived as a way to send messages to groups of actors, the -:class:`EventBus` has been generalized into a set of composable traits +`EventBus` has been generalized into a set of composable traits implementing a simple interface: -.. includecode:: ../../../akka-actor/src/main/scala/akka/event/EventBus.scala#event-bus-api +@@snip [EventBus.scala]../../../../../akka-actor/src/main/scala/akka/event/EventBus.scala) { #event-bus-api } -.. note:: +@@@ note - Please note that the EventBus does not preserve the sender of the - published messages. If you need a reference to the original sender - you have to provide it inside the message. +Please note that the EventBus does not preserve the sender of the +published messages. If you need a reference to the original sender +you have to provide it inside the message. -This mechanism is used in different places within Akka, e.g. the `Event Stream`_. +@@@ + +This mechanism is used in different places within Akka, e.g. the [Event Stream](#event-stream). Implementations can make use of the specific building blocks presented below. An event bus must define the following three abstract types: -- :class:`Event` is the type of all events published on that bus - -- :class:`Subscriber` is the type of subscribers allowed to register on that - event bus - -- :class:`Classifier` defines the classifier to be used in selecting - subscribers for dispatching events + * `Event` is the type of all events published on that bus + * `Subscriber` is the type of subscribers allowed to register on that +event bus + * `Classifier` defines the classifier to be used in selecting +subscribers for dispatching events The traits below are still generic in these types, but they need to be defined for any concrete implementation. -Classifiers -=========== +## Classifiers The classifiers presented here are part of the Akka distribution, but rolling your own in case you do not find a perfect match is not difficult, check the -implementation of the existing ones on `github <@github@/akka-actor/src/main/scala/akka/event/EventBus.scala>`_ +implementation of the existing ones on [github](@github@/akka-actor/src/main/scala/akka/event/EventBus.scala) -Lookup Classification ---------------------- +### Lookup Classification The simplest classification is just to extract an arbitrary classifier from each event and maintaining a set of subscribers for each possible classifier. This can be compared to tuning in on a radio station. The trait -:class:`LookupClassification` is still generic in that it abstracts over how to +`LookupClassification` is still generic in that it abstracts over how to compare subscribers and how exactly to classify. The necessary methods to be implemented are illustrated with the following example: -.. includecode:: code/docs/event/EventBusDocSpec.scala#lookup-bus +@@snip [EventBusDocSpec.scala](code/docs/event/EventBusDocSpec.scala) { #lookup-bus } A test for this implementation may look like this: -.. includecode:: code/docs/event/EventBusDocSpec.scala#lookup-bus-test +@@snip [EventBusDocSpec.scala](code/docs/event/EventBusDocSpec.scala) { #lookup-bus-test } This classifier is efficient in case no subscribers exist for a particular event. -Subchannel Classification -------------------------- +### Subchannel Classification If classifiers form a hierarchy and it is desired that subscription be possible not only at the leaf nodes, this classification may be just the right one. It @@ -72,11 +64,11 @@ classifier hierarchy. The necessary methods to be implemented are illustrated with the following example: -.. includecode:: code/docs/event/EventBusDocSpec.scala#subchannel-bus +@@snip [EventBusDocSpec.scala](code/docs/event/EventBusDocSpec.scala) { #subchannel-bus } A test for this implementation may look like this: -.. includecode:: code/docs/event/EventBusDocSpec.scala#subchannel-bus-test +@@snip [EventBusDocSpec.scala](code/docs/event/EventBusDocSpec.scala) { #subchannel-bus-test } This classifier is also efficient in case no subscribers are found for an event, but it uses conventional locking to synchronize an internal classifier @@ -84,8 +76,7 @@ cache, hence it is not well-suited to use cases in which subscriptions change with very high frequency (keep in mind that “opening” a classifier by sending the first message will also have to re-check all previous subscriptions). -Scanning Classification ------------------------ +### Scanning Classification The previous classifier was built for multi-classifier subscriptions which are strictly hierarchical, this classifier is useful if there are overlapping @@ -95,116 +86,113 @@ stations by geographical reachability (for old-school radio-wave transmission). The necessary methods to be implemented are illustrated with the following example: -.. includecode:: code/docs/event/EventBusDocSpec.scala#scanning-bus +@@snip [EventBusDocSpec.scala](code/docs/event/EventBusDocSpec.scala) { #scanning-bus } A test for this implementation may look like this: -.. includecode:: code/docs/event/EventBusDocSpec.scala#scanning-bus-test +@@snip [EventBusDocSpec.scala](code/docs/event/EventBusDocSpec.scala) { #scanning-bus-test } This classifier takes always a time which is proportional to the number of subscriptions, independent of how many actually match. -.. _actor-classification-scala: - -Actor Classification --------------------- + +### Actor Classification This classification was originally developed specifically for implementing -:ref:`DeathWatch `: subscribers as well as classifiers are of -type :class:`ActorRef`. +@ref:[DeathWatch](actors.md#deathwatch-scala): subscribers as well as classifiers are of +type `ActorRef`. -This classification requires an :class:`ActorSystem` in order to perform book-keeping +This classification requires an `ActorSystem` in order to perform book-keeping operations related to the subscribers being Actors, which can terminate without first unsubscribing from the EventBus. ManagedActorClassification maintains a system Actor which takes care of unsubscribing terminated actors automatically. The necessary methods to be implemented are illustrated with the following example: -.. includecode:: code/docs/event/EventBusDocSpec.scala#actor-bus +@@snip [EventBusDocSpec.scala](code/docs/event/EventBusDocSpec.scala) { #actor-bus } A test for this implementation may look like this: -.. includecode:: code/docs/event/EventBusDocSpec.scala#actor-bus-test +@@snip [EventBusDocSpec.scala](code/docs/event/EventBusDocSpec.scala) { #actor-bus-test } This classifier is still is generic in the event type, and it is efficient for all use cases. -.. _event-stream-scala: - -Event Stream -============ + +## Event Stream The event stream is the main event bus of each actor system: it is used for -carrying :ref:`log messages ` and `Dead Letters`_ and may be -used by the user code for other purposes as well. It uses `Subchannel -Classification`_ which enables registering to related sets of channels (as is -used for :class:`RemotingLifecycleEvent`). The following example demonstrates +carrying @ref:[log messages](logging.md) and [Dead Letters](#dead-letters) and may be +used by the user code for other purposes as well. It uses [Subchannel +Classification](#subchannel-classification) which enables registering to related sets of channels (as is +used for `RemotingLifecycleEvent`). The following example demonstrates how a simple subscription works: -.. includecode:: code/docs/event/LoggingDocSpec.scala#deadletters +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #deadletters } It is also worth pointing out that thanks to the way the subchannel classification is implemented in the event stream, it is possible to subscribe to a group of events, by subscribing to their common superclass as demonstrated in the following example: -.. includecode:: code/docs/event/LoggingDocSpec.scala#superclass-subscription-eventstream +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #superclass-subscription-eventstream } -Similarly to `Actor Classification`_, :class:`EventStream` will automatically remove subscribers when they terminate. +Similarly to [Actor Classification](#actor-classification), `EventStream` will automatically remove subscribers when they terminate. -.. note:: - The event stream is a *local facility*, meaning that it will *not* distribute events to other nodes in a clustered environment (unless you subscribe a Remote Actor to the stream explicitly). - If you need to broadcast events in an Akka cluster, *without* knowing your recipients explicitly (i.e. obtaining their ActorRefs), you may want to look into: :ref:`distributed-pub-sub-scala`. +@@@ note -Default Handlers ----------------- +The event stream is a *local facility*, meaning that it will *not* distribute events to other nodes in a clustered environment (unless you subscribe a Remote Actor to the stream explicitly). +If you need to broadcast events in an Akka cluster, *without* knowing your recipients explicitly (i.e. obtaining their ActorRefs), you may want to look into: @ref:[Distributed Publish Subscribe in Cluster](distributed-pub-sub.md). + +@@@ + +### Default Handlers Upon start-up the actor system creates and subscribes actors to the event stream for logging: these are the handlers which are configured for example in -``application.conf``: +`application.conf`: -.. code-block:: text - - akka { - loggers = ["akka.event.Logging$DefaultLogger"] - } +```text +akka { + loggers = ["akka.event.Logging$DefaultLogger"] +} +``` The handlers listed here by fully-qualified class name will be subscribed to all log event classes with priority higher than or equal to the configured log-level and their subscriptions are kept in sync when changing the log-level -at runtime:: +at runtime: - system.eventStream.setLogLevel(Logging.DebugLevel) +``` +system.eventStream.setLogLevel(Logging.DebugLevel) +``` This means that log events for a level which will not be logged are not typically not dispatched at all (unless manual subscriptions to the respective event class have been done) -Dead Letters ------------- +### Dead Letters -As described at :ref:`stopping-actors-scala`, messages queued when an actor +As described at @ref:[Stopping actors](actors.md#stopping-actors-scala), messages queued when an actor terminates or sent after its death are re-routed to the dead letter mailbox, -which by default will publish the messages wrapped in :class:`DeadLetter`. This +which by default will publish the messages wrapped in `DeadLetter`. This wrapper holds the original sender, receiver and message of the envelope which was redirected. -Some internal messages (marked with the :class:`DeadLetterSuppression` trait) will not end up as +Some internal messages (marked with the `DeadLetterSuppression` trait) will not end up as dead letters like normal messages. These are by design safe and expected to sometimes arrive at a terminated actor and since they are nothing to worry about, they are suppressed from the default dead letters logging mechanism. However, in case you find yourself in need of debugging these kinds of low level suppressed dead letters, it's still possible to subscribe to them explicitly: -.. includecode:: code/docs/event/LoggingDocSpec.scala#suppressed-deadletters +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #suppressed-deadletters } or all dead letters (including the suppressed ones): -.. includecode:: code/docs/event/LoggingDocSpec.scala#all-deadletters +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #all-deadletters } -Other Uses ----------- +### Other Uses The event stream is always there and ready to be used, just publish your own -events (it accepts ``AnyRef``) and subscribe listeners to the corresponding JVM -classes. - +events (it accepts `AnyRef`) and subscribe listeners to the corresponding JVM +classes. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/extending-akka.md b/akka-docs/src/main/paradox/scala/extending-akka.md index e2d36980eb..b7b03db8c1 100644 --- a/akka-docs/src/main/paradox/scala/extending-akka.md +++ b/akka-docs/src/main/paradox/scala/extending-akka.md @@ -1,107 +1,90 @@ -.. _extending-akka-scala: - -######################### - Akka Extensions -######################### - +# Akka Extensions If you want to add features to Akka, there is a very elegant, but powerful mechanism for doing so. -It's called Akka Extensions and is comprised of 2 basic components: an ``Extension`` and an ``ExtensionId``. +It's called Akka Extensions and is comprised of 2 basic components: an `Extension` and an `ExtensionId`. -Extensions will only be loaded once per ``ActorSystem``, which will be managed by Akka. -You can choose to have your Extension loaded on-demand or at ``ActorSystem`` creation time through the Akka configuration. +Extensions will only be loaded once per `ActorSystem`, which will be managed by Akka. +You can choose to have your Extension loaded on-demand or at `ActorSystem` creation time through the Akka configuration. Details on how to make that happens are below, in the "Loading from Configuration" section. -.. warning:: +@@@ warning - Since an extension is a way to hook into Akka itself, the implementor of the extension needs to - ensure the thread safety of his/her extension. +Since an extension is a way to hook into Akka itself, the implementor of the extension needs to +ensure the thread safety of his/her extension. -Building an Extension -===================== +@@@ + +## Building an Extension So let's create a sample extension that just lets us count the number of times something has happened. -First, we define what our ``Extension`` should do: +First, we define what our `Extension` should do: -.. includecode:: code/docs/extension/ExtensionDocSpec.scala - :include: extension +@@snip [ExtensionDocSpec.scala](code/docs/extension/ExtensionDocSpec.scala) { #extension } -Then we need to create an ``ExtensionId`` for our extension so we can grab a hold of it. +Then we need to create an `ExtensionId` for our extension so we can grab a hold of it. -.. includecode:: code/docs/extension/ExtensionDocSpec.scala - :include: extensionid +@@snip [ExtensionDocSpec.scala](code/docs/extension/ExtensionDocSpec.scala) { #extensionid } Wicked! Now all we need to do is to actually use it: -.. includecode:: code/docs/extension/ExtensionDocSpec.scala - :include: extension-usage +@@snip [ExtensionDocSpec.scala](code/docs/extension/ExtensionDocSpec.scala) { #extension-usage } Or from inside of an Akka Actor: -.. includecode:: code/docs/extension/ExtensionDocSpec.scala - :include: extension-usage-actor +@@snip [ExtensionDocSpec.scala](code/docs/extension/ExtensionDocSpec.scala) { #extension-usage-actor } You can also hide extension behind traits: -.. includecode:: code/docs/extension/ExtensionDocSpec.scala - :include: extension-usage-actor-trait +@@snip [ExtensionDocSpec.scala](code/docs/extension/ExtensionDocSpec.scala) { #extension-usage-actor-trait } That's all there is to it! -Loading from Configuration -========================== +## Loading from Configuration -To be able to load extensions from your Akka configuration you must add FQCNs of implementations of either ``ExtensionId`` or ``ExtensionIdProvider`` -in the ``akka.extensions`` section of the config you provide to your ``ActorSystem``. +To be able to load extensions from your Akka configuration you must add FQCNs of implementations of either `ExtensionId` or `ExtensionIdProvider` +in the `akka.extensions` section of the config you provide to your `ActorSystem`. -.. includecode:: code/docs/extension/ExtensionDocSpec.scala - :include: config +@@snip [ExtensionDocSpec.scala](code/docs/extension/ExtensionDocSpec.scala) { #config } -Applicability -============= +## Applicability The sky is the limit! -By the way, did you know that Akka's ``Typed Actors``, ``Serialization`` and other features are implemented as Akka Extensions? +By the way, did you know that Akka's `Typed Actors`, `Serialization` and other features are implemented as Akka Extensions? -.. _extending-akka-scala.settings: + +### Application specific settings -Application specific settings ------------------------------ - -The :ref:`configuration` can be used for application specific settings. A good practice is to place those settings in an Extension. +The configuration can be used for application specific settings. A good practice is to place those settings in an Extension. Sample configuration: -.. includecode:: code/docs/extension/SettingsExtensionDocSpec.scala - :include: config +@@snip [SettingsExtensionDocSpec.scala](code/docs/extension/SettingsExtensionDocSpec.scala) { #config } -The ``Extension``: - -.. includecode:: code/docs/extension/SettingsExtensionDocSpec.scala - :include: imports,extension,extensionid +The `Extension`: +@@snip [SettingsExtensionDocSpec.scala](code/docs/extension/SettingsExtensionDocSpec.scala) { #imports #extension #extensionid } Use it: -.. includecode:: code/docs/extension/SettingsExtensionDocSpec.scala - :include: extension-usage-actor +@@snip [SettingsExtensionDocSpec.scala](code/docs/extension/SettingsExtensionDocSpec.scala) { #extension-usage-actor } +## Library extensions -Library extensions -================== A third part library may register it's extension for auto-loading on actor system startup by appending it to -``akka.library-extensions`` in its ``reference.conf``. - -:: - - akka.library-extensions += "docs.extension.ExampleExtension" +`akka.library-extensions` in its `reference.conf`. +``` +akka.library-extensions += "docs.extension.ExampleExtension" +``` As there is no way to selectively remove such extensions, it should be used with care and only when there is no case where the user would ever want it disabled or have specific support for disabling such sub-features. One example where this could be important is in tests. -.. warning:: - The``akka.library-extensions`` must never be assigned (``= ["Extension"]``) instead of appending as this will break - the library-extension mechanism and make behavior depend on class path ordering. \ No newline at end of file +@@@ warning + +The``akka.library-extensions`` must never be assigned (`= ["Extension"]`) instead of appending as this will break +the library-extension mechanism and make behavior depend on class path ordering. + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/fault-tolerance-sample.md b/akka-docs/src/main/paradox/scala/fault-tolerance-sample.md index c3d9eca28c..6615862238 100644 --- a/akka-docs/src/main/paradox/scala/fault-tolerance-sample.md +++ b/akka-docs/src/main/paradox/scala/fault-tolerance-sample.md @@ -1,55 +1,38 @@ -.. _fault-tolerance-sample-scala: -Diagrams of the Fault Tolerance Sample ----------------------------------------------- + +# Diagrams of the Fault Tolerance Sample - - -.. image:: ../images/faulttolerancesample-normal-flow.png +![faulttolerancesample-normal-flow.png](../images/faulttolerancesample-normal-flow.png) *The above diagram illustrates the normal message flow.* **Normal flow:** -======= ================================================================================== -Step Description -======= ================================================================================== -1 The progress ``Listener`` starts the work. -2 The ``Worker`` schedules work by sending ``Do`` messages periodically to itself -3, 4, 5 When receiving ``Do`` the ``Worker`` tells the ``CounterService`` - to increment the counter, three times. The ``Increment`` message is forwarded - to the ``Counter``, which updates its counter variable and sends current value - to the ``Storage``. -6, 7 The ``Worker`` asks the ``CounterService`` of current value of the counter and pipes - the result back to the ``Listener``. -======= ================================================================================== +|Step | Description | +|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +|1 | The progress `Listener` starts the work. | +|2 | The `Worker` schedules work by sending `Do` messages periodically to itself | +|3, 4, 5 | When receiving `Do` the `Worker` tells the `CounterService` to increment the counter, three times. The `Increment` message is forwarded to the `Counter`, which updates its counter variable and sends current value to the `Storage`.| +|6, 7 | The `Worker` asks the `CounterService` of current value of the counter and pipes the result back to the `Listener`. | - -.. image:: ../images/faulttolerancesample-failure-flow.png +![faulttolerancesample-failure-flow.png](../images/faulttolerancesample-failure-flow.png) *The above diagram illustrates what happens in case of storage failure.* **Failure flow:** -=========== ================================================================================== -Step Description -=========== ================================================================================== -1 The ``Storage`` throws ``StorageException``. -2 The ``CounterService`` is supervisor of the ``Storage`` and restarts the - ``Storage`` when ``StorageException`` is thrown. -3, 4, 5, 6 The ``Storage`` continues to fail and is restarted. -7 After 3 failures and restarts within 5 seconds the ``Storage`` is stopped by its - supervisor, i.e. the ``CounterService``. -8 The ``CounterService`` is also watching the ``Storage`` for termination and - receives the ``Terminated`` message when the ``Storage`` has been stopped ... -9, 10, 11 and tells the ``Counter`` that there is no ``Storage``. -12 The ``CounterService`` schedules a ``Reconnect`` message to itself. -13, 14 When it receives the ``Reconnect`` message it creates a new ``Storage`` ... -15, 16 and tells the ``Counter`` to use the new ``Storage`` -=========== ================================================================================== +|Step | Description | +|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------| +|1 | The `Storage` throws `StorageException`. | +|2 | The `CounterService` is supervisor of the `Storage` and restarts the `Storage` when `StorageException` is thrown. | +|3, 4, 5, 6 | The `Storage` continues to fail and is restarted. | +|7 | After 3 failures and restarts within 5 seconds the `Storage` is stopped by its supervisor, i.e. the `CounterService`. | +|8 | The `CounterService` is also watching the `Storage` for termination and receives the `Terminated` message when the `Storage` has been stopped ...| +|9, 10, 11 | and tells the `Counter` that there is no `Storage`. | +|12 | The `CounterService` schedules a `Reconnect` message to itself. | +|13, 14 | When it receives the `Reconnect` message it creates a new `Storage` ... | +|15, 16 | and tells the `Counter` to use the new `Storage` | -Full Source Code of the Fault Tolerance Sample ------------------------------------------------------- - -.. includecode:: code/docs/actor/FaultHandlingDocSample.scala#all +# Full Source Code of the Fault Tolerance Sample +@@snip [FaultHandlingDocSample.scala](code/docs/actor/FaultHandlingDocSample.scala) { #all } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/fault-tolerance.md b/akka-docs/src/main/paradox/scala/fault-tolerance.md index 3550ca563f..e94884880d 100644 --- a/akka-docs/src/main/paradox/scala/fault-tolerance.md +++ b/akka-docs/src/main/paradox/scala/fault-tolerance.md @@ -1,15 +1,11 @@ -.. _fault-tolerance-scala: +# Fault Tolerance -Fault Tolerance -======================= - -As explained in :ref:`actor-systems` each actor is the supervisor of its +As explained in @ref:[Actor Systems](../general/actor-systems.md) each actor is the supervisor of its children, and as such each actor defines fault handling supervisor strategy. This strategy cannot be changed afterwards as it is an integral part of the actor system’s structure. -Fault Handling in Practice --------------------------- +## Fault Handling in Practice First, let us look at a sample that illustrates one way to handle data store errors, which is a typical source of failure in real world applications. Of course it depends @@ -20,23 +16,25 @@ Read the following source code. The inlined comments explain the different piece the fault handling and why they are added. It is also highly recommended to run this sample as it is easy to follow the log output to understand what is happening at runtime. -.. toctree:: +@@toc - fault-tolerance-sample +@@@ index -Creating a Supervisor Strategy ------------------------------- +* [fault-tolerance-sample](fault-tolerance-sample.md) + +@@@ + +## Creating a Supervisor Strategy The following sections explain the fault handling mechanism and alternatives in more depth. For the sake of demonstration let us consider the following strategy: -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: strategy +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #strategy } I have chosen a few well-known exception types in order to demonstrate the -application of the fault handling directives described in :ref:`supervision`. +application of the fault handling directives described in supervision. First off, it is a one-for-one strategy, meaning that each child is treated separately (an all-for-one strategy works very similarly, the only difference is that any decision is applied to all children of the supervisor, not only the @@ -46,139 +44,124 @@ that the respective limit does not apply, leaving the possibility to specify an absolute upper limit on the restarts or to make the restarts work infinitely. The child actor is stopped if the limit is exceeded. -The match statement which forms the bulk of the body is of type ``Decider``, -which is a ``PartialFunction[Throwable, Directive]``. This +The match statement which forms the bulk of the body is of type `Decider`, +which is a `PartialFunction[Throwable, Directive]`. This is the piece which maps child failure types to their corresponding directives. -.. note:: +@@@ note - If the strategy is declared inside the supervising actor (as opposed to - within a companion object) its decider has access to all internal state of - the actor in a thread-safe fashion, including obtaining a reference to the - currently failed child (available as the ``sender`` of the failure message). +If the strategy is declared inside the supervising actor (as opposed to +within a companion object) its decider has access to all internal state of +the actor in a thread-safe fashion, including obtaining a reference to the +currently failed child (available as the `sender` of the failure message). -Default Supervisor Strategy -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@@ -``Escalate`` is used if the defined strategy doesn't cover the exception that was thrown. +### Default Supervisor Strategy + +`Escalate` is used if the defined strategy doesn't cover the exception that was thrown. When the supervisor strategy is not defined for an actor the following exceptions are handled by default: -* ``ActorInitializationException`` will stop the failing child actor -* ``ActorKilledException`` will stop the failing child actor -* ``DeathPactException`` will stop the failing child actor -* ``Exception`` will restart the failing child actor -* Other types of ``Throwable`` will be escalated to parent actor + * `ActorInitializationException` will stop the failing child actor + * `ActorKilledException` will stop the failing child actor + * `DeathPactException` will stop the failing child actor + * `Exception` will restart the failing child actor + * Other types of `Throwable` will be escalated to parent actor If the exception escalate all the way up to the root guardian it will handle it in the same way as the default strategy defined above. You can combine your own strategy with the default strategy: -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: default-strategy-fallback +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #default-strategy-fallback } -Stopping Supervisor Strategy -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Stopping Supervisor Strategy Closer to the Erlang way is the strategy to just stop children when they fail and then take corrective action in the supervisor when DeathWatch signals the loss of the child. This strategy is also provided pre-packaged as -:obj:`SupervisorStrategy.stoppingStrategy` with an accompanying -:class:`StoppingSupervisorStrategy` configurator to be used when you want the -``"/user"`` guardian to apply it. +`SupervisorStrategy.stoppingStrategy` with an accompanying +`StoppingSupervisorStrategy` configurator to be used when you want the +`"/user"` guardian to apply it. -Logging of Actor Failures -^^^^^^^^^^^^^^^^^^^^^^^^^ +### Logging of Actor Failures -By default the ``SupervisorStrategy`` logs failures unless they are escalated. +By default the `SupervisorStrategy` logs failures unless they are escalated. Escalated failures are supposed to be handled, and potentially logged, at a level higher in the hierarchy. -You can mute the default logging of a ``SupervisorStrategy`` by setting -``loggingEnabled`` to ``false`` when instantiating it. Customized logging -can be done inside the ``Decider``. Note that the reference to the currently -failed child is available as the ``sender`` when the ``SupervisorStrategy`` is +You can mute the default logging of a `SupervisorStrategy` by setting +`loggingEnabled` to `false` when instantiating it. Customized logging +can be done inside the `Decider`. Note that the reference to the currently +failed child is available as the `sender` when the `SupervisorStrategy` is declared inside the supervising actor. -You may also customize the logging in your own ``SupervisorStrategy`` implementation -by overriding the ``logFailure`` method. +You may also customize the logging in your own `SupervisorStrategy` implementation +by overriding the `logFailure` method. -Supervision of Top-Level Actors -------------------------------- +## Supervision of Top-Level Actors -Toplevel actors means those which are created using ``system.actorOf()``, and -they are children of the :ref:`User Guardian `. There are no +Toplevel actors means those which are created using `system.actorOf()`, and +they are children of the @ref:[User Guardian](../general/supervision.md#user-guardian). There are no special rules applied in this case, the guardian simply applies the configured strategy. -Test Application ----------------- +## Test Application The following section shows the effects of the different directives in practice, where a test setup is needed. First off, we need a suitable supervisor: -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: supervisor +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #supervisor } This supervisor will be used to create a child, with which we can experiment: -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: child +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #child } -The test is easier by using the utilities described in :ref:`akka-testkit`. +The test is easier by using the utilities described in @ref:[Testing Actor Systems](testing.md). -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: testkit +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #testkit } Let us create actors: -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: create +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #create } -The first test shall demonstrate the ``Resume`` directive, so we try it out by +The first test shall demonstrate the `Resume` directive, so we try it out by setting some non-initial state in the actor and have it fail: -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: resume +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #resume } As you can see the value 42 survives the fault handling directive. Now, if we -change the failure to a more serious ``NullPointerException``, that will no +change the failure to a more serious `NullPointerException`, that will no longer be the case: -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: restart +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #restart } -And finally in case of the fatal ``IllegalArgumentException`` the child will be +And finally in case of the fatal `IllegalArgumentException` the child will be terminated by the supervisor: -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: stop +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #stop } Up to now the supervisor was completely unaffected by the child’s failure, -because the directives set did handle it. In case of an ``Exception``, this is not +because the directives set did handle it. In case of an `Exception`, this is not true anymore and the supervisor escalates the failure. -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: escalate-kill +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #escalate-kill } The supervisor itself is supervised by the top-level actor provided by the -:class:`ActorSystem`, which has the default policy to restart in case of all -``Exception`` cases (with the notable exceptions of -``ActorInitializationException`` and ``ActorKilledException``). Since the +`ActorSystem`, which has the default policy to restart in case of all +`Exception` cases (with the notable exceptions of +`ActorInitializationException` and `ActorKilledException`). Since the default directive in case of a restart is to kill all children, we expected our poor child not to survive this failure. In case this is not desired (which depends on the use case), we need to use a different supervisor which overrides this behavior. -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: supervisor2 +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #supervisor2 } With this parent, the child survives the escalated restart, as demonstrated in the last test: -.. includecode:: code/docs/actor/FaultHandlingDocSpec.scala - :include: escalate-restart - +@@snip [FaultHandlingDocSpec.scala](code/docs/actor/FaultHandlingDocSpec.scala) { #escalate-restart } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/fsm.md b/akka-docs/src/main/paradox/scala/fsm.md index a15095e924..9df24b7f85 100644 --- a/akka-docs/src/main/paradox/scala/fsm.md +++ b/akka-docs/src/main/paradox/scala/fsm.md @@ -1,279 +1,266 @@ -.. _fsm-scala: +# FSM -### -FSM -### - - -Overview -======== +## Overview The FSM (Finite State Machine) is available as a mixin for the Akka Actor and -is best described in the `Erlang design principles -`_ +is best described in the [Erlang design principles](http://www.erlang.org/documentation/doc-4.8.2/doc/design_principles/fsm.html) A FSM can be described as a set of relations of the form: - **State(S) x Event(E) -> Actions (A), State(S')** +> +**State(S) x Event(E) -> Actions (A), State(S')** These relations are interpreted as meaning: - *If we are in state S and the event E occurs, we should perform the actions A - and make a transition to the state S'.* +> +*If we are in state S and the event E occurs, we should perform the actions A +and make a transition to the state S'.* -A Simple Example -================ +## A Simple Example -To demonstrate most of the features of the :class:`FSM` trait, consider an +To demonstrate most of the features of the `FSM` trait, consider an actor which shall receive and queue messages while they arrive in a burst and send them on after the burst ended or a flush request is received. First, consider all of the below to use these import statements: -.. includecode:: code/docs/actor/FSMDocSpec.scala#simple-imports +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #simple-imports } The contract of our “Buncher” actor is that it accepts or produces the following messages: -.. includecode:: code/docs/actor/FSMDocSpec.scala#simple-events +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #simple-events } -``SetTarget`` is needed for starting it up, setting the destination for the -``Batches`` to be passed on; ``Queue`` will add to the internal queue while -``Flush`` will mark the end of a burst. +`SetTarget` is needed for starting it up, setting the destination for the +`Batches` to be passed on; `Queue` will add to the internal queue while +`Flush` will mark the end of a burst. -.. includecode:: code/docs/actor/FSMDocSpec.scala#simple-state +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #simple-state } -The actor can be in two states: no message queued (aka ``Idle``) or some -message queued (aka ``Active``). It will stay in the active state as long as +The actor can be in two states: no message queued (aka `Idle`) or some +message queued (aka `Active`). It will stay in the active state as long as messages keep arriving and no flush is requested. The internal state data of the actor is made up of the target actor reference to send the batches to and the actual queue of messages. Now let’s take a look at the skeleton for our FSM actor: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: simple-fsm - :exclude: transition-elided,unhandled-elided +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #simple-fsm } -The basic strategy is to declare the actor, mixing in the :class:`FSM` trait +The basic strategy is to declare the actor, mixing in the `FSM` trait and specifying the possible states and data values as type parameters. Within the body of the actor a DSL is used for declaring the state machine: - * :meth:`startWith` defines the initial state and initial data - * then there is one :meth:`when() { ... }` declaration per state to be - handled (could potentially be multiple ones, the passed - :class:`PartialFunction` will be concatenated using :meth:`orElse`) - * finally starting it up using :meth:`initialize`, which performs the - transition into the initial state and sets up timers (if required). +> + * `startWith` defines the initial state and initial data + * then there is one `when() { ... }` declaration per state to be +handled (could potentially be multiple ones, the passed +`PartialFunction` will be concatenated using `orElse`) + * finally starting it up using `initialize`, which performs the +transition into the initial state and sets up timers (if required). -In this case, we start out in the ``Idle`` and ``Uninitialized`` state, where -only the ``SetTarget()`` message is handled; ``stay`` prepares to end this -event’s processing for not leaving the current state, while the ``using`` -modifier makes the FSM replace the internal state (which is ``Uninitialized`` -at this point) with a fresh ``Todo()`` object containing the target actor -reference. The ``Active`` state has a state timeout declared, which means that -if no message is received for 1 second, a ``FSM.StateTimeout`` message will be -generated. This has the same effect as receiving the ``Flush`` command in this -case, namely to transition back into the ``Idle`` state and resetting the +In this case, we start out in the `Idle` and `Uninitialized` state, where +only the `SetTarget()` message is handled; `stay` prepares to end this +event’s processing for not leaving the current state, while the `using` +modifier makes the FSM replace the internal state (which is `Uninitialized` +at this point) with a fresh `Todo()` object containing the target actor +reference. The `Active` state has a state timeout declared, which means that +if no message is received for 1 second, a `FSM.StateTimeout` message will be +generated. This has the same effect as receiving the `Flush` command in this +case, namely to transition back into the `Idle` state and resetting the internal queue to the empty vector. But how do messages get queued? Since this shall work identically in both states, we make use of the fact that any event -which is not handled by the ``when()`` block is passed to the -``whenUnhandled()`` block: +which is not handled by the `when()` block is passed to the +`whenUnhandled()` block: -.. includecode:: code/docs/actor/FSMDocSpec.scala#unhandled-elided +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #unhandled-elided } -The first case handled here is adding ``Queue()`` requests to the internal -queue and going to the ``Active`` state (this does the obvious thing of staying -in the ``Active`` state if already there), but only if the FSM data are not -``Uninitialized`` when the ``Queue()`` event is received. Otherwise—and in all +The first case handled here is adding `Queue()` requests to the internal +queue and going to the `Active` state (this does the obvious thing of staying +in the `Active` state if already there), but only if the FSM data are not +`Uninitialized` when the `Queue()` event is received. Otherwise—and in all other non-handled cases—the second case just logs a warning and does not change the internal state. -The only missing piece is where the ``Batches`` are actually sent to the -target, for which we use the ``onTransition`` mechanism: you can declare +The only missing piece is where the `Batches` are actually sent to the +target, for which we use the `onTransition` mechanism: you can declare multiple such blocks and all of them will be tried for matching behavior in case a state transition occurs (i.e. only when the state actually changes). -.. includecode:: code/docs/actor/FSMDocSpec.scala#transition-elided +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #transition-elided } The transition callback is a partial function which takes as input a pair of states—the current and the next state. The FSM trait includes a convenience extractor for these in form of an arrow operator, which conveniently reminds you of the direction of the state change which is being matched. During the -state change, the old state data is available via ``stateData`` as shown, and -the new state data would be available as ``nextStateData``. +state change, the old state data is available via `stateData` as shown, and +the new state data would be available as `nextStateData`. -.. note:: - Same-state transitions can be implemented (when currently in state ``S``) using - ``goto(S)`` or ``stay()``. The difference between those being that ``goto(S)`` will - emit an event ``S->S`` event that can be handled by ``onTransition``, - whereas ``stay()`` will *not*. +@@@ note +Same-state transitions can be implemented (when currently in state `S`) using +`goto(S)` or `stay()`. The difference between those being that `goto(S)` will +emit an event `S->S` event that can be handled by `onTransition`, +whereas `stay()` will *not*. + +@@@ To verify that this buncher actually works, it is quite easy to write a test -using the :ref:`akka-testkit`, which is conveniently bundled with ScalaTest traits -into ``AkkaSpec``: +using the @ref:[Testing Actor Systems](testing.md), which is conveniently bundled with ScalaTest traits +into `AkkaSpec`: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: test-code - :exclude: fsm-code-elided +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #test-code } -Reference -========= +## Reference -The FSM Trait and Object ------------------------- +### The FSM Trait and Object -The :class:`FSM` trait inherits directly from :class:`Actor`, when you -extend :class:`FSM` you must be aware that an actor is actually created: +The `FSM` trait inherits directly from `Actor`, when you +extend `FSM` you must be aware that an actor is actually created: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: simple-fsm - :exclude: fsm-body +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #simple-fsm } -.. note:: +@@@ note - The FSM trait defines a ``receive`` method which handles internal messages - and passes everything else through to the FSM logic (according to the - current state). When overriding the ``receive`` method, keep in mind that - e.g. state timeout handling depends on actually passing the messages through - the FSM logic. +The FSM trait defines a `receive` method which handles internal messages +and passes everything else through to the FSM logic (according to the +current state). When overriding the `receive` method, keep in mind that +e.g. state timeout handling depends on actually passing the messages through +the FSM logic. -The :class:`FSM` trait takes two type parameters: +@@@ - #. the supertype of all state names, usually a sealed trait with case objects - extending it, - #. the type of the state data which are tracked by the :class:`FSM` module - itself. +The `FSM` trait takes two type parameters: -.. _fsm-philosophy: +> + 1. the supertype of all state names, usually a sealed trait with case objects +extending it, + 2. the type of the state data which are tracked by the `FSM` module +itself. -.. note:: +@@@ note - The state data together with the state name describe the internal state of - the state machine; if you stick to this scheme and do not add mutable fields - to the FSM class you have the advantage of making all changes of the - internal state explicit in a few well-known places. +The state data together with the state name describe the internal state of +the state machine; if you stick to this scheme and do not add mutable fields +to the FSM class you have the advantage of making all changes of the +internal state explicit in a few well-known places. -Defining States ---------------- +@@@ + +### Defining States A state is defined by one or more invocations of the method - :func:`when([, stateTimeout = ])(stateFunction)`. +> +`when([, stateTimeout = ])(stateFunction)`. The given name must be an object which is type-compatible with the first type -parameter given to the :class:`FSM` trait. This object is used as a hash key, -so you must ensure that it properly implements :meth:`equals` and -:meth:`hashCode`; in particular it must not be mutable. The easiest fit for +parameter given to the `FSM` trait. This object is used as a hash key, +so you must ensure that it properly implements `equals` and +`hashCode`; in particular it must not be mutable. The easiest fit for these requirements are case objects. -If the :meth:`stateTimeout` parameter is given, then all transitions into this +If the `stateTimeout` parameter is given, then all transitions into this state, including staying, receive this timeout by default. Initiating the transition with an explicit timeout may be used to override this default, see -`Initiating Transitions`_ for more information. The state timeout of any state +[Initiating Transitions](#initiating-transitions) for more information. The state timeout of any state may be changed during action processing with -:func:`setStateTimeout(state, duration)`. This enables runtime configuration +`setStateTimeout(state, duration)`. This enables runtime configuration e.g. via external message. -The :meth:`stateFunction` argument is a :class:`PartialFunction[Event, State]`, +The `stateFunction` argument is a `PartialFunction[Event, State]`, which is conveniently given using the partial function literal syntax as demonstrated below: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: when-syntax +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #when-syntax } -The :class:`Event(msg: Any, data: D)` case class is parameterized with the data +The `Event(msg: Any, data: D)` case class is parameterized with the data type held by the FSM for convenient pattern matching. -.. warning:: +@@@ warning - It is required that you define handlers for each of the possible FSM states, - otherwise there will be failures when trying to switch to undeclared states. +It is required that you define handlers for each of the possible FSM states, +otherwise there will be failures when trying to switch to undeclared states. + +@@@ It is recommended practice to declare the states as objects extending a -sealed trait and then verify that there is a ``when`` clause for each of the +sealed trait and then verify that there is a `when` clause for each of the states. If you want to leave the handling of a state “unhandled” (more below), it still needs to be declared like this: -.. includecode:: code/docs/actor/FSMDocSpec.scala#NullFunction +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #NullFunction } -Defining the Initial State --------------------------- +### Defining the Initial State Each FSM needs a starting point, which is declared using - :func:`startWith(state, data[, timeout])` +> +`startWith(state, data[, timeout])` The optionally given timeout argument overrides any specification given for the desired initial state. If you want to cancel a default timeout, use -:obj:`None`. +`None`. -Unhandled Events ----------------- +### Unhandled Events If a state doesn't handle a received event a warning is logged. If you want to do something else in this case you can specify that with -:func:`whenUnhandled(stateFunction)`: +`whenUnhandled(stateFunction)`: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: unhandled-syntax +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #unhandled-syntax } Within this handler the state of the FSM may be queried using the -:meth:`stateName` method. +`stateName` method. **IMPORTANT**: This handler is not stacked, meaning that each invocation of -:func:`whenUnhandled` replaces the previously installed handler. +`whenUnhandled` replaces the previously installed handler. -Initiating Transitions ----------------------- +### Initiating Transitions -The result of any :obj:`stateFunction` must be a definition of the next state -unless terminating the FSM, which is described in `Termination from Inside`_. +The result of any `stateFunction` must be a definition of the next state +unless terminating the FSM, which is described in [Termination from Inside](#termination-from-inside). The state definition can either be the current state, as described by the -:func:`stay` directive, or it is a different state as given by -:func:`goto(state)`. The resulting object allows further qualification by way +`stay` directive, or it is a different state as given by +`goto(state)`. The resulting object allows further qualification by way of the modifiers described in the following: -* :meth:`forMax(duration)` - - This modifier sets a state timeout on the next state. This means that a timer - is started which upon expiry sends a :obj:`StateTimeout` message to the FSM. - This timer is canceled upon reception of any other message in the meantime; - you can rely on the fact that the :obj:`StateTimeout` message will not be - processed after an intervening message. - - This modifier can also be used to override any default timeout which is - specified for the target state. If you want to cancel the default timeout, - use :obj:`Duration.Inf`. - -* :meth:`using(data)` - - This modifier replaces the old state data with the new data given. If you - follow the advice :ref:`above `, this is the only place where - internal state data are ever modified. - -* :meth:`replying(msg)` - - This modifier sends a reply to the currently processed message and otherwise - does not modify the state transition. + * + `forMax(duration)` + This modifier sets a state timeout on the next state. This means that a timer +is started which upon expiry sends a `StateTimeout` message to the FSM. +This timer is canceled upon reception of any other message in the meantime; +you can rely on the fact that the `StateTimeout` message will not be +processed after an intervening message. + This modifier can also be used to override any default timeout which is +specified for the target state. If you want to cancel the default timeout, +use `Duration.Inf`. + * + `using(data)` + This modifier replaces the old state data with the new data given. If you +follow the advice [above](#fsm-philosophy), this is the only place where +internal state data are ever modified. + * + `replying(msg)` + This modifier sends a reply to the currently processed message and otherwise +does not modify the state transition. All modifiers can be chained to achieve a nice and concise description: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: modifier-syntax +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #modifier-syntax } The parentheses are not actually needed in all cases, but they visually distinguish between modifiers and their arguments and therefore make the code even more pleasant to read for foreigners. -.. note:: +@@@ note - Please note that the ``return`` statement may not be used in :meth:`when` - blocks or similar; this is a Scala restriction. Either refactor your code - using ``if () ... else ...`` or move it into a method definition. +Please note that the `return` statement may not be used in `when` +blocks or similar; this is a Scala restriction. Either refactor your code +using `if () ... else ...` or move it into a method definition. -Monitoring Transitions ----------------------- +@@@ + +### Monitoring Transitions Transitions occur "between states" conceptually, which means after any actions you have put into the event handling block; this is obvious since the next @@ -282,218 +269,213 @@ not need to worry about the exact order with respect to setting the internal state variable, as everything within the FSM actor is running single-threaded anyway. -Internal Monitoring -^^^^^^^^^^^^^^^^^^^ +#### Internal Monitoring Up to this point, the FSM DSL has been centered on states and events. The dual view is to describe it as a series of transitions. This is enabled by the method - :func:`onTransition(handler)` +> +`onTransition(handler)` which associates actions with a transition instead of with a state and event. The handler is a partial function which takes a pair of states as input; no resulting state is needed as it is not possible to modify the transition in progress. -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: transition-syntax +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #transition-syntax } -The convenience extractor :obj:`->` enables decomposition of the pair of states +The convenience extractor `->` enables decomposition of the pair of states with a clear visual reminder of the transition's direction. As usual in pattern matches, an underscore may be used for irrelevant parts; alternatively you could bind the unconstrained state to a variable, e.g. for logging as shown in the last case. It is also possible to pass a function object accepting two states to -:func:`onTransition`, in case your transition handling logic is implemented as +`onTransition`, in case your transition handling logic is implemented as a method: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: alt-transition-syntax +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #alt-transition-syntax } The handlers registered with this method are stacked, so you can intersperse -:func:`onTransition` blocks with :func:`when` blocks as suits your design. It +`onTransition` blocks with `when` blocks as suits your design. It should be noted, however, that *all handlers will be invoked for each transition*, not only the first matching one. This is designed specifically so you can put all transition handling for a certain aspect into one place without having to worry about earlier declarations shadowing later ones; the actions are still executed in declaration order, though. -.. note:: +@@@ note - This kind of internal monitoring may be used to structure your FSM according - to transitions, so that for example the cancellation of a timer upon leaving - a certain state cannot be forgot when adding new target states. +This kind of internal monitoring may be used to structure your FSM according +to transitions, so that for example the cancellation of a timer upon leaving +a certain state cannot be forgot when adding new target states. -External Monitoring -^^^^^^^^^^^^^^^^^^^ +@@@ + +#### External Monitoring External actors may be registered to be notified of state transitions by -sending a message :class:`SubscribeTransitionCallBack(actorRef)`. The named -actor will be sent a :class:`CurrentState(self, stateName)` message immediately -and will receive :class:`Transition(actorRef, oldState, newState)` messages +sending a message `SubscribeTransitionCallBack(actorRef)`. The named +actor will be sent a `CurrentState(self, stateName)` message immediately +and will receive `Transition(actorRef, oldState, newState)` messages whenever a state change is triggered. -Please note that a state change includes the action of performing an ``goto(S)``, while -already being state ``S``. In that case the monitoring actor will be notified with an -``Transition(ref,S,S)`` message. This may be useful if your ``FSM`` should +Please note that a state change includes the action of performing an `goto(S)`, while +already being state `S`. In that case the monitoring actor will be notified with an +`Transition(ref,S,S)` message. This may be useful if your `FSM` should react on all (also same-state) transitions. In case you'd rather not emit events for same-state -transitions use ``stay()`` instead of ``goto(S)``. +transitions use `stay()` instead of `goto(S)`. External monitors may be unregistered by sending -:class:`UnsubscribeTransitionCallBack(actorRef)` to the ``FSM`` actor. +`UnsubscribeTransitionCallBack(actorRef)` to the `FSM` actor. Stopping a listener without unregistering will not remove the listener from the -subscription list; use :class:`UnsubscribeTransitionCallback` before stopping +subscription list; use `UnsubscribeTransitionCallback` before stopping the listener. -Transforming State ------------------- +### Transforming State -The partial functions supplied as argument to the ``when()`` blocks can be +The partial functions supplied as argument to the `when()` blocks can be transformed using Scala’s full supplement of functional programming tools. In order to retain type inference, there is a helper function which may be used in case some common handling logic shall be applied to different clauses: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: transform-syntax +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #transform-syntax } It goes without saying that the arguments to this method may also be stored, to be used several times, e.g. when applying the same transformation to several -``when()`` blocks: +`when()` blocks: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: alt-transform-syntax +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #alt-transform-syntax } -Timers ------- +### Timers -Besides state timeouts, FSM manages timers identified by :class:`String` names. +Besides state timeouts, FSM manages timers identified by `String` names. You may set a timer using - :func:`setTimer(name, msg, interval, repeat)` +> +`setTimer(name, msg, interval, repeat)` -where :obj:`msg` is the message object which will be sent after the duration -:obj:`interval` has elapsed. If :obj:`repeat` is :obj:`true`, then the timer is -scheduled at fixed rate given by the :obj:`interval` parameter. +where `msg` is the message object which will be sent after the duration +`interval` has elapsed. If `repeat` is `true`, then the timer is +scheduled at fixed rate given by the `interval` parameter. Any existing timer with the same name will automatically be canceled before adding the new timer. Timers may be canceled using - :func:`cancelTimer(name)` +> +`cancelTimer(name)` which is guaranteed to work immediately, meaning that the scheduled message will not be processed after this call even if the timer already fired and queued it. The status of any timer may be inquired with - :func:`isTimerActive(name)` +> +`isTimerActive(name)` These named timers complement state timeouts because they are not affected by intervening reception of other messages. -Termination from Inside ------------------------ +### Termination from Inside The FSM is stopped by specifying the result state as - :func:`stop([reason[, data]])` +> +`stop([reason[, data]])` -The reason must be one of :obj:`Normal` (which is the default), :obj:`Shutdown` -or :obj:`Failure(reason)`, and the second argument may be given to change the +The reason must be one of `Normal` (which is the default), `Shutdown` +or `Failure(reason)`, and the second argument may be given to change the state data which is available during termination handling. -.. note:: +@@@ note - It should be noted that :func:`stop` does not abort the actions and stop the - FSM immediately. The stop action must be returned from the event handler in - the same way as a state transition (but note that the ``return`` statement - may not be used within a :meth:`when` block). +It should be noted that `stop` does not abort the actions and stop the +FSM immediately. The stop action must be returned from the event handler in +the same way as a state transition (but note that the `return` statement +may not be used within a `when` block). -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: stop-syntax +@@@ -You can use :func:`onTermination(handler)` to specify custom code that is +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #stop-syntax } + +You can use `onTermination(handler)` to specify custom code that is executed when the FSM is stopped. The handler is a partial function which takes -a :class:`StopEvent(reason, stateName, stateData)` as argument: +a `StopEvent(reason, stateName, stateData)` as argument: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: termination-syntax +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #termination-syntax } -As for the :func:`whenUnhandled` case, this handler is not stacked, so each -invocation of :func:`onTermination` replaces the previously installed handler. +As for the `whenUnhandled` case, this handler is not stacked, so each +invocation of `onTermination` replaces the previously installed handler. -Termination from Outside ------------------------- +### Termination from Outside -When an :class:`ActorRef` associated to a FSM is stopped using the -:meth:`stop()` method, its :meth:`postStop` hook will be executed. The default -implementation by the :class:`FSM` trait is to execute the -:meth:`onTermination` handler if that is prepared to handle a -:obj:`StopEvent(Shutdown, ...)`. +When an `ActorRef` associated to a FSM is stopped using the +`stop()` method, its `postStop` hook will be executed. The default +implementation by the `FSM` trait is to execute the +`onTermination` handler if that is prepared to handle a +`StopEvent(Shutdown, ...)`. -.. warning:: +@@@ warning - In case you override :meth:`postStop` and want to have your - :meth:`onTermination` handler called, do not forget to call - ``super.postStop``. +In case you override `postStop` and want to have your +`onTermination` handler called, do not forget to call +`super.postStop`. -Testing and Debugging Finite State Machines -=========================================== +@@@ + +## Testing and Debugging Finite State Machines During development and for trouble shooting FSMs need care just as any other -actor. There are specialized tools available as described in :ref:`TestFSMRef` +actor. There are specialized tools available as described in @ref:[TestFSMRef](testing.md#testfsmref) and in the following. -Event Tracing -------------- +### Event Tracing -The setting ``akka.actor.debug.fsm`` in :ref:`configuration` enables logging of an -event trace by :class:`LoggingFSM` instances: +The setting `akka.actor.debug.fsm` in configuration enables logging of an +event trace by `LoggingFSM` instances: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: logging-fsm - :exclude: body-elided +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #logging-fsm } This FSM will log at DEBUG level: - * all processed events, including :obj:`StateTimeout` and scheduled timer - messages - * every setting and cancellation of named timers - * all state transitions +> + * all processed events, including `StateTimeout` and scheduled timer +messages + * every setting and cancellation of named timers + * all state transitions Life cycle changes and special messages can be logged as described for -:ref:`Actors `. +@ref:[Actors](testing.md#actor-logging-scala). -Rolling Event Log ------------------ +### Rolling Event Log -The :class:`LoggingFSM` trait adds one more feature to the FSM: a rolling event +The `LoggingFSM` trait adds one more feature to the FSM: a rolling event log which may be used during debugging (for tracing how the FSM entered a certain failure state) or for other creative uses: -.. includecode:: code/docs/actor/FSMDocSpec.scala - :include: logging-fsm +@@snip [FSMDocSpec.scala](code/docs/actor/FSMDocSpec.scala) { #logging-fsm } -The :meth:`logDepth` defaults to zero, which turns off the event log. +The `logDepth` defaults to zero, which turns off the event log. -.. warning:: +@@@ warning - The log buffer is allocated during actor creation, which is why the - configuration is done using a virtual method call. If you want to override - with a ``val``, make sure that its initialization happens before the - initializer of :class:`LoggingFSM` runs, and do not change the value returned - by ``logDepth`` after the buffer has been allocated. +The log buffer is allocated during actor creation, which is why the +configuration is done using a virtual method call. If you want to override +with a `val`, make sure that its initialization happens before the +initializer of `LoggingFSM` runs, and do not change the value returned +by `logDepth` after the buffer has been allocated. -The contents of the event log are available using method :meth:`getLog`, which -returns an :class:`IndexedSeq[LogEntry]` where the oldest entry is at index +@@@ + +The contents of the event log are available using method `getLog`, which +returns an `IndexedSeq[LogEntry]` where the oldest entry is at index zero. -Examples -======== +## Examples -A bigger FSM example contrasted with Actor's :meth:`become`/:meth:`unbecome` can be -downloaded as a ready to run `Akka FSM sample <@exampleCodeService@/akka-samples-fsm-scala>`_ +A bigger FSM example contrasted with Actor's `become`/`unbecome` can be +downloaded as a ready to run [Akka FSM sample](@exampleCodeService@/akka-samples-fsm-scala) together with a tutorial. The source code of this sample can be found in the -`Akka Samples Repository <@samples@/akka-sample-fsm-scala>`_. +[Akka Samples Repository](@samples@/akka-sample-fsm-scala). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/futures.md b/akka-docs/src/main/paradox/scala/futures.md index 705e620708..116aefcd5d 100644 --- a/akka-docs/src/main/paradox/scala/futures.md +++ b/akka-docs/src/main/paradox/scala/futures.md @@ -1,299 +1,252 @@ -.. _futures-scala: +# Futures -Futures -=============== +## Introduction - -Introduction ------------- - -In the Scala Standard Library, a `Future `_ is a data structure +In the Scala Standard Library, a [Future](http://en.wikipedia.org/wiki/Futures_and_promises) is a data structure used to retrieve the result of some concurrent operation. This result can be accessed synchronously (blocking) or asynchronously (non-blocking). -Execution Contexts ------------------- +## Execution Contexts -In order to execute callbacks and operations, Futures need something called an ``ExecutionContext``, -which is very similar to a ``java.util.concurrent.Executor``. if you have an ``ActorSystem`` in scope, -it will use its default dispatcher as the ``ExecutionContext``, or you can use the factory methods provided -by the ``ExecutionContext`` companion object to wrap ``Executors`` and ``ExecutorServices``, or even create your own. +In order to execute callbacks and operations, Futures need something called an `ExecutionContext`, +which is very similar to a `java.util.concurrent.Executor`. if you have an `ActorSystem` in scope, +it will use its default dispatcher as the `ExecutionContext`, or you can use the factory methods provided +by the `ExecutionContext` companion object to wrap `Executors` and `ExecutorServices`, or even create your own. -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: diy-execution-context +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #diy-execution-context } -Within Actors -^^^^^^^^^^^^^ +### Within Actors -Each actor is configured to be run on a :class:`MessageDispatcher`, and that -dispatcher doubles as an :class:`ExecutionContext`. If the nature of the Future +Each actor is configured to be run on a `MessageDispatcher`, and that +dispatcher doubles as an `ExecutionContext`. If the nature of the Future calls invoked by the actor matches or is compatible with the activities of that actor (e.g. all CPU bound and no latency requirements), then it may be easiest to reuse the dispatcher for running the Futures by importing -``context.dispatcher``. +`context.dispatcher`. -.. includecode:: code/docs/future/FutureDocSpec.scala#context-dispatcher - :exclude: receive-omitted +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #context-dispatcher } -Use With Actors ---------------- +## Use With Actors -There are generally two ways of getting a reply from an ``Actor``: the first is by a sent message (``actor ! msg``), -which only works if the original sender was an ``Actor``) and the second is through a ``Future``. +There are generally two ways of getting a reply from an `Actor`: the first is by a sent message (`actor ! msg`), +which only works if the original sender was an `Actor`) and the second is through a `Future`. -Using an ``Actor``\'s ``?`` method to send a message will return a ``Future``: +Using an `Actor`'s `?` method to send a message will return a `Future`: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: ask-blocking +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #ask-blocking } -This will cause the current thread to block and wait for the ``Actor`` to 'complete' the ``Future`` with it's reply. +This will cause the current thread to block and wait for the `Actor` to 'complete' the `Future` with it's reply. Blocking is discouraged though as it will cause performance problems. -The blocking operations are located in ``Await.result`` and ``Await.ready`` to make it easy to spot where blocking occurs. -Alternatives to blocking are discussed further within this documentation. Also note that the ``Future`` returned by -an ``Actor`` is a ``Future[Any]`` since an ``Actor`` is dynamic. That is why the ``asInstanceOf`` is used in the above sample. -When using non-blocking it is better to use the ``mapTo`` method to safely try to cast a ``Future`` to an expected type: +The blocking operations are located in `Await.result` and `Await.ready` to make it easy to spot where blocking occurs. +Alternatives to blocking are discussed further within this documentation. Also note that the `Future` returned by +an `Actor` is a `Future[Any]` since an `Actor` is dynamic. That is why the `asInstanceOf` is used in the above sample. +When using non-blocking it is better to use the `mapTo` method to safely try to cast a `Future` to an expected type: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: map-to +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #map-to } -The ``mapTo`` method will return a new ``Future`` that contains the result if the cast was successful, -or a ``ClassCastException`` if not. Handling ``Exception``\s will be discussed further within this documentation. +The `mapTo` method will return a new `Future` that contains the result if the cast was successful, +or a `ClassCastException` if not. Handling `Exception`s will be discussed further within this documentation. -To send the result of a ``Future`` to an ``Actor``, you can use the ``pipe`` construct: +To send the result of a `Future` to an `Actor`, you can use the `pipe` construct: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: pipe-to +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #pipe-to } -Use Directly ------------- +## Use Directly -A common use case within Akka is to have some computation performed concurrently without needing the extra utility of an ``Actor``. -If you find yourself creating a pool of ``Actor``\s for the sole reason of performing a calculation in parallel, +A common use case within Akka is to have some computation performed concurrently without needing the extra utility of an `Actor`. +If you find yourself creating a pool of `Actor`s for the sole reason of performing a calculation in parallel, there is an easier (and faster) way: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: future-eval +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #future-eval } -In the above code the block passed to ``Future`` will be executed by the default ``Dispatcher``, -with the return value of the block used to complete the ``Future`` (in this case, the result would be the string: "HelloWorld"). -Unlike a ``Future`` that is returned from an ``Actor``, this ``Future`` is properly typed, -and we also avoid the overhead of managing an ``Actor``. +In the above code the block passed to `Future` will be executed by the default `Dispatcher`, +with the return value of the block used to complete the `Future` (in this case, the result would be the string: "HelloWorld"). +Unlike a `Future` that is returned from an `Actor`, this `Future` is properly typed, +and we also avoid the overhead of managing an `Actor`. -You can also create already completed Futures using the ``Future`` companion, which can be either successes: +You can also create already completed Futures using the `Future` companion, which can be either successes: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: successful +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #successful } Or failures: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: failed +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #failed } -It is also possible to create an empty ``Promise``, to be filled later, and obtain the corresponding ``Future``: +It is also possible to create an empty `Promise`, to be filled later, and obtain the corresponding `Future`: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: promise +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #promise } -Functional Futures ------------------- +## Functional Futures -Scala's ``Future`` has several monadic methods that are very similar to the ones used by Scala's collections. +Scala's `Future` has several monadic methods that are very similar to the ones used by Scala's collections. These allow you to create 'pipelines' or 'streams' that the result will travel through. -Future is a Monad -^^^^^^^^^^^^^^^^^ +### Future is a Monad -The first method for working with ``Future`` functionally is ``map``. This method takes a ``Function`` -which performs some operation on the result of the ``Future``, and returning a new result. -The return value of the ``map`` method is another ``Future`` that will contain the new result: +The first method for working with `Future` functionally is `map`. This method takes a `Function` +which performs some operation on the result of the `Future`, and returning a new result. +The return value of the `map` method is another `Future` that will contain the new result: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: map +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #map } -In this example we are joining two strings together within a ``Future``. Instead of waiting for this to complete, -we apply our function that calculates the length of the string using the ``map`` method. -Now we have a second ``Future`` that will eventually contain an ``Int``. -When our original ``Future`` completes, it will also apply our function and complete the second ``Future`` with its result. -When we finally get the result, it will contain the number 10. Our original ``Future`` still contains the -string "HelloWorld" and is unaffected by the ``map``. +In this example we are joining two strings together within a `Future`. Instead of waiting for this to complete, +we apply our function that calculates the length of the string using the `map` method. +Now we have a second `Future` that will eventually contain an `Int`. +When our original `Future` completes, it will also apply our function and complete the second `Future` with its result. +When we finally get the result, it will contain the number 10. Our original `Future` still contains the +string "HelloWorld" and is unaffected by the `map`. -The ``map`` method is fine if we are modifying a single ``Future``, -but if 2 or more ``Future``\s are involved ``map`` will not allow you to combine them together: +The `map` method is fine if we are modifying a single `Future`, +but if 2 or more `Future`s are involved `map` will not allow you to combine them together: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: wrong-nested-map +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #wrong-nested-map } -``f3`` is a ``Future[Future[Int]]`` instead of the desired ``Future[Int]``. Instead, the ``flatMap`` method should be used: +`f3` is a `Future[Future[Int]]` instead of the desired `Future[Int]`. Instead, the `flatMap` method should be used: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: flat-map +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #flat-map } Composing futures using nested combinators it can sometimes become quite complicated and hard to read, in these cases using Scala's 'for comprehensions' usually yields more readable code. See next section for examples. -If you need to do conditional propagation, you can use ``filter``: +If you need to do conditional propagation, you can use `filter`: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: filter +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #filter } -For Comprehensions -^^^^^^^^^^^^^^^^^^ +### For Comprehensions -Since ``Future`` has a ``map``, ``filter`` and ``flatMap`` method it can be easily used in a 'for comprehension': +Since `Future` has a `map`, `filter` and `flatMap` method it can be easily used in a 'for comprehension': -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: for-comprehension +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #for-comprehension } Something to keep in mind when doing this is even though it looks like parts of the above example can run in parallel, each step of the for comprehension is run sequentially. This will happen on separate threads for each step but -there isn't much benefit over running the calculations all within a single ``Future``. -The real benefit comes when the ``Future``\s are created first, and then combining them together. +there isn't much benefit over running the calculations all within a single `Future`. +The real benefit comes when the `Future`s are created first, and then combining them together. -Composing Futures -^^^^^^^^^^^^^^^^^ +### Composing Futures -The example for comprehension above is an example of composing ``Future``\s. -A common use case for this is combining the replies of several ``Actor``\s into a single calculation -without resorting to calling ``Await.result`` or ``Await.ready`` to block for each result. -First an example of using ``Await.result``: +The example for comprehension above is an example of composing `Future`s. +A common use case for this is combining the replies of several `Actor`s into a single calculation +without resorting to calling `Await.result` or `Await.ready` to block for each result. +First an example of using `Await.result`: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: composing-wrong +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #composing-wrong } -.. warning:: +@@@ warning - ``Await.result`` and ``Await.ready`` are provided for exceptional situations where you **must** block, - a good rule of thumb is to only use them if you know why you **must** block. For all other cases, use - asynchronous composition as described below. +`Await.result` and `Await.ready` are provided for exceptional situations where you **must** block, +a good rule of thumb is to only use them if you know why you **must** block. For all other cases, use +asynchronous composition as described below. -Here we wait for the results from the first 2 ``Actor``\s before sending that result to the third ``Actor``. -We called ``Await.result`` 3 times, which caused our little program to block 3 times before getting our final result. +@@@ + +Here we wait for the results from the first 2 `Actor`s before sending that result to the third `Actor`. +We called `Await.result` 3 times, which caused our little program to block 3 times before getting our final result. Now compare that to this example: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: composing +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #composing } Here we have 2 actors processing a single message each. Once the 2 results are available -(note that we don't block to get these results!), they are being added together and sent to a third ``Actor``, +(note that we don't block to get these results!), they are being added together and sent to a third `Actor`, which replies with a string, which we assign to 'result'. This is fine when dealing with a known amount of Actors, but can grow unwieldy if we have more than a handful. -The ``sequence`` and ``traverse`` helper methods can make it easier to handle more complex use cases. -Both of these methods are ways of turning, for a subclass ``T`` of ``Traversable``, ``T[Future[A]]`` into a ``Future[T[A]]``. +The `sequence` and `traverse` helper methods can make it easier to handle more complex use cases. +Both of these methods are ways of turning, for a subclass `T` of `Traversable`, `T[Future[A]]` into a `Future[T[A]]`. For example: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: sequence-ask +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #sequence-ask } -To better explain what happened in the example, ``Future.sequence`` is taking the ``List[Future[Int]]`` -and turning it into a ``Future[List[Int]]``. We can then use ``map`` to work with the ``List[Int]`` directly, -and we find the sum of the ``List``. +To better explain what happened in the example, `Future.sequence` is taking the `List[Future[Int]]` +and turning it into a `Future[List[Int]]`. We can then use `map` to work with the `List[Int]` directly, +and we find the sum of the `List`. -The ``traverse`` method is similar to ``sequence``, but it takes a ``T[A]`` and a function ``A => Future[B]`` to return a ``Future[T[B]]``, -where ``T`` is again a subclass of Traversable. For example, to use ``traverse`` to sum the first 100 odd numbers: +The `traverse` method is similar to `sequence`, but it takes a `T[A]` and a function `A => Future[B]` to return a `Future[T[B]]`, +where `T` is again a subclass of Traversable. For example, to use `traverse` to sum the first 100 odd numbers: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: traverse +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #traverse } This is the same result as this example: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: sequence +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #sequence } -But it may be faster to use ``traverse`` as it doesn't have to create an intermediate ``List[Future[Int]]``. +But it may be faster to use `traverse` as it doesn't have to create an intermediate `List[Future[Int]]`. -Then there's a method that's called ``fold`` that takes a start-value, a sequence of ``Future``\s and a function +Then there's a method that's called `fold` that takes a start-value, a sequence of `Future`s and a function from the type of the start-value and the type of the futures and returns something with the same type as the start-value, and then applies the function to all elements in the sequence of futures, asynchronously, the execution will start when the last of the Futures is completed. -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: fold +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #fold } That's all it takes! +If the sequence passed to `fold` is empty, it will return the start-value, in the case above, that will be 0. +In some cases you don't have a start-value and you're able to use the value of the first completing `Future` in the sequence +as the start-value, you can use `reduce`, it works like this: -If the sequence passed to ``fold`` is empty, it will return the start-value, in the case above, that will be 0. -In some cases you don't have a start-value and you're able to use the value of the first completing ``Future`` in the sequence -as the start-value, you can use ``reduce``, it works like this: +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #reduce } -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: reduce - -Same as with ``fold``, the execution will be done asynchronously when the last of the ``Future`` is completed, +Same as with `fold`, the execution will be done asynchronously when the last of the `Future` is completed, you can also parallelize it by chunking your futures into sub-sequences and reduce them, and then reduce the reduced results again. -Callbacks ---------- +## Callbacks -Sometimes you just want to listen to a ``Future`` being completed, and react to that not by creating a new ``Future``, but by side-effecting. -For this Scala supports ``onComplete``, ``onSuccess`` and ``onFailure``, of which the last two are specializations of the first. +Sometimes you just want to listen to a `Future` being completed, and react to that not by creating a new `Future`, but by side-effecting. +For this Scala supports `onComplete`, `onSuccess` and `onFailure`, of which the last two are specializations of the first. -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: onSuccess +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #onSuccess } -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: onFailure +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #onFailure } -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: onComplete +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #onComplete } -Define Ordering ---------------- +## Define Ordering Since callbacks are executed in any order and potentially in parallel, it can be tricky at the times when you need sequential ordering of operations. -But there's a solution and it's name is ``andThen``. It creates a new ``Future`` with -the specified callback, a ``Future`` that will have the same result as the ``Future`` it's called on, +But there's a solution and it's name is `andThen`. It creates a new `Future` with +the specified callback, a `Future` that will have the same result as the `Future` it's called on, which allows for ordering like in the following sample: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: and-then +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #and-then } -Auxiliary Methods ------------------ +## Auxiliary Methods -``Future`` ``fallbackTo`` combines 2 Futures into a new ``Future``, and will hold the successful value of the second ``Future`` -if the first ``Future`` fails. +`Future` `fallbackTo` combines 2 Futures into a new `Future`, and will hold the successful value of the second `Future` +if the first `Future` fails. -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: fallback-to +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #fallback-to } -You can also combine two Futures into a new ``Future`` that will hold a tuple of the two Futures successful results, -using the ``zip`` operation. +You can also combine two Futures into a new `Future` that will hold a tuple of the two Futures successful results, +using the `zip` operation. -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: zip +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #zip } -Exceptions ----------- +## Exceptions -Since the result of a ``Future`` is created concurrently to the rest of the program, exceptions must be handled differently. -It doesn't matter if an ``Actor`` or the dispatcher is completing the ``Future``, -if an ``Exception`` is caught the ``Future`` will contain it instead of a valid result. -If a ``Future`` does contain an ``Exception``, calling ``Await.result`` will cause it to be thrown again so it can be handled properly. +Since the result of a `Future` is created concurrently to the rest of the program, exceptions must be handled differently. +It doesn't matter if an `Actor` or the dispatcher is completing the `Future`, +if an `Exception` is caught the `Future` will contain it instead of a valid result. +If a `Future` does contain an `Exception`, calling `Await.result` will cause it to be thrown again so it can be handled properly. -It is also possible to handle an ``Exception`` by returning a different result. -This is done with the ``recover`` method. For example: +It is also possible to handle an `Exception` by returning a different result. +This is done with the `recover` method. For example: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: recover +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #recover } -In this example, if the actor replied with a ``akka.actor.Status.Failure`` containing the ``ArithmeticException``, -our ``Future`` would have a result of 0. The ``recover`` method works very similarly to the standard try/catch blocks, -so multiple ``Exception``\s can be handled in this manner, and if an ``Exception`` is not handled this way -it will behave as if we hadn't used the ``recover`` method. +In this example, if the actor replied with a `akka.actor.Status.Failure` containing the `ArithmeticException`, +our `Future` would have a result of 0. The `recover` method works very similarly to the standard try/catch blocks, +so multiple `Exception`s can be handled in this manner, and if an `Exception` is not handled this way +it will behave as if we hadn't used the `recover` method. -You can also use the ``recoverWith`` method, which has the same relationship to ``recover`` as ``flatMap`` has to ``map``, +You can also use the `recoverWith` method, which has the same relationship to `recover` as `flatMap` has to `map`, and is use like this: -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: try-recover +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #try-recover } -After ------ +## After -``akka.pattern.after`` makes it easy to complete a ``Future`` with a value or exception after a timeout. +`akka.pattern.after` makes it easy to complete a `Future` with a value or exception after a timeout. -.. includecode:: code/docs/future/FutureDocSpec.scala - :include: after +@@snip [FutureDocSpec.scala](code/docs/future/FutureDocSpec.scala) { #after } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/hello-world.md b/akka-docs/src/main/paradox/scala/hello-world.md index 32628122f9..631ba4939c 100644 --- a/akka-docs/src/main/paradox/scala/hello-world.md +++ b/akka-docs/src/main/paradox/scala/hello-world.md @@ -1,21 +1,21 @@ -########################## -The Obligatory Hello World -########################## +# The Obligatory Hello World The actor based version of the tough problem of printing a -well-known greeting to the console is introduced in a ready to run `Akka Main sample <@exampleCodeService@/akka-samples-main-scala>`_ +well-known greeting to the console is introduced in a ready to run [Akka Main sample](@exampleCodeService@/akka-samples-main-scala) together with a tutorial. The source code of this sample can be found in the -`Akka Samples Repository <@samples@/akka-sample-main-scala>`_. +[Akka Samples Repository](@samples@/akka-sample-main-scala). -The tutorial illustrates the generic launcher class :class:`akka.Main` which expects only +The tutorial illustrates the generic launcher class `akka.Main` which expects only one command line argument: the class name of the application’s main actor. This main method will then create the infrastructure needed for running the actors, start the given main actor and arrange for the whole application to shut down once the main actor terminates. -There is also a `Gitter8 `_ template in the same problem domain -that is named `Hello Akka! `_. -It describes the basics of Akka in more depth. If you have `sbt` already installed, you can create a project -from this template by running:: +There is also a [Gitter8](http://www.foundweekends.org/giter8/) template in the same problem domain +that is named [Hello Akka!](https://github.com/akka/hello-akka.g8). +It describes the basics of Akka in more depth. If you have *sbt* already installed, you can create a project +from this template by running: - sbt new akka/hello-akka.g8 +``` +sbt new akka/hello-akka.g8 +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/howto.md b/akka-docs/src/main/paradox/scala/howto.md index 9628770593..9bd27ebcef 100644 --- a/akka-docs/src/main/paradox/scala/howto.md +++ b/akka-docs/src/main/paradox/scala/howto.md @@ -1,9 +1,4 @@ - -.. _howto-scala: - -###################### -HowTo: Common Patterns -###################### +# HowTo: Common Patterns This section lists common actor patterns which have been found to be useful, elegant or instructive. Anything is welcome, example topics being message @@ -11,20 +6,17 @@ routing strategies, supervision patterns, restart handling, etc. As a special bonus, additions to this section are marked with the contributor’s name, and it would be nice if every Akka user who finds a recurring pattern in his or her code could share it for the profit of all. Where applicable it might also make -sense to add to the ``akka.pattern`` package for creating an `OTP-like library -`_. +sense to add to the `akka.pattern` package for creating an [OTP-like library](http://www.erlang.org/doc/man_index.html). -Throttling Messages -=================== +## Throttling Messages Contributed by: Kaspar Fischer "A message throttler that ensures that messages are not sent out at too high a rate." -The pattern is described in `Throttling Messages in Akka 2 `_. +The pattern is described in [Throttling Messages in Akka 2](http://letitcrash.com/post/28901663062/throttling-messages-in-akka-2). -Balancing Workload Across Nodes -=============================== +## Balancing Workload Across Nodes Contributed by: Derek Wyatt @@ -32,10 +24,9 @@ Contributed by: Derek Wyatt stipulation that the Actors doing the work have distinct Mailboxes on remote nodes. In this post we’ll explore the implementation of such a concept." -The pattern is described `Balancing Workload across Nodes with Akka 2 `_. +The pattern is described [Balancing Workload across Nodes with Akka 2](http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2). -Work Pulling Pattern to throttle and distribute work, and prevent mailbox overflow -================================================================================== +## Work Pulling Pattern to throttle and distribute work, and prevent mailbox overflow Contributed by: Michael Pollmeier @@ -45,10 +36,9 @@ eventually become too full. It also let’s you distribute work around your clus scale dynamically scale and is completely non-blocking. This pattern is a specialisation of the above 'Balancing Workload Pattern'." -The pattern is described `Work Pulling Pattern to prevent mailbox overflow, throttle and distribute work `_. +The pattern is described [Work Pulling Pattern to prevent mailbox overflow, throttle and distribute work](http://www.michaelpollmeier.com/akka-work-pulling-pattern). -Ordered Termination -=================== +## Ordered Termination Contributed by: Derek Wyatt @@ -59,10 +49,9 @@ If an Actor has children that have order dependencies, then you might need to en a particular shutdown order of those children so that their postStop() methods get called in the right order." -The pattern is described `An Akka 2 Terminator `_. +The pattern is described [An Akka 2 Terminator](http://letitcrash.com/post/29773618510/an-akka-2-terminator). -Akka AMQP Proxies -================= +## Akka AMQP Proxies Contributed by: Fabrice Drouin @@ -70,10 +59,9 @@ Contributed by: Fabrice Drouin You still write “local” code, have very little to configure, and end up with a distributed, elastic, fault-tolerant grid where computing nodes can be written in nearly every programming language." -The pattern is described `Akka AMQP Proxies `_. +The pattern is described [Akka AMQP Proxies](http://letitcrash.com/post/29988753572/akka-amqp-proxies). -Shutdown Patterns in Akka 2 -=========================== +## Shutdown Patterns in Akka 2 Contributed by: Derek Wyatt @@ -85,10 +73,9 @@ She’s just plain mean. In this post, we’ll discuss why this is the case and provide you with a simple option for shutting down “at the right time”, as well as a not-so-simple-option for doing the exact same thing." -The pattern is described `Shutdown Patterns in Akka 2 `_. +The pattern is described [Shutdown Patterns in Akka 2](http://letitcrash.com/post/30165507578/shutdown-patterns-in-akka-2). -Distributed (in-memory) graph processing with Akka -================================================== +## Distributed (in-memory) graph processing with Akka Contributed by: Adelbert Chang @@ -96,10 +83,9 @@ Contributed by: Adelbert Chang and have become even more interesting in the context of online social networks such as Facebook and Twitter, whose underlying network structures are nicely represented by graphs." -The pattern is described `Distributed In-Memory Graph Processing with Akka `_. +The pattern is described [Distributed In-Memory Graph Processing with Akka](http://letitcrash.com/post/30257014291/distributed-in-memory-graph-processing-with-akka). -Case Study: An Auto-Updating Cache Using Actors -=============================================== +## Case Study: An Auto-Updating Cache Using Actors Contributed by: Eric Pederson @@ -109,10 +95,9 @@ The data in the backend system is constantly being updated so the caches need to Requests to the backend system need to be throttled. The caching system we built used Akka actors and Scala’s support for functions as first class objects." -The pattern is described `Case Study: An Auto-Updating Cache using Actors `_. +The pattern is described [Case Study: An Auto-Updating Cache using Actors](http://letitcrash.com/post/30509298968/case-study-an-auto-updating-cache-using-actors). -Discovering message flows in actor systems with the Spider Pattern -================================================================== +## Discovering message flows in actor systems with the Spider Pattern Contributed by: Raymond Roestenburg @@ -122,36 +107,38 @@ on several machines to find out what’s going on. I’m sure you have browsed t This is where the Spider pattern comes in." -The pattern is described `Discovering Message Flows in Actor System with the Spider Pattern `_. +The pattern is described [Discovering Message Flows in Actor System with the Spider Pattern](http://letitcrash.com/post/30585282971/discovering-message-flows-in-actor-systems-with-the). -Scheduling Periodic Messages -============================ +## Scheduling Periodic Messages This pattern describes how to schedule periodic messages to yourself in two different ways. The first way is to set up periodic message scheduling in the constructor of the actor, -and cancel that scheduled sending in ``postStop`` or else we might have multiple registered +and cancel that scheduled sending in `postStop` or else we might have multiple registered message sends to the same actor. -.. note:: +@@@ note - With this approach the scheduled periodic message send will be restarted with the actor on restarts. - This also means that the time period that elapses between two tick messages during a restart may drift - off based on when you restart the scheduled message sends relative to the time that the last message was - sent, and how long the initial delay is. Worst case scenario is ``interval`` plus ``initialDelay``. +With this approach the scheduled periodic message send will be restarted with the actor on restarts. +This also means that the time period that elapses between two tick messages during a restart may drift +off based on when you restart the scheduled message sends relative to the time that the last message was +sent, and how long the initial delay is. Worst case scenario is `interval` plus `initialDelay`. -.. includecode:: code/docs/pattern/SchedulerPatternSpec.scala#schedule-constructor +@@@ -The second variant sets up an initial one shot message send in the ``preStart`` method +@@snip [SchedulerPatternSpec.scala](code/docs/pattern/SchedulerPatternSpec.scala) { #schedule-constructor } + +The second variant sets up an initial one shot message send in the `preStart` method of the actor, and the then the actor when it receives this message sets up a new one shot -message send. You also have to override ``postRestart`` so we don't call ``preStart`` +message send. You also have to override `postRestart` so we don't call `preStart` and schedule the initial message send again. -.. note:: +@@@ note - With this approach we won't fill up the mailbox with tick messages if the actor is - under pressure, but only schedule a new tick message when we have seen the previous one. +With this approach we won't fill up the mailbox with tick messages if the actor is +under pressure, but only schedule a new tick message when we have seen the previous one. -.. includecode:: code/docs/pattern/SchedulerPatternSpec.scala#schedule-receive +@@@ +@@snip [SchedulerPatternSpec.scala](code/docs/pattern/SchedulerPatternSpec.scala) { #schedule-receive } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/http/index.md b/akka-docs/src/main/paradox/scala/http/index.md index ef38e38987..94f6f73117 100644 --- a/akka-docs/src/main/paradox/scala/http/index.md +++ b/akka-docs/src/main/paradox/scala/http/index.md @@ -1,5 +1,4 @@ -Akka HTTP Documentation (Scala) moved! -====================================== +# Akka HTTP Documentation (Scala) moved! Akka HTTP has been released as independent stable module (from Akka HTTP 10.x onwards). -The documentation is available under `doc.akka.io/akka-http/current/ `_. +The documentation is available under [doc.akka.io/akka-http/current/](http://doc.akka.io/docs/akka-http/current/scala.html). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/index-actors.md b/akka-docs/src/main/paradox/scala/index-actors.md index cc92986f34..e8a6b86369 100644 --- a/akka-docs/src/main/paradox/scala/index-actors.md +++ b/akka-docs/src/main/paradox/scala/index-actors.md @@ -1,20 +1,22 @@ -Actors -====== +# Actors -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - actors - typed - fault-tolerance - dispatchers - mailboxes - routing - fsm - persistence - persistence-schema-evolution - persistence-query - persistence-query-leveldb - testing - actordsl - typed-actors +@@@ index + +* [actors](actors.md) +* [typed](typed.md) +* [fault-tolerance](fault-tolerance.md) +* [dispatchers](dispatchers.md) +* [mailboxes](mailboxes.md) +* [routing](routing.md) +* [fsm](fsm.md) +* [persistence](persistence.md) +* [persistence-schema-evolution](persistence-schema-evolution.md) +* [persistence-query](persistence-query.md) +* [persistence-query-leveldb](persistence-query-leveldb.md) +* [testing](testing.md) +* [actordsl](actordsl.md) +* [typed-actors](typed-actors.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/index-futures.md b/akka-docs/src/main/paradox/scala/index-futures.md index 59b846d0d6..c0158c312f 100644 --- a/akka-docs/src/main/paradox/scala/index-futures.md +++ b/akka-docs/src/main/paradox/scala/index-futures.md @@ -1,8 +1,10 @@ -Futures and Agents -================== +# Futures and Agents -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - futures - agents +@@@ index + +* [futures](futures.md) +* [agents](agents.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/index-network.md b/akka-docs/src/main/paradox/scala/index-network.md index 33ed035cc8..0c7ef30707 100644 --- a/akka-docs/src/main/paradox/scala/index-network.md +++ b/akka-docs/src/main/paradox/scala/index-network.md @@ -1,21 +1,23 @@ -Networking -========== +# Networking -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - ../common/cluster - cluster-usage - cluster-singleton - distributed-pub-sub - cluster-client - cluster-sharding - cluster-metrics - distributed-data - remoting - remoting-artery - serialization - io - io-tcp - io-udp - camel +@@@ index + +* [../common/cluster](../common/cluster.md) +* [cluster-usage](cluster-usage.md) +* [cluster-singleton](cluster-singleton.md) +* [distributed-pub-sub](distributed-pub-sub.md) +* [cluster-client](cluster-client.md) +* [cluster-sharding](cluster-sharding.md) +* [cluster-metrics](cluster-metrics.md) +* [distributed-data](distributed-data.md) +* [remoting](remoting.md) +* [remoting-artery](remoting-artery.md) +* [serialization](serialization.md) +* [io](io.md) +* [io-tcp](io-tcp.md) +* [io-udp](io-udp.md) +* [camel](camel.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/index-utilities.md b/akka-docs/src/main/paradox/scala/index-utilities.md index 02e4ce5114..45fd046727 100644 --- a/akka-docs/src/main/paradox/scala/index-utilities.md +++ b/akka-docs/src/main/paradox/scala/index-utilities.md @@ -1,14 +1,15 @@ -Utilities -========= +# Utilities -.. toctree:: - :maxdepth: 2 +@@toc { depth=2 } - event-bus - logging - scheduler - ../common/duration - ../common/circuitbreaker - extending-akka - ../intro/deployment-scenarios +@@@ index +* [event-bus](event-bus.md) +* [logging](logging.md) +* [scheduler](scheduler.md) +* [../common/duration](../common/duration.md) +* [../common/circuitbreaker](../common/circuitbreaker.md) +* [extending-akka](extending-akka.md) +* [../intro/deployment-scenarios](../intro/deployment-scenarios.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/io-tcp.md b/akka-docs/src/main/paradox/scala/io-tcp.md index c8772f4bce..21a1156e4c 100644 --- a/akka-docs/src/main/paradox/scala/io-tcp.md +++ b/akka-docs/src/main/paradox/scala/io-tcp.md @@ -1,162 +1,154 @@ -.. _io-scala-tcp: - -Using TCP -========= +# Using TCP The code snippets through-out this section assume the following imports: -.. includecode:: code/docs/io/IODocSpec.scala#imports +@@snip [IODocSpec.scala](code/docs/io/IODocSpec.scala) { #imports } All of the Akka I/O APIs are accessed through manager objects. When using an I/O API, the first step is to acquire a -reference to the appropriate manager. The code below shows how to acquire a reference to the ``Tcp`` manager. +reference to the appropriate manager. The code below shows how to acquire a reference to the `Tcp` manager. -.. includecode:: code/docs/io/IODocSpec.scala#manager +@@snip [IODocSpec.scala](code/docs/io/IODocSpec.scala) { #manager } The manager is an actor that handles the underlying low level I/O resources (selectors, channels) and instantiates workers for specific tasks, such as listening to incoming connections. -Connecting ----------- +## Connecting -.. includecode:: code/docs/io/IODocSpec.scala#client +@@snip [IODocSpec.scala](code/docs/io/IODocSpec.scala) { #client } -The first step of connecting to a remote address is sending a :class:`Connect` +The first step of connecting to a remote address is sending a `Connect` message to the TCP manager; in addition to the simplest form shown above there -is also the possibility to specify a local :class:`InetSocketAddress` to bind +is also the possibility to specify a local `InetSocketAddress` to bind to and a list of socket options to apply. -.. note:: +@@@ note - The SO_NODELAY (TCP_NODELAY on Windows) socket option defaults to true in - Akka, independently of the OS default settings. This setting disables Nagle's - algorithm, considerably improving latency for most applications. This setting - could be overridden by passing ``SO.TcpNoDelay(false)`` in the list of socket - options of the ``Connect`` message. +The SO_NODELAY (TCP_NODELAY on Windows) socket option defaults to true in +Akka, independently of the OS default settings. This setting disables Nagle's +algorithm, considerably improving latency for most applications. This setting +could be overridden by passing `SO.TcpNoDelay(false)` in the list of socket +options of the `Connect` message. -The TCP manager will then reply either with a :class:`CommandFailed` or it will +@@@ + +The TCP manager will then reply either with a `CommandFailed` or it will spawn an internal actor representing the new connection. This new actor will -then send a :class:`Connected` message to the original sender of the -:class:`Connect` message. +then send a `Connected` message to the original sender of the +`Connect` message. -In order to activate the new connection a :class:`Register` message must be +In order to activate the new connection a `Register` message must be sent to the connection actor, informing that one about who shall receive data from the socket. Before this step is done the connection cannot be used, and there is an internal timeout after which the connection actor will shut itself -down if no :class:`Register` message is received. +down if no `Register` message is received. The connection actor watches the registered handler and closes the connection when that one terminates, thereby cleaning up all internal resources associated with that connection. -The actor in the example above uses :meth:`become` to switch from unconnected +The actor in the example above uses `become` to switch from unconnected to connected operation, demonstrating the commands and events which are -observed in that state. For a discussion on :class:`CommandFailed` see -`Throttling Reads and Writes`_ below. :class:`ConnectionClosed` is a trait, +observed in that state. For a discussion on `CommandFailed` see +[Throttling Reads and Writes](#throttling-reads-and-writes) below. `ConnectionClosed` is a trait, which marks the different connection close events. The last line handles all connection close events in the same way. It is possible to listen for more -fine-grained connection close events, see `Closing Connections`_ below. +fine-grained connection close events, see [Closing Connections](#closing-connections) below. -Accepting connections ---------------------- +## Accepting connections -.. includecode:: code/docs/io/IODocSpec.scala#server - :exclude: do-some-logging-or-setup +@@snip [IODocSpec.scala](code/docs/io/IODocSpec.scala) { #server } -To create a TCP server and listen for inbound connections, a :class:`Bind` +To create a TCP server and listen for inbound connections, a `Bind` command has to be sent to the TCP manager. This will instruct the TCP manager -to listen for TCP connections on a particular :class:`InetSocketAddress`; the -port may be specified as ``0`` in order to bind to a random port. +to listen for TCP connections on a particular `InetSocketAddress`; the +port may be specified as `0` in order to bind to a random port. -The actor sending the :class:`Bind` message will receive a :class:`Bound` +The actor sending the `Bind` message will receive a `Bound` message signaling that the server is ready to accept incoming connections; -this message also contains the :class:`InetSocketAddress` to which the socket +this message also contains the `InetSocketAddress` to which the socket was actually bound (i.e. resolved IP address and correct port number). From this point forward the process of handling connections is the same as for outgoing connections. The example demonstrates that handling the reads from a certain connection can be delegated to another actor by naming it as the -handler when sending the :class:`Register` message. Writes can be sent from any +handler when sending the `Register` message. Writes can be sent from any actor in the system to the connection actor (i.e. the actor which sent the -:class:`Connected` message). The simplistic handler is defined as: +`Connected` message). The simplistic handler is defined as: -.. includecode:: code/docs/io/IODocSpec.scala#simplistic-handler +@@snip [IODocSpec.scala](code/docs/io/IODocSpec.scala) { #simplistic-handler } For a more complete sample which also takes into account the possibility of -failures when sending please see `Throttling Reads and Writes`_ below. +failures when sending please see [Throttling Reads and Writes](#throttling-reads-and-writes) below. The only difference to outgoing connections is that the internal actor managing -the listen port—the sender of the :class:`Bound` message—watches the actor -which was named as the recipient for :class:`Connected` messages in the -:class:`Bind` message. When that actor terminates the listen port will be +the listen port—the sender of the `Bound` message—watches the actor +which was named as the recipient for `Connected` messages in the +`Bind` message. When that actor terminates the listen port will be closed and all resources associated with it will be released; existing connections will not be terminated at this point. -Closing connections -------------------- +## Closing connections -A connection can be closed by sending one of the commands ``Close``, ``ConfirmedClose`` or ``Abort`` to the connection +A connection can be closed by sending one of the commands `Close`, `ConfirmedClose` or `Abort` to the connection actor. -``Close`` will close the connection by sending a ``FIN`` message, but without waiting for confirmation from +`Close` will close the connection by sending a `FIN` message, but without waiting for confirmation from the remote endpoint. Pending writes will be flushed. If the close is successful, the listener will be notified with -``Closed``. +`Closed`. -``ConfirmedClose`` will close the sending direction of the connection by sending a ``FIN`` message, but data +`ConfirmedClose` will close the sending direction of the connection by sending a `FIN` message, but data will continue to be received until the remote endpoint closes the connection, too. Pending writes will be flushed. If the close is -successful, the listener will be notified with ``ConfirmedClosed``. +successful, the listener will be notified with `ConfirmedClosed`. -``Abort`` will immediately terminate the connection by sending a ``RST`` message to the remote endpoint. Pending -writes will be not flushed. If the close is successful, the listener will be notified with ``Aborted``. +`Abort` will immediately terminate the connection by sending a `RST` message to the remote endpoint. Pending +writes will be not flushed. If the close is successful, the listener will be notified with `Aborted`. -``PeerClosed`` will be sent to the listener if the connection has been closed by the remote endpoint. Per default, the +`PeerClosed` will be sent to the listener if the connection has been closed by the remote endpoint. Per default, the connection will then automatically be closed from this endpoint as well. To support half-closed connections set the -``keepOpenOnPeerClosed`` member of the ``Register`` message to ``true`` in which case the connection stays open until +`keepOpenOnPeerClosed` member of the `Register` message to `true` in which case the connection stays open until it receives one of the above close commands. -``ErrorClosed`` will be sent to the listener whenever an error happened that forced the connection to be closed. +`ErrorClosed` will be sent to the listener whenever an error happened that forced the connection to be closed. -All close notifications are sub-types of ``ConnectionClosed`` so listeners who do not need fine-grained close events +All close notifications are sub-types of `ConnectionClosed` so listeners who do not need fine-grained close events may handle all close events in the same way. -Writing to a connection ------------------------ +## Writing to a connection -Once a connection has been established data can be sent to it from any actor in the form of a ``Tcp.WriteCommand``. -``Tcp.WriteCommand`` is an abstract class with three concrete implementations: +Once a connection has been established data can be sent to it from any actor in the form of a `Tcp.WriteCommand`. +`Tcp.WriteCommand` is an abstract class with three concrete implementations: Tcp.Write - The simplest ``WriteCommand`` implementation which wraps a ``ByteString`` instance and an "ack" event. - A ``ByteString`` (as explained in :ref:`this section `) models one or more chunks of immutable - in-memory data with a maximum (total) size of 2 GB (2^31 bytes). +: The simplest `WriteCommand` implementation which wraps a `ByteString` instance and an "ack" event. +A `ByteString` (as explained in @ref:[this section](io.md#bytestring-scala)) models one or more chunks of immutable +in-memory data with a maximum (total) size of 2 GB (2^31 bytes). Tcp.WriteFile - If you want to send "raw" data from a file you can do so efficiently with the ``Tcp.WriteFile`` command. - This allows you do designate a (contiguous) chunk of on-disk bytes for sending across the connection without - the need to first load them into the JVM memory. As such ``Tcp.WriteFile`` can "hold" more than 2GB of data and - an "ack" event if required. +: If you want to send "raw" data from a file you can do so efficiently with the `Tcp.WriteFile` command. +This allows you do designate a (contiguous) chunk of on-disk bytes for sending across the connection without +the need to first load them into the JVM memory. As such `Tcp.WriteFile` can "hold" more than 2GB of data and +an "ack" event if required. Tcp.CompoundWrite - Sometimes you might want to group (or interleave) several ``Tcp.Write`` and/or ``Tcp.WriteFile`` commands into - one atomic write command which gets written to the connection in one go. The ``Tcp.CompoundWrite`` allows you - to do just that and offers three benefits: +: +Sometimes you might want to group (or interleave) several `Tcp.Write` and/or `Tcp.WriteFile` commands into +one atomic write command which gets written to the connection in one go. The `Tcp.CompoundWrite` allows you +to do just that and offers three benefits: + 1. As explained in the following section the TCP connection actor can only handle one single write command at a time. +By combining several writes into one `CompoundWrite` you can have them be sent across the connection with +minimum overhead and without the need to spoon feed them to the connection actor via an *ACK-based* message +protocol. + 2. Because a `WriteCommand` is atomic you can be sure that no other actor can "inject" other writes into your +series of writes if you combine them into one single `CompoundWrite`. In scenarios where several actors write +to the same connection this can be an important feature which can be somewhat hard to achieve otherwise. + 3. The "sub writes" of a `CompoundWrite` are regular `Write` or `WriteFile` commands that themselves can request +"ack" events. These ACKs are sent out as soon as the respective "sub write" has been completed. This allows you to +attach more than one ACK to a `Write` or `WriteFile` (by combining it with an empty write that itself requests +an ACK) or to have the connection actor acknowledge the progress of transmitting the `CompoundWrite` by sending +out intermediate ACKs at arbitrary points. - 1. As explained in the following section the TCP connection actor can only handle one single write command at a time. - By combining several writes into one ``CompoundWrite`` you can have them be sent across the connection with - minimum overhead and without the need to spoon feed them to the connection actor via an *ACK-based* message - protocol. - 2. Because a ``WriteCommand`` is atomic you can be sure that no other actor can "inject" other writes into your - series of writes if you combine them into one single ``CompoundWrite``. In scenarios where several actors write - to the same connection this can be an important feature which can be somewhat hard to achieve otherwise. - - 3. The "sub writes" of a ``CompoundWrite`` are regular ``Write`` or ``WriteFile`` commands that themselves can request - "ack" events. These ACKs are sent out as soon as the respective "sub write" has been completed. This allows you to - attach more than one ACK to a ``Write`` or ``WriteFile`` (by combining it with an empty write that itself requests - an ACK) or to have the connection actor acknowledge the progress of transmitting the ``CompoundWrite`` by sending - out intermediate ACKs at arbitrary points. - -Throttling Reads and Writes ---------------------------- +## Throttling Reads and Writes The basic model of the TCP connection actor is that it has no internal buffering (i.e. it can only process one write at a time, meaning it can buffer @@ -165,84 +157,81 @@ needs to be handled at the user level, for both writes and reads. For back-pressuring writes there are three modes of operation -* *ACK-based:* every :class:`Write` command carries an arbitrary object, and if - this object is not ``Tcp.NoAck`` then it will be returned to the sender of - the :class:`Write` upon successfully writing all contained data to the - socket. If no other write is initiated before having received this - acknowledgement then no failures can happen due to buffer overrun. - -* *NACK-based:* every write which arrives while a previous write is not yet - completed will be replied to with a :class:`CommandFailed` message containing - the failed write. Just relying on this mechanism requires the implemented - protocol to tolerate skipping writes (e.g. if each write is a valid message - on its own and it is not required that all are delivered). This mode is - enabled by setting the ``useResumeWriting`` flag to ``false`` within the - :class:`Register` message during connection activation. - -* *NACK-based with write suspending:* this mode is very similar to the - NACK-based one, but once a single write has failed no further writes will - succeed until a :class:`ResumeWriting` message is received. This message will - be answered with a :class:`WritingResumed` message once the last accepted - write has completed. If the actor driving the connection implements buffering - and resends the NACK’ed messages after having awaited the - :class:`WritingResumed` signal then every message is delivered exactly once - to the network socket. + * *ACK-based:* every `Write` command carries an arbitrary object, and if +this object is not `Tcp.NoAck` then it will be returned to the sender of +the `Write` upon successfully writing all contained data to the +socket. If no other write is initiated before having received this +acknowledgement then no failures can happen due to buffer overrun. + * *NACK-based:* every write which arrives while a previous write is not yet +completed will be replied to with a `CommandFailed` message containing +the failed write. Just relying on this mechanism requires the implemented +protocol to tolerate skipping writes (e.g. if each write is a valid message +on its own and it is not required that all are delivered). This mode is +enabled by setting the `useResumeWriting` flag to `false` within the +`Register` message during connection activation. + * *NACK-based with write suspending:* this mode is very similar to the +NACK-based one, but once a single write has failed no further writes will +succeed until a `ResumeWriting` message is received. This message will +be answered with a `WritingResumed` message once the last accepted +write has completed. If the actor driving the connection implements buffering +and resends the NACK’ed messages after having awaited the +`WritingResumed` signal then every message is delivered exactly once +to the network socket. These write back-pressure models (with the exception of the second which is rather specialised) are demonstrated in complete examples below. The full and contiguous source is -available `on GitHub <@github@/akka-docs/rst/scala/code/docs/io/EchoServer.scala>`_. +available [on GitHub](@github@/akka-docs/rst/scala/code/docs/io/EchoServer.scala). For back-pressuring reads there are two modes of operation -* *Push-reading:* in this mode the connection actor sends the registered reader actor - incoming data as soon as available as :class:`Received` events. Whenever the reader actor - wants to signal back-pressure to the remote TCP endpoint it can send a :class:`SuspendReading` - message to the connection actor to indicate that it wants to suspend the - reception of new data. No :class:`Received` events will arrive until a corresponding - :class:`ResumeReading` is sent indicating that the receiver actor is ready again. + * *Push-reading:* in this mode the connection actor sends the registered reader actor +incoming data as soon as available as `Received` events. Whenever the reader actor +wants to signal back-pressure to the remote TCP endpoint it can send a `SuspendReading` +message to the connection actor to indicate that it wants to suspend the +reception of new data. No `Received` events will arrive until a corresponding +`ResumeReading` is sent indicating that the receiver actor is ready again. + * *Pull-reading:* after sending a `Received` event the connection +actor automatically suspends accepting data from the socket until the reader actor signals +with a `ResumeReading` message that it is ready to process more input data. Hence +new data is "pulled" from the connection by sending `ResumeReading` messages. -* *Pull-reading:* after sending a :class:`Received` event the connection - actor automatically suspends accepting data from the socket until the reader actor signals - with a :class:`ResumeReading` message that it is ready to process more input data. Hence - new data is "pulled" from the connection by sending :class:`ResumeReading` messages. +@@@ note -.. note:: +It should be obvious that all these flow control schemes only work between +one writer/reader and one connection actor; as soon as multiple actors send write +commands to a single connection no consistent result can be achieved. - It should be obvious that all these flow control schemes only work between - one writer/reader and one connection actor; as soon as multiple actors send write - commands to a single connection no consistent result can be achieved. +@@@ -ACK-Based Write Back-Pressure ------------------------------ +## ACK-Based Write Back-Pressure For proper function of the following example it is important to configure the connection to remain half-open when the remote side closed its writing end: -this allows the example :class:`EchoHandler` to write all outstanding data back +this allows the example `EchoHandler` to write all outstanding data back to the client before fully closing the connection. This is enabled using a flag -upon connection activation (observe the :class:`Register` message): +upon connection activation (observe the `Register` message): -.. includecode:: code/docs/io/EchoServer.scala#echo-manager +@@snip [EchoServer.scala](code/docs/io/EchoServer.scala) { #echo-manager } With this preparation let us dive into the handler itself: -.. includecode:: code/docs/io/EchoServer.scala#simple-echo-handler - :exclude: storage-omitted +@@snip [EchoServer.scala](code/docs/io/EchoServer.scala) { #simple-echo-handler } The principle is simple: when having written a chunk always wait for the -``Ack`` to come back before sending the next chunk. While waiting we switch +`Ack` to come back before sending the next chunk. While waiting we switch behavior such that new incoming data are buffered. The helper functions used are a bit lengthy but not complicated: -.. includecode:: code/docs/io/EchoServer.scala#simple-helpers +@@snip [EchoServer.scala](code/docs/io/EchoServer.scala) { #simple-helpers } -The most interesting part is probably the last: an ``Ack`` removes the oldest +The most interesting part is probably the last: an `Ack` removes the oldest data chunk from the buffer, and if that was the last chunk then we either close the connection (if the peer closed its half already) or return to the idle behavior; otherwise we just send the next buffered chunk and stay waiting for -the next ``Ack``. +the next `Ack`. Back-pressure can be propagated also across the reading side back to the writer -on the other end of the connection by sending the :class:`SuspendReading` +on the other end of the connection by sending the `SuspendReading` command to the connection actor. This will lead to no data being read from the socket anymore (although this does happen after a delay because it takes some time until the connection actor processes this command, hence appropriate @@ -252,43 +241,40 @@ the remote side from writing, filling up its write buffer, until finally the writer on the other side cannot push any data into the socket anymore. This is how end-to-end back-pressure is realized across a TCP connection. -NACK-Based Write Back-Pressure with Suspending ----------------------------------------------- +## NACK-Based Write Back-Pressure with Suspending -.. includecode:: code/docs/io/EchoServer.scala#echo-handler - :exclude: buffering,closing,storage-omitted +@@snip [EchoServer.scala](code/docs/io/EchoServer.scala) { #echo-handler } -The principle here is to keep writing until a :class:`CommandFailed` is +The principle here is to keep writing until a `CommandFailed` is received, using acknowledgements only to prune the resend buffer. When a such a failure was received, transition into a different state for handling and handle resending of all queued data: -.. includecode:: code/docs/io/EchoServer.scala#buffering +@@snip [EchoServer.scala](code/docs/io/EchoServer.scala) { #buffering } It should be noted that all writes which are currently buffered have also been sent to the connection actor upon entering this state, which means that the -:class:`ResumeWriting` message is enqueued after those writes, leading to the -reception of all outstanding :class:`CommandFailed` messages (which are ignored -in this state) before receiving the :class:`WritingResumed` signal. That latter +`ResumeWriting` message is enqueued after those writes, leading to the +reception of all outstanding `CommandFailed` messages (which are ignored +in this state) before receiving the `WritingResumed` signal. That latter message is sent by the connection actor only once the internally queued write has been fully completed, meaning that a subsequent write will not fail. This -is exploited by the :class:`EchoHandler` to switch to an ACK-based approach for +is exploited by the `EchoHandler` to switch to an ACK-based approach for the first ten writes after a failure before resuming the optimistic write-through behavior. -.. includecode:: code/docs/io/EchoServer.scala#closing +@@snip [EchoServer.scala](code/docs/io/EchoServer.scala) { #closing } Closing the connection while still sending all data is a bit more involved than in the ACK-based approach: the idea is to always send all outstanding messages and acknowledge all successful writes, and if a failure happens then switch -behavior to await the :class:`WritingResumed` event and start over. +behavior to await the `WritingResumed` event and start over. The helper functions are very similar to the ACK-based case: -.. includecode:: code/docs/io/EchoServer.scala#helpers +@@snip [EchoServer.scala](code/docs/io/EchoServer.scala) { #helpers } -Read Back-Pressure with Pull Mode ---------------------------------- +## Read Back-Pressure with Pull Mode When using push based reading, data coming from the socket is sent to the actor as soon as it is available. In the case of the previous Echo server example @@ -298,48 +284,46 @@ since the rate of writing might be slower than the rate of the arrival of new da With the Pull mode this buffer can be completely eliminated as the following snippet demonstrates: -.. includecode:: code/docs/io/ReadBackPressure.scala#pull-reading-echo +@@snip [ReadBackPressure.scala](code/docs/io/ReadBackPressure.scala) { #pull-reading-echo } The idea here is that reading is not resumed until the previous write has been completely acknowledged by the connection actor. Every pull mode connection actor starts from suspended state. To start the flow of data we send a -``ResumeReading`` in the ``preStart`` method to tell the connection actor that +`ResumeReading` in the `preStart` method to tell the connection actor that we are ready to receive the first chunk of data. Since we only resume reading when the previous data chunk has been completely written there is no need for maintaining a buffer. -To enable pull reading on an outbound connection the ``pullMode`` parameter of -the :class:`Connect` should be set to ``true``: +To enable pull reading on an outbound connection the `pullMode` parameter of +the `Connect` should be set to `true`: -.. includecode:: code/docs/io/ReadBackPressure.scala#pull-mode-connect +@@snip [ReadBackPressure.scala](code/docs/io/ReadBackPressure.scala) { #pull-mode-connect } -Pull Mode Reading for Inbound Connections -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Pull Mode Reading for Inbound Connections The previous section demonstrated how to enable pull reading mode for outbound connections but it is possible to create a listener actor with this mode of reading -by setting the ``pullMode`` parameter of the :class:`Bind` command to ``true``: +by setting the `pullMode` parameter of the `Bind` command to `true`: -.. includecode:: code/docs/io/ReadBackPressure.scala#pull-mode-bind +@@snip [ReadBackPressure.scala](code/docs/io/ReadBackPressure.scala) { #pull-mode-bind } One of the effects of this setting is that all connections accepted by this listener actor will use pull mode reading. Another effect of this setting is that in addition of setting all inbound connections to pull mode, accepting connections becomes pull based, too. This means that after handling -one (or more) :class:`Connected` events the listener actor has to be resumed by sending -it a :class:`ResumeAccepting` message. +one (or more) `Connected` events the listener actor has to be resumed by sending +it a `ResumeAccepting` message. Listener actors with pull mode start suspended so to start accepting connections -a :class:`ResumeAccepting` command has to be sent to the listener actor after binding was successful: +a `ResumeAccepting` command has to be sent to the listener actor after binding was successful: -.. includecode:: code/docs/io/ReadBackPressure.scala#pull-accepting +@@snip [ReadBackPressure.scala](code/docs/io/ReadBackPressure.scala) { #pull-accepting } After handling an incoming connection we need to resume accepting again: -.. includecode:: code/docs/io/ReadBackPressure.scala#pull-accepting-cont - -The :class:`ResumeAccepting` accepts a ``batchSize`` parameter that specifies how -many new connections are accepted before a next :class:`ResumeAccepting` message -is needed to resume handling of new connections. +@@snip [ReadBackPressure.scala](code/docs/io/ReadBackPressure.scala) { #pull-accepting-cont } +The `ResumeAccepting` accepts a `batchSize` parameter that specifies how +many new connections are accepted before a next `ResumeAccepting` message +is needed to resume handling of new connections. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/io-udp.md b/akka-docs/src/main/paradox/scala/io-udp.md index bdc1fd7792..134e3dc0bc 100644 --- a/akka-docs/src/main/paradox/scala/io-udp.md +++ b/akka-docs/src/main/paradox/scala/io-udp.md @@ -1,105 +1,101 @@ -.. _io-scala-udp: - -Using UDP -========= +# Using UDP UDP is a connectionless datagram protocol which offers two different ways of communication on the JDK level: +> * sockets which are free to send datagrams to any destination and receive - datagrams from any origin - +datagrams from any origin * sockets which are restricted to communication with one specific remote - socket address +socket address In the low-level API the distinction is made—confusingly—by whether or not -:meth:`connect` has been called on the socket (even when connect has been +`connect` has been called on the socket (even when connect has been called the protocol is still connectionless). These two forms of UDP usage are offered using distinct IO extensions described below. -Unconnected UDP ---------------- +## Unconnected UDP -Simple Send -^^^^^^^^^^^^ +### Simple Send -.. includecode:: code/docs/io/UdpDocSpec.scala#sender +@@snip [UdpDocSpec.scala](code/docs/io/UdpDocSpec.scala) { #sender } The simplest form of UDP usage is to just send datagrams without the need of getting a reply. To this end a “simple sender” facility is provided as demonstrated above. The UDP extension is queried using the -:class:`SimpleSender` message, which is answered by a :class:`SimpleSenderReady` +`SimpleSender` message, which is answered by a `SimpleSenderReady` notification. The sender of this message is the newly created sender actor which from this point onward can be used to send datagrams to arbitrary destinations; in this example it will just send any UTF-8 encoded -:class:`String` it receives to a predefined remote address. +`String` it receives to a predefined remote address. -.. note:: +@@@ note - The simple sender will not shut itself down because it cannot know when you - are done with it. You will need to send it a :class:`PoisonPill` when you - want to close the ephemeral port the sender is bound to. +The simple sender will not shut itself down because it cannot know when you +are done with it. You will need to send it a `PoisonPill` when you +want to close the ephemeral port the sender is bound to. -Bind (and Send) -^^^^^^^^^^^^^^^ +@@@ -.. includecode:: code/docs/io/UdpDocSpec.scala#listener +### Bind (and Send) + +@@snip [UdpDocSpec.scala](code/docs/io/UdpDocSpec.scala) { #listener } If you want to implement a UDP server which listens on a socket for incoming -datagrams then you need to use the :class:`Bind` command as shown above. The +datagrams then you need to use the `Bind` command as shown above. The local address specified may have a zero port in which case the operating system will automatically choose a free port and assign it to the new socket. Which -port was actually bound can be found out by inspecting the :class:`Bound` +port was actually bound can be found out by inspecting the `Bound` message. -The sender of the :class:`Bound` message is the actor which manages the new -socket. Sending datagrams is achieved by using the :class:`Send` message type -and the socket can be closed by sending a :class:`Unbind` command, in which -case the socket actor will reply with a :class:`Unbound` notification. +The sender of the `Bound` message is the actor which manages the new +socket. Sending datagrams is achieved by using the `Send` message type +and the socket can be closed by sending a `Unbind` command, in which +case the socket actor will reply with a `Unbound` notification. -Received datagrams are sent to the actor designated in the :class:`Bind` -message, whereas the :class:`Bound` message will be sent to the sender of the -:class:`Bind`. +Received datagrams are sent to the actor designated in the `Bind` +message, whereas the `Bound` message will be sent to the sender of the +`Bind`. -Connected UDP -------------- +## Connected UDP The service provided by the connection based UDP API is similar to the bind-and-send service we saw earlier, but the main difference is that a -connection is only able to send to the ``remoteAddress`` it was connected to, +connection is only able to send to the `remoteAddress` it was connected to, and will receive datagrams only from that address. -.. includecode:: code/docs/io/UdpDocSpec.scala#connected +@@snip [UdpDocSpec.scala](code/docs/io/UdpDocSpec.scala) { #connected } Consequently the example shown here looks quite similar to the previous one, the biggest difference is the absence of remote address information in -:class:`Send` and :class:`Received` messages. +`Send` and `Received` messages. -.. note:: - - There is a small performance benefit in using connection based UDP API over - the connectionless one. If there is a SecurityManager enabled on the system, - every connectionless message send has to go through a security check, while - in the case of connection-based UDP the security check is cached after - connect, thus writes do not suffer an additional performance penalty. +@@@ note -UDP Multicast ------------------------------------------- +There is a small performance benefit in using connection based UDP API over +the connectionless one. If there is a SecurityManager enabled on the system, +every connectionless message send has to go through a security check, while +in the case of connection-based UDP the security check is cached after +connect, thus writes do not suffer an additional performance penalty. -Akka provides a way to control various options of ``DatagramChannel`` through the -``akka.io.Inet.SocketOption`` interface. The example below shows +@@@ + +## UDP Multicast + +Akka provides a way to control various options of `DatagramChannel` through the +`akka.io.Inet.SocketOption` interface. The example below shows how to setup a receiver of multicast messages using IPv6 protocol. -To select a Protocol Family you must extend ``akka.io.Inet.DatagramChannelCreator`` -class which extends ``akka.io.Inet.SocketOption``. Provide custom logic -for opening a datagram channel by overriding :meth:`create` method. +To select a Protocol Family you must extend `akka.io.Inet.DatagramChannelCreator` +class which extends `akka.io.Inet.SocketOption`. Provide custom logic +for opening a datagram channel by overriding `create` method. -.. includecode:: code/docs/io/ScalaUdpMulticast.scala#inet6-protocol-family +@@snip [ScalaUdpMulticast.scala](code/docs/io/ScalaUdpMulticast.scala) { #inet6-protocol-family } Another socket option will be needed to join a multicast group. -.. includecode:: code/docs/io/ScalaUdpMulticast.scala#multicast-group +@@snip [ScalaUdpMulticast.scala](code/docs/io/ScalaUdpMulticast.scala) { #multicast-group } -Socket options must be provided to :class:`UdpMessage.Bind` message. +Socket options must be provided to `UdpMessage.Bind` message. -.. includecode:: code/docs/io/ScalaUdpMulticast.scala#bind +@@snip [ScalaUdpMulticast.scala](code/docs/io/ScalaUdpMulticast.scala) { #bind } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/io.md b/akka-docs/src/main/paradox/scala/io.md index 5cab5340e1..d068f912d5 100644 --- a/akka-docs/src/main/paradox/scala/io.md +++ b/akka-docs/src/main/paradox/scala/io.md @@ -1,14 +1,10 @@ -.. _io-scala: +# I/O -I/O -=========== +## Introduction -Introduction ------------- - -The ``akka.io`` package has been developed in collaboration between the Akka -and `spray.io`_ teams. Its design combines experiences from the -``spray-io`` module with improvements that were jointly developed for +The `akka.io` package has been developed in collaboration between the Akka +and [spray.io](http://spray.io) teams. Its design combines experiences from the +`spray-io` module with improvements that were jointly developed for more general consumption as an actor-based service. The guiding design goal for this I/O implementation was to reach extreme @@ -18,25 +14,23 @@ asynchronous. The API is meant to be a solid foundation for the implementation of network protocols and building higher abstractions; it is not meant to be a full-service high-level NIO wrapper for end users. -Terminology, Concepts ---------------------- +## Terminology, Concepts The I/O API is completely actor based, meaning that all operations are implemented with message passing instead of direct method calls. Every I/O driver (TCP, UDP) has a special actor, called a *manager* that serves as an entry point for the API. I/O is broken into several drivers. The manager for a particular driver -is accessible through the ``IO`` entry point. For example the following code -looks up the TCP manager and returns its ``ActorRef``: +is accessible through the `IO` entry point. For example the following code +looks up the TCP manager and returns its `ActorRef`: -.. includecode:: code/docs/io/IODocSpec.scala#manager +@@snip [IODocSpec.scala](code/docs/io/IODocSpec.scala) { #manager } The manager receives I/O command messages and instantiates worker actors in response. The worker actors present -themselves to the API user in the reply to the command that was sent. For example after a ``Connect`` command sent to +themselves to the API user in the reply to the command that was sent. For example after a `Connect` command sent to the TCP manager the manager creates an actor representing the TCP connection. All operations related to the given TCP -connections can be invoked by sending messages to the connection actor which announces itself by sending a ``Connected`` +connections can be invoked by sending messages to the connection actor which announces itself by sending a `Connected` message. -DeathWatch and Resource Management -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### DeathWatch and Resource Management I/O worker actors receive commands and also send out events. They usually need a user-side counterpart actor listening for these events (such events could be inbound connections, incoming bytes or acknowledgements for writes). These worker @@ -46,8 +40,7 @@ resources that it holds. This design makes the API more robust against resource Thanks to the completely actor based approach of the I/O API the opposite direction works as well: a user actor responsible for handling a connection can watch the connection actor to be notified if it unexpectedly terminates. -Write models (Ack, Nack) -^^^^^^^^^^^^^^^^^^^^^^^^ +### Write models (Ack, Nack) I/O devices have a maximum throughput which limits the frequency and size of writes. When an application tries to push more data than a device can handle, the driver has to buffer bytes until the device @@ -56,61 +49,56 @@ is able to write them. With buffering it is possible to handle short bursts of i Akka supports two types of flow control: -* *Ack-based*, where the driver notifies the writer when writes have succeeded. - -* *Nack-based*, where the driver notifies the writer when writes have failed. + * *Ack-based*, where the driver notifies the writer when writes have succeeded. + * *Nack-based*, where the driver notifies the writer when writes have failed. Each of these models is available in both the TCP and the UDP implementations of Akka I/O. -Individual writes can be acknowledged by providing an ack object in the write message (``Write`` in the case of TCP and -``Send`` for UDP). When the write is complete the worker will send the ack object to the writing actor. This can be +Individual writes can be acknowledged by providing an ack object in the write message (`Write` in the case of TCP and +`Send` for UDP). When the write is complete the worker will send the ack object to the writing actor. This can be used to implement *ack-based* flow control; sending new data only when old data has been acknowledged. If a write (or any other command) fails, the driver notifies the actor that sent the command with a special message -(``CommandFailed`` in the case of UDP and TCP). This message will also notify the writer of a failed write, serving as a +(`CommandFailed` in the case of UDP and TCP). This message will also notify the writer of a failed write, serving as a nack for that write. Please note, that in a nack-based flow-control setting the writer has to be prepared for the fact that the failed write might not be the most recent write it sent. For example, the failure notification for a write -``W1`` might arrive after additional write commands ``W2`` and ``W3`` have been sent. If the writer wants to resend any +`W1` might arrive after additional write commands `W2` and `W3` have been sent. If the writer wants to resend any nacked messages it may need to keep a buffer of pending messages. -.. warning:: - An acknowledged write does not mean acknowledged delivery or storage; receiving an ack for a write simply signals that - the I/O driver has successfully processed the write. The Ack/Nack protocol described here is a means of flow control - not error handling. In other words, data may still be lost, even if every write is acknowledged. +@@@ warning -.. _bytestring_scala: +An acknowledged write does not mean acknowledged delivery or storage; receiving an ack for a write simply signals that +the I/O driver has successfully processed the write. The Ack/Nack protocol described here is a means of flow control +not error handling. In other words, data may still be lost, even if every write is acknowledged. -ByteString -^^^^^^^^^^ +@@@ -To maintain isolation, actors should communicate with immutable objects only. ``ByteString`` is an + +### ByteString + +To maintain isolation, actors should communicate with immutable objects only. `ByteString` is an immutable container for bytes. It is used by Akka's I/O system as an efficient, immutable alternative -the traditional byte containers used for I/O on the JVM, such as ``Array[Byte]`` and ``ByteBuffer``. +the traditional byte containers used for I/O on the JVM, such as `Array[Byte]` and `ByteBuffer`. -``ByteString`` is a `rope-like `_ data structure that is immutable -and provides fast concatenation and slicing operations (perfect for I/O). When two ``ByteString``\s are concatenated -together they are both stored within the resulting ``ByteString`` instead of copying both to a new ``Array``. Operations -such as ``drop`` and ``take`` return ``ByteString``\s that still reference the original ``Array``, but just change the -offset and length that is visible. Great care has also been taken to make sure that the internal ``Array`` cannot be -modified. Whenever a potentially unsafe ``Array`` is used to create a new ``ByteString`` a defensive copy is created. If -you require a ``ByteString`` that only blocks as much memory as necessary for it's content, use the ``compact`` method to -get a ``CompactByteString`` instance. If the ``ByteString`` represented only a slice of the original array, this will +`ByteString` is a [rope-like](http://en.wikipedia.org/wiki/Rope_\(computer_science\)) data structure that is immutable +and provides fast concatenation and slicing operations (perfect for I/O). When two `ByteString`s are concatenated +together they are both stored within the resulting `ByteString` instead of copying both to a new `Array`. Operations +such as `drop` and `take` return `ByteString`s that still reference the original `Array`, but just change the +offset and length that is visible. Great care has also been taken to make sure that the internal `Array` cannot be +modified. Whenever a potentially unsafe `Array` is used to create a new `ByteString` a defensive copy is created. If +you require a `ByteString` that only blocks as much memory as necessary for it's content, use the `compact` method to +get a `CompactByteString` instance. If the `ByteString` represented only a slice of the original array, this will result in copying all bytes in that slice. -``ByteString`` inherits all methods from ``IndexedSeq``, and it also has some new ones. For more information, look up the ``akka.util.ByteString`` class and it's companion object in the ScalaDoc. +`ByteString` inherits all methods from `IndexedSeq`, and it also has some new ones. For more information, look up the `akka.util.ByteString` class and it's companion object in the ScalaDoc. -``ByteString`` also comes with its own optimized builder and iterator classes ``ByteStringBuilder`` and -``ByteIterator`` which provide extra features in addition to those of normal builders and iterators. +`ByteString` also comes with its own optimized builder and iterator classes `ByteStringBuilder` and +`ByteIterator` which provide extra features in addition to those of normal builders and iterators. -Compatibility with java.io -.......................... +#### Compatibility with java.io -A ``ByteStringBuilder`` can be wrapped in a ``java.io.OutputStream`` via the ``asOutputStream`` method. Likewise, ``ByteIterator`` can be wrapped in a ``java.io.InputStream`` via ``asInputStream``. Using these, ``akka.io`` applications can integrate legacy code based on ``java.io`` streams. +A `ByteStringBuilder` can be wrapped in a `java.io.OutputStream` via the `asOutputStream` method. Likewise, `ByteIterator` can be wrapped in a `java.io.InputStream` via `asInputStream`. Using these, `akka.io` applications can integrate legacy code based on `java.io` streams. -Architecture in-depth ---------------------- - -For further details on the design and internal architecture see :ref:`io-layer`. - -.. _spray.io: http://spray.io +## Architecture in-depth +For further details on the design and internal architecture see @ref:[I/O Layer Design](../dev/io-layer.md). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/logging.md b/akka-docs/src/main/paradox/scala/logging.md index 62c51d8776..6a40684078 100644 --- a/akka-docs/src/main/paradox/scala/logging.md +++ b/akka-docs/src/main/paradox/scala/logging.md @@ -1,8 +1,4 @@ -.. _logging-scala: - -################# - Logging -################# +# Logging Logging in Akka is not tied to a specific logging backend. By default log messages are printed to STDOUT, but you can plug-in a SLF4J logger or @@ -11,49 +7,47 @@ has minimal performance impact. Logging generally means IO and locks, which can slow down the operations of your code if it was performed synchronously. -How to Log -========== +## How to Log -Create a ``LoggingAdapter`` and use the ``error``, ``warning``, ``info``, or ``debug`` methods, +Create a `LoggingAdapter` and use the `error`, `warning`, `info`, or `debug` methods, as illustrated in this example: -.. includecode:: code/docs/event/LoggingDocSpec.scala - :include: my-actor +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #my-actor } -For convenience, you can mix in the ``log`` member into actors, instead of defining it as above. +For convenience, you can mix in the `log` member into actors, instead of defining it as above. -.. code-block:: scala +```scala +class MyActor extends Actor with akka.actor.ActorLogging { + ... +} +``` - class MyActor extends Actor with akka.actor.ActorLogging { - ... - } - -The second parameter to the ``Logging`` is the source of this logging channel. +The second parameter to the `Logging` is the source of this logging channel. The source object is translated to a String according to the following rules: - * if it is an Actor or ActorRef, its path is used - * in case of a String it is used as is - * in case of a class an approximation of its simpleName - * and in all other cases a compile error occurs unless an implicit - :class:`LogSource[T]` is in scope for the type in question. +> + * if it is an Actor or ActorRef, its path is used + * in case of a String it is used as is + * in case of a class an approximation of its simpleName + * and in all other cases a compile error occurs unless an implicit +`LogSource[T]` is in scope for the type in question. -The log message may contain argument placeholders ``{}``, which will be +The log message may contain argument placeholders `{}`, which will be substituted if the log level is enabled. Giving more arguments than placeholders results in a warning being appended to the log statement (i.e. on the same line with the same severity). You may pass an array as the only substitution argument to have its elements be treated individually: -.. includecode:: code/docs/event/LoggingDocSpec.scala#array +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #array } -The Java :class:`Class` of the log source is also included in the generated -:class:`LogEvent`. In case of a simple string this is replaced with a “marker” -class :class:`akka.event.DummyClassForStringSources` in order to allow special +The Java `Class` of the log source is also included in the generated +`LogEvent`. In case of a simple string this is replaced with a “marker” +class `akka.event.DummyClassForStringSources` in order to allow special treatment of this case, e.g. in the SLF4J event listener which will then use the string instead of the class’ name for looking up the logger instance to use. -Logging of Dead Letters ------------------------ +### Logging of Dead Letters By default messages sent to dead letters are logged at info level. Existence of dead letters does not necessarily indicate a problem, but they are logged by default for the sake of caution. @@ -63,476 +57,490 @@ logged. During system shutdown it is likely that you see dead letters, since pen messages in the actor mailboxes are sent to dead letters. You can also disable logging of dead letters during shutdown. -.. code-block:: ruby - - akka { - log-dead-letters = 10 - log-dead-letters-during-shutdown = on - } +```ruby +akka { + log-dead-letters = 10 + log-dead-letters-during-shutdown = on +} +``` To customize the logging further or take other actions for dead letters you can subscribe -to the :ref:`event-stream-scala`. +to the @ref:[Event Stream](event-bus.md#event-stream-scala). -Auxiliary logging options -------------------------- +### Auxiliary logging options Akka has a few configuration options for very low level debugging. These make more sense in development than in production. You almost definitely need to have logging set to DEBUG to use any of the options below: -.. code-block:: ruby - - akka { - loglevel = "DEBUG" - } +```ruby +akka { + loglevel = "DEBUG" +} +``` This config option is very good if you want to know what config settings are loaded by Akka: -.. code-block:: ruby - - akka { - # Log the complete configuration at INFO level when the actor system is started. - # This is useful when you are uncertain of what configuration is used. - log-config-on-start = on - } +```ruby +akka { + # Log the complete configuration at INFO level when the actor system is started. + # This is useful when you are uncertain of what configuration is used. + log-config-on-start = on +} +``` If you want very detailed logging of user-level messages then wrap your actors' behaviors with -``akka.event.LoggingReceive`` and enable the ``receive`` option: +`akka.event.LoggingReceive` and enable the `receive` option: -.. code-block:: ruby - - akka { - actor { - debug { - # enable function of LoggingReceive, which is to log any received message at - # DEBUG level - receive = on - } - } +```ruby +akka { + actor { + debug { + # enable function of LoggingReceive, which is to log any received message at + # DEBUG level + receive = on } + } +} +``` If you want very detailed logging of all automatically received messages that are processed by Actors: -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of all AutoReceiveMessages (Kill, PoisonPill et.c.) - autoreceive = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of all AutoReceiveMessages (Kill, PoisonPill et.c.) + autoreceive = on } + } +} +``` If you want very detailed logging of all lifecycle changes of Actors (restarts, deaths etc): -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of actor lifecycle changes - lifecycle = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of actor lifecycle changes + lifecycle = on } + } +} +``` If you want unhandled messages logged at DEBUG: -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of unhandled messages - unhandled = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of unhandled messages + unhandled = on } + } +} +``` If you want very detailed logging of all events, transitions and timers of FSM Actors that extend LoggingFSM: -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of all LoggingFSMs for events, transitions and timers - fsm = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of all LoggingFSMs for events, transitions and timers + fsm = on } + } +} +``` If you want to monitor subscriptions (subscribe/unsubscribe) on the ActorSystem.eventStream: -.. code-block:: ruby - - akka { - actor { - debug { - # enable DEBUG logging of subscription changes on the eventStream - event-stream = on - } - } +```ruby +akka { + actor { + debug { + # enable DEBUG logging of subscription changes on the eventStream + event-stream = on } + } +} +``` -Auxiliary remote logging options --------------------------------- +### Auxiliary remote logging options If you want to see all messages that are sent through remoting at DEBUG log level, use the following config option. Note that this logs the messages as they are sent by the transport layer, not by an actor. -.. code-block:: ruby - - akka { - remote { - # If this is "on", Akka will log all outbound messages at DEBUG level, - # if off then they are not logged - log-sent-messages = on - } - } +```ruby +akka { + remote { + # If this is "on", Akka will log all outbound messages at DEBUG level, + # if off then they are not logged + log-sent-messages = on + } +} +``` If you want to see all messages that are received through remoting at DEBUG log level, use the following config option. Note that this logs the messages as they are received by the transport layer, not by an actor. -.. code-block:: ruby - - akka { - remote { - # If this is "on", Akka will log all inbound messages at DEBUG level, - # if off then they are not logged - log-received-messages = on - } - } +```ruby +akka { + remote { + # If this is "on", Akka will log all inbound messages at DEBUG level, + # if off then they are not logged + log-received-messages = on + } +} +``` If you want to see message types with payload size in bytes larger than a specified limit at INFO log level: -.. code-block:: ruby +```ruby +akka { + remote { + # Logging of message types with payload size in bytes larger than + # this value. Maximum detected size per message type is logged once, + # with an increase threshold of 10%. + # By default this feature is turned off. Activate it by setting the property to + # a value in bytes, such as 1000b. Note that for all messages larger than this + # limit there will be extra performance and scalability cost. + log-frame-size-exceeding = 1000b + } +} +``` - akka { - remote { - # Logging of message types with payload size in bytes larger than - # this value. Maximum detected size per message type is logged once, - # with an increase threshold of 10%. - # By default this feature is turned off. Activate it by setting the property to - # a value in bytes, such as 1000b. Note that for all messages larger than this - # limit there will be extra performance and scalability cost. - log-frame-size-exceeding = 1000b - } - } +Also see the logging options for TestKit: @ref:[actor.logging-scala](testing.md#actor-logging-scala). -Also see the logging options for TestKit: :ref:`actor.logging-scala`. - -Translating Log Source to String and Class ------------------------------------------- +### Translating Log Source to String and Class The rules for translating the source object to the source string and class -which are inserted into the :class:`LogEvent` during runtime are implemented +which are inserted into the `LogEvent` during runtime are implemented using implicit parameters and thus fully customizable: simply create your own -instance of :class:`LogSource[T]` and have it in scope when creating the +instance of `LogSource[T]` and have it in scope when creating the logger. -.. includecode:: code/docs/event/LoggingDocSpec.scala#my-source +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #my-source } This example creates a log source which mimics traditional usage of Java loggers, which are based upon the originating object’s class name as log -category. The override of :meth:`getClazz` is only included for demonstration +category. The override of `getClazz` is only included for demonstration purposes as it contains exactly the default behavior. -.. note:: +@@@ note - You may also create the string representation up front and pass that in as - the log source, but be aware that then the :class:`Class[_]` which will be - put in the :class:`LogEvent` is - :class:`akka.event.DummyClassForStringSources`. +You may also create the string representation up front and pass that in as +the log source, but be aware that then the `Class[_]` which will be +put in the `LogEvent` is +`akka.event.DummyClassForStringSources`. - The SLF4J event listener treats this case specially (using the actual string - to look up the logger instance to use instead of the class’ name), and you - might want to do this also in case you implement your own logging adapter. +The SLF4J event listener treats this case specially (using the actual string +to look up the logger instance to use instead of the class’ name), and you +might want to do this also in case you implement your own logging adapter. -Turn Off Logging ----------------- +@@@ -To turn off logging you can configure the log levels to be ``OFF`` like this. +### Turn Off Logging -.. code-block:: ruby +To turn off logging you can configure the log levels to be `OFF` like this. - akka { - stdout-loglevel = "OFF" - loglevel = "OFF" - } +```ruby +akka { + stdout-loglevel = "OFF" + loglevel = "OFF" +} +``` -The ``stdout-loglevel`` is only in effect during system startup and shutdown, and setting -it to ``OFF`` as well, ensures that nothing gets logged during system startup or shutdown. +The `stdout-loglevel` is only in effect during system startup and shutdown, and setting +it to `OFF` as well, ensures that nothing gets logged during system startup or shutdown. -Loggers -======= +## Loggers Logging is performed asynchronously through an event bus. Log events are processed by an event handler actor that receives the log events in the same order they were emitted. -.. note:: - The event handler actor does not have a bounded inbox and is run on the default dispatcher. This means - that logging extreme amounts of data may affect your application badly. This can be somewhat mitigated by - using an async logging backend though. (See :ref:`slf4j-directly-scala`) +@@@ note + +The event handler actor does not have a bounded inbox and is run on the default dispatcher. This means +that logging extreme amounts of data may affect your application badly. This can be somewhat mitigated by +using an async logging backend though. (See [Using the SLF4J API directly](#slf4j-directly-scala)) + +@@@ You can configure which event handlers are created at system start-up and listen to logging events. That is done using the -``loggers`` element in the :ref:`configuration`. +`loggers` element in the configuration. Here you can also define the log level. More fine grained filtering based on the log source -can be implemented in a custom ``LoggingFilter``, which can be defined in the ``logging-filter`` +can be implemented in a custom `LoggingFilter`, which can be defined in the `logging-filter` configuration property. -.. code-block:: ruby - - akka { - # Loggers to register at boot time (akka.event.Logging$DefaultLogger logs - # to STDOUT) - loggers = ["akka.event.Logging$DefaultLogger"] - # Options: OFF, ERROR, WARNING, INFO, DEBUG - loglevel = "DEBUG" - } +```ruby +akka { + # Loggers to register at boot time (akka.event.Logging$DefaultLogger logs + # to STDOUT) + loggers = ["akka.event.Logging$DefaultLogger"] + # Options: OFF, ERROR, WARNING, INFO, DEBUG + loglevel = "DEBUG" +} +``` The default one logs to STDOUT and is registered by default. It is not intended -to be used for production. There is also an :ref:`slf4j-scala` +to be used for production. There is also an [SLF4J](#slf4j-scala) logger available in the 'akka-slf4j' module. Example of creating a listener: -.. includecode:: code/docs/event/LoggingDocSpec.scala - :include: my-event-listener +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #my-event-listener } -Logging to stdout during startup and shutdown -============================================= +## Logging to stdout during startup and shutdown -When the actor system is starting up and shutting down the configured ``loggers`` are not used. +When the actor system is starting up and shutting down the configured `loggers` are not used. Instead log messages are printed to stdout (System.out). The default log level for this -stdout logger is ``WARNING`` and it can be silenced completely by setting -``akka.stdout-loglevel=OFF``. +stdout logger is `WARNING` and it can be silenced completely by setting +`akka.stdout-loglevel=OFF`. -.. _slf4j-scala: + +## SLF4J -SLF4J -===== +Akka provides a logger for [SL4FJ](http://www.slf4j.org/). This module is available in the 'akka-slf4j.jar'. +It has a single dependency: the slf4j-api jar. In your runtime, you also need a SLF4J backend. We recommend [Logback](http://logback.qos.ch/): -Akka provides a logger for `SL4FJ `_. This module is available in the 'akka-slf4j.jar'. -It has a single dependency: the slf4j-api jar. In your runtime, you also need a SLF4J backend. We recommend `Logback `_: +> +```scala +libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.3" +``` - .. code-block:: scala - - libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.3" - - -You need to enable the Slf4jLogger in the ``loggers`` element in -the :ref:`configuration`. Here you can also define the log level of the event bus. +You need to enable the Slf4jLogger in the `loggers` element in +the configuration. Here you can also define the log level of the event bus. More fine grained log levels can be defined in the configuration of the SLF4J backend -(e.g. logback.xml). You should also define ``akka.event.slf4j.Slf4jLoggingFilter`` in -the ``logging-filter`` configuration property. It will filter the log events using the backend +(e.g. logback.xml). You should also define `akka.event.slf4j.Slf4jLoggingFilter` in +the `logging-filter` configuration property. It will filter the log events using the backend configuration (e.g. logback.xml) before they are published to the event bus. -.. warning:: - If you set the ``loglevel`` to a higher level than "DEBUG", any DEBUG events will be filtered - out already at the source and will never reach the logging backend, regardless of how the backend - is configured. +@@@ warning -.. code-block:: ruby +If you set the `loglevel` to a higher level than "DEBUG", any DEBUG events will be filtered +out already at the source and will never reach the logging backend, regardless of how the backend +is configured. - akka { - loggers = ["akka.event.slf4j.Slf4jLogger"] - loglevel = "DEBUG" - logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" - } +@@@ + +```ruby +akka { + loggers = ["akka.event.slf4j.Slf4jLogger"] + loglevel = "DEBUG" + logging-filter = "akka.event.slf4j.Slf4jLoggingFilter" +} +``` One gotcha is that the timestamp is attributed in the event handler, not when actually doing the logging. The SLF4J logger selected for each log event is chosen based on the -:class:`Class[_]` of the log source specified when creating the -:class:`LoggingAdapter`, unless that was given directly as a string in which -case that string is used (i.e. ``LoggerFactory.getLogger(c: Class[_])`` is used in -the first case and ``LoggerFactory.getLogger(s: String)`` in the second). +`Class[_]` of the log source specified when creating the +`LoggingAdapter`, unless that was given directly as a string in which +case that string is used (i.e. `LoggerFactory.getLogger(c: Class[_])` is used in +the first case and `LoggerFactory.getLogger(s: String)` in the second). -.. note:: +@@@ note - Beware that the actor system’s name is appended to a :class:`String` log - source if the LoggingAdapter was created giving an :class:`ActorSystem` to - the factory. If this is not intended, give a :class:`LoggingBus` instead as - shown below: +Beware that the actor system’s name is appended to a `String` log +source if the LoggingAdapter was created giving an `ActorSystem` to +the factory. If this is not intended, give a `LoggingBus` instead as +shown below: -.. code-block:: scala +@@@ - val log = Logging(system.eventStream, "my.nice.string") +```scala +val log = Logging(system.eventStream, "my.nice.string") +``` -.. _slf4j-directly-scala: + +### Using the SLF4J API directly -Using the SLF4J API directly ----------------------------- If you use the SLF4J API directly in your application, remember that the logging operations will block while the underlying infrastructure writes the log statements. This can be avoided by configuring the logging implementation to use -a non-blocking appender. Logback provides `AsyncAppender `_ -that does this. It also contains a feature which will drop ``INFO`` and ``DEBUG`` messages if the logging +a non-blocking appender. Logback provides [AsyncAppender](http://logback.qos.ch/manual/appenders.html#AsyncAppender) +that does this. It also contains a feature which will drop `INFO` and `DEBUG` messages if the logging load is high. -Logging Thread, Akka Source and Actor System in MDC ---------------------------------------------------- +### Logging Thread, Akka Source and Actor System in MDC Since the logging is done asynchronously the thread in which the logging was performed is captured in -Mapped Diagnostic Context (MDC) with attribute name ``sourceThread``. -With Logback the thread name is available with ``%X{sourceThread}`` specifier within the pattern layout configuration:: +Mapped Diagnostic Context (MDC) with attribute name `sourceThread`. +With Logback the thread name is available with `%X{sourceThread}` specifier within the pattern layout configuration: - - - %date{ISO8601} %-5level %logger{36} %X{sourceThread} - %msg%n - - +``` + + + %date{ISO8601} %-5level %logger{36} %X{sourceThread} - %msg%n + + +``` -.. note:: +@@@ note - It will probably be a good idea to use the ``sourceThread`` MDC value also in - non-Akka parts of the application in order to have this property consistently - available in the logs. +It will probably be a good idea to use the `sourceThread` MDC value also in +non-Akka parts of the application in order to have this property consistently +available in the logs. + +@@@ Another helpful facility is that Akka captures the actor’s address when instantiating a logger within it, meaning that the full instance identification is available for associating log messages e.g. with members of a router. This -information is available in the MDC with attribute name ``akkaSource``:: +information is available in the MDC with attribute name `akkaSource`: - - - %date{ISO8601} %-5level %logger{36} %X{akkaSource} - %msg%n - - +``` + + + %date{ISO8601} %-5level %logger{36} %X{akkaSource} - %msg%n + + +``` Finally, the actor system in which the logging was performed -is available in the MDC with attribute name ``sourceActorSystem``:: +is available in the MDC with attribute name `sourceActorSystem`: - - - %date{ISO8601} %-5level %logger{36} %X{sourceActorSystem} - %msg%n - - +``` + + + %date{ISO8601} %-5level %logger{36} %X{sourceActorSystem} - %msg%n + + +``` For more details on what this attribute contains—also for non-actors—please see -`How to Log`_. +[How to Log](#how-to-log). - -More accurate timestamps for log output in MDC ----------------------------------------------- +### More accurate timestamps for log output in MDC Akka's logging is asynchronous which means that the timestamp of a log entry is taken from when the underlying logger implementation is called, which can be surprising at first. -If you want to more accurately output the timestamp, use the MDC attribute ``akkaTimestamp``:: +If you want to more accurately output the timestamp, use the MDC attribute `akkaTimestamp`: - - - %X{akkaTimestamp} %-5level %logger{36} %X{akkaSource} - %msg%n - - +``` + + + %X{akkaTimestamp} %-5level %logger{36} %X{akkaSource} - %msg%n + + +``` -MDC values defined by the application -------------------------------------- +### MDC values defined by the application -One useful feature available in Slf4j is `MDC `_, +One useful feature available in Slf4j is [MDC](http://logback.qos.ch/manual/mdc.html), Akka has a way to let the application specify custom values, you just need to get a -specialized :class:`LoggingAdapter`, the :class:`DiagnosticLoggingAdapter`. In order to +specialized `LoggingAdapter`, the `DiagnosticLoggingAdapter`. In order to get it you can use the factory, providing an Actor as logSource: -.. code-block:: scala - - // Within your Actor - val log: DiagnosticLoggingAdapter = Logging(this); +```scala +// Within your Actor +val log: DiagnosticLoggingAdapter = Logging(this); +``` Once you have the logger, you just need to add the custom values before you log something. This way, the values will be put in the SLF4J MDC right before appending the log and removed after. -.. note:: +@@@ note - The cleanup (removal) should be done in the actor at the end, - otherwise, next message will log with same mdc values, - if it is not set to a new map. Use ``log.clearMDC()``. +The cleanup (removal) should be done in the actor at the end, +otherwise, next message will log with same mdc values, +if it is not set to a new map. Use `log.clearMDC()`. -.. includecode:: code/docs/event/LoggingDocSpec.scala#mdc +@@@ -For convenience, you can mix in the ``log`` member into actors, instead of defining it as above. -This trait also lets you override ``def mdc(msg: Any): MDC`` for specifying MDC values +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #mdc } + +For convenience, you can mix in the `log` member into actors, instead of defining it as above. +This trait also lets you override `def mdc(msg: Any): MDC` for specifying MDC values depending on current message and lets you forget about the cleanup as well, since it already does it for you. -.. includecode:: code/docs/event/LoggingDocSpec.scala - :include: mdc-actor +@@snip [LoggingDocSpec.scala](code/docs/event/LoggingDocSpec.scala) { #mdc-actor } -Now, the values will be available in the MDC, so you can use them in the layout pattern:: +Now, the values will be available in the MDC, so you can use them in the layout pattern: - - - - %-5level %logger{36} [req: %X{requestId}, visitor: %X{visitorId}] - %msg%n - - - +``` + + + + %-5level %logger{36} [req: %X{requestId}, visitor: %X{visitorId}] - %msg%n + + + +``` - -Using Markers -------------- +### Using Markers Some logging libraries allow, in addition to MDC data, attaching so called "markers" to log statements. These are used to filter out rare and special events, for example you might want to mark logs that detect -some malicious activity and mark them with a ``SECURITY`` tag, and in your appender configuration make these +some malicious activity and mark them with a `SECURITY` tag, and in your appender configuration make these trigger emails and other notifications immediately. -Markers are available through the LoggingAdapters, when obtained via ``Logging.withMarker``. -The first argument passed into all log calls then should be a ``akka.event.LogMarker``. +Markers are available through the LoggingAdapters, when obtained via `Logging.withMarker`. +The first argument passed into all log calls then should be a `akka.event.LogMarker`. -The slf4j bridge provided by akka in ``akka-slf4j`` will automatically pick up this marker value and make it available to SLF4J. -For example you could use it like this:: +The slf4j bridge provided by akka in `akka-slf4j` will automatically pick up this marker value and make it available to SLF4J. +For example you could use it like this: - %date{ISO8601} [%marker][%level] [%msg]%n +``` +%date{ISO8601} [%marker][%level] [%msg]%n +``` -A more advanced (including most Akka added information) example pattern would be:: +A more advanced (including most Akka added information) example pattern would be: - %date{ISO8601} level=[%level] marker=[%marker] logger=[%logger] akkaSource=[%X{akkaSource}] sourceActorSystem=[%X{sourceActorSystem}] sourceThread=[%X{sourceThread}] mdc=[ticket-#%X{ticketNumber}: %X{ticketDesc}] - msg=[%msg]%n----%n +``` +%date{ISO8601} level=[%level] marker=[%marker] logger=[%logger] akkaSource=[%X{akkaSource}] sourceActorSystem=[%X{sourceActorSystem}] sourceThread=[%X{sourceThread}] mdc=[ticket-#%X{ticketNumber}: %X{ticketDesc}] - msg=[%msg]%n----%n +``` -.. _jul-scala: + +## java.util.logging -java.util.logging -================= +Akka includes a logger for [java.util.logging](https://docs.oracle.com/javase/8/docs/api/java/util/logging/package-summary.html#package.description). -Akka includes a logger for `java.util.logging `_. - -You need to enable the ``akka.event.jul.JavaLogger`` in the ``loggers`` element in -the :ref:`configuration`. Here you can also define the log level of the event bus. +You need to enable the `akka.event.jul.JavaLogger` in the `loggers` element in +the configuration. Here you can also define the log level of the event bus. More fine grained log levels can be defined in the configuration of the logging backend. -You should also define ``akka.event.jul.JavaLoggingFilter`` in -the ``logging-filter`` configuration property. It will filter the log events using the backend +You should also define `akka.event.jul.JavaLoggingFilter` in +the `logging-filter` configuration property. It will filter the log events using the backend configuration before they are published to the event bus. -.. warning:: - If you set the ``loglevel`` to a higher level than "DEBUG", any DEBUG events will be filtered - out already at the source and will never reach the logging backend, regardless of how the backend - is configured. +@@@ warning -.. code-block:: ruby +If you set the `loglevel` to a higher level than "DEBUG", any DEBUG events will be filtered +out already at the source and will never reach the logging backend, regardless of how the backend +is configured. - akka { - loglevel = DEBUG - loggers = ["akka.event.jul.JavaLogger"] - logging-filter = "akka.event.jul.JavaLoggingFilter" - } +@@@ + +```ruby +akka { + loglevel = DEBUG + loggers = ["akka.event.jul.JavaLogger"] + logging-filter = "akka.event.jul.JavaLoggingFilter" +} +``` One gotcha is that the timestamp is attributed in the event handler, not when actually doing the logging. -The ``java.util.logging.Logger`` selected for each log event is chosen based on the -:class:`Class[_]` of the log source specified when creating the -:class:`LoggingAdapter`, unless that was given directly as a string in which -case that string is used (i.e. ``LoggerFactory.getLogger(c: Class[_])`` is used in -the first case and ``LoggerFactory.getLogger(s: String)`` in the second). +The `java.util.logging.Logger` selected for each log event is chosen based on the +`Class[_]` of the log source specified when creating the +`LoggingAdapter`, unless that was given directly as a string in which +case that string is used (i.e. `LoggerFactory.getLogger(c: Class[_])` is used in +the first case and `LoggerFactory.getLogger(s: String)` in the second). -.. note:: +@@@ note - Beware that the actor system’s name is appended to a :class:`String` log - source if the LoggingAdapter was created giving an :class:`ActorSystem` to - the factory. If this is not intended, give a :class:`LoggingBus` instead as - shown below: +Beware that the actor system’s name is appended to a `String` log +source if the LoggingAdapter was created giving an `ActorSystem` to +the factory. If this is not intended, give a `LoggingBus` instead as +shown below: -.. code-block:: scala +@@@ - val log = Logging(system.eventStream, "my.nice.string") +```scala +val log = Logging(system.eventStream, "my.nice.string") +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/mailboxes.md b/akka-docs/src/main/paradox/scala/mailboxes.md index dd09d53a7e..6287ae5298 100644 --- a/akka-docs/src/main/paradox/scala/mailboxes.md +++ b/akka-docs/src/main/paradox/scala/mailboxes.md @@ -1,334 +1,271 @@ -.. _mailboxes-scala: +# Mailboxes -Mailboxes -######### - -An Akka ``Mailbox`` holds the messages that are destined for an ``Actor``. -Normally each ``Actor`` has its own mailbox, but with for example a ``BalancingPool`` +An Akka `Mailbox` holds the messages that are destined for an `Actor`. +Normally each `Actor` has its own mailbox, but with for example a `BalancingPool` all routees will share a single mailbox instance. -Mailbox Selection -================= +## Mailbox Selection -Requiring a Message Queue Type for an Actor -------------------------------------------- +### Requiring a Message Queue Type for an Actor It is possible to require a certain type of message queue for a certain type of actor -by having that actor extend the parameterized trait :class:`RequiresMessageQueue`. Here is +by having that actor extend the parameterized trait `RequiresMessageQueue`. Here is an example: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#required-mailbox-class +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #required-mailbox-class } -The type parameter to the :class:`RequiresMessageQueue` trait needs to be mapped to a mailbox in +The type parameter to the `RequiresMessageQueue` trait needs to be mapped to a mailbox in configuration like this: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala - :include: bounded-mailbox-config,required-mailbox-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #bounded-mailbox-config #required-mailbox-config } -Now every time you create an actor of type :class:`MyBoundedActor` it will try to get a bounded +Now every time you create an actor of type `MyBoundedActor` it will try to get a bounded mailbox. If the actor has a different mailbox configured in deployment, either directly or via a dispatcher with a specified mailbox type, then that will override this mapping. -.. note:: +@@@ note - The type of the queue in the mailbox created for an actor will be checked against the required type in the - trait and if the queue doesn't implement the required type then actor creation will fail. +The type of the queue in the mailbox created for an actor will be checked against the required type in the +trait and if the queue doesn't implement the required type then actor creation will fail. -Requiring a Message Queue Type for a Dispatcher ------------------------------------------------ +@@@ + +### Requiring a Message Queue Type for a Dispatcher A dispatcher may also have a requirement for the mailbox type used by the actors running on it. An example is the BalancingDispatcher which requires a message queue that is thread-safe for multiple concurrent consumers. Such a requirement is formulated within the dispatcher configuration section like -this:: +this: - my-dispatcher { - mailbox-requirement = org.example.MyInterface - } +``` +my-dispatcher { + mailbox-requirement = org.example.MyInterface +} +``` The given requirement names a class or interface which will then be ensured to be a supertype of the message queue’s implementation. In case of a conflict—e.g. if the actor requires a mailbox type which does not satisfy this requirement—then actor creation will fail. -How the Mailbox Type is Selected --------------------------------- +### How the Mailbox Type is Selected -When an actor is created, the :class:`ActorRefProvider` first determines the +When an actor is created, the `ActorRefProvider` first determines the dispatcher which will execute it. Then the mailbox is determined as follows: -1. If the actor’s deployment configuration section contains a ``mailbox`` key - then that names a configuration section describing the mailbox type to be - used. + 1. If the actor’s deployment configuration section contains a `mailbox` key +then that names a configuration section describing the mailbox type to be +used. + 2. If the actor’s `Props` contains a mailbox selection—i.e. `withMailbox` +was called on it—then that names a configuration section describing the +mailbox type to be used. + 3. If the dispatcher’s configuration section contains a `mailbox-type` key +the same section will be used to configure the mailbox type. + 4. If the actor requires a mailbox type as described above then the mapping for +that requirement will be used to determine the mailbox type to be used; if +that fails then the dispatcher’s requirement—if any—will be tried instead. + 5. If the dispatcher requires a mailbox type as described above then the +mapping for that requirement will be used to determine the mailbox type to +be used. + 6. The default mailbox `akka.actor.default-mailbox` will be used. -2. If the actor’s ``Props`` contains a mailbox selection—i.e. ``withMailbox`` - was called on it—then that names a configuration section describing the - mailbox type to be used. - -3. If the dispatcher’s configuration section contains a ``mailbox-type`` key - the same section will be used to configure the mailbox type. - -4. If the actor requires a mailbox type as described above then the mapping for - that requirement will be used to determine the mailbox type to be used; if - that fails then the dispatcher’s requirement—if any—will be tried instead. - -5. If the dispatcher requires a mailbox type as described above then the - mapping for that requirement will be used to determine the mailbox type to - be used. - -6. The default mailbox ``akka.actor.default-mailbox`` will be used. - -Default Mailbox ---------------- +### Default Mailbox When the mailbox is not specified as described above the default mailbox is used. By default it is an unbounded mailbox, which is backed by a -``java.util.concurrent.ConcurrentLinkedQueue``. +`java.util.concurrent.ConcurrentLinkedQueue`. -``SingleConsumerOnlyUnboundedMailbox`` is an even more efficient mailbox, and +`SingleConsumerOnlyUnboundedMailbox` is an even more efficient mailbox, and it can be used as the default mailbox, but it cannot be used with a BalancingDispatcher. -Configuration of ``SingleConsumerOnlyUnboundedMailbox`` as default mailbox:: +Configuration of `SingleConsumerOnlyUnboundedMailbox` as default mailbox: - akka.actor.default-mailbox { - mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" - } +``` +akka.actor.default-mailbox { + mailbox-type = "akka.dispatch.SingleConsumerOnlyUnboundedMailbox" +} +``` -Which Configuration is passed to the Mailbox Type -------------------------------------------------- +### Which Configuration is passed to the Mailbox Type -Each mailbox type is implemented by a class which extends :class:`MailboxType` -and takes two constructor arguments: a :class:`ActorSystem.Settings` object and -a :class:`Config` section. The latter is computed by obtaining the named +Each mailbox type is implemented by a class which extends `MailboxType` +and takes two constructor arguments: a `ActorSystem.Settings` object and +a `Config` section. The latter is computed by obtaining the named configuration section from the actor system’s configuration, overriding its -``id`` key with the configuration path of the mailbox type and adding a +`id` key with the configuration path of the mailbox type and adding a fall-back to the default mailbox configuration section. -Builtin Mailbox Implementations -=============================== +## Builtin Mailbox Implementations Akka comes shipped with a number of mailbox implementations: -* **UnboundedMailbox** (default) - - - The default mailbox - - - Backed by a ``java.util.concurrent.ConcurrentLinkedQueue`` - - - Blocking: No - - - Bounded: No - - - Configuration name: ``"unbounded"`` or ``"akka.dispatch.UnboundedMailbox"`` - -* **SingleConsumerOnlyUnboundedMailbox** - - This queue may or may not be faster than the default one depending on your use-case—be sure to benchmark properly! - - - Backed by a Multiple-Producer Single-Consumer queue, cannot be used with ``BalancingDispatcher`` - - - Blocking: No - - - Bounded: No - - - Configuration name: ``"akka.dispatch.SingleConsumerOnlyUnboundedMailbox"`` - -* **NonBlockingBoundedMailbox** - - - Backed by a very efficient Multiple-Producer Single-Consumer queue - - - Blocking: No (discards overflowing messages into deadLetters) - - - Bounded: Yes - - - Configuration name: ``"akka.dispatch.NonBlockingBoundedMailbox"`` - -* **UnboundedControlAwareMailbox** - - - Delivers messages that extend ``akka.dispatch.ControlMessage`` with higher priority - - - Backed by two ``java.util.concurrent.ConcurrentLinkedQueue`` - - - Blocking: No - - - Bounded: No - - - Configuration name: "akka.dispatch.UnboundedControlAwareMailbox" - -* **UnboundedPriorityMailbox** - - - Backed by a ``java.util.concurrent.PriorityBlockingQueue`` - - - Delivery order for messages of equal priority is undefined - contrast with the UnboundedStablePriorityMailbox - - - Blocking: No - - - Bounded: No - - - Configuration name: "akka.dispatch.UnboundedPriorityMailbox" - -* **UnboundedStablePriorityMailbox** - - - Backed by a ``java.util.concurrent.PriorityBlockingQueue`` wrapped in an ``akka.util.PriorityQueueStabilizer`` - - - FIFO order is preserved for messages of equal priority - contrast with the UnboundedPriorityMailbox - - - Blocking: No - - - Bounded: No - - - Configuration name: "akka.dispatch.UnboundedStablePriorityMailbox" + * + **UnboundedMailbox** (default) + * The default mailbox + * Backed by a `java.util.concurrent.ConcurrentLinkedQueue` + * Blocking: No + * Bounded: No + * Configuration name: `"unbounded"` or `"akka.dispatch.UnboundedMailbox"` + * + **SingleConsumerOnlyUnboundedMailbox** + This queue may or may not be faster than the default one depending on your use-case—be sure to benchmark properly! + * Backed by a Multiple-Producer Single-Consumer queue, cannot be used with `BalancingDispatcher` + * Blocking: No + * Bounded: No + * Configuration name: `"akka.dispatch.SingleConsumerOnlyUnboundedMailbox"` + * + **NonBlockingBoundedMailbox** + * Backed by a very efficient Multiple-Producer Single-Consumer queue + * Blocking: No (discards overflowing messages into deadLetters) + * Bounded: Yes + * Configuration name: `"akka.dispatch.NonBlockingBoundedMailbox"` + * + **UnboundedControlAwareMailbox** + * Delivers messages that extend `akka.dispatch.ControlMessage` with higher priority + * Backed by two `java.util.concurrent.ConcurrentLinkedQueue` + * Blocking: No + * Bounded: No + * Configuration name: "akka.dispatch.UnboundedControlAwareMailbox" + * + **UnboundedPriorityMailbox** + * Backed by a `java.util.concurrent.PriorityBlockingQueue` + * Delivery order for messages of equal priority is undefined - contrast with the UnboundedStablePriorityMailbox + * Blocking: No + * Bounded: No + * Configuration name: "akka.dispatch.UnboundedPriorityMailbox" + * + **UnboundedStablePriorityMailbox** + * Backed by a `java.util.concurrent.PriorityBlockingQueue` wrapped in an `akka.util.PriorityQueueStabilizer` + * FIFO order is preserved for messages of equal priority - contrast with the UnboundedPriorityMailbox + * Blocking: No + * Bounded: No + * Configuration name: "akka.dispatch.UnboundedStablePriorityMailbox" Other bounded mailbox implementations which will block the sender if the capacity is reached and -configured with non-zero ``mailbox-push-timeout-time``. +configured with non-zero `mailbox-push-timeout-time`. -.. note:: The following mailboxes should only be used with zero ``mailbox-push-timeout-time``. +@@@ note -* **BoundedMailbox** +The following mailboxes should only be used with zero `mailbox-push-timeout-time`. - - Backed by a ``java.util.concurrent.LinkedBlockingQueue`` +@@@ - - Blocking: Yes if used with non-zero ``mailbox-push-timeout-time``, otherwise No + * **BoundedMailbox** + * Backed by a `java.util.concurrent.LinkedBlockingQueue` + * Blocking: Yes if used with non-zero `mailbox-push-timeout-time`, otherwise No + * Bounded: Yes + * Configuration name: "bounded" or "akka.dispatch.BoundedMailbox" + * **BoundedPriorityMailbox** + * Backed by a `java.util.PriorityQueue` wrapped in an `akka.util.BoundedBlockingQueue` + * Delivery order for messages of equal priority is undefined - contrast with the `BoundedStablePriorityMailbox` + * Blocking: Yes if used with non-zero `mailbox-push-timeout-time`, otherwise No + * Bounded: Yes + * Configuration name: `"akka.dispatch.BoundedPriorityMailbox"` + * **BoundedStablePriorityMailbox** + * Backed by a `java.util.PriorityQueue` wrapped in an `akka.util.PriorityQueueStabilizer` and an `akka.util.BoundedBlockingQueue` + * FIFO order is preserved for messages of equal priority - contrast with the BoundedPriorityMailbox + * Blocking: Yes if used with non-zero `mailbox-push-timeout-time`, otherwise No + * Bounded: Yes + * Configuration name: "akka.dispatch.BoundedStablePriorityMailbox" + * **BoundedControlAwareMailbox** + * Delivers messages that extend `akka.dispatch.ControlMessage` with higher priority + * Backed by two `java.util.concurrent.ConcurrentLinkedQueue` and blocking on enqueue if capacity has been reached + * Blocking: Yes if used with non-zero `mailbox-push-timeout-time`, otherwise No + * Bounded: Yes + * Configuration name: "akka.dispatch.BoundedControlAwareMailbox" - - Bounded: Yes +## Mailbox configuration examples - - Configuration name: "bounded" or "akka.dispatch.BoundedMailbox" - -* **BoundedPriorityMailbox** - - - Backed by a ``java.util.PriorityQueue`` wrapped in an ``akka.util.BoundedBlockingQueue`` - - - Delivery order for messages of equal priority is undefined - contrast with the ``BoundedStablePriorityMailbox`` - - - Blocking: Yes if used with non-zero ``mailbox-push-timeout-time``, otherwise No - - - Bounded: Yes - - - Configuration name: ``"akka.dispatch.BoundedPriorityMailbox"`` - -* **BoundedStablePriorityMailbox** - - - Backed by a ``java.util.PriorityQueue`` wrapped in an ``akka.util.PriorityQueueStabilizer`` and an ``akka.util.BoundedBlockingQueue`` - - - FIFO order is preserved for messages of equal priority - contrast with the BoundedPriorityMailbox - - - Blocking: Yes if used with non-zero ``mailbox-push-timeout-time``, otherwise No - - - Bounded: Yes - - - Configuration name: "akka.dispatch.BoundedStablePriorityMailbox" - -* **BoundedControlAwareMailbox** - - - Delivers messages that extend ``akka.dispatch.ControlMessage`` with higher priority - - - Backed by two ``java.util.concurrent.ConcurrentLinkedQueue`` and blocking on enqueue if capacity has been reached - - - Blocking: Yes if used with non-zero ``mailbox-push-timeout-time``, otherwise No - - - Bounded: Yes - - - Configuration name: "akka.dispatch.BoundedControlAwareMailbox" - - -Mailbox configuration examples -============================== - -PriorityMailbox ---------------- +### PriorityMailbox How to create a PriorityMailbox: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#prio-mailbox +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #prio-mailbox } And then add it to the configuration: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#prio-dispatcher-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #prio-dispatcher-config } And then an example on how you would use it: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#prio-dispatcher +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #prio-dispatcher } It is also possible to configure a mailbox type directly like this: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala - :include: prio-mailbox-config,mailbox-deployment-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #prio-mailbox-config #mailbox-deployment-config } And then use it either from deployment like this: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#defining-mailbox-in-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #defining-mailbox-in-config } Or code like this: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#defining-mailbox-in-code +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #defining-mailbox-in-code } -ControlAwareMailbox -------------------- +### ControlAwareMailbox -A ``ControlAwareMailbox`` can be very useful if an actor needs to be able to receive control messages +A `ControlAwareMailbox` can be very useful if an actor needs to be able to receive control messages immediately no matter how many other messages are already in its mailbox. It can be configured like this: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#control-aware-mailbox-config +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #control-aware-mailbox-config } -Control messages need to extend the ``ControlMessage`` trait: +Control messages need to extend the `ControlMessage` trait: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#control-aware-mailbox-messages +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #control-aware-mailbox-messages } And then an example on how you would use it: -.. includecode:: ../scala/code/docs/dispatcher/DispatcherDocSpec.scala#control-aware-dispatcher +@@snip [DispatcherDocSpec.scala](../scala/code/docs/dispatcher/DispatcherDocSpec.scala) { #control-aware-dispatcher } -Creating your own Mailbox type -============================== +## Creating your own Mailbox type An example is worth a thousand quacks: -.. includecode:: ../scala/code/docs/dispatcher/MyUnboundedMailbox.scala#mailbox-implementation-example +@@snip [MyUnboundedMailbox.scala](../scala/code/docs/dispatcher/MyUnboundedMailbox.scala) { #mailbox-implementation-example } And then you just specify the FQCN of your MailboxType as the value of the "mailbox-type" in the dispatcher configuration, or the mailbox configuration. -.. note:: +@@@ note - Make sure to include a constructor which takes - ``akka.actor.ActorSystem.Settings`` and ``com.typesafe.config.Config`` - arguments, as this constructor is invoked reflectively to construct your - mailbox type. The config passed in as second argument is that section from - the configuration which describes the dispatcher or mailbox setting using - this mailbox type; the mailbox type will be instantiated once for each - dispatcher or mailbox setting using it. +Make sure to include a constructor which takes +`akka.actor.ActorSystem.Settings` and `com.typesafe.config.Config` +arguments, as this constructor is invoked reflectively to construct your +mailbox type. The config passed in as second argument is that section from +the configuration which describes the dispatcher or mailbox setting using +this mailbox type; the mailbox type will be instantiated once for each +dispatcher or mailbox setting using it. + +@@@ You can also use the mailbox as a requirement on the dispatcher like this: -.. includecode:: code/docs/dispatcher/DispatcherDocSpec.scala#custom-mailbox-config-java - +@@snip [DispatcherDocSpec.scala](code/docs/dispatcher/DispatcherDocSpec.scala) { #custom-mailbox-config-java } Or by defining the requirement on your actor class like this: -.. includecode:: code/docs/dispatcher/DispatcherDocSpec.scala#require-mailbox-on-actor +@@snip [DispatcherDocSpec.scala](code/docs/dispatcher/DispatcherDocSpec.scala) { #require-mailbox-on-actor } +## Special Semantics of `system.actorOf` -Special Semantics of ``system.actorOf`` -======================================= - -In order to make ``system.actorOf`` both synchronous and non-blocking while -keeping the return type :class:`ActorRef` (and the semantics that the returned +In order to make `system.actorOf` both synchronous and non-blocking while +keeping the return type `ActorRef` (and the semantics that the returned ref is fully functional), special handling takes place for this case. Behind the scenes, a hollow kind of actor reference is constructed, which is sent to the system’s guardian actor who actually creates the actor and its context and puts those inside the reference. Until that has happened, messages sent to the -:class:`ActorRef` will be queued locally, and only upon swapping the real +`ActorRef` will be queued locally, and only upon swapping the real filling in will they be transferred into the real mailbox. Thus, -.. code-block:: scala - - val props: Props = ... - // this actor uses MyCustomMailbox, which is assumed to be a singleton - system.actorOf(props.withDispatcher("myCustomMailbox")) ! "bang" - assert(MyCustomMailbox.instance.getLastEnqueuedMessage == "bang") +```scala +val props: Props = ... +// this actor uses MyCustomMailbox, which is assumed to be a singleton +system.actorOf(props.withDispatcher("myCustomMailbox")) ! "bang" +assert(MyCustomMailbox.instance.getLastEnqueuedMessage == "bang") +``` will probably fail; you will have to allow for some time to pass and retry the -check à la :meth:`TestKit.awaitCond`. +check à la `TestKit.awaitCond`. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/persistence-query-leveldb.md b/akka-docs/src/main/paradox/scala/persistence-query-leveldb.md index e7ff2f76c1..02be924484 100644 --- a/akka-docs/src/main/paradox/scala/persistence-query-leveldb.md +++ b/akka-docs/src/main/paradox/scala/persistence-query-leveldb.md @@ -1,107 +1,99 @@ -.. _persistence-query-leveldb-scala: +# Persistence Query for LevelDB -############################# -Persistence Query for LevelDB -############################# - -This is documentation for the LevelDB implementation of the :ref:`persistence-query-scala` API. +This is documentation for the LevelDB implementation of the @ref:[Persistence Query](persistence-query.md) API. Note that implementations for other journals may have different semantics. - -Dependencies -============ -Akka persistence LevelDB query implementation is bundled in the ``akka-persistence-query`` artifact. -Make sure that you have the following dependency in your project:: +## Dependencies - "com.typesafe.akka" %% "akka-persistence-query" % "@version@" @crossString@ +Akka persistence LevelDB query implementation is bundled in the `akka-persistence-query` artifact. +Make sure that you have the following dependency in your project: -How to get the ReadJournal -========================== +``` +"com.typesafe.akka" %% "akka-persistence-query" % "@version@" @crossString@ +``` -The ``ReadJournal`` is retrieved via the ``akka.persistence.query.PersistenceQuery`` +## How to get the ReadJournal + +The `ReadJournal` is retrieved via the `akka.persistence.query.PersistenceQuery` extension: -.. includecode:: code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala#get-read-journal +@@snip [LeveldbPersistenceQueryDocSpec.scala](code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala) { #get-read-journal } -Supported Queries -================= +## Supported Queries -EventsByPersistenceIdQuery and CurrentEventsByPersistenceIdQuery ----------------------------------------------------------------- +### EventsByPersistenceIdQuery and CurrentEventsByPersistenceIdQuery -``eventsByPersistenceId`` is used for retrieving events for a specific ``PersistentActor`` -identified by ``persistenceId``. +`eventsByPersistenceId` is used for retrieving events for a specific `PersistentActor` +identified by `persistenceId`. -.. includecode:: code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala#EventsByPersistenceId +@@snip [LeveldbPersistenceQueryDocSpec.scala](code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala) { #EventsByPersistenceId } -You can retrieve a subset of all events by specifying ``fromSequenceNr`` and ``toSequenceNr`` -or use ``0L`` and ``Long.MaxValue`` respectively to retrieve all events. Note that -the corresponding sequence number of each event is provided in the ``EventEnvelope``, +You can retrieve a subset of all events by specifying `fromSequenceNr` and `toSequenceNr` +or use `0L` and `Long.MaxValue` respectively to retrieve all events. Note that +the corresponding sequence number of each event is provided in the `EventEnvelope`, which makes it possible to resume the stream at a later point from a given sequence number. The returned event stream is ordered by sequence number, i.e. the same order as the -``PersistentActor`` persisted the events. The same prefix of stream elements (in same order) +`PersistentActor` persisted the events. The same prefix of stream elements (in same order) are returned for multiple executions of the query, except for when events have been deleted. The stream is not completed when it reaches the end of the currently stored events, but it continues to push new events when new events are persisted. Corresponding query that is completed when it reaches the end of the currently -stored events is provided by ``currentEventsByPersistenceId``. +stored events is provided by `currentEventsByPersistenceId`. The LevelDB write journal is notifying the query side as soon as events are persisted, but for efficiency reasons the query side retrieves the events in batches that sometimes can -be delayed up to the configured ``refresh-interval`` or given ``RefreshInterval`` +be delayed up to the configured `refresh-interval` or given `RefreshInterval` hint. The stream is completed with failure if there is a failure in executing the query in the backend journal. -AllPersistenceIdsQuery and CurrentPersistenceIdsQuery ------------------------------------------------------ +### AllPersistenceIdsQuery and CurrentPersistenceIdsQuery -``allPersistenceIds`` is used for retrieving all ``persistenceIds`` of all persistent actors. +`allPersistenceIds` is used for retrieving all `persistenceIds` of all persistent actors. -.. includecode:: code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala#AllPersistenceIds +@@snip [LeveldbPersistenceQueryDocSpec.scala](code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala) { #AllPersistenceIds } The returned event stream is unordered and you can expect different order for multiple executions of the query. -The stream is not completed when it reaches the end of the currently used `persistenceIds`, -but it continues to push new `persistenceIds` when new persistent actors are created. +The stream is not completed when it reaches the end of the currently used *persistenceIds*, +but it continues to push new *persistenceIds* when new persistent actors are created. Corresponding query that is completed when it reaches the end of the -currently used `persistenceIds` is provided by ``currentPersistenceIds``. +currently used *persistenceIds* is provided by `currentPersistenceIds`. -The LevelDB write journal is notifying the query side as soon as new ``persistenceIds`` are +The LevelDB write journal is notifying the query side as soon as new `persistenceIds` are created and there is no periodic polling or batching involved in this query. The stream is completed with failure if there is a failure in executing the query in the backend journal. -EventsByTag and CurrentEventsByTag ----------------------------------- +### EventsByTag and CurrentEventsByTag -``eventsByTag`` is used for retrieving events that were marked with a given tag, e.g. +`eventsByTag` is used for retrieving events that were marked with a given tag, e.g. all domain events of an Aggregate Root type. -.. includecode:: code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala#EventsByTag +@@snip [LeveldbPersistenceQueryDocSpec.scala](code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala) { #EventsByTag } -To tag events you create an :ref:`event-adapters-scala` that wraps the events in a ``akka.persistence.journal.Tagged`` -with the given ``tags``. +To tag events you create an @ref:[Event Adapters](persistence.md#event-adapters-scala) that wraps the events in a `akka.persistence.journal.Tagged` +with the given `tags`. -.. includecode:: code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala#tagger +@@snip [LeveldbPersistenceQueryDocSpec.scala](code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala) { #tagger } -You can use ``NoOffset`` to retrieve all events with a given tag or retrieve a subset of all -events by specifying a ``Sequence`` ``offset``. The ``offset`` corresponds to an ordered sequence number for +You can use `NoOffset` to retrieve all events with a given tag or retrieve a subset of all +events by specifying a `Sequence` `offset`. The `offset` corresponds to an ordered sequence number for the specific tag. Note that the corresponding offset of each event is provided in the -``EventEnvelope``, which makes it possible to resume the stream at a later point from a given offset. +`EventEnvelope`, which makes it possible to resume the stream at a later point from a given offset. -The ``offset`` is exclusive, i.e. the event with the exact same sequence number will not be included -in the returned stream. This means that you can use the offset that is returned in ``EventEnvelope`` -as the ``offset`` parameter in a subsequent query. +The `offset` is exclusive, i.e. the event with the exact same sequence number will not be included +in the returned stream. This means that you can use the offset that is returned in `EventEnvelope` +as the `offset` parameter in a subsequent query. -In addition to the ``offset`` the ``EventEnvelope`` also provides ``persistenceId`` and ``sequenceNr`` -for each event. The ``sequenceNr`` is the sequence number for the persistent actor with the -``persistenceId`` that persisted the event. The ``persistenceId`` + ``sequenceNr`` is an unique +In addition to the `offset` the `EventEnvelope` also provides `persistenceId` and `sequenceNr` +for each event. The `sequenceNr` is the sequence number for the persistent actor with the +`persistenceId` that persisted the event. The `persistenceId` + `sequenceNr` is an unique identifier for the event. The returned event stream is ordered by the offset (tag sequence number), which corresponds @@ -109,30 +101,31 @@ to the same order as the write journal stored the events. The same stream elemen are returned for multiple executions of the query. Deleted events are not deleted from the tagged event stream. -.. note:: +@@@ note - Events deleted using ``deleteMessages(toSequenceNr)`` are not deleted from the "tagged stream". +Events deleted using `deleteMessages(toSequenceNr)` are not deleted from the "tagged stream". + +@@@ The stream is not completed when it reaches the end of the currently stored events, but it continues to push new events when new events are persisted. Corresponding query that is completed when it reaches the end of the currently -stored events is provided by ``currentEventsByTag``. +stored events is provided by `currentEventsByTag`. The LevelDB write journal is notifying the query side as soon as tagged events are persisted, but for efficiency reasons the query side retrieves the events in batches that sometimes can -be delayed up to the configured ``refresh-interval`` or given ``RefreshInterval`` +be delayed up to the configured `refresh-interval` or given `RefreshInterval` hint. The stream is completed with failure if there is a failure in executing the query in the backend journal. -Configuration -============= +## Configuration Configuration settings can be defined in the configuration section with the -absolute path corresponding to the identifier, which is ``"akka.persistence.query.journal.leveldb"`` -for the default ``LeveldbReadJournal.Identifier``. +absolute path corresponding to the identifier, which is `"akka.persistence.query.journal.leveldb"` +for the default `LeveldbReadJournal.Identifier`. It can be configured with the following properties: -.. includecode:: ../../../akka-persistence-query/src/main/resources/reference.conf#query-leveldb +@@snip [reference.conf]../../../../../akka-persistence-query/src/main/resources/reference.conf) { #query-leveldb } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/persistence-query.md b/akka-docs/src/main/paradox/scala/persistence-query.md index 7173b5b047..aab6ad8510 100644 --- a/akka-docs/src/main/paradox/scala/persistence-query.md +++ b/akka-docs/src/main/paradox/scala/persistence-query.md @@ -1,10 +1,6 @@ -.. _persistence-query-scala: +# Persistence Query -################# -Persistence Query -################# - -Akka persistence query complements :ref:`persistence-scala` by providing a universal asynchronous stream based +Akka persistence query complements @ref:[Persistence](persistence.md) by providing a universal asynchronous stream based query interface that various journal plugins can implement in order to expose their query capabilities. The most typical use case of persistence query is implementing the so-called query side (also known as "read side") @@ -14,15 +10,15 @@ side of an application, however it can help to migrate data from the write side simple scenarios Persistence Query may be powerful enough to fulfill the query needs of your app, however we highly recommend (in the spirit of CQRS) of splitting up the write/read sides into separate datastores as the need arises. -Dependencies -============ +## Dependencies -Akka persistence query is a separate jar file. Make sure that you have the following dependency in your project:: +Akka persistence query is a separate jar file. Make sure that you have the following dependency in your project: - "com.typesafe.akka" %% "akka-persistence-query" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-persistence-query" % "@version@" @crossString@ +``` -Design overview -=============== +## Design overview Akka persistence query is purposely designed to be a very loosely specified API. This is in order to keep the provided APIs general enough for each journal implementation to be able to expose its best @@ -35,120 +31,119 @@ Refer to your journal's plugins documentation for details on which queries and s While Akka Persistence Query does not provide actual implementations of ReadJournals, it defines a number of pre-defined query types for the most common query scenarios, that most journals are likely to implement (however they are not required to). -Read Journals -============= +## Read Journals -In order to issue queries one has to first obtain an instance of a ``ReadJournal``. -Read journals are implemented as `Community plugins`_, each targeting a specific datastore (for example Cassandra or JDBC -databases). For example, given a library that provides a ``akka.persistence.query.my-read-journal`` obtaining the related +In order to issue queries one has to first obtain an instance of a `ReadJournal`. +Read journals are implemented as [Community plugins](http://akka.io/community/#plugins-to-akka-persistence-query), each targeting a specific datastore (for example Cassandra or JDBC +databases). For example, given a library that provides a `akka.persistence.query.my-read-journal` obtaining the related journal is as simple as: -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#basic-usage +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #basic-usage } Journal implementers are encouraged to put this identifier in a variable known to the user, such that one can access it via -``readJournalFor[NoopJournal](NoopJournal.identifier)``, however this is not enforced. +`readJournalFor[NoopJournal](NoopJournal.identifier)`, however this is not enforced. -Read journal implementations are available as `Community plugins`_. +Read journal implementations are available as [Community plugins](http://akka.io/community/#plugins-to-akka-persistence-query). +### Predefined queries -Predefined queries ------------------- Akka persistence query comes with a number of query interfaces built in and suggests Journal implementors to implement them according to the semantics described below. It is important to notice that while these query types are very common a journal is not obliged to implement all of them - for example because in a given journal such query would be significantly inefficient. -.. note:: - Refer to the documentation of the :class:`ReadJournal` plugin you are using for a specific list of supported query types. - For example, Journal plugins should document their stream completion strategies. +@@@ note + +Refer to the documentation of the `ReadJournal` plugin you are using for a specific list of supported query types. +For example, Journal plugins should document their stream completion strategies. + +@@@ The predefined queries are: -AllPersistenceIdsQuery and CurrentPersistenceIdsQuery -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### AllPersistenceIdsQuery and CurrentPersistenceIdsQuery -``allPersistenceIds`` which is designed to allow users to subscribe to a stream of all persistent ids in the system. +`allPersistenceIds` which is designed to allow users to subscribe to a stream of all persistent ids in the system. By default this stream should be assumed to be a "live" stream, which means that the journal should keep emitting new persistence ids as they come into the system: -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#all-persistence-ids-live +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #all-persistence-ids-live } -If your usage does not require a live stream, you can use the ``currentPersistenceIds`` query: +If your usage does not require a live stream, you can use the `currentPersistenceIds` query: -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#all-persistence-ids-snap +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #all-persistence-ids-snap } -EventsByPersistenceIdQuery and CurrentEventsByPersistenceIdQuery -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### EventsByPersistenceIdQuery and CurrentEventsByPersistenceIdQuery -``eventsByPersistenceId`` is a query equivalent to replaying a :ref:`PersistentActor `, +`eventsByPersistenceId` is a query equivalent to replaying a @ref:[PersistentActor](persistence.md#event-sourcing-scala), however, since it is a stream it is possible to keep it alive and watch for additional incoming events persisted by the -persistent actor identified by the given ``persistenceId``. +persistent actor identified by the given `persistenceId`. -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#events-by-persistent-id +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #events-by-persistent-id } Most journals will have to revert to polling in order to achieve this, -which can typically be configured with a ``refresh-interval`` configuration property. +which can typically be configured with a `refresh-interval` configuration property. -If your usage does not require a live stream, you can use the ``currentEventsByPersistenceId`` query. +If your usage does not require a live stream, you can use the `currentEventsByPersistenceId` query. -EventsByTag and CurrentEventsByTag -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### EventsByTag and CurrentEventsByTag -``eventsByTag`` allows querying events regardless of which ``persistenceId`` they are associated with. This query is hard to +`eventsByTag` allows querying events regardless of which `persistenceId` they are associated with. This query is hard to implement in some journals or may need some additional preparation of the used data store to be executed efficiently. The goal of this query is to allow querying for all events which are "tagged" with a specific tag. That includes the use case to query all domain events of an Aggregate Root type. Please refer to your read journal plugin's documentation to find out if and how it is supported. -Some journals may support tagging of events via an :ref:`event-adapters-scala` that wraps the events in a -``akka.persistence.journal.Tagged`` with the given ``tags``. The journal may support other ways of doing tagging - again, +Some journals may support tagging of events via an @ref:[Event Adapters](persistence.md#event-adapters-scala) that wraps the events in a +`akka.persistence.journal.Tagged` with the given `tags`. The journal may support other ways of doing tagging - again, how exactly this is implemented depends on the used journal. Here is an example of such a tagging event adapter: -.. includecode:: code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala#tagger +@@snip [LeveldbPersistenceQueryDocSpec.scala](code/docs/persistence/query/LeveldbPersistenceQueryDocSpec.scala) { #tagger } -.. note:: - A very important thing to keep in mind when using queries spanning multiple persistenceIds, such as ``EventsByTag`` - is that the order of events at which the events appear in the stream rarely is guaranteed (or stable between materializations). +@@@ note - Journals *may* choose to opt for strict ordering of the events, and should then document explicitly what kind of ordering - guarantee they provide - for example "*ordered by timestamp ascending, independently of persistenceId*" is easy to achieve - on relational databases, yet may be hard to implement efficiently on plain key-value datastores. +A very important thing to keep in mind when using queries spanning multiple persistenceIds, such as `EventsByTag` +is that the order of events at which the events appear in the stream rarely is guaranteed (or stable between materializations). + +Journals *may* choose to opt for strict ordering of the events, and should then document explicitly what kind of ordering +guarantee they provide - for example "*ordered by timestamp ascending, independently of persistenceId*" is easy to achieve +on relational databases, yet may be hard to implement efficiently on plain key-value datastores. + +@@@ In the example below we query all events which have been tagged (we assume this was performed by the write-side using an -:ref:`EventAdapter `, or that the journal is smart enough that it can figure out what we mean by this -tag - for example if the journal stored the events as json it may try to find those with the field ``tag`` set to this value etc.). +@ref:[EventAdapter](persistence.md#event-adapters-scala), or that the journal is smart enough that it can figure out what we mean by this +tag - for example if the journal stored the events as json it may try to find those with the field `tag` set to this value etc.). -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#events-by-tag +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #events-by-tag } -As you can see, we can use all the usual stream combinators available from :ref:`streams-scala` on the resulting query stream, -including for example taking the first 10 and cancelling the stream. It is worth pointing out that the built-in ``EventsByTag`` -query has an optionally supported offset parameter (of type ``Long``) which the journals can use to implement resumable-streams. +As you can see, we can use all the usual stream combinators available from @ref:[Streams](stream/index.md) on the resulting query stream, +including for example taking the first 10 and cancelling the stream. It is worth pointing out that the built-in `EventsByTag` +query has an optionally supported offset parameter (of type `Long`) which the journals can use to implement resumable-streams. For example a journal may be able to use a WHERE clause to begin the read starting from a specific row, or in a datastore that is able to order events by insertion time it could treat the Long as a timestamp and select only older events. -If your usage does not require a live stream, you can use the ``currentEventsByTag`` query. +If your usage does not require a live stream, you can use the `currentEventsByTag` query. -Materialized values of queries ------------------------------- -Journals are able to provide additional information related to a query by exposing :ref:`materialized-values-quick-scala`, -which are a feature of :ref:`streams-scala` that allows to expose additional values at stream materialization time. +### Materialized values of queries + +Journals are able to provide additional information related to a query by exposing @ref:[Materialized values](stream/stream-quickstart.md#materialized-values-quick-scala), +which are a feature of @ref:[Streams](stream/index.md) that allows to expose additional values at stream materialization time. More advanced query journals may use this technique to expose information about the character of the materialized stream, for example if it's finite or infinite, strictly ordered or not ordered at all. The materialized value type -is defined as the second type parameter of the returned ``Source``, which allows journals to provide users with their +is defined as the second type parameter of the returned `Source`, which allows journals to provide users with their specialised query object, as demonstrated in the sample below: -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#advanced-journal-query-types +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #advanced-journal-query-types } -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#advanced-journal-query-definition +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #advanced-journal-query-definition } -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#advanced-journal-query-usage +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #advanced-journal-query-usage } -.. _Community plugins: http://akka.io/community/#plugins-to-akka-persistence-query +## Performance and denormalization -Performance and denormalization -=============================== -When building systems using :ref:`event-sourcing-scala` and CQRS (`Command & Query Responsibility Segregation`_) techniques +When building systems using @ref:[Event sourcing](persistence.md#event-sourcing-scala) and CQRS ([Command & Query Responsibility Segregation](https://msdn.microsoft.com/en-us/library/jj554200.aspx)) techniques it is tremendously important to realise that the write-side has completely different needs from the read-side, and separating those concerns into datastores that are optimised for either side makes it possible to offer the best experience for the write and read sides independently. @@ -162,104 +157,97 @@ to figure out best bidding strategies and trends – this often requires some ki for example SQL or writing Spark jobs to analyse the data. Therefore the data stored in the write-side needs to be projected into the other read-optimised datastore. -.. note:: - When referring to **Materialized Views** in Akka Persistence think of it as "some persistent storage of the result of a Query". - In other words, it means that the view is created once, in order to be afterwards queried multiple times, as in this format - it may be more efficient or interesting to query it (instead of the source events directly). +@@@ note -Materialize view to Reactive Streams compatible datastore ---------------------------------------------------------- +When referring to **Materialized Views** in Akka Persistence think of it as "some persistent storage of the result of a Query". +In other words, it means that the view is created once, in order to be afterwards queried multiple times, as in this format +it may be more efficient or interesting to query it (instead of the source events directly). -If the read datastore exposes a `Reactive Streams`_ interface then implementing a simple projection +@@@ + +### Materialize view to Reactive Streams compatible datastore + +If the read datastore exposes a [Reactive Streams](http://reactive-streams.org) interface then implementing a simple projection is as simple as, using the read-journal and feeding it into the databases driver interface, for example like so: -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#projection-into-different-store-rs +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #projection-into-different-store-rs } -.. _Reactive Streams: http://reactive-streams.org +### Materialize view using mapAsync -Materialize view using mapAsync -------------------------------- - -If the target database does not provide a reactive streams ``Subscriber`` that can perform writes, +If the target database does not provide a reactive streams `Subscriber` that can perform writes, you may have to implement the write logic using plain functions or Actors instead. In case your write logic is state-less and you just need to convert the events from one data type to another before writing into the alternative datastore, then the projection is as simple as: -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#projection-into-different-store-simple +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #projection-into-different-store-simple } -Resumable projections ---------------------- +### Resumable projections Sometimes you may need to implement "resumable" projections, that will not start from the beginning of time each time -when run. In this case you will need to store the sequence number (or ``offset``) of the processed event and use it +when run. In this case you will need to store the sequence number (or `offset`) of the processed event and use it the next time this projection is started. This pattern is not built-in, however is rather simple to implement yourself. The example below additionally highlights how you would use Actors to implement the write side, in case you need to do some complex logic that would be best handled inside an Actor before persisting the event into the other datastore: -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#projection-into-different-store-actor-run +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #projection-into-different-store-actor-run } -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#projection-into-different-store-actor +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #projection-into-different-store-actor } -.. _Command & Query Responsibility Segregation: https://msdn.microsoft.com/en-us/library/jj554200.aspx + +## Query plugins -.. _read-journal-plugin-api-scala: +Query plugins are various (mostly community driven) `ReadJournal` implementations for all kinds +of available datastores. The complete list of available plugins is maintained on the Akka Persistence Query [Community Plugins](http://akka.io/community/#plugins-to-akka-persistence-query) page. -Query plugins -============= - -Query plugins are various (mostly community driven) :class:`ReadJournal` implementations for all kinds -of available datastores. The complete list of available plugins is maintained on the Akka Persistence Query `Community Plugins`_ page. - -The plugin for LevelDB is described in :ref:`persistence-query-leveldb-scala`. +The plugin for LevelDB is described in @ref:[Persistence Query for LevelDB](persistence-query-leveldb.md). This section aims to provide tips and guide plugin developers through implementing a custom query plugin. Most users will not need to implement journals themselves, except if targeting a not yet supported datastore. -.. note:: - Since different data stores provide different query capabilities journal plugins **must extensively document** - their exposed semantics as well as handled query scenarios. +@@@ note -ReadJournal plugin API ----------------------- +Since different data stores provide different query capabilities journal plugins **must extensively document** +their exposed semantics as well as handled query scenarios. -A read journal plugin must implement ``akka.persistence.query.ReadJournalProvider`` which -creates instances of ``akka.persistence.query.scaladsl.ReadJournal`` and -``akka.persistence.query.javaadsl.ReadJournal``. The plugin must implement both the ``scaladsl`` -and the ``javadsl`` traits because the ``akka.stream.scaladsl.Source`` and -``akka.stream.javadsl.Source`` are different types and even though those types can easily be converted +@@@ + +### ReadJournal plugin API + +A read journal plugin must implement `akka.persistence.query.ReadJournalProvider` which +creates instances of `akka.persistence.query.scaladsl.ReadJournal` and +`akka.persistence.query.javaadsl.ReadJournal`. The plugin must implement both the `scaladsl` +and the `javadsl` traits because the `akka.stream.scaladsl.Source` and +`akka.stream.javadsl.Source` are different types and even though those types can easily be converted to each other it is most convenient for the end user to get access to the Java or Scala directly. As illustrated below one of the implementations can delegate to the other. Below is a simple journal implementation: -.. includecode:: code/docs/persistence/query/PersistenceQueryDocSpec.scala#my-read-journal +@@snip [PersistenceQueryDocSpec.scala](code/docs/persistence/query/PersistenceQueryDocSpec.scala) { #my-read-journal } -And the ``eventsByTag`` could be backed by such an Actor for example: +And the `eventsByTag` could be backed by such an Actor for example: -.. includecode:: code/docs/persistence/query/MyEventsByTagPublisher.scala#events-by-tag-publisher +@@snip [MyEventsByTagPublisher.scala](code/docs/persistence/query/MyEventsByTagPublisher.scala) { #events-by-tag-publisher } -The ``ReadJournalProvider`` class must have a constructor with one of these signatures: +The `ReadJournalProvider` class must have a constructor with one of these signatures: -* constructor with a ``ExtendedActorSystem`` parameter, a ``com.typesafe.config.Config`` parameter, and a ``String`` parameter for the config path -* constructor with a ``ExtendedActorSystem`` parameter, and a ``com.typesafe.config.Config`` parameter -* constructor with one ``ExtendedActorSystem`` parameter -* constructor without parameters + * constructor with a `ExtendedActorSystem` parameter, a `com.typesafe.config.Config` parameter, and a `String` parameter for the config path + * constructor with a `ExtendedActorSystem` parameter, and a `com.typesafe.config.Config` parameter + * constructor with one `ExtendedActorSystem` parameter + * constructor without parameters The plugin section of the actor system's config will be passed in the config constructor parameter. The config path -of the plugin is passed in the ``String`` parameter. +of the plugin is passed in the `String` parameter. If the underlying datastore only supports queries that are completed when they reach the end of the "result set", the journal has to submit new queries after a while in order to support "infinite" event streams that include events stored after the initial query has completed. It is recommended that the plugin use a configuration property named -``refresh-interval`` for defining such a refresh interval. - -Plugin TCK ----------- - -TODO, not available yet. +`refresh-interval` for defining such a refresh interval. +### Plugin TCK +TODO, not available yet. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/persistence-schema-evolution.md b/akka-docs/src/main/paradox/scala/persistence-schema-evolution.md index ae34c58558..cc2f7ecf68 100644 --- a/akka-docs/src/main/paradox/scala/persistence-schema-evolution.md +++ b/akka-docs/src/main/paradox/scala/persistence-schema-evolution.md @@ -1,10 +1,6 @@ -.. _persistence-schema-evolution-scala: +# Persistence - Schema Evolution -############################## -Persistence - Schema Evolution -############################## - -When working on long running projects using :ref:`persistence-scala`, or any kind of `Event Sourcing`_ architectures, +When working on long running projects using @ref:[Persistence](persistence.md), or any kind of [Event Sourcing](http://martinfowler.com/eaaDev/EventSourcing.html) architectures, schema evolution becomes one of the more important technical aspects of developing your application. The requirements as well as our own understanding of the business domain may (and will) change in time. @@ -15,61 +11,59 @@ lifecycle that could mean that no-one is really using it actively. In this chapter we will investigate various schema evolution strategies and techniques from which you can pick and choose the ones that match your domain and challenge at hand. -.. note:: - This page proposes a number of possible solutions to the schema evolution problem and explains how some of the - utilities Akka provides can be used to achieve this, it is by no means a complete (closed) set of solutions. +@@@ note - Sometimes, based on the capabilities of your serialization formats, you may be able to evolve your schema in - different ways than outlined in the sections below. If you discover useful patterns or techniques for schema - evolution feel free to submit Pull Requests to this page to extend it. +This page proposes a number of possible solutions to the schema evolution problem and explains how some of the +utilities Akka provides can be used to achieve this, it is by no means a complete (closed) set of solutions. +Sometimes, based on the capabilities of your serialization formats, you may be able to evolve your schema in +different ways than outlined in the sections below. If you discover useful patterns or techniques for schema +evolution feel free to submit Pull Requests to this page to extend it. -Schema evolution in event-sourced systems -========================================= +@@@ + +## Schema evolution in event-sourced systems In recent years we have observed a tremendous move towards immutable append-only datastores, with event-sourcing being the prime technique successfully being used in these settings. For an excellent overview why and how immutable data makes scalability -and systems design much simpler you may want to read Pat Helland's excellent `Immutability Changes Everything`_ whitepaper. +and systems design much simpler you may want to read Pat Helland's excellent [Immutability Changes Everything](http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf) whitepaper. -Since with `Event Sourcing`_ the **events are immutable** and usually never deleted – the way schema evolution is handled +Since with [Event Sourcing](http://martinfowler.com/eaaDev/EventSourcing.html) the **events are immutable** and usually never deleted – the way schema evolution is handled differs from how one would go about it in a mutable database setting (e.g. in typical CRUD database applications). The system needs to be able to continue to work in the presence of "old" events which were stored under the "old" schema. We also want to limit complexity in the business logic layer, exposing a consistent view over all of the events of a given -type to :class:`PersistentActor` s and :ref:`persistence queries `. This allows the business logic layer to focus on solving business problems +type to `PersistentActor` s and @ref:[persistence queries](persistence-query.md). This allows the business logic layer to focus on solving business problems instead of having to explicitly deal with different schemas. - In summary, schema evolution in event sourced systems exposes the following characteristics: - - Allow the system to continue operating without large scale migrations to be applied, - - Allow the system to read "old" events from the underlying storage, however present them in a "new" view to the application logic, - - Transparently promote events to the latest versions during recovery (or queries) such that the business logic need not consider multiple versions of events +: + * Allow the system to continue operating without large scale migrations to be applied, + * Allow the system to read "old" events from the underlying storage, however present them in a "new" view to the application logic, + * Transparently promote events to the latest versions during recovery (or queries) such that the business logic need not consider multiple versions of events -.. _Immutability Changes Everything: http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf -.. _Event Sourcing: http://martinfowler.com/eaaDev/EventSourcing.html -Types of schema evolution -------------------------- +### Types of schema evolution + Before we explain the various techniques that can be used to safely evolve the schema of your persistent events over time, we first need to define what the actual problem is, and what the typical styles of changes are. Since events are never deleted, we need to have a way to be able to replay (read) old events, in such way -that does not force the ``PersistentActor`` to be aware of all possible versions of an event that it may have +that does not force the `PersistentActor` to be aware of all possible versions of an event that it may have persisted in the past. Instead, we want the Actors to work on some form of "latest" version of the event and provide some means of either converting old "versions" of stored events into this "latest" event type, or constantly evolve the event definition - in a backwards compatible way - such that the new deserialization code can still read old events. The most common schema changes you will likely are: -- :ref:`adding a field to an event type `, -- :ref:`remove or rename field in event type `, -- :ref:`remove event type `, -- :ref:`split event into multiple smaller events `. + * [adding a field to an event type](#add-field-scala), + * [remove or rename field in event type](#rename-field-scala), + * [remove event type](#remove-event-class-scala), + * [split event into multiple smaller events](#split-large-event-into-smaller-scala). The following sections will explain some patterns which can be used to safely evolve your schema when facing those changes. -Picking the right serialization format -====================================== +## Picking the right serialization format Picking the serialization format is a very important decision you will have to make while building your application. It affects which kind of evolutions are simple (or hard) to do, how much work is required to add a new datatype, and, @@ -82,158 +76,151 @@ an event-log from one serialization format to another one, however it may be a m to perform this on a live system. Binary serialization formats that we have seen work well for long-lived applications include the very flexible IDL based: -`Google Protobuf`_, `Apache Thrift`_ or `Apache Avro`_. Avro schema evolution is more "entire schema" based, instead of +[Google Protobuf](https://developers.google.com/protocol-buffers), [Apache Thrift](https://thrift.apache.org/) or [Apache Avro](https://avro.apache.org). Avro schema evolution is more "entire schema" based, instead of single fields focused like in protobuf or thrift, and usually requires using some kind of schema registry. Users who want their data to be human-readable directly in the write-side -datastore may opt to use plain-old `JSON`_ as the storage format, though that comes at a cost of lacking support for schema +datastore may opt to use plain-old [JSON](http://json.org) as the storage format, though that comes at a cost of lacking support for schema evolution and relatively large marshalling latency. There are plenty excellent blog posts explaining the various trade-offs between popular serialization formats, -one post we would like to highlight is the very well illustrated `Schema evolution in Avro, Protocol Buffers and Thrift`_ +one post we would like to highlight is the very well illustrated [Schema evolution in Avro, Protocol Buffers and Thrift](http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html) by Martin Kleppmann. -.. _Google Protobuf: https://developers.google.com/protocol-buffers -.. _Apache Avro: https://avro.apache.org -.. _JSON: http://json.org -.. _Schema evolution in Avro, Protocol Buffers and Thrift: http://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html +### Provided default serializers -Provided default serializers ----------------------------- - -Akka Persistence provides `Google Protocol Buffers`_ based serializers (using :ref:`Akka Serialization `) -for it's own message types such as ``PersistentRepr``, ``AtomicWrite`` and snapshots. Journal plugin implementations +Akka Persistence provides [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) based serializers (using @ref:[Akka Serialization](serialization.md)) +for it's own message types such as `PersistentRepr`, `AtomicWrite` and snapshots. Journal plugin implementations *may* choose to use those provided serializers, or pick a serializer which suits the underlying database better. -.. note:: - Serialization is **NOT** handled automatically by Akka Persistence itself. Instead, it only provides the above described - serializers, and in case a ``AsyncWriteJournal`` plugin implementation chooses to use them directly, the above serialization - scheme will be used. +@@@ note - Please refer to your write journal's documentation to learn more about how it handles serialization! +Serialization is **NOT** handled automatically by Akka Persistence itself. Instead, it only provides the above described +serializers, and in case a `AsyncWriteJournal` plugin implementation chooses to use them directly, the above serialization +scheme will be used. - For example, some journals may choose to not use Akka Serialization *at all* and instead store the data in a format - that is more "native" for the underlying datastore, e.g. using JSON or some other kind of format that the target - datastore understands directly. +Please refer to your write journal's documentation to learn more about how it handles serialization! + +For example, some journals may choose to not use Akka Serialization *at all* and instead store the data in a format +that is more "native" for the underlying datastore, e.g. using JSON or some other kind of format that the target +datastore understands directly. + +@@@ The below figure explains how the default serialization scheme works, and how it fits together with serializing the -user provided message itself, which we will from here on refer to as the ``payload`` (highlighted in yellow): +user provided message itself, which we will from here on refer to as the `payload` (highlighted in yellow): -.. figure:: ../images/persistent-message-envelope.png - :align: center +![persistent-message-envelope.png](../images/persistent-message-envelope.png) +> +Akka Persistence provided serializers wrap the user payload in an envelope containing all persistence-relevant information. +**If the Journal uses provided Protobuf serializers for the wrapper types (e.g. PersistentRepr), then the payload will +be serialized using the user configured serializer, and if none is provided explicitly, Java serialization will be used for it.** - Akka Persistence provided serializers wrap the user payload in an envelope containing all persistence-relevant information. - **If the Journal uses provided Protobuf serializers for the wrapper types (e.g. PersistentRepr), then the payload will - be serialized using the user configured serializer, and if none is provided explicitly, Java serialization will be used for it.** +The blue colored regions of the `PersistentMessage` indicate what is serialized using the generated protocol buffers +serializers, and the yellow payload indicates the user provided event (by calling `persist(payload)(...)`). +As you can see, the `PersistentMessage` acts as an envelope around the payload, adding various fields related to the +origin of the event (`persistenceId`, `sequenceNr` and more). -The blue colored regions of the ``PersistentMessage`` indicate what is serialized using the generated protocol buffers -serializers, and the yellow payload indicates the user provided event (by calling ``persist(payload)(...)``). -As you can see, the ``PersistentMessage`` acts as an envelope around the payload, adding various fields related to the -origin of the event (``persistenceId``, ``sequenceNr`` and more). - -More advanced techniques (e.g. :ref:`remove-event-class-scala`) will dive into using the manifests for increasing the +More advanced techniques (e.g. [Remove event class and ignore events](#remove-event-class-scala)) will dive into using the manifests for increasing the flexibility of the persisted vs. exposed types even more. However for now we will focus on the simpler evolution techniques, concerning simply configuring the payload serializers. -By default the ``payload`` will be serialized using Java Serialization. This is fine for testing and initial phases +By default the `payload` will be serialized using Java Serialization. This is fine for testing and initial phases of your development (while you're still figuring out things and the data will not need to stay persisted forever). However, once you move to production you should really *pick a different serializer for your payloads*. -.. warning:: - Do not rely on Java serialization (which will be picked by Akka by default if you don't specify any serializers) - for *serious* application development! It does not lean itself well to evolving schemas over long periods of time, - and its performance is also not very high (it never was designed for high-throughput scenarios). +@@@ warning -.. _Google Protocol Buffers: https://developers.google.com/protocol-buffers/ -.. _Apache Thrift: https://thrift.apache.org/ +Do not rely on Java serialization (which will be picked by Akka by default if you don't specify any serializers) +for *serious* application development! It does not lean itself well to evolving schemas over long periods of time, +and its performance is also not very high (it never was designed for high-throughput scenarios). -Configuring payload serializers -------------------------------- -This section aims to highlight the complete basics on how to define custom serializers using :ref:`Akka Serialization `. +@@@ + +### Configuring payload serializers + +This section aims to highlight the complete basics on how to define custom serializers using @ref:[Akka Serialization](serialization.md). Many journal plugin implementations use Akka Serialization, thus it is tremendously important to understand how to configure it to work with your event classes. -.. note:: - Read the :ref:`Akka Serialization ` docs to learn more about defining custom serializers, - to improve performance and maintainability of your system. Do not depend on Java serialization for production deployments. +@@@ note + +Read the @ref:[Akka Serialization](serialization.md) docs to learn more about defining custom serializers, +to improve performance and maintainability of your system. Do not depend on Java serialization for production deployments. + +@@@ The below snippet explains in the minimal amount of lines how a custom serializer can be registered. For more in-depth explanations on how serialization picks the serializer to use etc, please refer to its documentation. First we start by defining our domain model class, here representing a person: -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#simplest-custom-serializer-model +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #simplest-custom-serializer-model } -Next we implement a serializer (or extend an existing one to be able to handle the new ``Person`` class): +Next we implement a serializer (or extend an existing one to be able to handle the new `Person` class): -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#simplest-custom-serializer +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #simplest-custom-serializer } -And finally we register the serializer and bind it to handle the ``docs.persistence.Person`` class: +And finally we register the serializer and bind it to handle the `docs.persistence.Person` class: -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#simplest-custom-serializer-config +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #simplest-custom-serializer-config } Deserialization will be performed by the same serializer which serialized the message initially -because of the ``identifier`` being stored together with the message. +because of the `identifier` being stored together with the message. -Please refer to the :ref:`Akka Serialization ` documentation for more advanced use of serializers, -especially the :ref:`string-manifest-serializer-scala` section since it is very useful for Persistence based applications +Please refer to the @ref:[Akka Serialization](serialization.md) documentation for more advanced use of serializers, +especially the @ref:[Serializer with String Manifest](serialization.md#string-manifest-serializer-scala) section since it is very useful for Persistence based applications dealing with schema evolutions, as we will see in some of the examples below. -Schema evolution in action -========================== +## Schema evolution in action In this section we will discuss various schema evolution techniques using concrete examples and explaining some of the various options one might go about handling the described situation. The list below is by no means a complete guide, so feel free to adapt these techniques depending on your serializer's capabilities and/or other domain specific limitations. -.. _add-field-scala: - -Add fields ----------- + +### Add fields **Situation:** -You need to add a field to an existing message type. For example, a ``SeatReserved(letter:String, row:Int)`` now +You need to add a field to an existing message type. For example, a `SeatReserved(letter:String, row:Int)` now needs to have an associated code which indicates if it is a window or aisle seat. **Solution:** Adding fields is the most common change you'll need to apply to your messages so make sure the serialization format you picked for your payloads can handle it apropriately, i.e. such changes should be *binary compatible*. -This is easily achieved using the right serializer toolkit – we recommend something like `Google Protocol Buffers`_ or -`Apache Thrift`_ however other tools may fit your needs just as well – picking a serializer backend is something +This is easily achieved using the right serializer toolkit – we recommend something like [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) or +[Apache Thrift](https://thrift.apache.org/) however other tools may fit your needs just as well – picking a serializer backend is something you should research before picking one to run with. In the following examples we will be using protobuf, mostly because we are familiar with it, it does its job well and Akka is using it internally as well. While being able to read messages with missing fields is half of the solution, you also need to deal with the missing -values somehow. This is usually modeled as some kind of default value, or by representing the field as an ``Option[T]`` +values somehow. This is usually modeled as some kind of default value, or by representing the field as an `Option[T]` See below for an example how reading an optional field from a serialized protocol buffers message might look like. -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#protobuf-read-optional-model +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #protobuf-read-optional-model } Next we prepare an protocol definition using the protobuf Interface Description Language, which we'll use to generate the serializer code to be used on the Akka Serialization layer (notice that the schema aproach allows us to easily rename fields, as long as the numeric identifiers of the fields do not change): -.. includecode:: ../../src/main/protobuf/FlightAppModels.proto#protobuf-read-optional-proto +@@snip [FlightAppModels.proto]../../protobuf/FlightAppModels.proto) { #protobuf-read-optional-proto } The serializer implementation uses the protobuf generated classes to marshall the payloads. -Optional fields can be handled explicitly or missing values by calling the ``has...`` methods on the protobuf object, -which we do for ``seatType`` in order to use a ``Unknown`` type in case the event was stored before we had introduced +Optional fields can be handled explicitly or missing values by calling the `has...` methods on the protobuf object, +which we do for `seatType` in order to use a `Unknown` type in case the event was stored before we had introduced the field to this event type: -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#protobuf-read-optional +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #protobuf-read-optional } -.. _rename-field-scala: - -Rename fields -------------- + +### Rename fields **Situation:** -When first designing the system the ``SeatReserved`` event featured a ``code`` field. -After some time you discover that what was originally called ``code`` actually means ``seatNr``, thus the model +When first designing the system the `SeatReserved` event featured a `code` field. +After some time you discover that what was originally called `code` actually means `seatNr`, thus the model should be changed to reflect this concept more accurately. - **Solution 1 - using IDL based serializers:** First, we will discuss the most efficient way of dealing with such kinds of schema changes – IDL based serializers. @@ -246,57 +233,57 @@ representation of the message. This is one of the advantages of schema based ser add the overhead of having to maintain the schema. When using serializers like this, no additional code change (except renaming the field and method used during serialization) is needed to perform such evolution: -.. figure:: ../images/persistence-serializer-rename.png - :align: center - +![persistence-serializer-rename.png](../images/persistence-serializer-rename.png) +> This is how such a rename would look in protobuf: -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#protobuf-rename-proto +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #protobuf-rename-proto } It is important to learn about the strengths and limitations of your serializers, in order to be able to move swiftly and refactor your models fearlessly as you go on with the project. -.. note:: - Learn in-depth about the serialization engine you're using as it will impact how you can aproach schema evolution. +@@@ note - Some operations are "free" in certain serialization formats (more often than not: removing/adding optional fields, - sometimes renaming fields etc.), while some other operations are strictly not possible. +Learn in-depth about the serialization engine you're using as it will impact how you can aproach schema evolution. + +Some operations are "free" in certain serialization formats (more often than not: removing/adding optional fields, +sometimes renaming fields etc.), while some other operations are strictly not possible. + +@@@ **Solution 2 - by manually handling the event versions:** Another solution, in case your serialization format does not support renames as easily as the above mentioned formats, -is versioning your schema. For example, you could have made your events carry an additional field called ``_version`` -which was set to ``1`` (because it was the initial schema), and once you change the schema you bump this number to ``2``, +is versioning your schema. For example, you could have made your events carry an additional field called `_version` +which was set to `1` (because it was the initial schema), and once you change the schema you bump this number to `2`, and write an adapter which can perform the rename. This approach is popular when your serialization format is something like JSON, where renames can not be performed automatically by the serializer. You can do these kinds of "promotions" either manually (as shown in the example below) -or using a library like `Stamina`_ which helps to create those ``V1->V2->V3->...->Vn`` promotion chains without much boilerplate. +or using a library like [Stamina](https://github.com/scalapenos/stamina) which helps to create those `V1->V2->V3->...->Vn` promotion chains without much boilerplate. -.. figure:: ../images/persistence-manual-rename.png - :align: center +![persistence-manual-rename.png](../images/persistence-manual-rename.png) +> +The following snippet showcases how one could apply renames if working with plain JSON (using `spray.json.JsObject`): -The following snippet showcases how one could apply renames if working with plain JSON (using ``spray.json.JsObject``): - -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#rename-plain-json +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #rename-plain-json } As you can see, manually handling renames induces some boilerplate onto the EventAdapter, however much of it you will find is common infrastructure code that can be either provided by an external library (for promotion management) or put together in a simple helper trait. -.. note:: - The technique of versioning events and then promoting them to the latest version using JSON transformations - can of course be applied to more than just field renames – it also applies to adding fields and all kinds of - changes in the message format. +@@@ note -.. _Stamina: https://github.com/scalapenos/stamina +The technique of versioning events and then promoting them to the latest version using JSON transformations +can of course be applied to more than just field renames – it also applies to adding fields and all kinds of +changes in the message format. -.. _remove-event-class-scala: +@@@ -Remove event class and ignore events ------------------------------------- + +### Remove event class and ignore events **Situation:** -While investigating app performance you notice that insane amounts of ``CustomerBlinked`` events are being stored +While investigating app performance you notice that insane amounts of `CustomerBlinked` events are being stored for every customer each time he/she blinks. Upon investigation you decide that the event does not add any value and should be deleted. You still have to be able to replay from a journal which contains those old CustomerBlinked events though. @@ -304,14 +291,12 @@ and should be deleted. You still have to be able to replay from a journal which The problem of removing an event type from the domain model is not as much its removal, as the implications for the recovery mechanisms that this entails. For example, a naive way of filtering out certain kinds of events from -being delivered to a recovering ``PersistentActor`` is pretty simple, as one can simply filter them out in an :ref:`EventAdapter `: +being delivered to a recovering `PersistentActor` is pretty simple, as one can simply filter them out in an @ref:[EventAdapter](persistence.md#event-adapters-scala): - -.. figure:: ../images/persistence-drop-event.png - :align: center - - The ``EventAdapter`` can drop old events (**O**) by emitting an empty :class:`EventSeq`. - Other events can simply be passed through (**E**). +![persistence-drop-event.png](../images/persistence-drop-event.png) +> +The `EventAdapter` can drop old events (**O**) by emitting an empty `EventSeq`. +Other events can simply be passed through (**E**). This however does not address the underlying cost of having to deserialize all the events during recovery, even those which will be filtered out by the adapter. In the next section we will improve the above explained mechanism @@ -321,42 +306,40 @@ during a recovery containing lots of such events (without actually having to del **Improved solution - deserialize into tombstone:** In the just described technique we have saved the PersistentActor from receiving un-wanted events by filtering them -out in the ``EventAdapter``, however the event itself still was deserialized and loaded into memory. +out in the `EventAdapter`, however the event itself still was deserialized and loaded into memory. This has two notable *downsides*: - - first, that the deserialization was actually performed, so we spent some of out time budget on the - deserialization, even though the event does not contribute anything to the persistent actors state. - - second, that we are *unable to remove the event class* from the system – since the serializer still needs to create - the actuall instance of it, as it does not know it will not be used. +> + * first, that the deserialization was actually performed, so we spent some of out time budget on the +deserialization, even though the event does not contribute anything to the persistent actors state. + * second, that we are *unable to remove the event class* from the system – since the serializer still needs to create +the actuall instance of it, as it does not know it will not be used. The solution to these problems is to use a serializer that is aware of that event being no longer needed, and can notice this before starting to deserialize the object. This aproach allows us to *remove the original class from our classpath*, which makes for less "old" classes lying around in the project. -This can for example be implemented by using an ``SerializerWithStringManifest`` -(documented in depth in :ref:`string-manifest-serializer-scala`). By looking at the string manifest, the serializer can notice +This can for example be implemented by using an `SerializerWithStringManifest` +(documented in depth in @ref:[Serializer with String Manifest](serialization.md#string-manifest-serializer-scala)). By looking at the string manifest, the serializer can notice that the type is no longer needed, and skip the deserialization all-together: -.. figure:: ../images/persistence-drop-event-serializer.png - :align: center - - The serializer is aware of the old event types that need to be skipped (**O**), and can skip deserializing them alltogether - by simply returning a "tombstone" (**T**), which the EventAdapter converts into an empty EventSeq. - Other events (**E**) can simply be passed through. +![persistence-drop-event-serializer.png](../images/persistence-drop-event-serializer.png) +> +The serializer is aware of the old event types that need to be skipped (**O**), and can skip deserializing them alltogether +by simply returning a "tombstone" (**T**), which the EventAdapter converts into an empty EventSeq. +Other events (**E**) can simply be passed through. The serializer detects that the string manifest points to a removed event type and skips attempting to deserialize it: -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#string-serializer-skip-deleved-event-by-manifest +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #string-serializer-skip-deleved-event-by-manifest } -The EventAdapter we implemented is aware of ``EventDeserializationSkipped`` events (our "Tombstones"), -and emits and empty ``EventSeq`` whenever such object is encoutered: +The EventAdapter we implemented is aware of `EventDeserializationSkipped` events (our "Tombstones"), +and emits and empty `EventSeq` whenever such object is encoutered: -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#string-serializer-skip-deleved-event-by-manifest-adapter +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #string-serializer-skip-deleved-event-by-manifest-adapter } -.. _detach-domain-from-data-model-scala: - -Detach domain model from data model ------------------------------------ + +### Detach domain model from data model **Situation:** You want to separate the application model (often called the "*domain model*") completely from the models used to @@ -364,7 +347,7 @@ persist the corresponding events (the "*data model*"). For example because the d independently of the domain model. Another situation where this technique may be useful is when your serialization tool of choice requires generated -classes to be used for serialization and deserialization of objects, like for example `Google Protocol Buffers`_ do, +classes to be used for serialization and deserialization of objects, like for example [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) do, yet you do not want to leak this implementation detail into the domain model itself, which you'd like to model as plain Scala case classes. @@ -374,75 +357,74 @@ classes which very often may be less user-friendly yet highly optimised for thro (like the classes generated by protobuf for example), it is possible to use a simple EventAdapter which maps between these types in a 1:1 style as illustrated below: -.. figure:: ../images/persistence-detach-models.png - :align: center - - Domain events (**A**) are adapted to the data model events (**D**) by the ``EventAdapter``. - The data model can be a format natively understood by the journal, such that it can store it more efficiently or - include additional data for the event (e.g. tags), for ease of later querying. +![persistence-detach-models.png](../images/persistence-detach-models.png) +> +Domain events (**A**) are adapted to the data model events (**D**) by the `EventAdapter`. +The data model can be a format natively understood by the journal, such that it can store it more efficiently or +include additional data for the event (e.g. tags), for ease of later querying. We will use the following domain and data models to showcase how the separation can be implemented by the adapter: -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#detach-models +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #detach-models } -The :class:`EventAdapter` takes care of converting from one model to the other one (in both directions), +The `EventAdapter` takes care of converting from one model to the other one (in both directions), alowing the models to be completely detached from each other, such that they can be optimised independently as long as the mapping logic is able to convert between them: -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#detach-models-adapter +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #detach-models-adapter } The same technique could also be used directly in the Serializer if the end result of marshalling is bytes. Then the serializer can simply convert the bytes do the domain object by using the generated protobuf builders. -.. _store-human-readable-scala: + +### Store events as human-readable data model -Store events as human-readable data model ------------------------------------------ **Situation:** You want to keep your persisted events in a human-readable format, for example JSON. **Solution:** -This is a special case of the :ref:`detach-domain-from-data-model-scala` pattern, and thus requires some co-operation +This is a special case of the [Detach domain model from data model](#detach-domain-from-data-model-scala) pattern, and thus requires some co-operation from the Journal implementation to achieve this. An example of a Journal which may implement this pattern is MongoDB, however other databases such as PostgreSQL and Cassandra could also do it because of their built-in JSON capabilities. -In this aproach, the :class:`EventAdapter` is used as the marshalling layer: it serializes the events to/from JSON. -The journal plugin notices that the incoming event type is JSON (for example by performing a ``match`` on the incoming +In this aproach, the `EventAdapter` is used as the marshalling layer: it serializes the events to/from JSON. +The journal plugin notices that the incoming event type is JSON (for example by performing a `match` on the incoming event) and stores the incoming object directly. -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#detach-models-adapter-json +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #detach-models-adapter-json } -.. note:: - This technique only applies if the Akka Persistence plugin you are using provides this capability. - Check the documentation of your favourite plugin to see if it supports this style of persistence. +@@@ note - If it doesn't, you may want to skim the `list of existing journal plugins`_, just in case some other plugin - for your favourite datastore *does* provide this capability. +This technique only applies if the Akka Persistence plugin you are using provides this capability. +Check the documentation of your favourite plugin to see if it supports this style of persistence. + +If it doesn't, you may want to skim the [list of existing journal plugins](http://akka.io/community/#journal-plugins), just in case some other plugin +for your favourite datastore *does* provide this capability. + +@@@ **Alternative solution:** In fact, an AsyncWriteJournal implementation could natively decide to not use binary serialization at all, -and *always* serialize the incoming messages as JSON - in which case the ``toJournal`` implementation of the -:class:`EventAdapter` would be an identity function, and the ``fromJournal`` would need to de-serialize messages +and *always* serialize the incoming messages as JSON - in which case the `toJournal` implementation of the +`EventAdapter` would be an identity function, and the `fromJournal` would need to de-serialize messages from JSON. -.. note:: - If in need of human-readable events on the *write-side* of your application reconsider whether preparing materialized views - using :ref:`persistence-query-scala` would not be an efficient way to go about this, without compromising the - write-side's throughput characteristics. +@@@ note - If indeed you want to use a human-readable representation on the write-side, pick a Persistence plugin - that provides that functionality, or – implement one yourself. +If in need of human-readable events on the *write-side* of your application reconsider whether preparing materialized views +using @ref:[Persistence Query](persistence-query.md) would not be an efficient way to go about this, without compromising the +write-side's throughput characteristics. +If indeed you want to use a human-readable representation on the write-side, pick a Persistence plugin +that provides that functionality, or – implement one yourself. -.. _list of existing journal plugins: http://akka.io/community/#journal-plugins +@@@ -.. _split-large-event-into-smaller-scala: - -Split large event into fine-grained events ------------------------------------------- + +### Split large event into fine-grained events **Situation:** While refactoring your domain events, you find that one of the events has become too large (coarse-grained) @@ -453,20 +435,19 @@ Let us consider a situation where an event represents "user details changed". Af event is too coarse, and needs to be split into "user name changed" and "user address changed", because somehow users keep changing their usernames a lot and we'd like to keep this as a separate event. -The write side change is very simple, we simply persist ``UserNameChanged`` or ``UserAddressChanged`` depending -on what the user actually intended to change (instead of the composite ``UserDetailsChanged`` that we had in version 1 +The write side change is very simple, we simply persist `UserNameChanged` or `UserAddressChanged` depending +on what the user actually intended to change (instead of the composite `UserDetailsChanged` that we had in version 1 of our model). -.. figure:: ../images/persistence-event-adapter-1-n.png - :align: center +![persistence-event-adapter-1-n.png](../images/persistence-event-adapter-1-n.png) +> +The `EventAdapter` splits the incoming event into smaller more fine grained events during recovery. - The ``EventAdapter`` splits the incoming event into smaller more fine grained events during recovery. - -During recovery however, we now need to convert the old ``V1`` model into the ``V2`` representation of the change. -Depending if the old event contains a name change, we either emit the ``UserNameChanged`` or we don't, +During recovery however, we now need to convert the old `V1` model into the `V2` representation of the change. +Depending if the old event contains a name change, we either emit the `UserNameChanged` or we don't, and the address change is handled similarily: -.. includecode:: code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala#split-events-during-recovery +@@snip [PersistenceSchemaEvolutionDocSpec.scala](code/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala) { #split-events-during-recovery } -By returning an :class:`EventSeq` from the event adapter, the recovered event can be converted to multiple events before -being delivered to the persistent actor. +By returning an `EventSeq` from the event adapter, the recovered event can be converted to multiple events before +being delivered to the persistent actor. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/persistence.md b/akka-docs/src/main/paradox/scala/persistence.md index b094b91334..9fa290dd95 100644 --- a/akka-docs/src/main/paradox/scala/persistence.md +++ b/akka-docs/src/main/paradox/scala/persistence.md @@ -1,8 +1,4 @@ -.. _persistence-scala: - -########### -Persistence -########### +# Persistence Akka persistence enables stateful actors to persist their internal state so that it can be recovered when an actor is started, restarted after a JVM crash or by a supervisor, or migrated in a cluster. The key concept behind Akka @@ -13,56 +9,49 @@ changes to these actors from which they can rebuild internal state. This can be or starting from a snapshot which can dramatically reduce recovery times. Akka persistence also provides point-to-point communication with at-least-once message delivery semantics. -Akka persistence is inspired by and the official replacement of the `eventsourced`_ library. It follows the same -concepts and architecture of `eventsourced`_ but significantly differs on API and implementation level. See also -:ref:`migration-eventsourced-2.3` +Akka persistence is inspired by and the official replacement of the [eventsourced](https://github.com/eligosource/eventsourced) library. It follows the same +concepts and architecture of [eventsourced](https://github.com/eligosource/eventsourced) but significantly differs on API and implementation level. See also +@ref:[migration-eventsourced-2.3](../project/migration-guide-eventsourced-2.3.x.md) -.. _eventsourced: https://github.com/eligosource/eventsourced +## Dependencies -Dependencies -============ +Akka persistence is a separate jar file. Make sure that you have the following dependency in your project: -Akka persistence is a separate jar file. Make sure that you have the following dependency in your project:: - - "com.typesafe.akka" %% "akka-persistence" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-persistence" % "@version@" @crossString@ +``` The Akka persistence extension comes with few built-in persistence plugins, including in-memory heap based journal, local file-system based snapshot-store and LevelDB based journal. -LevelDB based plugins will require the following additional dependency declaration:: +LevelDB based plugins will require the following additional dependency declaration: - "org.iq80.leveldb" % "leveldb" % "0.7" - "org.fusesource.leveldbjni" % "leveldbjni-all" % "1.8" +``` +"org.iq80.leveldb" % "leveldb" % "0.7" +"org.fusesource.leveldbjni" % "leveldbjni-all" % "1.8" +``` -Architecture -============ +## Architecture -* *PersistentActor*: Is a persistent, stateful actor. It is able to persist events to a journal and can react to - them in a thread-safe manner. It can be used to implement both *command* as well as *event sourced* actors. - When a persistent actor is started or restarted, journaled messages are replayed to that actor so that it can - recover internal state from these messages. + * *PersistentActor*: Is a persistent, stateful actor. It is able to persist events to a journal and can react to +them in a thread-safe manner. It can be used to implement both *command* as well as *event sourced* actors. +When a persistent actor is started or restarted, journaled messages are replayed to that actor so that it can +recover internal state from these messages. + * *AtLeastOnceDelivery*: To send messages with at-least-once delivery semantics to destinations, also in +case of sender and receiver JVM crashes. + * *AsyncWriteJournal*: A journal stores the sequence of messages sent to a persistent actor. An application can control which messages +are journaled and which are received by the persistent actor without being journaled. Journal maintains *highestSequenceNr* that is increased on each message. +The storage backend of a journal is pluggable. The persistence extension comes with a "leveldb" journal plugin, which writes to the local filesystem. +Replicated journals are available as [Community plugins](http://akka.io/community/). + * *Snapshot store*: A snapshot store persists snapshots of a persistent actor's or a view's internal state. Snapshots are +used for optimizing recovery times. The storage backend of a snapshot store is pluggable. +The persistence extension comes with a "local" snapshot storage plugin, which writes to the local filesystem. +Replicated snapshot stores are available as [Community plugins](http://akka.io/community/). -* *AtLeastOnceDelivery*: To send messages with at-least-once delivery semantics to destinations, also in - case of sender and receiver JVM crashes. + +## Event sourcing -* *AsyncWriteJournal*: A journal stores the sequence of messages sent to a persistent actor. An application can control which messages - are journaled and which are received by the persistent actor without being journaled. Journal maintains *highestSequenceNr* that is increased on each message. - The storage backend of a journal is pluggable. The persistence extension comes with a "leveldb" journal plugin, which writes to the local filesystem. - Replicated journals are available as `Community plugins`_. - -* *Snapshot store*: A snapshot store persists snapshots of a persistent actor's or a view's internal state. Snapshots are - used for optimizing recovery times. The storage backend of a snapshot store is pluggable. - The persistence extension comes with a "local" snapshot storage plugin, which writes to the local filesystem. - Replicated snapshot stores are available as `Community plugins`_. - -.. _Community plugins: http://akka.io/community/ - -.. _event-sourcing-scala: - -Event sourcing -============== - -The basic idea behind `Event Sourcing`_ is quite simple. A persistent actor receives a (non-persistent) command +The basic idea behind [Event Sourcing](http://martinfowler.com/eaaDev/EventSourcing.html) is quite simple. A persistent actor receives a (non-persistent) command which is first validated if it can be applied to the current state. Here validation can mean anything, from simple inspection of a command message's fields up to a conversation with several external services, for example. If validation succeeds, events are generated from the command, representing the effect of the command. These events @@ -71,23 +60,21 @@ needs to be recovered, only the persisted events are replayed of which we know t In other words, events cannot fail when being replayed to a persistent actor, in contrast to commands. Event sourced actors may of course also process commands that do not change application state such as query commands for example. -.. _Event Sourcing: http://martinfowler.com/eaaDev/EventSourcing.html +Akka persistence supports event sourcing with the `PersistentActor` trait. An actor that extends this trait uses the +`persist` method to persist and handle events. The behavior of a `PersistentActor` +is defined by implementing `receiveRecover` and `receiveCommand`. This is demonstrated in the following example. -Akka persistence supports event sourcing with the ``PersistentActor`` trait. An actor that extends this trait uses the -``persist`` method to persist and handle events. The behavior of a ``PersistentActor`` -is defined by implementing ``receiveRecover`` and ``receiveCommand``. This is demonstrated in the following example. +@@snip [PersistentActorExample.scala](code/docs/persistence/PersistentActorExample.scala) { #persistent-actor-example } -.. includecode:: code/docs/persistence/PersistentActorExample.scala#persistent-actor-example +The example defines two data types, `Cmd` and `Evt` to represent commands and events, respectively. The +`state` of the `ExamplePersistentActor` is a list of persisted event data contained in `ExampleState`. -The example defines two data types, ``Cmd`` and ``Evt`` to represent commands and events, respectively. The -``state`` of the ``ExamplePersistentActor`` is a list of persisted event data contained in ``ExampleState``. - -The persistent actor's ``receiveRecover`` method defines how ``state`` is updated during recovery by handling ``Evt`` -and ``SnapshotOffer`` messages. The persistent actor's ``receiveCommand`` method is a command handler. In this example, +The persistent actor's `receiveRecover` method defines how `state` is updated during recovery by handling `Evt` +and `SnapshotOffer` messages. The persistent actor's `receiveCommand` method is a command handler. In this example, a command is handled by generating an event which is then persisted and handled. Events are persisted by calling -``persist`` with an event (or a sequence of events) as first argument and an event handler as second argument. +`persist` with an event (or a sequence of events) as first argument and an event handler as second argument. -The ``persist`` method persists events asynchronously and the event handler is executed for successfully persisted +The `persist` method persists events asynchronously and the event handler is executed for successfully persisted events. Successfully persisted events are internally sent back to the persistent actor as individual messages that trigger event handler executions. An event handler may close over persistent actor state and mutate it. The sender of a persisted event is the sender of the corresponding command. This allows event handlers to reply to the sender of a command @@ -96,50 +83,51 @@ event is the sender of the corresponding command. This allows event handlers to The main responsibility of an event handler is changing persistent actor state using event data and notifying others about successful state changes by publishing events. -When persisting events with ``persist`` it is guaranteed that the persistent actor will not receive further commands between -the ``persist`` call and the execution(s) of the associated event handler. This also holds for multiple ``persist`` -calls in context of a single command. Incoming messages are :ref:`stashed ` until the ``persist`` +When persisting events with `persist` it is guaranteed that the persistent actor will not receive further commands between +the `persist` call and the execution(s) of the associated event handler. This also holds for multiple `persist` +calls in context of a single command. Incoming messages are [stashed](#internal-stash-scala) until the `persist` is completed. -If persistence of an event fails, ``onPersistFailure`` will be invoked (logging the error by default), +If persistence of an event fails, `onPersistFailure` will be invoked (logging the error by default), and the actor will unconditionally be stopped. If persistence of an event is rejected before it is -stored, e.g. due to serialization error, ``onPersistRejected`` will be invoked (logging a warning +stored, e.g. due to serialization error, `onPersistRejected` will be invoked (logging a warning by default) and the actor continues with the next message. The easiest way to run this example yourself is to download the ready to run -`Akka Persistence Sample with Scala <@exampleCodeService@/akka-samples-persistence-scala>`_ -together with the tutorial. It contains instructions on how to run the ``PersistentActorExample``. -The source code of this sample can be found in the `Akka Samples Repository <@samples@/akka-sample-persistence-scala>`_. +[Akka Persistence Sample with Scala](@exampleCodeService@/akka-samples-persistence-scala) +together with the tutorial. It contains instructions on how to run the `PersistentActorExample`. +The source code of this sample can be found in the [Akka Samples Repository](@samples@/akka-sample-persistence-scala). -.. note:: +@@@ note - It's also possible to switch between different command handlers during normal processing and recovery - with ``context.become()`` and ``context.unbecome()``. To get the actor into the same state after - recovery you need to take special care to perform the same state transitions with ``become`` and - ``unbecome`` in the ``receiveRecover`` method as you would have done in the command handler. - Note that when using ``become`` from ``receiveRecover`` it will still only use the ``receiveRecover`` - behavior when replaying the events. When replay is completed it will use the new behavior. +It's also possible to switch between different command handlers during normal processing and recovery +with `context.become()` and `context.unbecome()`. To get the actor into the same state after +recovery you need to take special care to perform the same state transitions with `become` and +`unbecome` in the `receiveRecover` method as you would have done in the command handler. +Note that when using `become` from `receiveRecover` it will still only use the `receiveRecover` +behavior when replaying the events. When replay is completed it will use the new behavior. -.. _persistence-id-scala: +@@@ -Identifiers ------------ + +### Identifiers A persistent actor must have an identifier that doesn't change across different actor incarnations. -The identifier must be defined with the ``persistenceId`` method. +The identifier must be defined with the `persistenceId` method. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#persistence-id-override +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #persistence-id-override } -.. note:: - ``persistenceId`` must be unique to a given entity in the journal (database table/keyspace). - When replaying messages persisted to the journal, you query messages with a ``persistenceId``. - So, if two different entities share the same ``persistenceId``, message-replaying - behavior is corrupted. +@@@ note -.. _recovery-scala: +`persistenceId` must be unique to a given entity in the journal (database table/keyspace). +When replaying messages persisted to the journal, you query messages with a `persistenceId`. +So, if two different entities share the same `persistenceId`, message-replaying +behavior is corrupted. -Recovery --------- +@@@ + + +### Recovery By default, a persistent actor is automatically recovered on start and on restart by replaying journaled messages. New messages sent to a persistent actor during recovery do not interfere with replayed messages. @@ -147,28 +135,31 @@ They are stashed and received by a persistent actor after recovery phase complet The number of concurrent recoveries of recoveries that can be in progress at the same time is limited to not overload the system and the backend data store. When exceeding the limit the actors will wait -until other recoveries have been completed. This is configured by:: +until other recoveries have been completed. This is configured by: - akka.persistence.max-concurrent-recoveries = 50 +``` +akka.persistence.max-concurrent-recoveries = 50 +``` -.. note:: - Accessing the ``sender()`` for replayed messages will always result in a ``deadLetters`` reference, - as the original sender is presumed to be long gone. If you indeed have to notify an actor during - recovery in the future, store its ``ActorPath`` explicitly in your persisted events. +@@@ note -.. _recovery-custom-scala: +Accessing the `sender()` for replayed messages will always result in a `deadLetters` reference, +as the original sender is presumed to be long gone. If you indeed have to notify an actor during +recovery in the future, store its `ActorPath` explicitly in your persisted events. -Recovery customization -^^^^^^^^^^^^^^^^^^^^^^ +@@@ -Applications may also customise how recovery is performed by returning a customised ``Recovery`` object -in the ``recovery`` method of a ``PersistentActor``, + +#### Recovery customization -To skip loading snapshots and replay all events you can use ``SnapshotSelectionCriteria.None``. +Applications may also customise how recovery is performed by returning a customised `Recovery` object +in the `recovery` method of a `PersistentActor`, + +To skip loading snapshots and replay all events you can use `SnapshotSelectionCriteria.None`. This can be useful if snapshot serialization format has changed in an incompatible way. It should typically not be used when events have been deleted. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#recovery-no-snap +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #recovery-no-snap } Another example, which can be fun for experiments but probably not in a real application, is setting an upper bound to the replay which allows the actor to be replayed to a certain point "in the past" @@ -176,217 +167,225 @@ instead to its most up to date state. Note that after that it is a bad idea to p events because a later recovery will probably be confused by the new events that follow the events that were previously skipped. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#recovery-custom +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #recovery-custom } -Recovery can be disabled by returning ``Recovery.none()`` in the ``recovery`` method of a ``PersistentActor``: +Recovery can be disabled by returning `Recovery.none()` in the `recovery` method of a `PersistentActor`: -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#recovery-disabled +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #recovery-disabled } -Recovery status -^^^^^^^^^^^^^^^ +#### Recovery status A persistent actor can query its own recovery status via the methods -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#recovery-status +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #recovery-status } Sometimes there is a need for performing additional initialization when the recovery has completed before processing any other message sent to the persistent actor. -The persistent actor will receive a special :class:`RecoveryCompleted` message right after recovery +The persistent actor will receive a special `RecoveryCompleted` message right after recovery and before any other received messages. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#recovery-completed +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #recovery-completed } -The actor will always receive a :class:`RecoveryCompleted` message, even if there are no events +The actor will always receive a `RecoveryCompleted` message, even if there are no events in the journal and the snapshot store is empty, or if it's a new persistent actor with a previously -unused ``persistenceId``. +unused `persistenceId`. -If there is a problem with recovering the state of the actor from the journal, ``onRecoveryFailure`` +If there is a problem with recovering the state of the actor from the journal, `onRecoveryFailure` is called (logging the error by default) and the actor will be stopped. -.. _internal-stash-scala: + +### Internal stash -Internal stash --------------- - -The persistent actor has a private :ref:`stash ` for internally caching incoming messages during -:ref:`recovery ` or the ``persist\persistAll`` method persisting events. You can still use/inherit from the -``Stash`` interface. The internal stash cooperates with the normal stash by hooking into ``unstashAll`` method and +The persistent actor has a private @ref:[stash](actors.md#stash-scala) for internally caching incoming messages during +[recovery](#recovery-scala) or the `persist\persistAll` method persisting events. You can still use/inherit from the +`Stash` interface. The internal stash cooperates with the normal stash by hooking into `unstashAll` method and making sure messages are unstashed properly to the internal stash to maintain ordering guarantees. You should be careful to not send more messages to a persistent actor than it can keep up with, otherwise the number -of stashed messages will grow without bounds. It can be wise to protect against ``OutOfMemoryError`` by defining a -maximum stash capacity in the mailbox configuration:: +of stashed messages will grow without bounds. It can be wise to protect against `OutOfMemoryError` by defining a +maximum stash capacity in the mailbox configuration: - akka.actor.default-mailbox.stash-capacity=10000 +``` +akka.actor.default-mailbox.stash-capacity=10000 +``` Note that the stash capacity is per actor. If you have many persistent actors, e.g. when using cluster sharding, you may need to define a small stash capacity to ensure that the total number of stashed messages in the system doesn't consume too much memory. Additionally, the persistent actor defines three strategies to handle failure when the -internal stash capacity is exceeded. The default overflow strategy is the ``ThrowOverflowExceptionStrategy``, which -discards the current received message and throws a ``StashOverflowException``, causing actor restart if the default -supervision strategy is used. You can override the ``internalStashOverflowStrategy`` method to return -``DiscardToDeadLetterStrategy`` or ``ReplyToStrategy`` for any "individual" persistent actor, or define the "default" -for all persistent actors by providing FQCN, which must be a subclass of ``StashOverflowStrategyConfigurator``, in the -persistence configuration:: +internal stash capacity is exceeded. The default overflow strategy is the `ThrowOverflowExceptionStrategy`, which +discards the current received message and throws a `StashOverflowException`, causing actor restart if the default +supervision strategy is used. You can override the `internalStashOverflowStrategy` method to return +`DiscardToDeadLetterStrategy` or `ReplyToStrategy` for any "individual" persistent actor, or define the "default" +for all persistent actors by providing FQCN, which must be a subclass of `StashOverflowStrategyConfigurator`, in the +persistence configuration: - akka.persistence.internal-stash-overflow-strategy= - "akka.persistence.ThrowExceptionConfigurator" +``` +akka.persistence.internal-stash-overflow-strategy= + "akka.persistence.ThrowExceptionConfigurator" +``` -The ``DiscardToDeadLetterStrategy`` strategy also has a pre-packaged companion configurator -``akka.persistence.DiscardConfigurator``. +The `DiscardToDeadLetterStrategy` strategy also has a pre-packaged companion configurator +`akka.persistence.DiscardConfigurator`. -You can also query the default strategy via the Akka persistence extension singleton:: +You can also query the default strategy via the Akka persistence extension singleton: - Persistence(context.system).defaultInternalStashOverflowStrategy +``` +Persistence(context.system).defaultInternalStashOverflowStrategy +``` -.. note:: - The bounded mailbox should be avoided in the persistent actor, by which the messages come from storage backends may - be discarded. You can use bounded stash instead of it. +@@@ note -.. _persist-async-scala: +The bounded mailbox should be avoided in the persistent actor, by which the messages come from storage backends may +be discarded. You can use bounded stash instead of it. -Relaxed local consistency requirements and high throughput use-cases --------------------------------------------------------------------- +@@@ -If faced with relaxed local consistency requirements and high throughput demands sometimes ``PersistentActor`` and its -``persist`` may not be enough in terms of consuming incoming Commands at a high rate, because it has to wait until all + +### Relaxed local consistency requirements and high throughput use-cases + +If faced with relaxed local consistency requirements and high throughput demands sometimes `PersistentActor` and its +`persist` may not be enough in terms of consuming incoming Commands at a high rate, because it has to wait until all Events related to a given Command are processed in order to start processing the next Command. While this abstraction is very useful for most cases, sometimes you may be faced with relaxed requirements about consistency – for example you may want to process commands as fast as you can, assuming that the Event will eventually be persisted and handled properly in the background, retroactively reacting to persistence failures if needed. -The ``persistAsync`` method provides a tool for implementing high-throughput persistent actors. It will *not* +The `persistAsync` method provides a tool for implementing high-throughput persistent actors. It will *not* stash incoming Commands while the Journal is still working on persisting and/or user code is executing event callbacks. In the below example, the event callbacks may be called "at any time", even after the next Command has been processed. The ordering between events is still guaranteed ("evt-b-1" will be sent after "evt-a-2", which will be sent after "evt-a-1" etc.). -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#persist-async +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #persist-async } -.. note:: - In order to implement the pattern known as "*command sourcing*" simply call ``persistAsync(cmd)(...)`` right away on all incoming - messages and handle them in the callback. +@@@ note -.. warning:: - The callback will not be invoked if the actor is restarted (or stopped) in between the call to - ``persistAsync`` and the journal has confirmed the write. +In order to implement the pattern known as "*command sourcing*" simply call `persistAsync(cmd)(...)` right away on all incoming +messages and handle them in the callback. -.. _defer-scala: +@@@ -Deferring actions until preceding persist handlers have executed ----------------------------------------------------------------- +@@@ warning -Sometimes when working with ``persistAsync`` or ``persist`` you may find that it would be nice to define some actions in terms of -''happens-after the previous ``persistAsync``/``persist`` handlers have been invoked''. ``PersistentActor`` provides an utility method -called ``deferAsync``, which works similarly to ``persistAsync`` yet does not persist the passed in event. It is recommended to +The callback will not be invoked if the actor is restarted (or stopped) in between the call to +`persistAsync` and the journal has confirmed the write. + +@@@ + + +### Deferring actions until preceding persist handlers have executed + +Sometimes when working with `persistAsync` or `persist` you may find that it would be nice to define some actions in terms of +''happens-after the previous `persistAsync`/`persist` handlers have been invoked''. `PersistentActor` provides an utility method +called `deferAsync`, which works similarly to `persistAsync` yet does not persist the passed in event. It is recommended to use it for *read* operations, and actions which do not have corresponding events in your domain model. Using this method is very similar to the persist family of methods, yet it does **not** persist the passed in event. It will be kept in memory and used when invoking the handler. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#defer +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #defer } -Notice that the ``sender()`` is **safe** to access in the handler callback, and will be pointing to the original sender -of the command for which this ``deferAsync`` handler was called. +Notice that the `sender()` is **safe** to access in the handler callback, and will be pointing to the original sender +of the command for which this `deferAsync` handler was called. The calling side will get the responses in this (guaranteed) order: -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#defer-caller +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #defer-caller } -You can also call ``deferAsync`` with ``persist``. +You can also call `deferAsync` with `persist`. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#defer-with-persist +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #defer-with-persist } -.. warning:: - The callback will not be invoked if the actor is restarted (or stopped) in between the call to - ``deferAsync`` and the journal has processed and confirmed all preceding writes. +@@@ warning -.. _nested-persist-calls-scala: +The callback will not be invoked if the actor is restarted (or stopped) in between the call to +`deferAsync` and the journal has processed and confirmed all preceding writes. -Nested persist calls --------------------- -It is possible to call ``persist`` and ``persistAsync`` inside their respective callback blocks and they will properly -retain both the thread safety (including the right value of ``sender()``) as well as stashing guarantees. +@@@ + + +### Nested persist calls + +It is possible to call `persist` and `persistAsync` inside their respective callback blocks and they will properly +retain both the thread safety (including the right value of `sender()`) as well as stashing guarantees. In general it is encouraged to create command handlers which do not need to resort to nested event persisting, however there are situations where it may be useful. It is important to understand the ordering of callback execution in -those situations, as well as their implication on the stashing behaviour (that ``persist()`` enforces). In the following +those situations, as well as their implication on the stashing behaviour (that `persist()` enforces). In the following example two persist calls are issued, and each of them issues another persist inside its callback: -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#nested-persist-persist +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #nested-persist-persist } -When sending two commands to this ``PersistentActor``, the persist handlers will be executed in the following order: +When sending two commands to this `PersistentActor`, the persist handlers will be executed in the following order: -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#nested-persist-persist-caller +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #nested-persist-persist-caller } First the "outer layer" of persist calls is issued and their callbacks are applied. After these have successfully completed, the inner callbacks will be invoked (once the events they are persisting have been confirmed to be persisted by the journal). Only after all these handlers have been successfully invoked will the next command be delivered to the persistent Actor. -In other words, the stashing of incoming commands that is guaranteed by initially calling ``persist()`` on the outer layer -is extended until all nested ``persist`` callbacks have been handled. +In other words, the stashing of incoming commands that is guaranteed by initially calling `persist()` on the outer layer +is extended until all nested `persist` callbacks have been handled. -It is also possible to nest ``persistAsync`` calls, using the same pattern: +It is also possible to nest `persistAsync` calls, using the same pattern: -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#nested-persistAsync-persistAsync +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #nested-persistAsync-persistAsync } In this case no stashing is happening, yet events are still persisted and callbacks are executed in the expected order: -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#nested-persistAsync-persistAsync-caller +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #nested-persistAsync-persistAsync-caller } -While it is possible to nest mixed ``persist`` and ``persistAsync`` with keeping their respective semantics +While it is possible to nest mixed `persist` and `persistAsync` with keeping their respective semantics it is not a recommended practice, as it may lead to overly complex nesting. -.. warning:: - While it is possible to nest ``persist`` calls within one another, - it is *not* legal call ``persist`` from any other Thread than the Actors message processing Thread. - For example, it is not legal to call ``persist`` from Futures! Doing so will break the guarantees - that the persist methods aim to provide. Always call ``persist`` and ``persistAsync`` from within - the Actor's receive block (or methods synchronously invoked from there). +@@@ warning -.. _failures-scala: +While it is possible to nest `persist` calls within one another, +it is *not* legal call `persist` from any other Thread than the Actors message processing Thread. +For example, it is not legal to call `persist` from Futures! Doing so will break the guarantees +that the persist methods aim to provide. Always call `persist` and `persistAsync` from within +the Actor's receive block (or methods synchronously invoked from there). -Failures --------- +@@@ -If persistence of an event fails, ``onPersistFailure`` will be invoked (logging the error by default), + +### Failures + +If persistence of an event fails, `onPersistFailure` will be invoked (logging the error by default), and the actor will unconditionally be stopped. The reason that it cannot resume when persist fails is that it is unknown if the event was actually persisted or not, and therefore it is in an inconsistent state. Restarting on persistent failures will most likely fail anyway since the journal is probably unavailable. It is better to stop the -actor and after a back-off timeout start it again. The ``akka.pattern.BackoffSupervisor`` actor +actor and after a back-off timeout start it again. The `akka.pattern.BackoffSupervisor` actor is provided to support such restarts. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#backoff +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #backoff } If persistence of an event is rejected before it is stored, e.g. due to serialization error, -``onPersistRejected`` will be invoked (logging a warning by default), and the actor continues with +`onPersistRejected` will be invoked (logging a warning by default), and the actor continues with next message. If there is a problem with recovering the state of the actor from the journal when the actor is -started, ``onRecoveryFailure`` is called (logging the error by default), and the actor will be stopped. +started, `onRecoveryFailure` is called (logging the error by default), and the actor will be stopped. Note that failure to load snapshot is also treated like this, but you can disable loading of snapshots -if you for example know that serialization format has changed in an incompatible way, see :ref:`recovery-custom-scala`. +if you for example know that serialization format has changed in an incompatible way, see [Recovery customization](#recovery-custom-scala). -Atomic writes -------------- +### Atomic writes Each event is of course stored atomically, but it is also possible to store several events atomically by -using the ``persistAll`` or ``persistAllAsync`` method. That means that all events passed to that method +using the `persistAll` or `persistAllAsync` method. That means that all events passed to that method are stored or none of them are stored if there is an error. The recovery of a persistent actor will therefore never be done partially with only a subset of events persisted by -`persistAll`. +*persistAll*. -Some journals may not support atomic writes of several events and they will then reject the ``persistAll`` -command, i.e. ``onPersistRejected`` is called with an exception (typically ``UnsupportedOperationException``). +Some journals may not support atomic writes of several events and they will then reject the `persistAll` +command, i.e. `onPersistRejected` is called with an exception (typically `UnsupportedOperationException`). -.. _batch-writes: + +### Batch writes -Batch writes ------------- - -In order to optimize throughput when using ``persistAsync``, a persistent actor +In order to optimize throughput when using `persistAsync`, a persistent actor internally batches events to be stored under high load before writing them to the journal (as a single batch). The batch size is dynamically determined by how many events are emitted during the time of a journal round-trip: after @@ -394,466 +393,462 @@ sending a batch to the journal no further batch can be sent before confirmation has been received that the previous batch has been written. Batch writes are never timer-based which keeps latencies at a minimum. -Message deletion ----------------- +### Message deletion It is possible to delete all messages (journaled by a single persistent actor) up to a specified sequence number; -Persistent actors may call the ``deleteMessages`` method to this end. +Persistent actors may call the `deleteMessages` method to this end. Deleting messages in event sourcing based applications is typically either not used at all, or used in conjunction with -:ref:`snapshotting `, i.e. after a snapshot has been successfully stored, a ``deleteMessages(toSequenceNr)`` +[snapshotting](#snapshots), i.e. after a snapshot has been successfully stored, a `deleteMessages(toSequenceNr)` up until the sequence number of the data held by that snapshot can be issued to safely delete the previous events while still having access to the accumulated state during replays - by loading the snapshot. -.. warning:: - If you are using :ref:`persistence-query-scala`, query results may be missing deleted messages in a journal, - depending on how deletions are implemented in the journal plugin. - Unless you use a plugin which still shows deleted messages in persistence query results, - you have to design your application so that it is not affected by missing messages. +@@@ warning -The result of the ``deleteMessages`` request is signaled to the persistent actor with a ``DeleteMessagesSuccess`` -message if the delete was successful or a ``DeleteMessagesFailure`` message if it failed. +If you are using @ref:[Persistence Query](persistence-query.md), query results may be missing deleted messages in a journal, +depending on how deletions are implemented in the journal plugin. +Unless you use a plugin which still shows deleted messages in persistence query results, +you have to design your application so that it is not affected by missing messages. -Message deletion doesn't affect the highest sequence number of the journal, even if all messages were deleted from it after ``deleteMessages`` invocation. +@@@ + +The result of the `deleteMessages` request is signaled to the persistent actor with a `DeleteMessagesSuccess` +message if the delete was successful or a `DeleteMessagesFailure` message if it failed. + +Message deletion doesn't affect the highest sequence number of the journal, even if all messages were deleted from it after `deleteMessages` invocation. + +### Persistence status handling -Persistence status handling ---------------------------- Persisting, deleting, and replaying messages can either succeed or fail. -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ -| **Method** | **Success** | **Failure / Rejection** | **After failure handler invoked** | -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ -| ``persist`` / ``persistAsync`` | persist handler invoked | ``onPersistFailure`` | Actor is stopped. | -| | +-------------------------------+-----------------------------------+ -| | | ``onPersistRejected`` | No automatic actions. | -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ -| ``recovery`` | ``RecoveryCompleted`` | ``onRecoveryFailure`` | Actor is stopped. | -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ -| ``deleteMessages`` | ``DeleteMessagesSuccess`` | ``DeleteMessagesFailure`` | No automatic actions. | -+---------------------------------+-----------------------------+-------------------------------+-----------------------------------+ +|**Method** | **Success** | +|`persist` / `persistAsync` | persist handler invoked| +|`onPersistRejected` | No automatic actions. | +|`recovery` | `RecoveryCompleted` | +|`deleteMessages` | `DeleteMessagesSuccess`| -The most important operations (``persist`` and ``recovery``) have failure handlers modelled as explicit callbacks which -the user can override in the ``PersistentActor``. The default implementations of these handlers emit a log message -(``error`` for persist/recovery failures, and ``warning`` for others), logging the failure cause and information about +The most important operations (`persist` and `recovery`) have failure handlers modelled as explicit callbacks which +the user can override in the `PersistentActor`. The default implementations of these handlers emit a log message +(`error` for persist/recovery failures, and `warning` for others), logging the failure cause and information about which message caused the failure. For critical failures, such as recovery or persisting events failing, the persistent actor will be stopped after the failure handler is invoked. This is because if the underlying journal implementation is signalling persistence failures it is most likely either failing completely or overloaded and restarting right-away and trying to persist the event again will most -likely not help the journal recover – as it would likely cause a `Thundering herd problem`_, as many persistent actors -would restart and try to persist their events again. Instead, using a ``BackoffSupervisor`` (as described in :ref:`failures-scala`) which +likely not help the journal recover – as it would likely cause a [Thundering herd problem](https://en.wikipedia.org/wiki/Thundering_herd_problem), as many persistent actors +would restart and try to persist their events again. Instead, using a `BackoffSupervisor` (as described in [Failures](#failures-scala)) which implements an exponential-backoff strategy which allows for more breathing room for the journal to recover between restarts of the persistent actor. -.. note:: - Journal implementations may choose to implement a retry mechanism, e.g. such that only after a write fails N number - of times a persistence failure is signalled back to the user. In other words, once a journal returns a failure, - it is considered *fatal* by Akka Persistence, and the persistent actor which caused the failure will be stopped. +@@@ note - Check the documentation of the journal implementation you are using for details if/how it is using this technique. +Journal implementations may choose to implement a retry mechanism, e.g. such that only after a write fails N number +of times a persistence failure is signalled back to the user. In other words, once a journal returns a failure, +it is considered *fatal* by Akka Persistence, and the persistent actor which caused the failure will be stopped. -.. _Thundering herd problem: https://en.wikipedia.org/wiki/Thundering_herd_problem +Check the documentation of the journal implementation you are using for details if/how it is using this technique. -.. _safe-shutdown-scala: +@@@ -Safely shutting down persistent actors --------------------------------------- + +### Safely shutting down persistent actors Special care should be given when shutting down persistent actors from the outside. -With normal Actors it is often acceptable to use the special :ref:`PoisonPill ` message +With normal Actors it is often acceptable to use the special @ref:[PoisonPill](actors.md#poison-pill-scala) message to signal to an Actor that it should stop itself once it receives this message – in fact this message is handled automatically by Akka, leaving the target actor no way to refuse stopping itself when given a poison pill. -This can be dangerous when used with :class:`PersistentActor` due to the fact that incoming commands are *stashed* while -the persistent actor is awaiting confirmation from the Journal that events have been written when ``persist()`` was used. +This can be dangerous when used with `PersistentActor` due to the fact that incoming commands are *stashed* while +the persistent actor is awaiting confirmation from the Journal that events have been written when `persist()` was used. Since the incoming commands will be drained from the Actor's mailbox and put into its internal stash while awaiting the confirmation (thus, before calling the persist handlers) the Actor **may receive and (auto)handle the PoisonPill before it processes the other messages which have been put into its stash**, causing a pre-mature shutdown of the Actor. -.. warning:: - Consider using explicit shut-down messages instead of :class:`PoisonPill` when working with persistent actors. +@@@ warning + +Consider using explicit shut-down messages instead of `PoisonPill` when working with persistent actors. + +@@@ The example below highlights how messages arrive in the Actor's mailbox and how they interact with its internal stashing -mechanism when ``persist()`` is used. Notice the early stop behaviour that occurs when ``PoisonPill`` is used: +mechanism when `persist()` is used. Notice the early stop behaviour that occurs when `PoisonPill` is used: -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#safe-shutdown -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#safe-shutdown-example-bad -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#safe-shutdown-example-good +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #safe-shutdown } -.. _replay-filter-scala: +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #safe-shutdown-example-bad } + +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #safe-shutdown-example-good } + + +### Replay Filter -Replay Filter -------------- There could be cases where event streams are corrupted and multiple writers (i.e. multiple persistent actor instances) journaled different messages with the same sequence number. In such a case, you can configure how you filter replayed messages from multiple writers, upon recovery. -In your configuration, under the ``akka.persistence.journal.xxx.replay-filter`` section (where ``xxx`` is your journal plugin id), -you can select the replay filter ``mode`` from one of the following values: +In your configuration, under the `akka.persistence.journal.xxx.replay-filter` section (where `xxx` is your journal plugin id), +you can select the replay filter `mode` from one of the following values: -* repair-by-discard-old -* fail -* warn -* off + * repair-by-discard-old + * fail + * warn + * off -For example, if you configure the replay filter for leveldb plugin, it looks like this:: +For example, if you configure the replay filter for leveldb plugin, it looks like this: - # The replay filter can detect a corrupt event stream by inspecting - # sequence numbers and writerUuid when replaying events. - akka.persistence.journal.leveldb.replay-filter { - # What the filter should do when detecting invalid events. - # Supported values: - # `repair-by-discard-old` : discard events from old writers, - # warning is logged - # `fail` : fail the replay, error is logged - # `warn` : log warning but emit events untouched - # `off` : disable this feature completely - mode = repair-by-discard-old - } +``` +# The replay filter can detect a corrupt event stream by inspecting +# sequence numbers and writerUuid when replaying events. +akka.persistence.journal.leveldb.replay-filter { + # What the filter should do when detecting invalid events. + # Supported values: + # `repair-by-discard-old` : discard events from old writers, + # warning is logged + # `fail` : fail the replay, error is logged + # `warn` : log warning but emit events untouched + # `off` : disable this feature completely + mode = repair-by-discard-old +} +``` -.. _snapshots: - -Snapshots -========= + +## Snapshots Snapshots can dramatically reduce recovery times of persistent actors and views. The following discusses snapshots in context of persistent actors but this is also applicable to persistent views. -Persistent actors can save snapshots of internal state by calling the ``saveSnapshot`` method. If saving of a snapshot -succeeds, the persistent actor receives a ``SaveSnapshotSuccess`` message, otherwise a ``SaveSnapshotFailure`` message +Persistent actors can save snapshots of internal state by calling the `saveSnapshot` method. If saving of a snapshot +succeeds, the persistent actor receives a `SaveSnapshotSuccess` message, otherwise a `SaveSnapshotFailure` message -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#save-snapshot +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #save-snapshot } -where ``metadata`` is of type ``SnapshotMetadata``: +where `metadata` is of type `SnapshotMetadata`: -.. includecode:: ../../../akka-persistence/src/main/scala/akka/persistence/SnapshotProtocol.scala#snapshot-metadata +@@snip [SnapshotProtocol.scala]../../../../../akka-persistence/src/main/scala/akka/persistence/SnapshotProtocol.scala) { #snapshot-metadata } -During recovery, the persistent actor is offered a previously saved snapshot via a ``SnapshotOffer`` message from +During recovery, the persistent actor is offered a previously saved snapshot via a `SnapshotOffer` message from which it can initialize internal state. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#snapshot-offer +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #snapshot-offer } -The replayed messages that follow the ``SnapshotOffer`` message, if any, are younger than the offered snapshot. +The replayed messages that follow the `SnapshotOffer` message, if any, are younger than the offered snapshot. They finally recover the persistent actor to its current (i.e. latest) state. In general, a persistent actor is only offered a snapshot if that persistent actor has previously saved one or more snapshots -and at least one of these snapshots matches the ``SnapshotSelectionCriteria`` that can be specified for recovery. +and at least one of these snapshots matches the `SnapshotSelectionCriteria` that can be specified for recovery. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#snapshot-criteria +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #snapshot-criteria } -If not specified, they default to ``SnapshotSelectionCriteria.Latest`` which selects the latest (= youngest) snapshot. -To disable snapshot-based recovery, applications should use ``SnapshotSelectionCriteria.None``. A recovery where no -saved snapshot matches the specified ``SnapshotSelectionCriteria`` will replay all journaled messages. +If not specified, they default to `SnapshotSelectionCriteria.Latest` which selects the latest (= youngest) snapshot. +To disable snapshot-based recovery, applications should use `SnapshotSelectionCriteria.None`. A recovery where no +saved snapshot matches the specified `SnapshotSelectionCriteria` will replay all journaled messages. -.. note:: - In order to use snapshots, a default snapshot-store (``akka.persistence.snapshot-store.plugin``) must be configured, - or the ``PersistentActor`` can pick a snapshot store explicitly by overriding ``def snapshotPluginId: String``. +@@@ note - Since it is acceptable for some applications to not use any snapshotting, it is legal to not configure a snapshot store. - However, Akka will log a warning message when this situation is detected and then continue to operate until - an actor tries to store a snapshot, at which point the operation will fail (by replying with an ``SaveSnapshotFailure`` for example). +In order to use snapshots, a default snapshot-store (`akka.persistence.snapshot-store.plugin`) must be configured, +or the `PersistentActor` can pick a snapshot store explicitly by overriding `def snapshotPluginId: String`. - Note that :ref:`cluster_sharding_scala` is using snapshots, so if you use Cluster Sharding you need to define a snapshot store plugin. +Since it is acceptable for some applications to not use any snapshotting, it is legal to not configure a snapshot store. +However, Akka will log a warning message when this situation is detected and then continue to operate until +an actor tries to store a snapshot, at which point the operation will fail (by replying with an `SaveSnapshotFailure` for example). -Snapshot deletion ------------------ +Note that @ref:[cluster_sharding_scala](cluster-sharding.md) is using snapshots, so if you use Cluster Sharding you need to define a snapshot store plugin. -A persistent actor can delete individual snapshots by calling the ``deleteSnapshot`` method with the sequence number of +@@@ + +### Snapshot deletion + +A persistent actor can delete individual snapshots by calling the `deleteSnapshot` method with the sequence number of when the snapshot was taken. -To bulk-delete a range of snapshots matching ``SnapshotSelectionCriteria``, -persistent actors should use the ``deleteSnapshots`` method. +To bulk-delete a range of snapshots matching `SnapshotSelectionCriteria`, +persistent actors should use the `deleteSnapshots` method. -Snapshot status handling ------------------------- +### Snapshot status handling Saving or deleting snapshots can either succeed or fail – this information is reported back to the persistent actor via status messages as illustrated in the following table. -============================================== ========================== ============================== -**Method** **Success** **Failure message** -============================================== ========================== ============================== -``saveSnapshot(Any)`` ``SaveSnapshotSuccess`` ``SaveSnapshotFailure`` -``deleteSnapshot(Long)`` ``DeleteSnapshotSuccess`` ``DeleteSnapshotFailure`` -``deleteSnapshots(SnapshotSelectionCriteria)`` ``DeleteSnapshotsSuccess`` ``DeleteSnapshotsFailure`` -============================================== ========================== ============================== +|**Method** | **Success** | **Failure message** | +|---------------------------------------------|--------------------------|-------------------------| +|`saveSnapshot(Any)` | `SaveSnapshotSuccess` | `SaveSnapshotFailure` | +|`deleteSnapshot(Long)` | `DeleteSnapshotSuccess` | `DeleteSnapshotFailure` | +|`deleteSnapshots(SnapshotSelectionCriteria)` | `DeleteSnapshotsSuccess` | `DeleteSnapshotsFailure`| If failure messages are left unhandled by the actor, a default warning log message will be logged for each incoming failure message. No default action is performed on the success messages, however you're free to handle them e.g. in order to delete an in memory representation of the snapshot, or in the case of failure to attempt save the snapshot again. -.. _at-least-once-delivery-scala: + +## At-Least-Once Delivery -At-Least-Once Delivery -====================== - -To send messages with at-least-once delivery semantics to destinations you can mix-in ``AtLeastOnceDelivery`` -trait to your ``PersistentActor`` on the sending side. It takes care of re-sending messages when they +To send messages with at-least-once delivery semantics to destinations you can mix-in `AtLeastOnceDelivery` +trait to your `PersistentActor` on the sending side. It takes care of re-sending messages when they have not been confirmed within a configurable timeout. The state of the sending actor, including which messages have been sent that have not been confirmed by the recipient must be persistent so that it can survive a crash of the sending actor -or JVM. The ``AtLeastOnceDelivery`` trait does not persist anything by itself. It is your +or JVM. The `AtLeastOnceDelivery` trait does not persist anything by itself. It is your responsibility to persist the intent that a message is sent and that a confirmation has been received. -.. note:: +@@@ note - At-least-once delivery implies that original message sending order is not always preserved, - and the destination may receive duplicate messages. - Semantics do not match those of a normal :class:`ActorRef` send operation: +At-least-once delivery implies that original message sending order is not always preserved, +and the destination may receive duplicate messages. +Semantics do not match those of a normal `ActorRef` send operation: - * it is not at-most-once delivery + * it is not at-most-once delivery + * message order for the same sender–receiver pair is not preserved due to +possible resends + * after a crash and restart of the destination messages are still +delivered to the new actor incarnation - * message order for the same sender–receiver pair is not preserved due to - possible resends +These semantics are similar to what an `ActorPath` represents (see +@ref:[Actor Lifecycle](actors.md#actor-lifecycle-scala)), therefore you need to supply a path and not a +reference when delivering messages. The messages are sent to the path with +an actor selection. - * after a crash and restart of the destination messages are still - delivered to the new actor incarnation +@@@ - These semantics are similar to what an :class:`ActorPath` represents (see - :ref:`actor-lifecycle-scala`), therefore you need to supply a path and not a - reference when delivering messages. The messages are sent to the path with - an actor selection. - -Use the ``deliver`` method to send a message to a destination. Call the ``confirmDelivery`` method +Use the `deliver` method to send a message to a destination. Call the `confirmDelivery` method when the destination has replied with a confirmation message. -Relationship between deliver and confirmDelivery ------------------------------------------------- +### Relationship between deliver and confirmDelivery -To send messages to the destination path, use the ``deliver`` method after you have persisted the intent +To send messages to the destination path, use the `deliver` method after you have persisted the intent to send the message. The destination actor must send back a confirmation message. When the sending actor receives this confirmation message you should persist the fact that the message was delivered successfully and then call -the ``confirmDelivery`` method. +the `confirmDelivery` method. -If the persistent actor is not currently recovering, the ``deliver`` method will send the message to -the destination actor. When recovering, messages will be buffered until they have been confirmed using ``confirmDelivery``. +If the persistent actor is not currently recovering, the `deliver` method will send the message to +the destination actor. When recovering, messages will be buffered until they have been confirmed using `confirmDelivery`. Once recovery has completed, if there are outstanding messages that have not been confirmed (during the message replay), the persistent actor will resend these before sending any other messages. -Deliver requires a ``deliveryIdToMessage`` function to pass the provided ``deliveryId`` into the message so that the correlation -between ``deliver`` and ``confirmDelivery`` is possible. The ``deliveryId`` must do the round trip. Upon receipt +Deliver requires a `deliveryIdToMessage` function to pass the provided `deliveryId` into the message so that the correlation +between `deliver` and `confirmDelivery` is possible. The `deliveryId` must do the round trip. Upon receipt of the message, the destination actor will send the same``deliveryId`` wrapped in a confirmation message back to the sender. -The sender will then use it to call ``confirmDelivery`` method to complete the delivery routine. +The sender will then use it to call `confirmDelivery` method to complete the delivery routine. -.. includecode:: code/docs/persistence/PersistenceDocSpec.scala#at-least-once-example +@@snip [PersistenceDocSpec.scala](code/docs/persistence/PersistenceDocSpec.scala) { #at-least-once-example } -The ``deliveryId`` generated by the persistence module is a strictly monotonically increasing sequence number +The `deliveryId` generated by the persistence module is a strictly monotonically increasing sequence number without gaps. The same sequence is used for all destinations of the actor, i.e. when sending to multiple -destinations the destinations will see gaps in the sequence. It is not possible to use custom ``deliveryId``. +destinations the destinations will see gaps in the sequence. It is not possible to use custom `deliveryId`. However, you can send a custom correlation identifier in the message to the destination. You must then retain -a mapping between the internal ``deliveryId`` (passed into the ``deliveryIdToMessage`` function) and your custom -correlation id (passed into the message). You can do this by storing such mapping in a ``Map(correlationId -> deliveryId)`` -from which you can retrieve the ``deliveryId`` to be passed into the ``confirmDelivery`` method once the receiver +a mapping between the internal `deliveryId` (passed into the `deliveryIdToMessage` function) and your custom +correlation id (passed into the message). You can do this by storing such mapping in a `Map(correlationId -> deliveryId)` +from which you can retrieve the `deliveryId` to be passed into the `confirmDelivery` method once the receiver of your message has replied with your custom correlation id. -The ``AtLeastOnceDelivery`` trait has a state consisting of unconfirmed messages and a +The `AtLeastOnceDelivery` trait has a state consisting of unconfirmed messages and a sequence number. It does not store this state itself. You must persist events corresponding to the -``deliver`` and ``confirmDelivery`` invocations from your ``PersistentActor`` so that the state can -be restored by calling the same methods during the recovery phase of the ``PersistentActor``. Sometimes +`deliver` and `confirmDelivery` invocations from your `PersistentActor` so that the state can +be restored by calling the same methods during the recovery phase of the `PersistentActor`. Sometimes these events can be derived from other business level events, and sometimes you must create separate events. -During recovery, calls to ``deliver`` will not send out messages, those will be sent later -if no matching ``confirmDelivery`` will have been performed. +During recovery, calls to `deliver` will not send out messages, those will be sent later +if no matching `confirmDelivery` will have been performed. -Support for snapshots is provided by ``getDeliverySnapshot`` and ``setDeliverySnapshot``. -The ``AtLeastOnceDeliverySnapshot`` contains the full delivery state, including unconfirmed messages. +Support for snapshots is provided by `getDeliverySnapshot` and `setDeliverySnapshot`. +The `AtLeastOnceDeliverySnapshot` contains the full delivery state, including unconfirmed messages. If you need a custom snapshot for other parts of the actor state you must also include the -``AtLeastOnceDeliverySnapshot``. It is serialized using protobuf with the ordinary Akka -serialization mechanism. It is easiest to include the bytes of the ``AtLeastOnceDeliverySnapshot`` +`AtLeastOnceDeliverySnapshot`. It is serialized using protobuf with the ordinary Akka +serialization mechanism. It is easiest to include the bytes of the `AtLeastOnceDeliverySnapshot` as a blob in your custom snapshot. -The interval between redelivery attempts is defined by the ``redeliverInterval`` method. -The default value can be configured with the ``akka.persistence.at-least-once-delivery.redeliver-interval`` +The interval between redelivery attempts is defined by the `redeliverInterval` method. +The default value can be configured with the `akka.persistence.at-least-once-delivery.redeliver-interval` configuration key. The method can be overridden by implementation classes to return non-default values. The maximum number of messages that will be sent at each redelivery burst is defined by the -``redeliveryBurstLimit`` method (burst frequency is half of the redelivery interval). If there's a lot of +`redeliveryBurstLimit` method (burst frequency is half of the redelivery interval). If there's a lot of unconfirmed messages (e.g. if the destination is not available for a long time), this helps to prevent an overwhelming amount of messages to be sent at once. The default value can be configured with the -``akka.persistence.at-least-once-delivery.redelivery-burst-limit`` configuration key. The method can be overridden +`akka.persistence.at-least-once-delivery.redelivery-burst-limit` configuration key. The method can be overridden by implementation classes to return non-default values. -After a number of delivery attempts a ``AtLeastOnceDelivery.UnconfirmedWarning`` message -will be sent to ``self``. The re-sending will still continue, but you can choose to call -``confirmDelivery`` to cancel the re-sending. The number of delivery attempts before emitting the -warning is defined by the ``warnAfterNumberOfUnconfirmedAttempts`` method. The default value can be -configured with the ``akka.persistence.at-least-once-delivery.warn-after-number-of-unconfirmed-attempts`` +After a number of delivery attempts a `AtLeastOnceDelivery.UnconfirmedWarning` message +will be sent to `self`. The re-sending will still continue, but you can choose to call +`confirmDelivery` to cancel the re-sending. The number of delivery attempts before emitting the +warning is defined by the `warnAfterNumberOfUnconfirmedAttempts` method. The default value can be +configured with the `akka.persistence.at-least-once-delivery.warn-after-number-of-unconfirmed-attempts` configuration key. The method can be overridden by implementation classes to return non-default values. -The ``AtLeastOnceDelivery`` trait holds messages in memory until their successful delivery has been confirmed. +The `AtLeastOnceDelivery` trait holds messages in memory until their successful delivery has been confirmed. The maximum number of unconfirmed messages that the actor is allowed to hold in memory -is defined by the ``maxUnconfirmedMessages`` method. If this limit is exceed the ``deliver`` method will -not accept more messages and it will throw ``AtLeastOnceDelivery.MaxUnconfirmedMessagesExceededException``. -The default value can be configured with the ``akka.persistence.at-least-once-delivery.max-unconfirmed-messages`` +is defined by the `maxUnconfirmedMessages` method. If this limit is exceed the `deliver` method will +not accept more messages and it will throw `AtLeastOnceDelivery.MaxUnconfirmedMessagesExceededException`. +The default value can be configured with the `akka.persistence.at-least-once-delivery.max-unconfirmed-messages` configuration key. The method can be overridden by implementation classes to return non-default values. -.. _event-adapters-scala: - -Event Adapters -============== + +## Event Adapters In long running projects using event sourcing sometimes the need arises to detach the data model from the domain model completely. Event Adapters help in situations where: -- **Version Migrations** – existing events stored in *Version 1* should be "upcasted" to a new *Version 2* representation, - and the process of doing so involves actual code, not just changes on the serialization layer. For these scenarios - the ``toJournal`` function is usually an identity function, however the ``fromJournal`` is implemented as - ``v1.Event=>v2.Event``, performing the neccessary mapping inside the fromJournal method. - This technique is sometimes refered to as "upcasting" in other CQRS libraries. -- **Separating Domain and Data models** – thanks to EventAdapters it is possible to completely separate the domain model - from the model used to persist data in the Journals. For example one may want to use case classes in the - domain model, however persist their protocol-buffer (or any other binary serialization format) counter-parts to the Journal. - A simple ``toJournal:MyModel=>MyDataModel`` and ``fromJournal:MyDataModel=>MyModel`` adapter can be used to implement this feature. -- **Journal Specialized Data Types** – exposing data types understood by the underlying Journal, for example for data stores which - understand JSON it is possible to write an EventAdapter ``toJournal:Any=>JSON`` such that the Journal can *directly* store the - json instead of serializing the object to its binary representation. + * **Version Migrations** – existing events stored in *Version 1* should be "upcasted" to a new *Version 2* representation, +and the process of doing so involves actual code, not just changes on the serialization layer. For these scenarios +the `toJournal` function is usually an identity function, however the `fromJournal` is implemented as +`v1.Event=>v2.Event`, performing the neccessary mapping inside the fromJournal method. +This technique is sometimes refered to as "upcasting" in other CQRS libraries. + * **Separating Domain and Data models** – thanks to EventAdapters it is possible to completely separate the domain model +from the model used to persist data in the Journals. For example one may want to use case classes in the +domain model, however persist their protocol-buffer (or any other binary serialization format) counter-parts to the Journal. +A simple `toJournal:MyModel=>MyDataModel` and `fromJournal:MyDataModel=>MyModel` adapter can be used to implement this feature. + * **Journal Specialized Data Types** – exposing data types understood by the underlying Journal, for example for data stores which +understand JSON it is possible to write an EventAdapter `toJournal:Any=>JSON` such that the Journal can *directly* store the +json instead of serializing the object to its binary representation. Implementing an EventAdapter is rather stright forward: -.. includecode:: code/docs/persistence/PersistenceEventAdapterDocSpec.scala#identity-event-adapter +@@snip [PersistenceEventAdapterDocSpec.scala](code/docs/persistence/PersistenceEventAdapterDocSpec.scala) { #identity-event-adapter } Then in order for it to be used on events coming to and from the journal you must bind it using the below configuration syntax: -.. includecode:: code/docs/persistence/PersistenceEventAdapterDocSpec.scala#event-adapters-config +@@snip [PersistenceEventAdapterDocSpec.scala](code/docs/persistence/PersistenceEventAdapterDocSpec.scala) { #event-adapters-config } -It is possible to bind multiple adapters to one class *for recovery*, in which case the ``fromJournal`` methods of all +It is possible to bind multiple adapters to one class *for recovery*, in which case the `fromJournal` methods of all bound adapters will be applied to a given matching event (in order of definition in the configuration). Since each adapter may -return from ``0`` to ``n`` adapted events (called as ``EventSeq``), each adapter can investigate the event and if it should +return from `0` to `n` adapted events (called as `EventSeq`), each adapter can investigate the event and if it should indeed adapt it return the adapted event(s) for it. Other adapters which do not have anything to contribute during this -adaptation simply return ``EventSeq.empty``. The adapted events are then delivered in-order to the ``PersistentActor`` during replay. +adaptation simply return `EventSeq.empty`. The adapted events are then delivered in-order to the `PersistentActor` during replay. -.. note:: - For more advanced schema evolution techniques refer to the :ref:`persistence-schema-evolution-scala` documentation. +@@@ note -.. _persistent-fsm: +For more advanced schema evolution techniques refer to the @ref:[Persistence - Schema Evolution](persistence-schema-evolution.md) documentation. -Persistent FSM -============== -``PersistentFSM`` handles the incoming messages in an FSM like fashion. +@@@ + + +## Persistent FSM + +`PersistentFSM` handles the incoming messages in an FSM like fashion. Its internal state is persisted as a sequence of changes, later referred to as domain events. Relationship between incoming messages, FSM's states and transitions, persistence of domain events is defined by a DSL. -A Simple Example ----------------- -To demonstrate the features of the ``PersistentFSM`` trait, consider an actor which represents a Web store customer. +### A Simple Example + +To demonstrate the features of the `PersistentFSM` trait, consider an actor which represents a Web store customer. The contract of our "WebStoreCustomerFSMActor" is that it accepts the following commands: -.. includecode:: ../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala#customer-commands +@@snip [PersistentFSMSpec.scala]../../../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala) { #customer-commands } -``AddItem`` sent when the customer adds an item to a shopping cart -``Buy`` - when the customer finishes the purchase -``Leave`` - when the customer leaves the store without purchasing anything -``GetCurrentCart`` allows to query the current state of customer's shopping cart +`AddItem` sent when the customer adds an item to a shopping cart +`Buy` - when the customer finishes the purchase +`Leave` - when the customer leaves the store without purchasing anything +`GetCurrentCart` allows to query the current state of customer's shopping cart The customer can be in one of the following states: -.. includecode:: ../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala#customer-states +@@snip [PersistentFSMSpec.scala]../../../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala) { #customer-states } -``LookingAround`` customer is browsing the site, but hasn't added anything to the shopping cart -``Shopping`` customer has recently added items to the shopping cart -``Inactive`` customer has items in the shopping cart, but hasn't added anything recently -``Paid`` customer has purchased the items +`LookingAround` customer is browsing the site, but hasn't added anything to the shopping cart +`Shopping` customer has recently added items to the shopping cart +`Inactive` customer has items in the shopping cart, but hasn't added anything recently +`Paid` customer has purchased the items -.. note:: +@@@ note - ``PersistentFSM`` states must inherit from trait ``PersistentFSM.FSMState`` and implement the - ``def identifier: String`` method. This is required in order to simplify the serialization of FSM states. - String identifiers should be unique! +`PersistentFSM` states must inherit from trait `PersistentFSM.FSMState` and implement the +`def identifier: String` method. This is required in order to simplify the serialization of FSM states. +String identifiers should be unique! + +@@@ Customer's actions are "recorded" as a sequence of "domain events" which are persisted. Those events are replayed on an actor's start in order to restore the latest customer's state: -.. includecode:: ../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala#customer-domain-events +@@snip [PersistentFSMSpec.scala]../../../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala) { #customer-domain-events } Customer state data represents the items in a customer's shopping cart: -.. includecode:: ../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala#customer-states-data +@@snip [PersistentFSMSpec.scala]../../../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala) { #customer-states-data } Here is how everything is wired together: -.. includecode:: ../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala#customer-fsm-body +@@snip [PersistentFSMSpec.scala]../../../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala) { #customer-fsm-body } -.. note:: +@@@ note - State data can only be modified directly on initialization. Later it's modified only as a result of applying domain events. - Override the ``applyEvent`` method to define how state data is affected by domain events, see the example below +State data can only be modified directly on initialization. Later it's modified only as a result of applying domain events. +Override the `applyEvent` method to define how state data is affected by domain events, see the example below -.. includecode:: ../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala#customer-apply-event +@@@ -``andThen`` can be used to define actions which will be executed following event's persistence - convenient for "side effects" like sending a message or logging. -Notice that actions defined in ``andThen`` block are not executed on recovery: +@@snip [PersistentFSMSpec.scala]../../../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala) { #customer-apply-event } -.. includecode:: ../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala#customer-andthen-example +`andThen` can be used to define actions which will be executed following event's persistence - convenient for "side effects" like sending a message or logging. +Notice that actions defined in `andThen` block are not executed on recovery: -A snapshot of state data can be persisted by calling the ``saveStateSnapshot()`` method: +@@snip [PersistentFSMSpec.scala]../../../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala) { #customer-andthen-example } -.. includecode:: ../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala#customer-snapshot-example +A snapshot of state data can be persisted by calling the `saveStateSnapshot()` method: + +@@snip [PersistentFSMSpec.scala]../../../../../akka-persistence/src/test/scala/akka/persistence/fsm/PersistentFSMSpec.scala) { #customer-snapshot-example } On recovery state data is initialized according to the latest available snapshot, then the remaining domain events are replayed, triggering the -``applyEvent`` method. +`applyEvent` method. -.. _storage-plugins: - -Storage plugins -=============== + +## Storage plugins Storage backends for journals and snapshot stores are pluggable in the Akka persistence extension. -A directory of persistence journal and snapshot store plugins is available at the Akka Community Projects page, see `Community plugins`_ +A directory of persistence journal and snapshot store plugins is available at the Akka Community Projects page, see [Community plugins](http://akka.io/community/) Plugins can be selected either by "default" for all persistent actors and views, or "individually", when a persistent actor or view defines its own set of plugins. -When a persistent actor or view does NOT override the ``journalPluginId`` and ``snapshotPluginId`` methods, -the persistence extension will use the "default" journal and snapshot-store plugins configured in ``reference.conf``:: +When a persistent actor or view does NOT override the `journalPluginId` and `snapshotPluginId` methods, +the persistence extension will use the "default" journal and snapshot-store plugins configured in `reference.conf`: - akka.persistence.journal.plugin = "" - akka.persistence.snapshot-store.plugin = "" +``` +akka.persistence.journal.plugin = "" +akka.persistence.snapshot-store.plugin = "" +``` -However, these entries are provided as empty "", and require explicit user configuration via override in the user ``application.conf``. -For an example of a journal plugin which writes messages to LevelDB see :ref:`local-leveldb-journal`. -For an example of a snapshot store plugin which writes snapshots as individual files to the local filesystem see :ref:`local-snapshot-store`. +However, these entries are provided as empty "", and require explicit user configuration via override in the user `application.conf`. +For an example of a journal plugin which writes messages to LevelDB see [Local LevelDB journal](#local-leveldb-journal). +For an example of a snapshot store plugin which writes snapshots as individual files to the local filesystem see [Local snapshot store](#local-snapshot-store). Applications can provide their own plugins by implementing a plugin API and activating them by configuration. Plugin development requires the following imports: -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#plugin-imports +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #plugin-imports } -Eager initialization of persistence plugin ------------------------------------------- +### Eager initialization of persistence plugin By default, persistence plugins are started on-demand, as they are used. In some case, however, it might be beneficial -to start a certain plugin eagerly. In order to do that, you should first add the ``akka.persistence.Persistence`` -under the ``akka.extensions`` key. Then, specify the IDs of plugins you wish to start automatically under -``akka.persistence.journal.auto-start-journals`` and ``akka.persistence.snapshot-store.auto-start-snapshot-stores``. +to start a certain plugin eagerly. In order to do that, you should first add the `akka.persistence.Persistence` +under the `akka.extensions` key. Then, specify the IDs of plugins you wish to start automatically under +`akka.persistence.journal.auto-start-journals` and `akka.persistence.snapshot-store.auto-start-snapshot-stores`. -.. _journal-plugin-api: + +### Journal plugin API -Journal plugin API ------------------- +A journal plugin extends `AsyncWriteJournal`. -A journal plugin extends ``AsyncWriteJournal``. +`AsyncWriteJournal` is an actor and the methods to be implemented are: -``AsyncWriteJournal`` is an actor and the methods to be implemented are: - -.. includecode:: ../../../akka-persistence/src/main/scala/akka/persistence/journal/AsyncWriteJournal.scala#journal-plugin-api +@@snip [AsyncWriteJournal.scala]../../../../../akka-persistence/src/main/scala/akka/persistence/journal/AsyncWriteJournal.scala) { #journal-plugin-api } If the storage backend API only supports synchronous, blocking writes, the methods should be implemented as: -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#sync-journal-plugin-api +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #sync-journal-plugin-api } -A journal plugin must also implement the methods defined in ``AsyncRecovery`` for replays and sequence number recovery: +A journal plugin must also implement the methods defined in `AsyncRecovery` for replays and sequence number recovery: -.. includecode:: ../../../akka-persistence/src/main/scala/akka/persistence/journal/AsyncRecovery.scala#journal-plugin-api +@@snip [AsyncRecovery.scala]../../../../../akka-persistence/src/main/scala/akka/persistence/journal/AsyncRecovery.scala) { #journal-plugin-api } A journal plugin can be activated with the following minimal configuration: -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#journal-plugin-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #journal-plugin-config } The journal plugin instance is an actor so the methods corresponding to requests from persistent actors are executed sequentially. It may delegate to asynchronous libraries, spawn futures, or delegate to other @@ -861,28 +856,27 @@ actors to achive parallelism. The journal plugin class must have a constructor with one of these signatures: -* constructor with one ``com.typesafe.config.Config`` parameter and a ``String`` parameter for the config path -* constructor with one ``com.typesafe.config.Config`` parameter -* constructor without parameters + * constructor with one `com.typesafe.config.Config` parameter and a `String` parameter for the config path + * constructor with one `com.typesafe.config.Config` parameter + * constructor without parameters The plugin section of the actor system's config will be passed in the config constructor parameter. The config path -of the plugin is passed in the ``String`` parameter. +of the plugin is passed in the `String` parameter. -The ``plugin-dispatcher`` is the dispatcher used for the plugin actor. If not specified, it defaults to -``akka.persistence.dispatchers.default-plugin-dispatcher``. +The `plugin-dispatcher` is the dispatcher used for the plugin actor. If not specified, it defaults to +`akka.persistence.dispatchers.default-plugin-dispatcher`. Don't run journal tasks/futures on the system default dispatcher, since that might starve other tasks. -Snapshot store plugin API -------------------------- +### Snapshot store plugin API -A snapshot store plugin must extend the ``SnapshotStore`` actor and implement the following methods: +A snapshot store plugin must extend the `SnapshotStore` actor and implement the following methods: -.. includecode:: ../../../akka-persistence/src/main/scala/akka/persistence/snapshot/SnapshotStore.scala#snapshot-store-plugin-api +@@snip [SnapshotStore.scala]../../../../../akka-persistence/src/main/scala/akka/persistence/snapshot/SnapshotStore.scala) { #snapshot-store-plugin-api } A snapshot store plugin can be activated with the following minimal configuration: -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#snapshot-store-plugin-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #snapshot-store-plugin-config } The snapshot store instance is an actor so the methods corresponding to requests from persistent actors are executed sequentially. It may delegate to asynchronous libraries, spawn futures, or delegate to other @@ -890,238 +884,236 @@ actors to achive parallelism. The snapshot store plugin class must have a constructor with one of these signatures: -* constructor with one ``com.typesafe.config.Config`` parameter and a ``String`` parameter for the config path -* constructor with one ``com.typesafe.config.Config`` parameter -* constructor without parameters + * constructor with one `com.typesafe.config.Config` parameter and a `String` parameter for the config path + * constructor with one `com.typesafe.config.Config` parameter + * constructor without parameters The plugin section of the actor system's config will be passed in the config constructor parameter. The config path -of the plugin is passed in the ``String`` parameter. +of the plugin is passed in the `String` parameter. -The ``plugin-dispatcher`` is the dispatcher used for the plugin actor. If not specified, it defaults to -``akka.persistence.dispatchers.default-plugin-dispatcher``. +The `plugin-dispatcher` is the dispatcher used for the plugin actor. If not specified, it defaults to +`akka.persistence.dispatchers.default-plugin-dispatcher`. Don't run snapshot store tasks/futures on the system default dispatcher, since that might starve other tasks. -Plugin TCK ----------- -In order to help developers build correct and high quality storage plugins, we provide a Technology Compatibility Kit (`TCK `_ for short). +### Plugin TCK -The TCK is usable from Java as well as Scala projects. For Scala you need to include the akka-persistence-tck dependency:: +In order to help developers build correct and high quality storage plugins, we provide a Technology Compatibility Kit ([TCK](http://en.wikipedia.org/wiki/Technology_Compatibility_Kit) for short). - "com.typesafe.akka" %% "akka-persistence-tck" % "@version@" % "test" +The TCK is usable from Java as well as Scala projects. For Scala you need to include the akka-persistence-tck dependency: -To include the Journal TCK tests in your test suite simply extend the provided ``JournalSpec``: +``` +"com.typesafe.akka" %% "akka-persistence-tck" % "@version@" % "test" +``` -.. includecode:: ./code/docs/persistence/PersistencePluginDocSpec.scala#journal-tck-scala +To include the Journal TCK tests in your test suite simply extend the provided `JournalSpec`: -Please note that some of the tests are optional, and by overriding the ``supports...`` methods you give the +@@snip [PersistencePluginDocSpec.scala](./code/docs/persistence/PersistencePluginDocSpec.scala) { #journal-tck-scala } + +Please note that some of the tests are optional, and by overriding the `supports...` methods you give the TCK the needed information about which tests to run. You can implement these methods using boolean falues or the -provided ``CapabilityFlag.on`` / ``CapabilityFlag.off`` values. +provided `CapabilityFlag.on` / `CapabilityFlag.off` values. -We also provide a simple benchmarking class ``JournalPerfSpec`` which includes all the tests that ``JournalSpec`` +We also provide a simple benchmarking class `JournalPerfSpec` which includes all the tests that `JournalSpec` has, and also performs some longer operations on the Journal while printing its performance stats. While it is NOT aimed to provide a proper benchmarking environment it can be used to get a rough feel about your journal's performance in the most typical scenarios. -In order to include the ``SnapshotStore`` TCK tests in your test suite simply extend the ``SnapshotStoreSpec``: +In order to include the `SnapshotStore` TCK tests in your test suite simply extend the `SnapshotStoreSpec`: -.. includecode:: ./code/docs/persistence/PersistencePluginDocSpec.scala#snapshot-store-tck-scala +@@snip [PersistencePluginDocSpec.scala](./code/docs/persistence/PersistencePluginDocSpec.scala) { #snapshot-store-tck-scala } In case your plugin requires some setting up (starting a mock database, removing temporary files etc.) you can override the -``beforeAll`` and ``afterAll`` methods to hook into the tests lifecycle: +`beforeAll` and `afterAll` methods to hook into the tests lifecycle: -.. includecode:: ./code/docs/persistence/PersistencePluginDocSpec.scala#journal-tck-before-after-scala +@@snip [PersistencePluginDocSpec.scala](./code/docs/persistence/PersistencePluginDocSpec.scala) { #journal-tck-before-after-scala } We *highly recommend* including these specifications in your test suite, as they cover a broad range of cases you might have otherwise forgotten to test for when writing a plugin from scratch. -.. _pre-packaged-plugins: + +## Pre-packaged plugins -Pre-packaged plugins -==================== + +### Local LevelDB journal -.. _local-leveldb-journal: - -Local LevelDB journal ---------------------- - -The LevelDB journal plugin config entry is ``akka.persistence.journal.leveldb``. It writes messages to a local LevelDB +The LevelDB journal plugin config entry is `akka.persistence.journal.leveldb`. It writes messages to a local LevelDB instance. Enable this plugin by defining config property: -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#leveldb-plugin-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #leveldb-plugin-config } -LevelDB based plugins will also require the following additional dependency declaration:: +LevelDB based plugins will also require the following additional dependency declaration: - "org.iq80.leveldb" % "leveldb" % "0.7" - "org.fusesource.leveldbjni" % "leveldbjni-all" % "1.8" +``` +"org.iq80.leveldb" % "leveldb" % "0.7" +"org.fusesource.leveldbjni" % "leveldbjni-all" % "1.8" +``` -The default location of LevelDB files is a directory named ``journal`` in the current working +The default location of LevelDB files is a directory named `journal` in the current working directory. This location can be changed by configuration where the specified path can be relative or absolute: -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#journal-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #journal-config } With this plugin, each actor system runs its own private LevelDB instance. - -.. _shared-leveldb-journal: - -Shared LevelDB journal ----------------------- + +### Shared LevelDB journal A LevelDB instance can also be shared by multiple actor systems (on the same or on different nodes). This, for example, allows persistent actors to failover to a backup node and continue using the shared journal instance from the backup node. -.. warning:: +@@@ warning - A shared LevelDB instance is a single point of failure and should therefore only be used for testing - purposes. Highly-available, replicated journals are available as `Community plugins`_. +A shared LevelDB instance is a single point of failure and should therefore only be used for testing +purposes. Highly-available, replicated journals are available as [Community plugins](http://akka.io/community/). -.. note:: +@@@ - This plugin has been supplanted by :ref:`Persistence Plugin Proxy`. +@@@ note -A shared LevelDB instance is started by instantiating the ``SharedLeveldbStore`` actor. +This plugin has been supplanted by [Persistence Plugin Proxy](#persistence-plugin-proxy). -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#shared-store-creation +@@@ -By default, the shared instance writes journaled messages to a local directory named ``journal`` in the current +A shared LevelDB instance is started by instantiating the `SharedLeveldbStore` actor. + +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #shared-store-creation } + +By default, the shared instance writes journaled messages to a local directory named `journal` in the current working directory. The storage location can be changed by configuration: -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#shared-store-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #shared-store-config } -Actor systems that use a shared LevelDB store must activate the ``akka.persistence.journal.leveldb-shared`` +Actor systems that use a shared LevelDB store must activate the `akka.persistence.journal.leveldb-shared` plugin. -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#shared-journal-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #shared-journal-config } -This plugin must be initialized by injecting the (remote) ``SharedLeveldbStore`` actor reference. Injection is -done by calling the ``SharedLeveldbJournal.setStore`` method with the actor reference as argument. +This plugin must be initialized by injecting the (remote) `SharedLeveldbStore` actor reference. Injection is +done by calling the `SharedLeveldbJournal.setStore` method with the actor reference as argument. -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#shared-store-usage +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #shared-store-usage } Internal journal commands (sent by persistent actors) are buffered until injection completes. Injection is idempotent i.e. only the first injection is used. + +### Local snapshot store -.. _local-snapshot-store: - -Local snapshot store --------------------- - -The local snapshot store plugin config entry is ``akka.persistence.snapshot-store.local``. It writes snapshot files to +The local snapshot store plugin config entry is `akka.persistence.snapshot-store.local`. It writes snapshot files to the local filesystem. Enable this plugin by defining config property: -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#leveldb-snapshot-plugin-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #leveldb-snapshot-plugin-config } -The default storage location is a directory named ``snapshots`` in the current working +The default storage location is a directory named `snapshots` in the current working directory. This can be changed by configuration where the specified path can be relative or absolute: -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#snapshot-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #snapshot-config } Note that it is not mandatory to specify a snapshot store plugin. If you don't use snapshots you don't have to configure it. - -.. _persistence-plugin-proxy: - -Persistence Plugin Proxy ------------------------- + +### Persistence Plugin Proxy A persistence plugin proxy allows sharing of journals and snapshot stores across multiple actor systems (on the same or on different nodes). This, for example, allows persistent actors to failover to a backup node and continue using the shared journal instance from the backup node. The proxy works by forwarding all the journal/snapshot store messages to a single, shared, persistence plugin instance, and therefor supports any use case supported by the proxied plugin. -.. warning:: +@@@ warning - A shared journal/snapshot store is a single point of failure and should therefore only be used for testing - purposes. Highly-available, replicated persistence plugins are available as `Community plugins`_. +A shared journal/snapshot store is a single point of failure and should therefore only be used for testing +purposes. Highly-available, replicated persistence plugins are available as [Community plugins](http://akka.io/community/). -The journal and snapshot store proxies are controlled via the ``akka.persistence.journal.proxy`` and -``akka.persistence.snapshot-store.proxy`` configuration entries, respectively. Set the ``target-journal-plugin`` or -``target-snapshot-store-plugin`` keys to the underlying plugin you wish to use (for example: -``akka.persistence.journal.leveldb``). The ``start-target-journal`` and ``start-target-snapshot-store`` keys should be -set to ``on`` in exactly one actor system - this is the system that will instantiate the shared persistence plugin. -Next, the proxy needs to be told how to find the shared plugin. This can be done by setting the ``target-journal-address`` -and ``target-snapshot-store-address`` configuration keys, or programmatically by calling the -``PersistencePluginProxy.setTargetLocation`` method. +@@@ -.. note:: +The journal and snapshot store proxies are controlled via the `akka.persistence.journal.proxy` and +`akka.persistence.snapshot-store.proxy` configuration entries, respectively. Set the `target-journal-plugin` or +`target-snapshot-store-plugin` keys to the underlying plugin you wish to use (for example: +`akka.persistence.journal.leveldb`). The `start-target-journal` and `start-target-snapshot-store` keys should be +set to `on` in exactly one actor system - this is the system that will instantiate the shared persistence plugin. +Next, the proxy needs to be told how to find the shared plugin. This can be done by setting the `target-journal-address` +and `target-snapshot-store-address` configuration keys, or programmatically by calling the +`PersistencePluginProxy.setTargetLocation` method. - Akka starts extensions lazily when they are required, and this includes the proxy. This means that in order for the - proxy to work, the persistence plugin on the target node must be instantiated. This can be done by instantiating the - ``PersistencePluginProxyExtension`` :ref:`extension`, or by calling the ``PersistencePluginProxy.start`` method. +@@@ note -.. note:: +Akka starts extensions lazily when they are required, and this includes the proxy. This means that in order for the +proxy to work, the persistence plugin on the target node must be instantiated. This can be done by instantiating the +`PersistencePluginProxyExtension` @ref:[extension](extending-akka.md), or by calling the `PersistencePluginProxy.start` method. - The proxied persistence plugin can (and should) be configured using its original configuration keys. +@@@ +@@@ note -.. _custom-serialization: +The proxied persistence plugin can (and should) be configured using its original configuration keys. -Custom serialization -==================== +@@@ -Serialization of snapshots and payloads of ``Persistent`` messages is configurable with Akka's -:ref:`serialization-scala` infrastructure. For example, if an application wants to serialize + +## Custom serialization -* payloads of type ``MyPayload`` with a custom ``MyPayloadSerializer`` and -* snapshots of type ``MySnapshot`` with a custom ``MySnapshotSerializer`` +Serialization of snapshots and payloads of `Persistent` messages is configurable with Akka's +@ref:[Serialization](serialization.md) infrastructure. For example, if an application wants to serialize + + * payloads of type `MyPayload` with a custom `MyPayloadSerializer` and + * snapshots of type `MySnapshot` with a custom `MySnapshotSerializer` it must add -.. includecode:: code/docs/persistence/PersistenceSerializerDocSpec.scala#custom-serializer-config +@@snip [PersistenceSerializerDocSpec.scala](code/docs/persistence/PersistenceSerializerDocSpec.scala) { #custom-serializer-config } to the application configuration. If not specified, a default serializer is used. -For more advanced schema evolution techniques refer to the :ref:`persistence-schema-evolution-scala` documentation. +For more advanced schema evolution techniques refer to the @ref:[Persistence - Schema Evolution](persistence-schema-evolution.md) documentation. -Testing -======= +## Testing -When running tests with LevelDB default settings in ``sbt``, make sure to set ``fork := true`` in your sbt project. Otherwise, you'll see an ``UnsatisfiedLinkError``. Alternatively, you can switch to a LevelDB Java port by setting +When running tests with LevelDB default settings in `sbt`, make sure to set `fork := true` in your sbt project. Otherwise, you'll see an `UnsatisfiedLinkError`. Alternatively, you can switch to a LevelDB Java port by setting -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#native-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #native-config } or -.. includecode:: code/docs/persistence/PersistencePluginDocSpec.scala#shared-store-native-config +@@snip [PersistencePluginDocSpec.scala](code/docs/persistence/PersistencePluginDocSpec.scala) { #shared-store-native-config } in your Akka configuration. The LevelDB Java port is for testing purposes only. -.. warning:: - It is not possible to test persistence provided classes (i.e. :ref:`PersistentActor ` - and :ref:`AtLeastOnceDelivery `) using ``TestActorRef`` due to its *synchronous* nature. - These traits need to be able to perform asynchronous tasks in the background in order to handle internal persistence - related events. +@@@ warning - When testing Persistence based projects always rely on :ref:`asynchronous messaging using the TestKit `. +It is not possible to test persistence provided classes (i.e. [PersistentActor](#event-sourcing-scala) +and [AtLeastOnceDelivery](#at-least-once-delivery-scala)) using `TestActorRef` due to its *synchronous* nature. +These traits need to be able to perform asynchronous tasks in the background in order to handle internal persistence +related events. -Configuration -============= +When testing Persistence based projects always rely on @ref:[asynchronous messaging using the TestKit](testing.md#async-integration-testing-scala). + +@@@ + +## Configuration There are several configuration properties for the persistence module, please refer -to the :ref:`reference configuration `. +to the @ref:[reference configuration](../general/configuration.md#config-akka-persistence). -Multiple persistence plugin configurations -========================================== +## Multiple persistence plugin configurations By default, a persistent actor or view will use the "default" journal and snapshot store plugins -configured in the following sections of the ``reference.conf`` configuration resource: +configured in the following sections of the `reference.conf` configuration resource: -.. includecode:: code/docs/persistence/PersistenceMultiDocSpec.scala#default-config +@@snip [PersistenceMultiDocSpec.scala](code/docs/persistence/PersistenceMultiDocSpec.scala) { #default-config } -Note that in this case the actor or view overrides only the ``persistenceId`` method: +Note that in this case the actor or view overrides only the `persistenceId` method: -.. includecode:: code/docs/persistence/PersistenceMultiDocSpec.scala#default-plugins +@@snip [PersistenceMultiDocSpec.scala](code/docs/persistence/PersistenceMultiDocSpec.scala) { #default-plugins } -When the persistent actor or view overrides the ``journalPluginId`` and ``snapshotPluginId`` methods, +When the persistent actor or view overrides the `journalPluginId` and `snapshotPluginId` methods, the actor or view will be serviced by these specific persistence plugins instead of the defaults: -.. includecode:: code/docs/persistence/PersistenceMultiDocSpec.scala#override-plugins +@@snip [PersistenceMultiDocSpec.scala](code/docs/persistence/PersistenceMultiDocSpec.scala) { #override-plugins } -Note that ``journalPluginId`` and ``snapshotPluginId`` must refer to properly configured ``reference.conf`` -plugin entries with a standard ``class`` property as well as settings which are specific for those plugins, i.e.: +Note that `journalPluginId` and `snapshotPluginId` must refer to properly configured `reference.conf` +plugin entries with a standard `class` property as well as settings which are specific for those plugins, i.e.: -.. includecode:: code/docs/persistence/PersistenceMultiDocSpec.scala#override-config +@@snip [PersistenceMultiDocSpec.scala](code/docs/persistence/PersistenceMultiDocSpec.scala) { #override-config } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/remoting-artery.md b/akka-docs/src/main/paradox/scala/remoting-artery.md index 9b69feabde..8d4d72eef6 100644 --- a/akka-docs/src/main/paradox/scala/remoting-artery.md +++ b/akka-docs/src/main/paradox/scala/remoting-artery.md @@ -1,97 +1,101 @@ -.. _remoting-artery-scala: +# Remoting (codename Artery) -########################## -Remoting (codename Artery) -########################## +@@@ note -.. note:: +This page describes the @ref:[may change](../common/may-change.md) remoting subsystem, codenamed *Artery* that will eventually replace the +old remoting implementation. For the current stable remoting system please refer to @ref:[Remoting](remoting.md). - This page describes the :ref:`may change ` remoting subsystem, codenamed *Artery* that will eventually replace the - old remoting implementation. For the current stable remoting system please refer to :ref:`remoting-scala`. +@@@ Remoting enables Actor systems on different hosts or JVMs to communicate with each other. By enabling remoting the system will start listening on a provided network address and also gains the ability to connect to other systems through the network. From the application's perspective there is no API difference between local or remote -systems, :class:`ActorRef` instances that point to remote systems look exactly the same as local ones: they can be +systems, `ActorRef` instances that point to remote systems look exactly the same as local ones: they can be sent messages to, watched, etc. -Every :class:`ActorRef` contains hostname and port information and can be passed around even on the network. This means -that on a network every :class:`ActorRef` is a unique identifier of an actor on that network. +Every `ActorRef` contains hostname and port information and can be passed around even on the network. This means +that on a network every `ActorRef` is a unique identifier of an actor on that network. Remoting is not a server-client technology. All systems using remoting can contact any other system on the network -if they possess an :class:`ActorRef` pointing to those system. This means that every system that is remoting enabled +if they possess an `ActorRef` pointing to those system. This means that every system that is remoting enabled acts as a "server" to which arbitrary systems on the same network can connect to. -What is new in Artery ---------------------- +## What is new in Artery Artery is a reimplementation of the old remoting module aimed at improving performance and stability. It is mostly backwards compatible with the old implementation and it is a drop-in replacement in many cases. Main features of Artery compared to the previous implementation: -* Based on `Aeron `_ (UDP) instead of TCP -* Focused on high-throughput, low-latency communication -* Isolation of internal control messages from user messages improving stability and reducing false failure detection - in case of heavy traffic by using a dedicated subchannel. -* Mostly allocation-free operation -* Support for a separate subchannel for large messages to avoid interference with smaller messages -* Compression of actor paths on the wire to reduce overhead for smaller messages -* Support for faster serialization/deserialization using ByteBuffers directly -* Built-in Flight-Recorder to help debugging implementation issues without polluting users logs with implementaiton - specific events -* Providing protocol stability across major Akka versions to support rolling updates of large-scale systems + * Based on [Aeron](https://github.com/real-logic/Aeron) (UDP) instead of TCP + * Focused on high-throughput, low-latency communication + * Isolation of internal control messages from user messages improving stability and reducing false failure detection +in case of heavy traffic by using a dedicated subchannel. + * Mostly allocation-free operation + * Support for a separate subchannel for large messages to avoid interference with smaller messages + * Compression of actor paths on the wire to reduce overhead for smaller messages + * Support for faster serialization/deserialization using ByteBuffers directly + * Built-in Flight-Recorder to help debugging implementation issues without polluting users logs with implementaiton +specific events + * Providing protocol stability across major Akka versions to support rolling updates of large-scale systems The main incompatible change from the previous implementation that the protocol field of the string representation of an -:class:`ActorRef` is always `akka` instead of the previously used `akka.tcp` or `akka.ssl.tcp`. Configuration properties +`ActorRef` is always *akka* instead of the previously used *akka.tcp* or *akka.ssl.tcp*. Configuration properties are also different. -Preparing your ActorSystem for Remoting ---------------------------------------- +## Preparing your ActorSystem for Remoting -The Akka remoting is a separate jar file. Make sure that you have the following dependency in your project:: +The Akka remoting is a separate jar file. Make sure that you have the following dependency in your project: - "com.typesafe.akka" %% "akka-remote" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-remote" % "@version@" @crossString@ +``` To enable remote capabilities in your Akka project you should, at a minimum, add the following changes -to your ``application.conf`` file:: +to your `application.conf` file: - akka { - actor { - provider = remote - } - remote { - artery { - enabled = on - canonical.hostname = "127.0.0.1" - canonical.port = 25520 - } +``` +akka { + actor { + provider = remote + } + remote { + artery { + enabled = on + canonical.hostname = "127.0.0.1" + canonical.port = 25520 } } +} +``` As you can see in the example above there are four things you need to add to get started: -* Change provider from ``local`` to ``remote`` -* Enable Artery to use it as the remoting implementation -* Add host name - the machine you want to run the actor system on; this host - name is exactly what is passed to remote systems in order to identify this - system and consequently used for connecting back to this system if need be, - hence set it to a reachable IP address or resolvable name in case you want to - communicate across the network. -* Add port number - the port the actor system should listen on, set to 0 to have it chosen automatically + * Change provider from `local` to `remote` + * Enable Artery to use it as the remoting implementation + * Add host name - the machine you want to run the actor system on; this host +name is exactly what is passed to remote systems in order to identify this +system and consequently used for connecting back to this system if need be, +hence set it to a reachable IP address or resolvable name in case you want to +communicate across the network. + * Add port number - the port the actor system should listen on, set to 0 to have it chosen automatically -.. note:: +@@@ note - The port number needs to be unique for each actor system on the same machine even if the actor - systems have different names. This is because each actor system has its own networking subsystem - listening for connections and handling messages as not to interfere with other actor systems. +The port number needs to be unique for each actor system on the same machine even if the actor +systems have different names. This is because each actor system has its own networking subsystem +listening for connections and handling messages as not to interfere with other actor systems. + +@@@ The example above only illustrates the bare minimum of properties you have to add to enable remoting. -All settings are described in :ref:`remote-configuration-artery-scala`. +All settings are described in [Remote Configuration](#remote-configuration-artery-scala). -.. note:: - Aeron requires 64bit JVM to work reliably. +@@@ note -Canonical address -^^^^^^^^^^^^^^^^^ +Aeron requires 64bit JVM to work reliably. + +@@@ + +### Canonical address In order to remoting to work properly, where each system can send messages to any other system on the same network (for example a system forwards a message to a third system, and the third replies directly to the sender system) @@ -100,147 +104,154 @@ unique name of the system and will be used by other systems to open a connection that if a host has multiple names (different DNS records pointing to the same IP address) then only one of these can be *canonical*. If a message arrives to a system but it contains a different hostname than the expected canonical name then the message will be dropped. If multiple names for a system would be allowed, then equality checks among -:class:`ActorRef` instances would no longer to be trusted and this would violate the fundamental assumption that +`ActorRef` instances would no longer to be trusted and this would violate the fundamental assumption that an actor has a globally unique reference on a given network. As a consequence, this also means that localhost addresses -(e.g. `127.0.0.1`) cannot be used in general (apart from local development) since they are not unique addresses in a +(e.g. *127.0.0.1*) cannot be used in general (apart from local development) since they are not unique addresses in a real network. In cases, where Network Address Translation (NAT) is used or other network bridging is involved, it is important to configure the system so that it understands that there is a difference between his externally visible, canonical -address and between the host-port pair that is used to listen for connections. See :ref:`remote-configuration-nat-artery-scala` +address and between the host-port pair that is used to listen for connections. See [Akka behind NAT or in a Docker container](#remote-configuration-nat-artery-scala) for details. -Acquiring references to remote actors -------------------------------------- +## Acquiring references to remote actors -In order to communicate with an actor, it is necessary to have its :class:`ActorRef`. In the local case it is usually -the creator of the actor (the caller of ``actorOf()``) is who gets the :class:`ActorRef` for an actor that it can +In order to communicate with an actor, it is necessary to have its `ActorRef`. In the local case it is usually +the creator of the actor (the caller of `actorOf()`) is who gets the `ActorRef` for an actor that it can then send to other actors. In other words: -* An Actor can get a remote Actor's reference simply by receiving a message from it (as it's available as `sender()` then), - or inside of a remote message (e.g. `PleaseReply(message: String, remoteActorRef: ActorRef)`) + * An Actor can get a remote Actor's reference simply by receiving a message from it (as it's available as *sender()* then), +or inside of a remote message (e.g. *PleaseReply(message: String, remoteActorRef: ActorRef)*) Alternatively, an actor can look up another located at a known path using -:class:`ActorSelection`. These methods are available even in remoting enabled systems: +`ActorSelection`. These methods are available even in remoting enabled systems: -* Remote Lookup : used to look up an actor on a remote node with ``actorSelection(path)`` -* Remote Creation : used to create an actor on a remote node with ``actorOf(Props(...), actorName)`` + * Remote Lookup : used to look up an actor on a remote node with `actorSelection(path)` + * Remote Creation : used to create an actor on a remote node with `actorOf(Props(...), actorName)` In the next sections the two alternatives are described in detail. +### Looking up Remote Actors -Looking up Remote Actors -^^^^^^^^^^^^^^^^^^^^^^^^ +`actorSelection(path)` will obtain an `ActorSelection` to an Actor on a remote node, e.g.: -``actorSelection(path)`` will obtain an ``ActorSelection`` to an Actor on a remote node, e.g.:: +``` +val selection = + context.actorSelection("akka://actorSystemName@10.0.0.1:25520/user/actorName") +``` - val selection = - context.actorSelection("akka://actorSystemName@10.0.0.1:25520/user/actorName") +As you can see from the example above the following pattern is used to find an actor on a remote node: -As you can see from the example above the following pattern is used to find an actor on a remote node:: +``` +akka://@:/ +``` - akka://@:/ +@@@ note -.. note:: +Unlike with earlier remoting, the protocol field is always *akka* as pluggable transports are no longer supported. - Unlike with earlier remoting, the protocol field is always `akka` as pluggable transports are no longer supported. +@@@ -Once you obtained a selection to the actor you can interact with it in the same way you would with a local actor, e.g.:: +Once you obtained a selection to the actor you can interact with it in the same way you would with a local actor, e.g.: - selection ! "Pretty awesome feature" +``` +selection ! "Pretty awesome feature" +``` -To acquire an :class:`ActorRef` for an :class:`ActorSelection` you need to -send a message to the selection and use the ``sender`` reference of the reply from -the actor. There is a built-in ``Identify`` message that all Actors will understand -and automatically reply to with a ``ActorIdentity`` message containing the -:class:`ActorRef`. This can also be done with the ``resolveOne`` method of -the :class:`ActorSelection`, which returns a ``Future`` of the matching -:class:`ActorRef`. +To acquire an `ActorRef` for an `ActorSelection` you need to +send a message to the selection and use the `sender` reference of the reply from +the actor. There is a built-in `Identify` message that all Actors will understand +and automatically reply to with a `ActorIdentity` message containing the +`ActorRef`. This can also be done with the `resolveOne` method of +the `ActorSelection`, which returns a `Future` of the matching +`ActorRef`. -For more details on how actor addresses and paths are formed and used, please refer to :ref:`addressing`. +For more details on how actor addresses and paths are formed and used, please refer to @ref:[Actor References, Paths and Addresses](../general/addressing.md). -.. note:: +@@@ note - Message sends to actors that are actually in the sending actor system do not - get delivered via the remote actor ref provider. They're delivered directly, - by the local actor ref provider. +Message sends to actors that are actually in the sending actor system do not +get delivered via the remote actor ref provider. They're delivered directly, +by the local actor ref provider. - Aside from providing better performance, this also means that if the hostname - you configure remoting to listen as cannot actually be resolved from within - the very same actor system, such messages will (perhaps counterintuitively) - be delivered just fine. +Aside from providing better performance, this also means that if the hostname +you configure remoting to listen as cannot actually be resolved from within +the very same actor system, such messages will (perhaps counterintuitively) +be delivered just fine. +@@@ -Creating Actors Remotely -^^^^^^^^^^^^^^^^^^^^^^^^ +### Creating Actors Remotely If you want to use the creation functionality in Akka remoting you have to further amend the -``application.conf`` file in the following way (only showing deployment section):: +`application.conf` file in the following way (only showing deployment section): - akka { - actor { - deployment { - /sampleActor { - remote = "akka://sampleActorSystem@127.0.0.1:2553" - } +``` +akka { + actor { + deployment { + /sampleActor { + remote = "akka://sampleActorSystem@127.0.0.1:2553" } } } +} +``` -The configuration above instructs Akka to react when an actor with path ``/sampleActor`` is created, i.e. -using ``system.actorOf(Props(...), "sampleActor")``. This specific actor will not be directly instantiated, +The configuration above instructs Akka to react when an actor with path `/sampleActor` is created, i.e. +using `system.actorOf(Props(...), "sampleActor")`. This specific actor will not be directly instantiated, but instead the remote daemon of the remote system will be asked to create the actor, -which in this sample corresponds to ``sampleActorSystem@127.0.0.1:2553``. +which in this sample corresponds to `sampleActorSystem@127.0.0.1:2553`. Once you have configured the properties above you would do the following in code: -.. includecode:: code/docs/remoting/RemoteDeploymentDocSpec.scala#sample-actor +@@snip [RemoteDeploymentDocSpec.scala](code/docs/remoting/RemoteDeploymentDocSpec.scala) { #sample-actor } -The actor class ``SampleActor`` has to be available to the runtimes using it, i.e. the classloader of the +The actor class `SampleActor` has to be available to the runtimes using it, i.e. the classloader of the actor systems has to have a JAR containing the class. -.. note:: +@@@ note - In order to ensure serializability of ``Props`` when passing constructor - arguments to the actor being created, do not make the factory an inner class: - this will inherently capture a reference to its enclosing object, which in - most cases is not serializable. It is best to create a factory method in the - companion object of the actor’s class. +In order to ensure serializability of `Props` when passing constructor +arguments to the actor being created, do not make the factory an inner class: +this will inherently capture a reference to its enclosing object, which in +most cases is not serializable. It is best to create a factory method in the +companion object of the actor’s class. - Serializability of all Props can be tested by setting the configuration item - ``akka.actor.serialize-creators=on``. Only Props whose ``deploy`` has - ``LocalScope`` are exempt from this check. +Serializability of all Props can be tested by setting the configuration item +`akka.actor.serialize-creators=on`. Only Props whose `deploy` has +`LocalScope` are exempt from this check. + +@@@ You can use asterisks as wildcard matches for the actor paths, so you could specify: -``/*/sampleActor`` and that would match all ``sampleActor`` on that level in the hierarchy. +`/*/sampleActor` and that would match all `sampleActor` on that level in the hierarchy. You can also use wildcard in the last position to match all actors at a certain level: -``/someParent/*``. Non-wildcard matches always have higher priority to match than wildcards, so: -``/foo/bar`` is considered **more specific** than ``/foo/*`` and only the highest priority match is used. -Please note that it **cannot** be used to partially match section, like this: ``/foo*/bar``, ``/f*o/bar`` etc. +`/someParent/*`. Non-wildcard matches always have higher priority to match than wildcards, so: +`/foo/bar` is considered **more specific** than `/foo/*` and only the highest priority match is used. +Please note that it **cannot** be used to partially match section, like this: `/foo*/bar`, `/f*o/bar` etc. -Programmatic Remote Deployment -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Programmatic Remote Deployment To allow dynamically deployed systems, it is also possible to include -deployment configuration in the :class:`Props` which are used to create an +deployment configuration in the `Props` which are used to create an actor: this information is the equivalent of a deployment section from the configuration file, and if both are given, the external configuration takes precedence. With these imports: -.. includecode:: code/docs/remoting/RemoteDeploymentDocSpec.scala#import +@@snip [RemoteDeploymentDocSpec.scala](code/docs/remoting/RemoteDeploymentDocSpec.scala) { #import } and a remote address like this: -.. includecode:: code/docs/remoting/RemoteDeploymentDocSpec.scala#make-address-artery +@@snip [RemoteDeploymentDocSpec.scala](code/docs/remoting/RemoteDeploymentDocSpec.scala) { #make-address-artery } you can advise the system to create a child on that remote node like so: -.. includecode:: code/docs/remoting/RemoteDeploymentDocSpec.scala#deploy +@@snip [RemoteDeploymentDocSpec.scala](code/docs/remoting/RemoteDeploymentDocSpec.scala) { #deploy } -Remote deployment whitelist -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Remote deployment whitelist As remote deployment can potentially be abused by both users and even attackers a whitelist feature is available to guard the ActorSystem from deploying unexpected actors. Please note that remote deployment @@ -248,104 +259,107 @@ is *not* remote code loading, the Actors class to be deployed onto a remote syst remote system. This still however may pose a security risk, and one may want to restrict remote deployment to only a specific set of known actors by enabling the whitelist feature. -To enable remote deployment whitelisting set the ``akka.remote.deployment.enable-whitelist`` value to ``on``. +To enable remote deployment whitelisting set the `akka.remote.deployment.enable-whitelist` value to `on`. The list of allowed classes has to be configured on the "remote" system, in other words on the system onto which others will be attempting to remote deploy Actors. That system, locally, knows best which Actors it should or should not allow others to remote deploy onto it. The full settings section may for example look like this: -.. includecode:: ../../../akka-remote/src/test/scala/akka/remote/RemoteDeploymentWhitelistSpec.scala#whitelist-config +@@snip [RemoteDeploymentWhitelistSpec.scala]../../../../../akka-remote/src/test/scala/akka/remote/RemoteDeploymentWhitelistSpec.scala) { #whitelist-config } Actor classes not included in the whitelist will not be allowed to be remote deployed onto this system. -.. _remote-security-scala-artery: + +## Remote Security -Remote Security ---------------- - -An ``ActorSystem`` should not be exposed via Akka Remote (Artery) over plain Aeron/UDP to an untrusted network (e.g. internet). +An `ActorSystem` should not be exposed via Akka Remote (Artery) over plain Aeron/UDP to an untrusted network (e.g. internet). It should be protected by network security, such as a firewall. There is currently no support for encryption with Artery so if network security is not considered as enough protection the classic remoting with -:ref:`TLS and mutual authentication ` should be used. +@ref:[TLS and mutual authentication](remoting.md#remote-tls-scala) should be used. Best practice is that Akka remoting nodes should only be accessible from the adjacent network. -It is also security best practice to :ref:`disable the Java serializer ` because of -its multiple `known attack surfaces `_. +It is also security best practice to @ref:[disable the Java serializer](../java/remoting-artery.md#disable-java-serializer-java-artery) because of +its multiple [known attack surfaces](https://community.hpe.com/t5/Security-Research/The-perils-of-Java-deserialization/ba-p/6838995). -Untrusted Mode -^^^^^^^^^^^^^^ +### Untrusted Mode As soon as an actor system can connect to another remotely, it may in principle send any possible message to any actor contained within that remote system. One -example may be sending a :class:`PoisonPill` to the system guardian, shutting +example may be sending a `PoisonPill` to the system guardian, shutting that system down. This is not always desired, and it can be disabled with the -following setting:: +following setting: - akka.remote.artery.untrusted-mode = on +``` +akka.remote.artery.untrusted-mode = on +``` This disallows sending of system messages (actor life-cycle commands, -DeathWatch, etc.) and any message extending :class:`PossiblyHarmful` to the +DeathWatch, etc.) and any message extending `PossiblyHarmful` to the system on which this flag is set. Should a client send them nonetheless they are dropped and logged (at DEBUG level in order to reduce the possibilities for -a denial of service attack). :class:`PossiblyHarmful` covers the predefined -messages like :class:`PoisonPill` and :class:`Kill`, but it can also be added +a denial of service attack). `PossiblyHarmful` covers the predefined +messages like `PoisonPill` and `Kill`, but it can also be added as a marker trait to user-defined messages. -.. warning:: +@@@ warning - Untrusted mode does not give full protection against attacks by itself. - It makes it slightly harder to perform malicious or unintended actions but - it should be complemented with :ref:`disabled Java serializer `. - Additional protection can be achieved when running in an untrusted network by - network security (e.g. firewalls). +Untrusted mode does not give full protection against attacks by itself. +It makes it slightly harder to perform malicious or unintended actions but +it should be complemented with [disabled Java serializer](#disable-java-serializer-scala-artery). +Additional protection can be achieved when running in an untrusted network by +network security (e.g. firewalls). + +@@@ Messages sent with actor selection are by default discarded in untrusted mode, but permission to receive actor selection messages can be granted to specific actors -defined in configuration:: +defined in configuration: - akka.remote.artery..trusted-selection-paths = ["/user/receptionist", "/user/namingService"] +``` +akka.remote.artery..trusted-selection-paths = ["/user/receptionist", "/user/namingService"] +``` -The actual message must still not be of type :class:`PossiblyHarmful`. +The actual message must still not be of type `PossiblyHarmful`. In summary, the following operations are ignored by a system configured in untrusted mode when incoming via the remoting layer: -* remote deployment (which also means no remote supervision) -* remote DeathWatch -* ``system.stop()``, :class:`PoisonPill`, :class:`Kill` -* sending any message which extends from the :class:`PossiblyHarmful` marker - interface, which includes :class:`Terminated` -* messages sent with actor selection, unless destination defined in ``trusted-selection-paths``. + * remote deployment (which also means no remote supervision) + * remote DeathWatch + * `system.stop()`, `PoisonPill`, `Kill` + * sending any message which extends from the `PossiblyHarmful` marker +interface, which includes `Terminated` + * messages sent with actor selection, unless destination defined in `trusted-selection-paths`. -.. note:: +@@@ note - Enabling the untrusted mode does not remove the capability of the client to - freely choose the target of its message sends, which means that messages not - prohibited by the above rules can be sent to any actor in the remote system. - It is good practice for a client-facing system to only contain a well-defined - set of entry point actors, which then forward requests (possibly after - performing validation) to another actor system containing the actual worker - actors. If messaging between these two server-side systems is done using - local :class:`ActorRef` (they can be exchanged safely between actor systems - within the same JVM), you can restrict the messages on this interface by - marking them :class:`PossiblyHarmful` so that a client cannot forge them. +Enabling the untrusted mode does not remove the capability of the client to +freely choose the target of its message sends, which means that messages not +prohibited by the above rules can be sent to any actor in the remote system. +It is good practice for a client-facing system to only contain a well-defined +set of entry point actors, which then forward requests (possibly after +performing validation) to another actor system containing the actual worker +actors. If messaging between these two server-side systems is done using +local `ActorRef` (they can be exchanged safely between actor systems +within the same JVM), you can restrict the messages on this interface by +marking them `PossiblyHarmful` so that a client cannot forge them. +@@@ -Quarantine ----------- +## Quarantine Akka remoting is using Aeron as underlying message transport. Aeron is using UDP and adds among other things reliable delivery and session semantics, very similar to TCP. This means that -the order of the messages are preserved, which is needed for the :ref:`Actor message ordering guarantees `. +the order of the messages are preserved, which is needed for the @ref:[Actor message ordering guarantees](../general/message-delivery-reliability.md#message-ordering). Under normal circumstances all messages will be delivered but there are cases when messages may not be delivered to the destination: -* during a network partition and the Aeron session is broken, this automatically recovered once the partition is over -* when sending too many messages without flow control and thereby filling up the outbound send queue (``outbound-message-queue-size`` config) -* if serialization or deserialization of a message fails (only that message will be dropped) -* if an unexpected exception occurs in the remoting infrastructure + * during a network partition and the Aeron session is broken, this automatically recovered once the partition is over + * when sending too many messages without flow control and thereby filling up the outbound send queue (`outbound-message-queue-size` config) + * if serialization or deserialization of a message fails (only that message will be dropped) + * if an unexpected exception occurs in the remoting infrastructure -In short, Actor message delivery is “at-most-once” as described in :ref:`message-delivery-reliability` +In short, Actor message delivery is “at-most-once” as described in @ref:[Message Delivery Reliability](../general/message-delivery-reliability.md) Some messages in Akka are called system messages and those cannot be dropped because that would result in an inconsistent state between the systems. Such messages are used for essentially two features; remote death @@ -355,79 +369,79 @@ association with the destination system is irrecoverable failed, and Terminated actors on the remote system. It is placed in a so called quarantined state. Quarantine usually does not happen if remote watch or remote deployment is not used. -Each ``ActorSystem`` instance has an unique identifier (UID), which is important for differentiating between +Each `ActorSystem` instance has an unique identifier (UID), which is important for differentiating between incarnations of a system when it is restarted with the same hostname and port. It is the specific incarnation (UID) that is quarantined. The only way to recover from this state is to restart one of the actor systems. Messages that are sent to and received from a quarantined system will be dropped. However, it is possible to -send messages with ``actorSelection`` to the address of a quarantined system, which is useful to probe if the +send messages with `actorSelection` to the address of a quarantined system, which is useful to probe if the system has been restarted. An association will be quarantined when: -* Cluster node is removed from the cluster membership. -* Remote failure detector triggers, i.e. remote watch is used. This is different when :ref:`Akka Cluster ` - is used. The unreachable observation by the cluster failure detector can go back to reachable if the network - partition heals. A cluster member is not quarantined when the failure detector triggers. -* Overflow of the system message delivery buffer, e.g. because of too many ``watch`` requests at the same time - (``system-message-buffer-size`` config). -* Unexpected exception occurs in the control subchannel of the remoting infrastructure. + * Cluster node is removed from the cluster membership. + * Remote failure detector triggers, i.e. remote watch is used. This is different when @ref:[Akka Cluster](cluster-usage.md) +is used. The unreachable observation by the cluster failure detector can go back to reachable if the network +partition heals. A cluster member is not quarantined when the failure detector triggers. + * Overflow of the system message delivery buffer, e.g. because of too many `watch` requests at the same time +(`system-message-buffer-size` config). + * Unexpected exception occurs in the control subchannel of the remoting infrastructure. -The UID of the ``ActorSystem`` is exchanged in a two-way handshake when the first message is sent to +The UID of the `ActorSystem` is exchanged in a two-way handshake when the first message is sent to a destination. The handshake will be retried until the other system replies and no other messages will pass through until the handshake is completed. If the handshake cannot be established within a timeout -(``handshake-timeout`` config) the association is stopped (freeing up resources). Queued messages will be +(`handshake-timeout` config) the association is stopped (freeing up resources). Queued messages will be dropped if the handshake cannot be established. It will not be quarantined, because the UID is unknown. New handshake attempt will start when next message is sent to the destination. Handshake requests are actually also sent periodically to be able to establish a working connection when the destination system has been restarted. -Watching Remote Actors -^^^^^^^^^^^^^^^^^^^^^^ +### Watching Remote Actors Watching a remote actor is API wise not different than watching a local actor, as described in -:ref:`deathwatch-scala`. However, it is important to note, that unlike in the local case, remoting has to handle +@ref:[Lifecycle Monitoring aka DeathWatch](actors.md#deathwatch-scala). However, it is important to note, that unlike in the local case, remoting has to handle when a remote actor does not terminate in a graceful way sending a system message to notify the watcher actor about the event, but instead being hosted on a system which stopped abruptly (crashed). These situations are handled by the built-in failure detector. -Failure Detector -^^^^^^^^^^^^^^^^ +### Failure Detector -Under the hood remote death watch uses heartbeat messages and a failure detector to generate ``Terminated`` +Under the hood remote death watch uses heartbeat messages and a failure detector to generate `Terminated` message from network failures and JVM crashes, in addition to graceful termination of watched actor. The heartbeat arrival times is interpreted by an implementation of -`The Phi Accrual Failure Detector `_. +[The Phi Accrual Failure Detector](http://www.jaist.ac.jp/~defago/files/pdf/IS_RR_2004_010.pdf). The suspicion level of failure is given by a value called *phi*. The basic idea of the phi failure detector is to express the value of *phi* on a scale that is dynamically adjusted to reflect current network conditions. -The value of *phi* is calculated as:: +The value of *phi* is calculated as: - phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` +phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` where F is the cumulative distribution function of a normal distribution with mean and standard deviation estimated from historical heartbeat inter-arrival times. -In the :ref:`remote-configuration-artery-scala` you can adjust the ``akka.remote.watch-failure-detector.threshold`` +In the [Remote Configuration](#remote-configuration-artery-scala) you can adjust the `akka.remote.watch-failure-detector.threshold` to define when a *phi* value is considered to be a failure. -A low ``threshold`` is prone to generate many false positives but ensures -a quick detection in the event of a real crash. Conversely, a high ``threshold`` +A low `threshold` is prone to generate many false positives but ensures +a quick detection in the event of a real crash. Conversely, a high `threshold` generates fewer mistakes but needs more time to detect actual crashes. The -default ``threshold`` is 10 and is appropriate for most situations. However in +default `threshold` is 10 and is appropriate for most situations. However in cloud environments, such as Amazon EC2, the value could be increased to 12 in order to account for network issues that sometimes occur on such platforms. The following chart illustrates how *phi* increase with increasing time since the previous heartbeat. -.. image:: ../images/phi1.png +![phi1.png](../images/phi1.png) Phi is calculated from the mean and standard deviation of historical inter arrival times. The previous chart is an example for standard deviation @@ -435,32 +449,29 @@ of 200 ms. If the heartbeats arrive with less deviation the curve becomes steepe i.e. it is possible to determine failure more quickly. The curve looks like this for a standard deviation of 100 ms. -.. image:: ../images/phi2.png +![phi2.png](../images/phi2.png) To be able to survive sudden abnormalities, such as garbage collection pauses and transient network failures the failure detector is configured with a margin, -``akka.remote.watch-failure-detector.acceptable-heartbeat-pause``. You may want to -adjust the :ref:`remote-configuration-artery-scala` of this depending on you environment. -This is how the curve looks like for ``acceptable-heartbeat-pause`` configured to +`akka.remote.watch-failure-detector.acceptable-heartbeat-pause`. You may want to +adjust the [Remote Configuration](#remote-configuration-artery-scala) of this depending on you environment. +This is how the curve looks like for `acceptable-heartbeat-pause` configured to 3 seconds. -.. image:: ../images/phi3.png +![phi3.png](../images/phi3.png) -Serialization -------------- +## Serialization -When using remoting for actors you must ensure that the ``props`` and ``messages`` used for +When using remoting for actors you must ensure that the `props` and `messages` used for those actors are serializable. Failing to do so will cause the system to behave in an unintended way. -For more information please see :ref:`serialization-scala`. +For more information please see @ref:[Serialization](serialization.md). -.. _remote-bytebuffer-serialization-scala: + +### ByteBuffer based serialization -ByteBuffer based serialization -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Artery introduces a new serialization mechanism which allows the ``ByteBufferSerializer`` to directly write into a -shared :class:`java.nio.ByteBuffer` instead of being forced to allocate and return an ``Array[Byte]`` for each serialized +Artery introduces a new serialization mechanism which allows the `ByteBufferSerializer` to directly write into a +shared `java.nio.ByteBuffer` instead of being forced to allocate and return an `Array[Byte]` for each serialized message. For high-throughput messaging this API change can yield significant performance benefits, so we recommend changing your serializers to use this new mechanism. @@ -468,61 +479,61 @@ This new API also plays well with new versions of Google Protocol Buffers and ot the ability to serialize directly into and from ByteBuffers. As the new feature only changes how bytes are read and written, and the rest of the serialization infrastructure -remained the same, we recommend reading the :ref:`serialization-scala` documentation first. +remained the same, we recommend reading the @ref:[Serialization](serialization.md) documentation first. -Implementing an :class:`akka.serialization.ByteBufferSerializer` works the same way as any other serializer, +Implementing an `akka.serialization.ByteBufferSerializer` works the same way as any other serializer, -.. includecode:: ../../../akka-actor/src/main/scala/akka/serialization/Serializer.scala#ByteBufferSerializer +@@snip [Serializer.scala]../../../../../akka-actor/src/main/scala/akka/serialization/Serializer.scala) { #ByteBufferSerializer } Implementing a serializer for Artery is therefore as simple as implementing this interface, and binding the serializer -as usual (which is explained in :ref:`serialization-scala`). +as usual (which is explained in @ref:[Serialization](serialization.md)). -Implementations should typically extend ``SerializerWithStringManifest`` and in addition to the ``ByteBuffer`` based -``toBinary`` and ``fromBinary`` methods also implement the array based ``toBinary`` and ``fromBinary`` methods. -The array based methods will be used when ``ByteBuffer`` is not used, e.g. in Akka Persistence. +Implementations should typically extend `SerializerWithStringManifest` and in addition to the `ByteBuffer` based +`toBinary` and `fromBinary` methods also implement the array based `toBinary` and `fromBinary` methods. +The array based methods will be used when `ByteBuffer` is not used, e.g. in Akka Persistence. Note that the array based methods can be implemented by delegation like this: -.. includecode:: code/docs/actor/ByteBufferSerializerDocSpec.scala#bytebufserializer-with-manifest +@@snip [ByteBufferSerializerDocSpec.scala](code/docs/actor/ByteBufferSerializerDocSpec.scala) { #bytebufserializer-with-manifest } -.. _disable-java-serializer-scala-artery: - -Disabling the Java Serializer -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +### Disabling the Java Serializer It is possible to completely disable Java Serialization for the entire Actor system. -Java serialization is known to be slow and `prone to attacks -`_ +Java serialization is known to be slow and [prone to attacks](https://community.hpe.com/t5/Security-Research/The-perils-of-Java-deserialization/ba-p/6838995) of various kinds - it never was designed for high throughput messaging after all. However, it is very convenient to use, thus it remained the default serialization mechanism that Akka used to serialize user messages as well as some of its internal messages in previous versions. -Since the release of Artery, Akka internals do not rely on Java serialization anymore (exceptions to that being ``java.lang.Throwable`` and "remote deployment"). +Since the release of Artery, Akka internals do not rely on Java serialization anymore (exceptions to that being `java.lang.Throwable` and "remote deployment"). -.. note:: - Akka does not use Java Serialization for any of its internal messages. - It is highly encouraged to disable java serialization, so please plan to do so at the earliest possibility you have in your project. +@@@ note - One may think that network bandwidth and latency limit the performance of remote messaging, but serialization is a more typical bottleneck. +Akka does not use Java Serialization for any of its internal messages. +It is highly encouraged to disable java serialization, so please plan to do so at the earliest possibility you have in your project. + +One may think that network bandwidth and latency limit the performance of remote messaging, but serialization is a more typical bottleneck. + +@@@ For user messages, the default serializer, implemented using Java serialization, remains available and enabled. We do however recommend to disable it entirely and utilise a proper serialization library instead in order effectively utilise the improved performance and ability for rolling deployments using Artery. Libraries that we recommend to use include, -but are not limited to, `Kryo`_ by using the `akka-kryo-serialization`_ library or `Google Protocol Buffers`_ if you want +but are not limited to, [Kryo](https://github.com/EsotericSoftware/kryo) by using the [akka-kryo-serialization](https://github.com/romix/akka-kryo-serialization) library or [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) if you want more control over the schema evolution of your messages. In order to completely disable Java Serialization in your Actor system you need to add the following configuration to -your ``application.conf``: +your `application.conf`: -.. code-block:: ruby +```ruby +akka.actor.allow-java-serialization = off +``` - akka.actor.allow-java-serialization = off - -This will completely disable the use of ``akka.serialization.JavaSerialization`` by the -Akka Serialization extension, instead ``DisabledJavaSerializer`` will +This will completely disable the use of `akka.serialization.JavaSerialization` by the +Akka Serialization extension, instead `DisabledJavaSerializer` will be inserted which will fail explicitly if attempts to use java serialization are made. -It will also enable the above mentioned `enable-additional-serialization-bindings`. +It will also enable the above mentioned *enable-additional-serialization-bindings*. The log messages emitted by such serializer SHOULD be be treated as potential attacks which the serializer prevented, as they MAY indicate an external operator @@ -532,46 +543,37 @@ The attempts are logged with the SECURITY marker. Please note that this option does not stop you from manually invoking java serialization. Please note that this means that you will have to configure different serializers which will able to handle all of your -remote messages. Please refer to the :ref:`serialization-scala` documentation as well as :ref:`ByteBuffer based serialization ` to learn how to do this. +remote messages. Please refer to the @ref:[Serialization](serialization.md) documentation as well as [ByteBuffer based serialization](#remote-bytebuffer-serialization-scala) to learn how to do this. -.. _Kryo: https://github.com/EsotericSoftware/kryo -.. _akka-kryo-serialization: https://github.com/romix/akka-kryo-serialization -.. _Google Protocol Buffers: https://developers.google.com/protocol-buffers/ +## Routers with Remote Destinations -Routers with Remote Destinations --------------------------------- - -It is absolutely feasible to combine remoting with :ref:`routing-scala`. +It is absolutely feasible to combine remoting with @ref:[Routing](routing.md). A pool of remote deployed routees can be configured as: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-pool-artery +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-pool-artery } -This configuration setting will clone the actor defined in the ``Props`` of the ``remotePool`` 10 +This configuration setting will clone the actor defined in the `Props` of the `remotePool` 10 times and deploy it evenly distributed across the two given target nodes. A group of remote actors can be configured as: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-group-artery +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-group-artery } This configuration setting will send messages to the defined remote actor paths. It requires that you create the destination actors on the remote nodes with matching paths. That is not done by the router. -.. _remote-sample-scala-artery: + +## Remoting Sample -Remoting Sample ---------------- - -You can download a ready to run `remoting sample <@exampleCodeService@/akka-samples-remote-scala>`_ +You can download a ready to run [remoting sample](@exampleCodeService@/akka-samples-remote-scala) together with a tutorial for a more hands-on experience. The source code of this sample can be found in the -`Akka Samples Repository <@samples@/akka-sample-remote-scala>`_. +[Akka Samples Repository](@samples@/akka-sample-remote-scala). -Performance tuning ------------------- +## Performance tuning -Dedicated subchannel for large messages -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Dedicated subchannel for large messages All the communication between user defined remote actors are isolated from the channel of Akka internal messages so a large user message cannot block an urgent system message. While this provides good isolation for Akka services, all @@ -583,31 +585,32 @@ helpful to separate actors that have different QoS requirements: large messages Akka remoting provides a dedicated channel for large messages if configured. Since actor message ordering must not be violated the channel is actually dedicated for *actors* instead of messages, to ensure all of the messages arrive in send order. It is possible to assign actors on given paths to use this dedicated channel by using -path patterns that have to be specified in the actor system's configuration on both the sending and the receiving side:: +path patterns that have to be specified in the actor system's configuration on both the sending and the receiving side: - akka.remote.artery.large-message-destinations = [ - "/user/largeMessageActor", - "/user/largeMessagesGroup/*", - "/user/anotherGroup/*/largeMesssages", - "/user/thirdGroup/**", - ] +``` +akka.remote.artery.large-message-destinations = [ + "/user/largeMessageActor", + "/user/largeMessagesGroup/*", + "/user/anotherGroup/*/largeMesssages", + "/user/thirdGroup/**", +] +``` This means that all messages sent to the following actors will pass through the dedicated, large messages channel: -* ``/user/largeMessageActor`` -* ``/user/largeMessageActorGroup/actor1`` -* ``/user/largeMessageActorGroup/actor2`` -* ``/user/anotherGroup/actor1/largeMessages`` -* ``/user/anotherGroup/actor2/largeMessages`` -* ``/user/thirdGroup/actor3/`` -* ``/user/thirdGroup/actor4/actor5`` + * `/user/largeMessageActor` + * `/user/largeMessageActorGroup/actor1` + * `/user/largeMessageActorGroup/actor2` + * `/user/anotherGroup/actor1/largeMessages` + * `/user/anotherGroup/actor2/largeMessages` + * `/user/thirdGroup/actor3/` + * `/user/thirdGroup/actor4/actor5` Messages destined for actors not matching any of these patterns are sent using the default channel as before. -External, shared Aeron media driver -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### External, shared Aeron media driver -The Aeron transport is running in a so called `media driver `_. +The Aeron transport is running in a so called [media driver](https://github.com/real-logic/Aeron/wiki/Media-Driver-Operation). By default, Akka starts the media driver embedded in the same JVM process as application. This is convenient and simplifies operational concerns by only having one process to start and monitor. @@ -617,96 +620,109 @@ same machine it can therefore be wise to share the media driver by running it as The media driver has also different resource usage characteristics than a normal application and it can therefore be more efficient and stable to run the media driver as a separate process. -Given that Aeron jar files are in the classpath the standalone media driver can be started with:: +Given that Aeron jar files are in the classpath the standalone media driver can be started with: - java io.aeron.driver.MediaDriver +``` +java io.aeron.driver.MediaDriver +``` -The needed classpath:: +The needed classpath: - Agrona-0.5.4.jar:aeron-driver-1.0.1.jar:aeron-client-1.0.1.jar +``` +Agrona-0.5.4.jar:aeron-driver-1.0.1.jar:aeron-client-1.0.1.jar +``` -You find those jar files on `maven central `_, or you can create a +You find those jar files on [maven central](http://search.maven.org/), or you can create a package with your preferred build tool. -You can pass `Aeron properties `_ as -command line `-D` system properties:: +You can pass [Aeron properties](https://github.com/real-logic/Aeron/wiki/Configuration-Options) as +command line *-D* system properties: - -Daeron.dir=/dev/shm/aeron +``` +-Daeron.dir=/dev/shm/aeron +``` -You can also define Aeron properties in a file:: +You can also define Aeron properties in a file: - java io.aeron.driver.MediaDriver config/aeron.properties +``` +java io.aeron.driver.MediaDriver config/aeron.properties +``` -An example of such a properties file:: +An example of such a properties file: - aeron.mtu.length=16384 - aeron.socket.so_sndbuf=2097152 - aeron.socket.so_rcvbuf=2097152 - aeron.rcv.buffer.length=16384 - aeron.rcv.initial.window.length=2097152 - agrona.disable.bounds.checks=true +``` +aeron.mtu.length=16384 +aeron.socket.so_sndbuf=2097152 +aeron.socket.so_rcvbuf=2097152 +aeron.rcv.buffer.length=16384 +aeron.rcv.initial.window.length=2097152 +agrona.disable.bounds.checks=true - aeron.threading.mode=SHARED_NETWORK +aeron.threading.mode=SHARED_NETWORK - # low latency settings - #aeron.threading.mode=DEDICATED - #aeron.sender.idle.strategy=org.agrona.concurrent.BusySpinIdleStrategy - #aeron.receiver.idle.strategy=org.agrona.concurrent.BusySpinIdleStrategy +# low latency settings +#aeron.threading.mode=DEDICATED +#aeron.sender.idle.strategy=org.agrona.concurrent.BusySpinIdleStrategy +#aeron.receiver.idle.strategy=org.agrona.concurrent.BusySpinIdleStrategy - # use same director in akka.remote.artery.advanced.aeron-dir config - # of the Akka application - aeron.dir=/dev/shm/aeron +# use same director in akka.remote.artery.advanced.aeron-dir config +# of the Akka application +aeron.dir=/dev/shm/aeron +``` -Read more about the media driver in the `Aeron documentation `_. +Read more about the media driver in the [Aeron documentation](https://github.com/real-logic/Aeron/wiki/Media-Driver-Operation). To use the external media driver from the Akka application you need to define the following two -configuration properties:: +configuration properties: - akka.remote.artery.advanced { - embedded-media-driver = off - aeron-dir = /dev/shm/aeron - } +``` +akka.remote.artery.advanced { + embedded-media-driver = off + aeron-dir = /dev/shm/aeron +} +``` -The ``aeron-dir`` must match the directory you started the media driver with, i.e. the ``aeron.dir`` property. +The `aeron-dir` must match the directory you started the media driver with, i.e. the `aeron.dir` property. Several Akka applications can then be configured to use the same media driver by pointing to the same directory. Note that if the media driver process is stopped the Akka applications that are using it will also be stopped. -Aeron Tuning -^^^^^^^^^^^^ +### Aeron Tuning -See Aeron documentation about `Performance Testing `_. +See Aeron documentation about [Performance Testing](https://github.com/real-logic/Aeron/wiki/Performance-Testing). -Fine-tuning CPU usage latency tradeoff -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Fine-tuning CPU usage latency tradeoff Artery has been designed for low latency and as a result it can be CPU hungry when the system is mostly idle. This is not always desirable. It is possible to tune the tradeoff between CPU usage and latency with the following configuration: - # Values can be from 1 to 10, where 10 strongly prefers low latency - # and 1 strongly prefers less CPU usage - akka.remote.artery.advanced.idle-cpu-level = 1 +> +# Values can be from 1 to 10, where 10 strongly prefers low latency +# and 1 strongly prefers less CPU usage +akka.remote.artery.advanced.idle-cpu-level = 1 By setting this value to a lower number, it tells Akka to do longer "sleeping" periods on its thread dedicated -for `spin-waiting `_ and hence reducing CPU load when there is no +for [spin-waiting](https://en.wikipedia.org/wiki/Busy_waiting) and hence reducing CPU load when there is no immediate task to execute at the cost of a longer reaction time to an event when it actually happens. It is worth to be noted though that during a continuously high-throughput period this setting makes not much difference as the thread mostly has tasks to execute. This also means that under high throughput (but below maximum capacity) the system might have less latency than at low message rates. -Internal Event Log for Debugging (Flight Recorder) --------------------------------------------------- +## Internal Event Log for Debugging (Flight Recorder) -.. note:: - In this version (@version@) the flight-recorder is disabled by default because there is no automatic - file name and path calculation implemented to make it possible to reuse the same file for every restart of - the same actor system without clashing with files produced by other systems (possibly running on the same machine). - Currently, you have to set the path and file names yourself to avoid creating an unbounded number - of files and enable flight recorder manually by adding `akka.remote.artery.advanced.flight-recorder.enabled=on` to - your configuration file. This a limitation of the current version and will not be necessary in the future. +@@@ note + +In this version (@version@) the flight-recorder is disabled by default because there is no automatic +file name and path calculation implemented to make it possible to reuse the same file for every restart of +the same actor system without clashing with files produced by other systems (possibly running on the same machine). +Currently, you have to set the path and file names yourself to avoid creating an unbounded number +of files and enable flight recorder manually by adding *akka.remote.artery.advanced.flight-recorder.enabled=on* to +your configuration file. This a limitation of the current version and will not be necessary in the future. + +@@@ Emitting event information (logs) from internals is always a tradeoff. The events that are usable for the Akka developers are usually too low level to be of any use for users and usually need to be fine-grained enough @@ -722,54 +738,51 @@ more insight into the nature of the failure. There are various important features of this event log: -* Flight Recorder produces a fixed size file completely encapsulating log rotation. This means that this - file will never grow in size and will not cause any unexpected disk space shortage in production. -* This file is crash resistant, i.e. its contents can be recovered even if the JVM hosting the :class:`ActorSystem` - crashes unexpectedly. -* Very low overhead, specialized, binary logging that has no significant overhead and can be safely left enabled - for production systems. + * Flight Recorder produces a fixed size file completely encapsulating log rotation. This means that this +file will never grow in size and will not cause any unexpected disk space shortage in production. + * This file is crash resistant, i.e. its contents can be recovered even if the JVM hosting the `ActorSystem` +crashes unexpectedly. + * Very low overhead, specialized, binary logging that has no significant overhead and can be safely left enabled +for production systems. -The location of the file can be controlled via the `akka.remote.artery.advanced.flight-recoder.destination` setting (see -:ref:`config-akka-remote-artery` for details). By default, a file with the `.afr` extension is produced in the temporary +The location of the file can be controlled via the *akka.remote.artery.advanced.flight-recoder.destination* setting (see +@ref:[akka-remote (artery)](../general/configuration.md#config-akka-remote-artery) for details). By default, a file with the *.afr* extension is produced in the temporary directory of the operating system. In cases where the flight recorder casuses issues, it can be disabled by adding the -setting `akka.remote.artery.advanced.flight-recorder.enabled=off`, although this is not recommended. +setting *akka.remote.artery.advanced.flight-recorder.enabled=off*, although this is not recommended. -.. _remote-configuration-artery-scala: - -Remote Configuration --------------------- + +## Remote Configuration There are lots of configuration properties that are related to remoting in Akka. We refer to the -:ref:`reference configuration ` for more information. +@ref:[reference configuration](../general/configuration.md#config-akka-remote-artery) for more information. -.. note:: +@@@ note - Setting properties like the listening IP and port number programmatically is - best done by using something like the following: +Setting properties like the listening IP and port number programmatically is +best done by using something like the following: - .. includecode:: ../java/code/jdocs/remoting/RemoteDeploymentDocTest.java#programmatic-artery +@@snip [RemoteDeploymentDocTest.java](../java/code/jdocs/remoting/RemoteDeploymentDocTest.java) { #programmatic-artery } +@@@ -.. _remote-configuration-nat-artery-scala: - -Akka behind NAT or in a Docker container -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +### Akka behind NAT or in a Docker container In setups involving Network Address Translation (NAT), Load Balancers or Docker containers the hostname and port pair that Akka binds to will be different than the "logical" host name and port pair that is used to connect to the system from the outside. This requires special configuration that sets both the logical and the bind pairs for remoting. -.. code-block:: ruby +```ruby +akka { + remote { + artery { + canonical.hostname = my.domain.com # external (logical) hostname + canonical.port = 8000 # external (logical) port - akka { - remote { - artery { - canonical.hostname = my.domain.com # external (logical) hostname - canonical.port = 8000 # external (logical) port - - bind.hostname = local.address # internal (bind) hostname - bind.port = 25520 # internal (bind) port - } - } - } + bind.hostname = local.address # internal (bind) hostname + bind.port = 25520 # internal (bind) port + } + } +} +``` \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/remoting.md b/akka-docs/src/main/paradox/scala/remoting.md index 66a5716e1c..b7727e2faf 100644 --- a/akka-docs/src/main/paradox/scala/remoting.md +++ b/akka-docs/src/main/paradox/scala/remoting.md @@ -1,183 +1,198 @@ -.. _remoting-scala: +# Remoting -Remoting -######## +For an introduction of remoting capabilities of Akka please see [Location Transparency](). -For an introduction of remoting capabilities of Akka please see :ref:`remoting`. +@@@ note -.. note:: +As explained in that chapter Akka remoting is designed for communication in a +peer-to-peer fashion and it has limitations for client-server setups. In +particular Akka Remoting does not work transparently with Network Address Translation, +Load Balancers, or in Docker containers. For symmetric communication in these situations +network and/or Akka configuration will have to be changed as described in +[Akka behind NAT or in a Docker container](#remote-configuration-nat). - As explained in that chapter Akka remoting is designed for communication in a - peer-to-peer fashion and it has limitations for client-server setups. In - particular Akka Remoting does not work transparently with Network Address Translation, - Load Balancers, or in Docker containers. For symmetric communication in these situations - network and/or Akka configuration will have to be changed as described in - :ref:`remote-configuration-nat`. +@@@ -Preparing your ActorSystem for Remoting -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## Preparing your ActorSystem for Remoting -The Akka remoting is a separate jar file. Make sure that you have the following dependency in your project:: +The Akka remoting is a separate jar file. Make sure that you have the following dependency in your project: - "com.typesafe.akka" %% "akka-remote" % "@version@" @crossString@ +``` +"com.typesafe.akka" %% "akka-remote" % "@version@" @crossString@ +``` To enable remote capabilities in your Akka project you should, at a minimum, add the following changes -to your ``application.conf`` file:: +to your `application.conf` file: - akka { - actor { - provider = remote - } - remote { - enabled-transports = ["akka.remote.netty.tcp"] - netty.tcp { - hostname = "127.0.0.1" - port = 2552 - } - } +``` +akka { + actor { + provider = remote } + remote { + enabled-transports = ["akka.remote.netty.tcp"] + netty.tcp { + hostname = "127.0.0.1" + port = 2552 + } + } +} +``` As you can see in the example above there are four things you need to add to get started: -* Change provider from ``local`` to ``remote`` -* Add host name - the machine you want to run the actor system on; this host - name is exactly what is passed to remote systems in order to identify this - system and consequently used for connecting back to this system if need be, - hence set it to a reachable IP address or resolvable name in case you want to - communicate across the network. -* Add port number - the port the actor system should listen on, set to 0 to have it chosen automatically + * Change provider from `local` to `remote` + * Add host name - the machine you want to run the actor system on; this host +name is exactly what is passed to remote systems in order to identify this +system and consequently used for connecting back to this system if need be, +hence set it to a reachable IP address or resolvable name in case you want to +communicate across the network. + * Add port number - the port the actor system should listen on, set to 0 to have it chosen automatically -.. note:: - The port number needs to be unique for each actor system on the same machine even if the actor - systems have different names. This is because each actor system has its own networking subsystem - listening for connections and handling messages as not to interfere with other actor systems. +@@@ note + +The port number needs to be unique for each actor system on the same machine even if the actor +systems have different names. This is because each actor system has its own networking subsystem +listening for connections and handling messages as not to interfere with other actor systems. + +@@@ The example above only illustrates the bare minimum of properties you have to add to enable remoting. -All settings are described in :ref:`remote-configuration-scala`. +All settings are described in [Remote Configuration](#remote-configuration-scala). -Types of Remote Interaction -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## Types of Remote Interaction Akka has two ways of using remoting: -* Lookup : used to look up an actor on a remote node with ``actorSelection(path)`` -* Creation : used to create an actor on a remote node with ``actorOf(Props(...), actorName)`` + * Lookup : used to look up an actor on a remote node with `actorSelection(path)` + * Creation : used to create an actor on a remote node with `actorOf(Props(...), actorName)` In the next sections the two alternatives are described in detail. -Looking up Remote Actors -^^^^^^^^^^^^^^^^^^^^^^^^ +## Looking up Remote Actors -``actorSelection(path)`` will obtain an ``ActorSelection`` to an Actor on a remote node, e.g.:: +`actorSelection(path)` will obtain an `ActorSelection` to an Actor on a remote node, e.g.: - val selection = - context.actorSelection("akka.tcp://actorSystemName@10.0.0.1:2552/user/actorName") +``` +val selection = + context.actorSelection("akka.tcp://actorSystemName@10.0.0.1:2552/user/actorName") +``` -As you can see from the example above the following pattern is used to find an actor on a remote node:: +As you can see from the example above the following pattern is used to find an actor on a remote node: - akka.://@:/ +``` +akka.://@:/ +``` -Once you obtained a selection to the actor you can interact with it in the same way you would with a local actor, e.g.:: +Once you obtained a selection to the actor you can interact with it in the same way you would with a local actor, e.g.: - selection ! "Pretty awesome feature" +``` +selection ! "Pretty awesome feature" +``` -To acquire an :class:`ActorRef` for an :class:`ActorSelection` you need to -send a message to the selection and use the ``sender`` reference of the reply from -the actor. There is a built-in ``Identify`` message that all Actors will understand -and automatically reply to with a ``ActorIdentity`` message containing the -:class:`ActorRef`. This can also be done with the ``resolveOne`` method of -the :class:`ActorSelection`, which returns a ``Future`` of the matching -:class:`ActorRef`. +To acquire an `ActorRef` for an `ActorSelection` you need to +send a message to the selection and use the `sender` reference of the reply from +the actor. There is a built-in `Identify` message that all Actors will understand +and automatically reply to with a `ActorIdentity` message containing the +`ActorRef`. This can also be done with the `resolveOne` method of +the `ActorSelection`, which returns a `Future` of the matching +`ActorRef`. -.. note:: +@@@ note - For more details on how actor addresses and paths are formed and used, please refer to :ref:`addressing`. +For more details on how actor addresses and paths are formed and used, please refer to @ref:[Actor References, Paths and Addresses](../general/addressing.md). -.. note:: +@@@ - Message sends to actors that are actually in the sending actor system do not - get delivered via the remote actor ref provider. They're delivered directly, - by the local actor ref provider. +@@@ note - Aside from providing better performance, this also means that if the hostname - you configure remoting to listen as cannot actually be resolved from within - the very same actor system, such messages will (perhaps counterintuitively) - be delivered just fine. +Message sends to actors that are actually in the sending actor system do not +get delivered via the remote actor ref provider. They're delivered directly, +by the local actor ref provider. -Creating Actors Remotely -^^^^^^^^^^^^^^^^^^^^^^^^ +Aside from providing better performance, this also means that if the hostname +you configure remoting to listen as cannot actually be resolved from within +the very same actor system, such messages will (perhaps counterintuitively) +be delivered just fine. + +@@@ + +## Creating Actors Remotely If you want to use the creation functionality in Akka remoting you have to further amend the -``application.conf`` file in the following way (only showing deployment section):: +`application.conf` file in the following way (only showing deployment section): - akka { - actor { - deployment { - /sampleActor { - remote = "akka.tcp://sampleActorSystem@127.0.0.1:2553" - } +``` +akka { + actor { + deployment { + /sampleActor { + remote = "akka.tcp://sampleActorSystem@127.0.0.1:2553" } } } +} +``` -The configuration above instructs Akka to react when an actor with path ``/sampleActor`` is created, i.e. -using ``system.actorOf(Props(...), "sampleActor")``. This specific actor will not be directly instantiated, +The configuration above instructs Akka to react when an actor with path `/sampleActor` is created, i.e. +using `system.actorOf(Props(...), "sampleActor")`. This specific actor will not be directly instantiated, but instead the remote daemon of the remote system will be asked to create the actor, -which in this sample corresponds to ``sampleActorSystem@127.0.0.1:2553``. +which in this sample corresponds to `sampleActorSystem@127.0.0.1:2553`. Once you have configured the properties above you would do the following in code: -.. includecode:: code/docs/remoting/RemoteDeploymentDocSpec.scala#sample-actor +@@snip [RemoteDeploymentDocSpec.scala](code/docs/remoting/RemoteDeploymentDocSpec.scala) { #sample-actor } -The actor class ``SampleActor`` has to be available to the runtimes using it, i.e. the classloader of the +The actor class `SampleActor` has to be available to the runtimes using it, i.e. the classloader of the actor systems has to have a JAR containing the class. -.. note:: +@@@ note - In order to ensure serializability of ``Props`` when passing constructor - arguments to the actor being created, do not make the factory an inner class: - this will inherently capture a reference to its enclosing object, which in - most cases is not serializable. It is best to create a factory method in the - companion object of the actor’s class. +In order to ensure serializability of `Props` when passing constructor +arguments to the actor being created, do not make the factory an inner class: +this will inherently capture a reference to its enclosing object, which in +most cases is not serializable. It is best to create a factory method in the +companion object of the actor’s class. - Serializability of all Props can be tested by setting the configuration item - ``akka.actor.serialize-creators=on``. Only Props whose ``deploy`` has - ``LocalScope`` are exempt from this check. +Serializability of all Props can be tested by setting the configuration item +`akka.actor.serialize-creators=on`. Only Props whose `deploy` has +`LocalScope` are exempt from this check. -.. note:: +@@@ - You can use asterisks as wildcard matches for the actor paths, so you could specify: - ``/*/sampleActor`` and that would match all ``sampleActor`` on that level in the hierarchy. - You can also use wildcard in the last position to match all actors at a certain level: - ``/someParent/*``. Non-wildcard matches always have higher priority to match than wildcards, so: - ``/foo/bar`` is considered **more specific** than ``/foo/*`` and only the highest priority match is used. - Please note that it **cannot** be used to partially match section, like this: ``/foo*/bar``, ``/f*o/bar`` etc. +@@@ note -Programmatic Remote Deployment ------------------------------- +You can use asterisks as wildcard matches for the actor paths, so you could specify: +`/*/sampleActor` and that would match all `sampleActor` on that level in the hierarchy. +You can also use wildcard in the last position to match all actors at a certain level: +`/someParent/*`. Non-wildcard matches always have higher priority to match than wildcards, so: +`/foo/bar` is considered **more specific** than `/foo/*` and only the highest priority match is used. +Please note that it **cannot** be used to partially match section, like this: `/foo*/bar`, `/f*o/bar` etc. + +@@@ + +### Programmatic Remote Deployment To allow dynamically deployed systems, it is also possible to include -deployment configuration in the :class:`Props` which are used to create an +deployment configuration in the `Props` which are used to create an actor: this information is the equivalent of a deployment section from the configuration file, and if both are given, the external configuration takes precedence. With these imports: -.. includecode:: code/docs/remoting/RemoteDeploymentDocSpec.scala#import +@@snip [RemoteDeploymentDocSpec.scala](code/docs/remoting/RemoteDeploymentDocSpec.scala) { #import } and a remote address like this: -.. includecode:: code/docs/remoting/RemoteDeploymentDocSpec.scala#make-address +@@snip [RemoteDeploymentDocSpec.scala](code/docs/remoting/RemoteDeploymentDocSpec.scala) { #make-address } you can advise the system to create a child on that remote node like so: -.. includecode:: code/docs/remoting/RemoteDeploymentDocSpec.scala#deploy +@@snip [RemoteDeploymentDocSpec.scala](code/docs/remoting/RemoteDeploymentDocSpec.scala) { #deploy } -.. _remote-deployment-whitelist-scala: - -Remote deployment whitelist ---------------------------- + +### Remote deployment whitelist As remote deployment can potentially be abused by both users and even attackers a whitelist feature is available to guard the ActorSystem from deploying unexpected actors. Please note that remote deployment @@ -185,79 +200,76 @@ is *not* remote code loading, the Actors class to be deployed onto a remote syst remote system. This still however may pose a security risk, and one may want to restrict remote deployment to only a specific set of known actors by enabling the whitelist feature. -To enable remote deployment whitelisting set the ``akka.remote.deployment.enable-whitelist`` value to ``on``. +To enable remote deployment whitelisting set the `akka.remote.deployment.enable-whitelist` value to `on`. The list of allowed classes has to be configured on the "remote" system, in other words on the system onto which others will be attempting to remote deploy Actors. That system, locally, knows best which Actors it should or should not allow others to remote deploy onto it. The full settings section may for example look like this: -.. includecode:: ../../../akka-remote/src/test/scala/akka/remote/RemoteDeploymentWhitelistSpec.scala#whitelist-config +@@snip [RemoteDeploymentWhitelistSpec.scala]../../../../../akka-remote/src/test/scala/akka/remote/RemoteDeploymentWhitelistSpec.scala) { #whitelist-config } Actor classes not included in the whitelist will not be allowed to be remote deployed onto this system. -Lifecycle and Failure Recovery Model -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +## Lifecycle and Failure Recovery Model -.. image:: ../images/association_lifecycle.png - :align: center - :width: 620 +![association_lifecycle.png](../images/association_lifecycle.png) Each link with a remote system can be in one of the four states as illustrated above. Before any communication -happens with a remote system at a given ``Address`` the state of the association is ``Idle``. The first time a message +happens with a remote system at a given `Address` the state of the association is `Idle`. The first time a message is attempted to be sent to the remote system or an inbound connection is accepted the state of the link transitions to -``Active`` denoting that the two systems has messages to send or receive and no failures were encountered so far. -When a communication failure happens and the connection is lost between the two systems the link becomes ``Gated``. +`Active` denoting that the two systems has messages to send or receive and no failures were encountered so far. +When a communication failure happens and the connection is lost between the two systems the link becomes `Gated`. In this state the system will not attempt to connect to the remote host and all outbound messages will be dropped. The time -while the link is in the ``Gated`` state is controlled by the setting ``akka.remote.retry-gate-closed-for``: -after this time elapses the link state transitions to ``Idle`` again. ``Gate`` is one-sided in the -sense that whenever a successful *inbound* connection is accepted from a remote system during ``Gate`` it automatically -transitions to ``Active`` and communication resumes immediately. +while the link is in the `Gated` state is controlled by the setting `akka.remote.retry-gate-closed-for`: +after this time elapses the link state transitions to `Idle` again. `Gate` is one-sided in the +sense that whenever a successful *inbound* connection is accepted from a remote system during `Gate` it automatically +transitions to `Active` and communication resumes immediately. In the face of communication failures that are unrecoverable because the state of the participating systems are inconsistent, -the remote system becomes ``Quarantined``. Unlike ``Gate``, quarantining is permanent and lasts until one of the systems -is restarted. After a restart communication can be resumed again and the link can become ``Active`` again. +the remote system becomes `Quarantined`. Unlike `Gate`, quarantining is permanent and lasts until one of the systems +is restarted. After a restart communication can be resumed again and the link can become `Active` again. -Watching Remote Actors -^^^^^^^^^^^^^^^^^^^^^^ +## Watching Remote Actors Watching a remote actor is not different than watching a local actor, as described in -:ref:`deathwatch-scala`. +@ref:[Lifecycle Monitoring aka DeathWatch](actors.md#deathwatch-scala). -Failure Detector ----------------- +### Failure Detector -Under the hood remote death watch uses heartbeat messages and a failure detector to generate ``Terminated`` +Under the hood remote death watch uses heartbeat messages and a failure detector to generate `Terminated` message from network failures and JVM crashes, in addition to graceful termination of watched actor. The heartbeat arrival times is interpreted by an implementation of -`The Phi Accrual Failure Detector `_. +[The Phi Accrual Failure Detector](http://www.jaist.ac.jp/~defago/files/pdf/IS_RR_2004_010.pdf). The suspicion level of failure is given by a value called *phi*. The basic idea of the phi failure detector is to express the value of *phi* on a scale that is dynamically adjusted to reflect current network conditions. -The value of *phi* is calculated as:: +The value of *phi* is calculated as: - phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` +phi = -log10(1 - F(timeSinceLastHeartbeat)) +``` where F is the cumulative distribution function of a normal distribution with mean and standard deviation estimated from historical heartbeat inter-arrival times. -In the :ref:`remote-configuration-scala` you can adjust the ``akka.remote.watch-failure-detector.threshold`` +In the [Remote Configuration](#remote-configuration-scala) you can adjust the `akka.remote.watch-failure-detector.threshold` to define when a *phi* value is considered to be a failure. -A low ``threshold`` is prone to generate many false positives but ensures -a quick detection in the event of a real crash. Conversely, a high ``threshold`` +A low `threshold` is prone to generate many false positives but ensures +a quick detection in the event of a real crash. Conversely, a high `threshold` generates fewer mistakes but needs more time to detect actual crashes. The -default ``threshold`` is 10 and is appropriate for most situations. However in +default `threshold` is 10 and is appropriate for most situations. However in cloud environments, such as Amazon EC2, the value could be increased to 12 in order to account for network issues that sometimes occur on such platforms. The following chart illustrates how *phi* increase with increasing time since the previous heartbeat. -.. image:: ../images/phi1.png +![phi1.png](../images/phi1.png) Phi is calculated from the mean and standard deviation of historical inter arrival times. The previous chart is an example for standard deviation @@ -265,32 +277,29 @@ of 200 ms. If the heartbeats arrive with less deviation the curve becomes steepe i.e. it is possible to determine failure more quickly. The curve looks like this for a standard deviation of 100 ms. -.. image:: ../images/phi2.png +![phi2.png](../images/phi2.png) To be able to survive sudden abnormalities, such as garbage collection pauses and transient network failures the failure detector is configured with a margin, -``akka.remote.watch-failure-detector.acceptable-heartbeat-pause``. You may want to -adjust the :ref:`remote-configuration-scala` of this depending on you environment. -This is how the curve looks like for ``acceptable-heartbeat-pause`` configured to +`akka.remote.watch-failure-detector.acceptable-heartbeat-pause`. You may want to +adjust the [Remote Configuration](#remote-configuration-scala) of this depending on you environment. +This is how the curve looks like for `acceptable-heartbeat-pause` configured to 3 seconds. -.. image:: ../images/phi3.png +![phi3.png](../images/phi3.png) -Serialization -^^^^^^^^^^^^^ +## Serialization -When using remoting for actors you must ensure that the ``props`` and ``messages`` used for +When using remoting for actors you must ensure that the `props` and `messages` used for those actors are serializable. Failing to do so will cause the system to behave in an unintended way. -For more information please see :ref:`serialization-scala`. +For more information please see @ref:[Serialization](serialization.md). -.. _disable-java-serializer-scala: + +### Disabling the Java Serializer -Disabling the Java Serializer ------------------------------ - -Since the ``2.4.11`` release of Akka it is possible to entirely disable the default Java Serialization mechanism. -Please note that :ref:`new remoting implementation (codename Artery) ` does not use Java +Since the `2.4.11` release of Akka it is possible to entirely disable the default Java Serialization mechanism. +Please note that @ref:[new remoting implementation (codename Artery)](remoting-artery.md) does not use Java serialization for internal messages by default. For compatibility reasons, the current remoting still uses Java serialization for some classes, however you can disable it in this remoting implementation as well by following the steps below. @@ -299,56 +308,61 @@ The first step is to enable some additional serializers that replace previous Ja messages. This is recommended also when you can't disable Java serialization completely. Those serializers are enabled with this configuration: -.. code-block:: ruby - - akka.actor { - # Set this to on to enable serialization-bindings define in - # additional-serialization-bindings. Those are by default not included - # for backwards compatibility reasons. They are enabled by default if - # akka.remote.artery.enabled=on. - enable-additional-serialization-bindings = on - } +```ruby +akka.actor { + # Set this to on to enable serialization-bindings define in + # additional-serialization-bindings. Those are by default not included + # for backwards compatibility reasons. They are enabled by default if + # akka.remote.artery.enabled=on. + enable-additional-serialization-bindings = on +} +``` The reason these are not enabled by default is wire-level compatibility between any 2.4.x Actor Systems. If you roll out a new cluster, all on the same Akka version that can enable these serializers it is recommended to -enable this setting. When using :ref:`remoting-artery-scala` these serializers are enabled by default. +enable this setting. When using @ref:[Remoting (codename Artery)](remoting-artery.md) these serializers are enabled by default. -.. warning:: - Please note that when enabling the additional-serialization-bindings when using the old remoting, - you must do so on all nodes participating in a cluster, otherwise the mis-aligned serialization - configurations will cause deserialization errors on the receiving nodes. +@@@ warning -Java serialization is known to be slow and `prone to attacks -`_ +Please note that when enabling the additional-serialization-bindings when using the old remoting, +you must do so on all nodes participating in a cluster, otherwise the mis-aligned serialization +configurations will cause deserialization errors on the receiving nodes. + +@@@ + +Java serialization is known to be slow and [prone to attacks](https://community.hpe.com/t5/Security-Research/The-perils-of-Java-deserialization/ba-p/6838995) of various kinds - it never was designed for high throughput messaging after all. However, it is very convenient to use, thus it remained the default serialization mechanism that Akka used to serialize user messages as well as some of its internal messages in previous versions. -Since the release of Artery, Akka internals do not rely on Java serialization anymore (one exception being ``java.lang.Throwable``). +Since the release of Artery, Akka internals do not rely on Java serialization anymore (one exception being `java.lang.Throwable`). -.. note:: - When using the new remoting implementation (codename Artery), Akka does not use Java Serialization for any of its internal messages. - It is highly encouraged to disable java serialization, so please plan to do so at the earliest possibility you have in your project. +@@@ note - One may think that network bandwidth and latency limit the performance of remote messaging, but serialization is a more typical bottleneck. +When using the new remoting implementation (codename Artery), Akka does not use Java Serialization for any of its internal messages. +It is highly encouraged to disable java serialization, so please plan to do so at the earliest possibility you have in your project. + +One may think that network bandwidth and latency limit the performance of remote messaging, but serialization is a more typical bottleneck. + +@@@ For user messages, the default serializer, implemented using Java serialization, remains available and enabled. We do however recommend to disable it entirely and utilise a proper serialization library instead in order effectively utilise the improved performance and ability for rolling deployments using Artery. Libraries that we recommend to use include, -but are not limited to, `Kryo`_ by using the `akka-kryo-serialization`_ library or `Google Protocol Buffers`_ if you want +but are not limited to, [Kryo](https://github.com/EsotericSoftware/kryo) by using the [akka-kryo-serialization](https://github.com/romix/akka-kryo-serialization) library or [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) if you want more control over the schema evolution of your messages. In order to completely disable Java Serialization in your Actor system you need to add the following configuration to -your ``application.conf``: +your `application.conf`: -.. code-block:: ruby +```ruby +akka.actor.allow-java-serialization = off +``` - akka.actor.allow-java-serialization = off - -This will completely disable the use of ``akka.serialization.JavaSerialization`` by the -Akka Serialization extension, instead ``DisabledJavaSerializer`` will +This will completely disable the use of `akka.serialization.JavaSerialization` by the +Akka Serialization extension, instead `DisabledJavaSerializer` will be inserted which will fail explicitly if attempts to use java serialization are made. -It will also enable the above mentioned ``enable-additional-serialization-bindings``. +It will also enable the above mentioned `enable-additional-serialization-bindings`. The log messages emitted by such serializer SHOULD be be treated as potential attacks which the serializer prevented, as they MAY indicate an external operator @@ -358,162 +372,160 @@ The attempts are logged with the SECURITY marker. Please note that this option does not stop you from manually invoking java serialization. Please note that this means that you will have to configure different serializers which will able to handle all of your -remote messages. Please refer to the :ref:`serialization-scala` documentation as well as :ref:`ByteBuffer based serialization ` to learn how to do this. +remote messages. Please refer to the @ref:[Serialization](serialization.md) documentation as well as @ref:[ByteBuffer based serialization](remoting-artery.md#remote-bytebuffer-serialization-scala) to learn how to do this. -.. _Kryo: https://github.com/EsotericSoftware/kryo -.. _akka-kryo-serialization: https://github.com/romix/akka-kryo-serialization -.. _Google Protocol Buffers: https://developers.google.com/protocol-buffers/ +## Routers with Remote Destinations -Routers with Remote Destinations -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It is absolutely feasible to combine remoting with :ref:`routing-scala`. +It is absolutely feasible to combine remoting with @ref:[Routing](routing.md). A pool of remote deployed routees can be configured as: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-pool +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-pool } -This configuration setting will clone the actor defined in the ``Props`` of the ``remotePool`` 10 +This configuration setting will clone the actor defined in the `Props` of the `remotePool` 10 times and deploy it evenly distributed across the two given target nodes. A group of remote actors can be configured as: -.. includecode:: ../scala/code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-group +@@snip [RouterDocSpec.scala](../scala/code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-group } This configuration setting will send messages to the defined remote actor paths. It requires that you create the destination actors on the remote nodes with matching paths. That is not done by the router. -.. _remote-sample-scala: + +## Remoting Sample -Remoting Sample -^^^^^^^^^^^^^^^ - -You can download a ready to run `remoting sample <@exampleCodeService@/akka-samples-remote-scala>`_ +You can download a ready to run [remoting sample](@exampleCodeService@/akka-samples-remote-scala) together with a tutorial for a more hands-on experience. The source code of this sample can be found in the -`Akka Samples Repository <@samples@/akka-sample-remote-scala>`_. +[Akka Samples Repository](@samples@/akka-sample-remote-scala). -Remote Events -------------- +### Remote Events It is possible to listen to events that occur in Akka Remote, and to subscribe/unsubscribe to these events -you simply register as listener to the below described types in on the ``ActorSystem.eventStream``. +you simply register as listener to the below described types in on the `ActorSystem.eventStream`. -.. note:: +@@@ note - To subscribe to any remote event, subscribe to - :meth:`RemotingLifecycleEvent`. To subscribe to events related only to - the lifecycle of associations, subscribe to - :meth:`akka.remote.AssociationEvent`. +To subscribe to any remote event, subscribe to +`RemotingLifecycleEvent`. To subscribe to events related only to +the lifecycle of associations, subscribe to +`akka.remote.AssociationEvent`. -.. note:: +@@@ - The use of term "Association" instead of "Connection" reflects that the - remoting subsystem may use connectionless transports, but an association - similar to transport layer connections is maintained between endpoints by - the Akka protocol. +@@@ note + +The use of term "Association" instead of "Connection" reflects that the +remoting subsystem may use connectionless transports, but an association +similar to transport layer connections is maintained between endpoints by +the Akka protocol. + +@@@ By default an event listener is registered which logs all of the events described below. This default was chosen to help setting up a system, but it is quite common to switch this logging off once that phase of the project is finished. -.. note:: +@@@ note - In order to switch off the logging, set - ``akka.remote.log-remote-lifecycle-events = off`` in your - ``application.conf``. +In order to switch off the logging, set +`akka.remote.log-remote-lifecycle-events = off` in your +`application.conf`. -To be notified when an association is over ("disconnected") listen to ``DisassociatedEvent`` which +@@@ + +To be notified when an association is over ("disconnected") listen to `DisassociatedEvent` which holds the direction of the association (inbound or outbound) and the addresses of the involved parties. -To be notified when an association is successfully established ("connected") listen to ``AssociatedEvent`` which +To be notified when an association is successfully established ("connected") listen to `AssociatedEvent` which holds the direction of the association (inbound or outbound) and the addresses of the involved parties. -To intercept errors directly related to associations, listen to ``AssociationErrorEvent`` which +To intercept errors directly related to associations, listen to `AssociationErrorEvent` which holds the direction of the association (inbound or outbound), the addresses of the involved parties and the -``Throwable`` cause. +`Throwable` cause. -To be notified when the remoting subsystem is ready to accept associations, listen to ``RemotingListenEvent`` which +To be notified when the remoting subsystem is ready to accept associations, listen to `RemotingListenEvent` which contains the addresses the remoting listens on. -To be notified when the current system is quarantined by the remote system, listen to ``ThisActorSystemQuarantinedEvent``, +To be notified when the current system is quarantined by the remote system, listen to `ThisActorSystemQuarantinedEvent`, which includes the addresses of local and remote ActorSystems. -To be notified when the remoting subsystem has been shut down, listen to ``RemotingShutdownEvent``. +To be notified when the remoting subsystem has been shut down, listen to `RemotingShutdownEvent`. -To intercept generic remoting related errors, listen to ``RemotingErrorEvent`` which holds the ``Throwable`` cause. +To intercept generic remoting related errors, listen to `RemotingErrorEvent` which holds the `Throwable` cause. -.. _remote-security-scala: + +## Remote Security -Remote Security -^^^^^^^^^^^^^^^ - -An ``ActorSystem`` should not be exposed via Akka Remote over plain TCP to an untrusted network (e.g. internet). +An `ActorSystem` should not be exposed via Akka Remote over plain TCP to an untrusted network (e.g. internet). It should be protected by network security, such as a firewall. If that is not considered as enough protection -:ref:`TLS with mutual authentication ` should be enabled. +[TLS with mutual authentication](#remote-tls-scala) should be enabled. -It is also security best-practice to :ref:`disable the Java serializer ` because of -its multiple `known attack surfaces `_. +It is also security best-practice to [disable the Java serializer](#disable-java-serializer-scala) because of +its multiple [known attack surfaces](https://community.hpe.com/t5/Security-Research/The-perils-of-Java-deserialization/ba-p/6838995). -.. _remote-tls-scala: + +### Configuring SSL/TLS for Akka Remoting -Configuring SSL/TLS for Akka Remoting -------------------------------------- +SSL can be used as the remote transport by adding `akka.remote.netty.ssl` to the `enabled-transport` configuration section. +An example of setting up the default Netty based SSL driver as default: -SSL can be used as the remote transport by adding ``akka.remote.netty.ssl`` to the ``enabled-transport`` configuration section. -An example of setting up the default Netty based SSL driver as default:: - - akka { - remote { - enabled-transports = [akka.remote.netty.ssl] - } +``` +akka { + remote { + enabled-transports = [akka.remote.netty.ssl] } +} +``` -Next the actual SSL/TLS parameters have to be configured:: +Next the actual SSL/TLS parameters have to be configured: - akka { - remote { - netty.ssl { - hostname = "127.0.0.1" - port = "3553" +``` +akka { + remote { + netty.ssl { + hostname = "127.0.0.1" + port = "3553" - security { - key-store = "/example/path/to/mykeystore.jks" - trust-store = "/example/path/to/mytruststore.jks" + security { + key-store = "/example/path/to/mykeystore.jks" + trust-store = "/example/path/to/mytruststore.jks" - key-store-password = "changeme" - key-password = "changeme" - trust-store-password = "changeme" + key-store-password = "changeme" + key-password = "changeme" + trust-store-password = "changeme" - protocol = "TLSv1.2" + protocol = "TLSv1.2" - enabled-algorithms = [TLS_DHE_RSA_WITH_AES_128_GCM_SHA256] + enabled-algorithms = [TLS_DHE_RSA_WITH_AES_128_GCM_SHA256] - random-number-generator = "AES128CounterSecureRNG" - } + random-number-generator = "AES128CounterSecureRNG" } } } +} +``` -According to `RFC 7525 `_ the recommended algorithms to use with TLS 1.2 (as of writing this document) are: +According to [RFC 7525](https://tools.ietf.org/html/rfc7525) the recommended algorithms to use with TLS 1.2 (as of writing this document) are: -- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 -- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 -- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 -- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 + * TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 + * TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 + * TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 + * TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 You should always check the latest information about security and algorithm recommendations though before you configure your system. Creating and working with keystores and certificates is well documented in the -`Generating X.509 Certificates `_ +[Generating X.509 Certificates](http://typesafehub.github.io/ssl-config/CertificateGeneration.html#using-keytool) section of Lightbend's SSL-Config library. -Since an Akka remoting is inherently :ref:`peer-to-peer ` both the key-store as well as trust-store +Since an Akka remoting is inherently @ref:[peer-to-peer](../general/remoting.md#symmetric-communication) both the key-store as well as trust-store need to be configured on each remoting node participating in the cluster. -The official `Java Secure Socket Extension documentation `_ -as well as the `Oracle documentation on creating KeyStore and TrustStores `_ +The official [Java Secure Socket Extension documentation](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html) +as well as the [Oracle documentation on creating KeyStore and TrustStores](https://docs.oracle.com/cd/E19509-01/820-3503/6nf1il6er/index.html) are both great resources to research when setting up security on the JVM. Please consult those resources when troubleshooting and configuring SSL. @@ -527,116 +539,123 @@ the other (the "server"). Note that if TLS is enabled with mutual authentication there is still a risk that an attacker can gain access to a valid certificate by compromising any node with certificates issued by the same internal PKI tree. -See also a description of the settings in the :ref:`remote-configuration-scala` section. +See also a description of the settings in the [Remote Configuration](#remote-configuration-scala) section. -.. note:: +@@@ note - When using SHA1PRNG on Linux it's recommended specify ``-Djava.security.egd=file:/dev/urandom`` as argument - to the JVM to prevent blocking. It is NOT as secure because it reuses the seed. +When using SHA1PRNG on Linux it's recommended specify `-Djava.security.egd=file:/dev/urandom` as argument +to the JVM to prevent blocking. It is NOT as secure because it reuses the seed. -Untrusted Mode --------------- +@@@ + +### Untrusted Mode As soon as an actor system can connect to another remotely, it may in principle send any possible message to any actor contained within that remote system. One -example may be sending a :class:`PoisonPill` to the system guardian, shutting +example may be sending a `PoisonPill` to the system guardian, shutting that system down. This is not always desired, and it can be disabled with the -following setting:: +following setting: - akka.remote.untrusted-mode = on +``` +akka.remote.untrusted-mode = on +``` This disallows sending of system messages (actor life-cycle commands, -DeathWatch, etc.) and any message extending :class:`PossiblyHarmful` to the +DeathWatch, etc.) and any message extending `PossiblyHarmful` to the system on which this flag is set. Should a client send them nonetheless they are dropped and logged (at DEBUG level in order to reduce the possibilities for -a denial of service attack). :class:`PossiblyHarmful` covers the predefined -messages like :class:`PoisonPill` and :class:`Kill`, but it can also be added +a denial of service attack). `PossiblyHarmful` covers the predefined +messages like `PoisonPill` and `Kill`, but it can also be added as a marker trait to user-defined messages. -.. warning:: +@@@ warning - Untrusted mode does not give full protection against attacks by itself. - It makes it slightly harder to perform malicious or unintended actions but - it should be complemented with :ref:`disabled Java serializer `. - Additional protection can be achieved when running in an untrusted network by - network security (e.g. firewalls) and/or enabling :ref:`TLS with mutual - authentication `. +Untrusted mode does not give full protection against attacks by itself. +It makes it slightly harder to perform malicious or unintended actions but +it should be complemented with [disabled Java serializer](#disable-java-serializer-scala). +Additional protection can be achieved when running in an untrusted network by +network security (e.g. firewalls) and/or enabling TLS with mutual +authentication . + +@@@ Messages sent with actor selection are by default discarded in untrusted mode, but permission to receive actor selection messages can be granted to specific actors -defined in configuration:: +defined in configuration: - akka.remote.trusted-selection-paths = ["/user/receptionist", "/user/namingService"] +``` +akka.remote.trusted-selection-paths = ["/user/receptionist", "/user/namingService"] +``` -The actual message must still not be of type :class:`PossiblyHarmful`. +The actual message must still not be of type `PossiblyHarmful`. In summary, the following operations are ignored by a system configured in untrusted mode when incoming via the remoting layer: -* remote deployment (which also means no remote supervision) -* remote DeathWatch -* ``system.stop()``, :class:`PoisonPill`, :class:`Kill` -* sending any message which extends from the :class:`PossiblyHarmful` marker - interface, which includes :class:`Terminated` -* messages sent with actor selection, unless destination defined in ``trusted-selection-paths``. + * remote deployment (which also means no remote supervision) + * remote DeathWatch + * `system.stop()`, `PoisonPill`, `Kill` + * sending any message which extends from the `PossiblyHarmful` marker +interface, which includes `Terminated` + * messages sent with actor selection, unless destination defined in `trusted-selection-paths`. -.. note:: +@@@ note - Enabling the untrusted mode does not remove the capability of the client to - freely choose the target of its message sends, which means that messages not - prohibited by the above rules can be sent to any actor in the remote system. - It is good practice for a client-facing system to only contain a well-defined - set of entry point actors, which then forward requests (possibly after - performing validation) to another actor system containing the actual worker - actors. If messaging between these two server-side systems is done using - local :class:`ActorRef` (they can be exchanged safely between actor systems - within the same JVM), you can restrict the messages on this interface by - marking them :class:`PossiblyHarmful` so that a client cannot forge them. +Enabling the untrusted mode does not remove the capability of the client to +freely choose the target of its message sends, which means that messages not +prohibited by the above rules can be sent to any actor in the remote system. +It is good practice for a client-facing system to only contain a well-defined +set of entry point actors, which then forward requests (possibly after +performing validation) to another actor system containing the actual worker +actors. If messaging between these two server-side systems is done using +local `ActorRef` (they can be exchanged safely between actor systems +within the same JVM), you can restrict the messages on this interface by +marking them `PossiblyHarmful` so that a client cannot forge them. -.. _remote-configuration-scala: +@@@ -Remote Configuration -^^^^^^^^^^^^^^^^^^^^ + +## Remote Configuration There are lots of configuration properties that are related to remoting in Akka. We refer to the -:ref:`reference configuration ` for more information. +@ref:[reference configuration](../general/configuration.md#config-akka-remote) for more information. -.. note:: +@@@ note - Setting properties like the listening IP and port number programmatically is - best done by using something like the following: +Setting properties like the listening IP and port number programmatically is +best done by using something like the following: - .. includecode:: ../java/code/jdocs/remoting/RemoteDeploymentDocTest.java#programmatic +@@snip [RemoteDeploymentDocTest.java](../java/code/jdocs/remoting/RemoteDeploymentDocTest.java) { #programmatic } +@@@ -.. _remote-configuration-nat: - -Akka behind NAT or in a Docker container ----------------------------------------- + +### Akka behind NAT or in a Docker container In setups involving Network Address Translation (NAT), Load Balancers or Docker containers the hostname and port pair that Akka binds to will be different than the "logical" host name and port pair that is used to connect to the system from the outside. This requires special configuration that sets both the logical and the bind pairs for remoting. -.. code-block:: ruby +```ruby +akka { + remote { + netty.tcp { + hostname = my.domain.com # external (logical) hostname + port = 8000 # external (logical) port - akka { - remote { - netty.tcp { - hostname = my.domain.com # external (logical) hostname - port = 8000 # external (logical) port - - bind-hostname = local.address # internal (bind) hostname - bind-port = 2552 # internal (bind) port - } - } - } + bind-hostname = local.address # internal (bind) hostname + bind-port = 2552 # internal (bind) port + } + } +} +``` Keep in mind that local.address will most likely be in one of private network ranges: -* *10.0.0.0 - 10.255.255.255* (network class A) -* *172.16.0.0 - 172.31.255.255* (network class B) -* *192.168.0.0 - 192.168.255.255* (network class C) + * *10.0.0.0 - 10.255.255.255* (network class A) + * *172.16.0.0 - 172.31.255.255* (network class B) + * *192.168.0.0 - 192.168.255.255* (network class C) -For further details see [RFC 1597](https://tools.ietf.org/html/rfc1597) and [RFC 1918](https://tools.ietf.org/html/rfc1918). +For further details see [RFC 1597]([https://tools.ietf.org/html/rfc1597](https://tools.ietf.org/html/rfc1597)) and [RFC 1918]([https://tools.ietf.org/html/rfc1918](https://tools.ietf.org/html/rfc1918)). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/routing.md b/akka-docs/src/main/paradox/scala/routing.md index 5e8b5bb023..cd281b3fdc 100644 --- a/akka-docs/src/main/paradox/scala/routing.md +++ b/akka-docs/src/main/paradox/scala/routing.md @@ -1,136 +1,125 @@ - -.. _routing-scala: - -Routing -======= +# Routing Messages can be sent via a router to efficiently route them to destination actors, known as -its *routees*. A ``Router`` can be used inside or outside of an actor, and you can manage the +its *routees*. A `Router` can be used inside or outside of an actor, and you can manage the routees yourselves or use a self contained router actor with configuration capabilities. Different routing strategies can be used, according to your application's needs. Akka comes with several useful routing strategies right out of the box. But, as you will see in this chapter, it is -also possible to :ref:`create your own `. +also possible to [create your own](#custom-router-scala). -.. _simple-router-scala: + +## A Simple Router -A Simple Router -^^^^^^^^^^^^^^^ +The following example illustrates how to use a `Router` and manage the routees from within an actor. -The following example illustrates how to use a ``Router`` and manage the routees from within an actor. +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #router-in-actor } -.. includecode:: code/docs/routing/RouterDocSpec.scala#router-in-actor - -We create a ``Router`` and specify that it should use ``RoundRobinRoutingLogic`` when routing the +We create a `Router` and specify that it should use `RoundRobinRoutingLogic` when routing the messages to the routees. The routing logic shipped with Akka are: -* ``akka.routing.RoundRobinRoutingLogic`` -* ``akka.routing.RandomRoutingLogic`` -* ``akka.routing.SmallestMailboxRoutingLogic`` -* ``akka.routing.BroadcastRoutingLogic`` -* ``akka.routing.ScatterGatherFirstCompletedRoutingLogic`` -* ``akka.routing.TailChoppingRoutingLogic`` -* ``akka.routing.ConsistentHashingRoutingLogic`` + * `akka.routing.RoundRobinRoutingLogic` + * `akka.routing.RandomRoutingLogic` + * `akka.routing.SmallestMailboxRoutingLogic` + * `akka.routing.BroadcastRoutingLogic` + * `akka.routing.ScatterGatherFirstCompletedRoutingLogic` + * `akka.routing.TailChoppingRoutingLogic` + * `akka.routing.ConsistentHashingRoutingLogic` -We create the routees as ordinary child actors wrapped in ``ActorRefRoutee``. We watch +We create the routees as ordinary child actors wrapped in `ActorRefRoutee`. We watch the routees to be able to replace them if they are terminated. -Sending messages via the router is done with the ``route`` method, as is done for the ``Work`` messages +Sending messages via the router is done with the `route` method, as is done for the `Work` messages in the example above. -The ``Router`` is immutable and the ``RoutingLogic`` is thread safe; meaning that they can also be used +The `Router` is immutable and the `RoutingLogic` is thread safe; meaning that they can also be used outside of actors. -.. note:: +@@@ note - In general, any message sent to a router will be sent onwards to its routees, but there is one exception. - The special :ref:`broadcast-messages-scala` will send to *all* of a router's routees. - However, do not use :ref:`broadcast-messages-scala` when you use :ref:`balancing-pool-scala` for routees - as described in :ref:`router-special-messages-scala`. +In general, any message sent to a router will be sent onwards to its routees, but there is one exception. +The special [Broadcast Messages](#broadcast-messages-scala) will send to *all* of a router's routees. +However, do not use [Broadcast Messages](#broadcast-messages-scala) when you use [BalancingPool](#balancing-pool-scala) for routees +as described in [Specially Handled Messages](#router-special-messages-scala). +@@@ -A Router Actor -^^^^^^^^^^^^^^ +## A Router Actor A router can also be created as a self contained actor that manages the routees itself and loads routing logic and other settings from configuration. This type of router actor comes in two distinct flavors: -* Pool - The router creates routees as child actors and removes them from the router if they - terminate. - -* Group - The routee actors are created externally to the router and the router sends - messages to the specified path using actor selection, without watching for termination. + * Pool - The router creates routees as child actors and removes them from the router if they +terminate. + * Group - The routee actors are created externally to the router and the router sends +messages to the specified path using actor selection, without watching for termination. The settings for a router actor can be defined in configuration or programmatically. -In order to make an actor to make use of an externally configurable router the ``FromConfig`` props wrapper must be used +In order to make an actor to make use of an externally configurable router the `FromConfig` props wrapper must be used to denote that the actor accepts routing settings from configuration. This is in contrast with Remote Deployment where such marker props is not necessary. If the props of an actor is NOT wrapped in FromConfig it will ignore the router section of the deployment configuration. You send messages to the routees via the router actor in the same way as for ordinary actors, -i.e. via its ``ActorRef``. The router actor forwards messages onto its routees without changing +i.e. via its `ActorRef`. The router actor forwards messages onto its routees without changing the original sender. When a routee replies to a routed message, the reply will be sent to the original sender, not to the router actor. -.. note:: +@@@ note - In general, any message sent to a router will be sent onwards to its routees, but there are a - few exceptions. These are documented in the :ref:`router-special-messages-scala` section below. +In general, any message sent to a router will be sent onwards to its routees, but there are a +few exceptions. These are documented in the [Specially Handled Messages](#router-special-messages-scala) section below. -Pool ----- +@@@ -The following code and configuration snippets show how to create a :ref:`round-robin -` router that forwards messages to five ``Worker`` routees. The +### Pool + +The following code and configuration snippets show how to create a [round-robin](#round-robin-router-scala) router that forwards messages to five `Worker` routees. The routees will be created as the router's children. -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-round-robin-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-round-robin-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#round-robin-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #round-robin-pool-1 } Here is the same example, but with the router configuration provided programmatically instead of from configuration. -.. includecode:: code/docs/routing/RouterDocSpec.scala#round-robin-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #round-robin-pool-2 } -Remote Deployed Routees -*********************** +#### Remote Deployed Routees In addition to being able to create local actors as routees, you can instruct the router to deploy its created children on a set of remote hosts. Routees will be deployed in round-robin fashion. In order to deploy routees remotely, wrap the router configuration in a -``RemoteRouterConfig``, attaching the remote addresses of the nodes to deploy to. Remote -deployment requires the ``akka-remote`` module to be included in the classpath. +`RemoteRouterConfig`, attaching the remote addresses of the nodes to deploy to. Remote +deployment requires the `akka-remote` module to be included in the classpath. -.. includecode:: code/docs/routing/RouterDocSpec.scala#remoteRoutees +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #remoteRoutees } -Senders -******* +#### Senders -By default, when a routee sends a message, it will :ref:`implicitly set itself as the sender -`. +By default, when a routee sends a message, it will @ref:[implicitly set itself as the sender +](actors.md#actors-tell-sender-scala). -.. includecode:: code/docs/actor/ActorDocSpec.scala#reply-without-sender +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #reply-without-sender } However, it is often useful for routees to set the *router* as a sender. For example, you might want to set the router as the sender if you want to hide the details of the routees behind the router. The following code snippet shows how to set the parent router as sender. -.. includecode:: code/docs/actor/ActorDocSpec.scala#reply-with-sender +@@snip [ActorDocSpec.scala](code/docs/actor/ActorDocSpec.scala) { #reply-with-sender } - -Supervision -*********** +#### Supervision Routees that are created by a pool router will be created as the router's children. The router is therefore also the children's supervisor. The supervision strategy of the router actor can be configured with the -``supervisorStrategy`` property of the Pool. If no configuration is provided, routers default +`supervisorStrategy` property of the Pool. If no configuration is provided, routers default to a strategy of “always escalate”. This means that errors are passed up to the router's supervisor for handling. The router's supervisor will decide what to do about any errors. @@ -141,285 +130,278 @@ turn, will cause its children to stop and restart. It should be mentioned that the router's restart behavior has been overridden so that a restart, while still re-creating the children, will still preserve the same number of actors in the pool. -This means that if you have not specified :meth:`supervisorStrategy` of the router or its parent a +This means that if you have not specified `supervisorStrategy` of the router or its parent a failure in a routee will escalate to the parent of the router, which will by default restart the router, which will restart all routees (it uses Escalate and does not stop routees during restart). The reason -is to make the default behave such that adding :meth:`.withRouter` to a child’s definition does not +is to make the default behave such that adding `.withRouter` to a child’s definition does not change the supervision strategy applied to the child. This might be an inefficiency that you can avoid by specifying the strategy when defining the router. Setting the strategy is easily done: -.. includecode:: ../../../akka-actor-tests/src/test/scala/akka/routing/RoutingSpec.scala#supervision +@@snip [RoutingSpec.scala]../../../../../akka-actor-tests/src/test/scala/akka/routing/RoutingSpec.scala) { #supervision } -.. _note-router-terminated-children-scala: +@@@ note -.. note:: +If the child of a pool router terminates, the pool router will not automatically spawn +a new child. In the event that all children of a pool router have terminated the +router will terminate itself unless it is a dynamic router, e.g. using +a resizer. - If the child of a pool router terminates, the pool router will not automatically spawn - a new child. In the event that all children of a pool router have terminated the - router will terminate itself unless it is a dynamic router, e.g. using - a resizer. +@@@ -Group ------ +### Group Sometimes, rather than having the router actor create its routees, it is desirable to create routees separately and provide them to the router for its use. You can do this by passing an -paths of the routees to the router's configuration. Messages will be sent with ``ActorSelection`` +paths of the routees to the router's configuration. Messages will be sent with `ActorSelection` to these paths. The example below shows how to create a router by providing it with the path strings of three routee actors. -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-round-robin-group +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-round-robin-group } -.. includecode:: code/docs/routing/RouterDocSpec.scala#round-robin-group-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #round-robin-group-1 } Here is the same example, but with the router configuration provided programmatically instead of from configuration. -.. includecode:: code/docs/routing/RouterDocSpec.scala#round-robin-group-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #round-robin-group-2 } The routee actors are created externally from the router: -.. includecode:: code/docs/routing/RouterDocSpec.scala#create-workers +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #create-workers } -.. includecode:: code/docs/routing/RouterDocSpec.scala#create-worker-actors +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #create-worker-actors } The paths may contain protocol and address information for actors running on remote hosts. -Remoting requires the ``akka-remote`` module to be included in the classpath. +Remoting requires the `akka-remote` module to be included in the classpath. -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-remote-round-robin-group +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-remote-round-robin-group } -Router usage -^^^^^^^^^^^^ +## Router usage In this section we will describe how to create the different types of router actors. -The router actors in this section are created from within a top level actor named ``parent``. -Note that deployment paths in the configuration starts with ``/parent/`` followed by the name +The router actors in this section are created from within a top level actor named `parent`. +Note that deployment paths in the configuration starts with `/parent/` followed by the name of the router actor. -.. includecode:: code/docs/routing/RouterDocSpec.scala#create-parent +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #create-parent } -.. _round-robin-router-scala: + +### RoundRobinPool and RoundRobinGroup -RoundRobinPool and RoundRobinGroup ----------------------------------- - -Routes in a `round-robin `_ fashion to its routees. +Routes in a [round-robin](http://en.wikipedia.org/wiki/Round-robin) fashion to its routees. RoundRobinPool defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-round-robin-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-round-robin-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#round-robin-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #round-robin-pool-1 } RoundRobinPool defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala#round-robin-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #round-robin-pool-2 } RoundRobinGroup defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-round-robin-group +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-round-robin-group } -.. includecode:: code/docs/routing/RouterDocSpec.scala#round-robin-group-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #round-robin-group-1 } RoundRobinGroup defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala - :include: paths,round-robin-group-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #paths #round-robin-group-2 } -RandomPool and RandomGroup --------------------------- +### RandomPool and RandomGroup This router type selects one of its routees randomly for each message. RandomPool defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-random-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-random-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#random-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #random-pool-1 } RandomPool defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala#random-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #random-pool-2 } RandomGroup defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-random-group +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-random-group } -.. includecode:: code/docs/routing/RouterDocSpec.scala#random-group-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #random-group-1 } RandomGroup defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala - :include: paths,random-group-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #paths #random-group-2 } -.. _balancing-pool-scala: - -BalancingPool -------------- + +### BalancingPool A Router that will try to redistribute work from busy routees to idle routees. All routees share the same mailbox. -.. note:: +@@@ note - The BalancingPool has the property that its routees do not have truly distinct - identity: they have different names, but talking to them will not end up at the - right actor in most cases. Therefore you cannot use it for workflows that - require state to be kept within the routee, you would in this case have to - include the whole state in the messages. +The BalancingPool has the property that its routees do not have truly distinct +identity: they have different names, but talking to them will not end up at the +right actor in most cases. Therefore you cannot use it for workflows that +require state to be kept within the routee, you would in this case have to +include the whole state in the messages. - With a `SmallestMailboxPool`_ you can have a vertically scaling service that - can interact in a stateful fashion with other services in the back-end before - replying to the original client. The other advantage is that it does not place - a restriction on the message queue implementation as BalancingPool does. +With a [SmallestMailboxPool](#smallestmailboxpool) you can have a vertically scaling service that +can interact in a stateful fashion with other services in the back-end before +replying to the original client. The other advantage is that it does not place +a restriction on the message queue implementation as BalancingPool does. -.. note:: - Do not use :ref:`broadcast-messages-scala` when you use :ref:`balancing-pool-scala` for routers. - as described in :ref:`router-special-messages-scala`, +@@@ +@@@ note + +Do not use [Broadcast Messages](#broadcast-messages-scala) when you use [BalancingPool](#balancing-pool-scala) for routers. +as described in [Specially Handled Messages](#router-special-messages-scala), + +@@@ BalancingPool defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-balancing-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-balancing-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#balancing-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #balancing-pool-1 } BalancingPool defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala#balancing-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #balancing-pool-2 } Addition configuration for the balancing dispatcher, which is used by the pool, -can be configured in the ``pool-dispatcher`` section of the router deployment +can be configured in the `pool-dispatcher` section of the router deployment configuration. -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-balancing-pool2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-balancing-pool2 } -The ``BalancingPool`` automatically uses a special ``BalancingDispatcher`` for its +The `BalancingPool` automatically uses a special `BalancingDispatcher` for its routees - disregarding any dispatcher that is set on the routee Props object. This is needed in order to implement the balancing semantics via sharing the same mailbox by all the routees. While it is not possible to change the dispatcher used by the routees, it is possible -to fine tune the used *executor*. By default the ``fork-join-dispatcher`` is used and -can be configured as explained in :ref:`dispatchers-scala`. In situations where the +to fine tune the used *executor*. By default the `fork-join-dispatcher` is used and +can be configured as explained in @ref:[Dispatchers](dispatchers.md). In situations where the routees are expected to perform blocking operations it may be useful to replace it -with a ``thread-pool-executor`` hinting the number of allocated threads explicitly: +with a `thread-pool-executor` hinting the number of allocated threads explicitly: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-balancing-pool3 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-balancing-pool3 } -It is also possible to change the ``mailbox`` used by the balancing dispatcher for +It is also possible to change the `mailbox` used by the balancing dispatcher for scenarios where the default unbounded mailbox is not well suited. An example of such a scenario could arise whether there exists the need to manage priority for each message. You can then implement a priority mailbox and configure your dispatcher: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-balancing-pool4 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-balancing-pool4 } -.. note:: +@@@ note - Bear in mind that ``BalancingDispatcher`` requires a message queue that must be thread-safe for - multiple concurrent consumers. So it is mandatory for the message queue backing a custom mailbox - for this kind of dispatcher to implement akka.dispatch.MultipleConsumerSemantics. See details - on how to implement your custom mailbox in :ref:`mailboxes-scala`. +Bear in mind that `BalancingDispatcher` requires a message queue that must be thread-safe for +multiple concurrent consumers. So it is mandatory for the message queue backing a custom mailbox +for this kind of dispatcher to implement akka.dispatch.MultipleConsumerSemantics. See details +on how to implement your custom mailbox in @ref:[Mailboxes](mailboxes.md). + +@@@ There is no Group variant of the BalancingPool. -SmallestMailboxPool -------------------- +### SmallestMailboxPool A Router that tries to send to the non-suspended child routee with fewest messages in mailbox. The selection is done in this order: +> * pick any idle routee (not processing message) with empty mailbox * pick any routee with empty mailbox * pick routee with fewest pending messages in mailbox * pick any remote routee, remote actors are consider lowest priority, - since their mailbox size is unknown +since their mailbox size is unknown SmallestMailboxPool defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-smallest-mailbox-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-smallest-mailbox-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#smallest-mailbox-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #smallest-mailbox-pool-1 } SmallestMailboxPool defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala#smallest-mailbox-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #smallest-mailbox-pool-2 } There is no Group variant of the SmallestMailboxPool because the size of the mailbox and the internal dispatching state of the actor is not practically available from the paths of the routees. -BroadcastPool and BroadcastGroup --------------------------------- +### BroadcastPool and BroadcastGroup A broadcast router forwards the message it receives to *all* its routees. BroadcastPool defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-broadcast-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-broadcast-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#broadcast-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #broadcast-pool-1 } BroadcastPool defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala#broadcast-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #broadcast-pool-2 } BroadcastGroup defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-broadcast-group +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-broadcast-group } -.. includecode:: code/docs/routing/RouterDocSpec.scala#broadcast-group-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #broadcast-group-1 } BroadcastGroup defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala - :include: paths,broadcast-group-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #paths #broadcast-group-2 } -.. note:: +@@@ note - Broadcast routers always broadcast *every* message to their routees. If you do not want to - broadcast every message, then you can use a non-broadcasting router and use - :ref:`broadcast-messages-scala` as needed. +Broadcast routers always broadcast *every* message to their routees. If you do not want to +broadcast every message, then you can use a non-broadcasting router and use +[Broadcast Messages](#broadcast-messages-scala) as needed. +@@@ -ScatterGatherFirstCompletedPool and ScatterGatherFirstCompletedGroup --------------------------------------------------------------------- +### ScatterGatherFirstCompletedPool and ScatterGatherFirstCompletedGroup The ScatterGatherFirstCompletedRouter will send the message on to all its routees. It then waits for first reply it gets back. This result will be sent back to original sender. Other replies are discarded. It is expecting at least one reply within a configured duration, otherwise it will reply with -``akka.pattern.AskTimeoutException`` in a ``akka.actor.Status.Failure``. +`akka.pattern.AskTimeoutException` in a `akka.actor.Status.Failure`. ScatterGatherFirstCompletedPool defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-scatter-gather-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-scatter-gather-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#scatter-gather-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #scatter-gather-pool-1 } ScatterGatherFirstCompletedPool defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala#scatter-gather-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #scatter-gather-pool-2 } ScatterGatherFirstCompletedGroup defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-scatter-gather-group +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-scatter-gather-group } -.. includecode:: code/docs/routing/RouterDocSpec.scala#scatter-gather-group-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #scatter-gather-group-1 } ScatterGatherFirstCompletedGroup defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala - :include: paths,scatter-gather-group-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #paths #scatter-gather-group-2 } -TailChoppingPool and TailChoppingGroup --------------------------------------- +### TailChoppingPool and TailChoppingGroup The TailChoppingRouter will first send the message to one, randomly picked, routee and then after a small delay to a second routee (picked randomly from the remaining routees) and so on. @@ -429,215 +411,202 @@ The goal of this router is to decrease latency by performing redundant queries t one of the other actors may still be faster to respond than the initial one. This optimisation was described nicely in a blog post by Peter Bailis: -`Doing redundant work to speed up distributed queries `_. +[Doing redundant work to speed up distributed queries](http://www.bailis.org/blog/doing-redundant-work-to-speed-up-distributed-queries/). TailChoppingPool defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-tail-chopping-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-tail-chopping-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#tail-chopping-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #tail-chopping-pool-1 } TailChoppingPool defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala#tail-chopping-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #tail-chopping-pool-2 } TailChoppingGroup defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-tail-chopping-group +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-tail-chopping-group } -.. includecode:: code/docs/routing/RouterDocSpec.scala#tail-chopping-group-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #tail-chopping-group-1 } TailChoppingGroup defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala - :include: paths,tail-chopping-group-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #paths #tail-chopping-group-2 } -ConsistentHashingPool and ConsistentHashingGroup ------------------------------------------------- +### ConsistentHashingPool and ConsistentHashingGroup -The ConsistentHashingPool uses `consistent hashing `_ +The ConsistentHashingPool uses [consistent hashing](http://en.wikipedia.org/wiki/Consistent_hashing) to select a routee based on the sent message. This -`article `_ gives good +[article](http://weblogs.java.net/blog/tomwhite/archive/2007/11/consistent_hash.html) gives good insight into how consistent hashing is implemented. There is 3 ways to define what data to use for the consistent hash key. -* You can define ``hashMapping`` of the router to map incoming - messages to their consistent hash key. This makes the decision - transparent for the sender. + * You can define `hashMapping` of the router to map incoming +messages to their consistent hash key. This makes the decision +transparent for the sender. + * The messages may implement `akka.routing.ConsistentHashingRouter.ConsistentHashable`. +The key is part of the message and it's convenient to define it together +with the message definition. + * The messages can be wrapped in a `akka.routing.ConsistentHashingRouter.ConsistentHashableEnvelope` +to define what data to use for the consistent hash key. The sender knows +the key to use. -* The messages may implement ``akka.routing.ConsistentHashingRouter.ConsistentHashable``. - The key is part of the message and it's convenient to define it together - with the message definition. - -* The messages can be wrapped in a ``akka.routing.ConsistentHashingRouter.ConsistentHashableEnvelope`` - to define what data to use for the consistent hash key. The sender knows - the key to use. - These ways to define the consistent hash key can be use together and at -the same time for one router. The ``hashMapping`` is tried first. - +the same time for one router. The `hashMapping` is tried first. Code example: -.. includecode:: code/docs/routing/ConsistentHashingRouterDocSpec.scala#cache-actor +@@snip [ConsistentHashingRouterDocSpec.scala](code/docs/routing/ConsistentHashingRouterDocSpec.scala) { #cache-actor } -.. includecode:: code/docs/routing/ConsistentHashingRouterDocSpec.scala#consistent-hashing-router +@@snip [ConsistentHashingRouterDocSpec.scala](code/docs/routing/ConsistentHashingRouterDocSpec.scala) { #consistent-hashing-router } -In the above example you see that the ``Get`` message implements ``ConsistentHashable`` itself, -while the ``Entry`` message is wrapped in a ``ConsistentHashableEnvelope``. The ``Evict`` -message is handled by the ``hashMapping`` partial function. +In the above example you see that the `Get` message implements `ConsistentHashable` itself, +while the `Entry` message is wrapped in a `ConsistentHashableEnvelope`. The `Evict` +message is handled by the `hashMapping` partial function. ConsistentHashingPool defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-consistent-hashing-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-consistent-hashing-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#consistent-hashing-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #consistent-hashing-pool-1 } ConsistentHashingPool defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala#consistent-hashing-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #consistent-hashing-pool-2 } ConsistentHashingGroup defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-consistent-hashing-group +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-consistent-hashing-group } -.. includecode:: code/docs/routing/RouterDocSpec.scala#consistent-hashing-group-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #consistent-hashing-group-1 } ConsistentHashingGroup defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala - :include: paths,consistent-hashing-group-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #paths #consistent-hashing-group-2 } - -``virtual-nodes-factor`` is the number of virtual nodes per routee that is used in the +`virtual-nodes-factor` is the number of virtual nodes per routee that is used in the consistent hash node ring to make the distribution more uniform. -.. _router-special-messages-scala: - -Specially Handled Messages -^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## Specially Handled Messages Most messages sent to router actors will be forwarded according to the routers' routing logic. However there are a few types of messages that have special behavior. -Note that these special messages, except for the ``Broadcast`` message, are only handled by -self contained router actors and not by the ``akka.routing.Router`` component described -in :ref:`simple-router-scala`. +Note that these special messages, except for the `Broadcast` message, are only handled by +self contained router actors and not by the `akka.routing.Router` component described +in [A Simple Router](#simple-router-scala). -.. _broadcast-messages-scala: + +### Broadcast Messages -Broadcast Messages ------------------- - -A ``Broadcast`` message can be used to send a message to *all* of a router's routees. When a router -receives a ``Broadcast`` message, it will broadcast that message's *payload* to all routees, no +A `Broadcast` message can be used to send a message to *all* of a router's routees. When a router +receives a `Broadcast` message, it will broadcast that message's *payload* to all routees, no matter how that router would normally route its messages. -The example below shows how you would use a ``Broadcast`` message to send a very important message +The example below shows how you would use a `Broadcast` message to send a very important message to every routee of a router. -.. includecode:: code/docs/routing/RouterDocSpec.scala#broadcastDavyJonesWarning +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #broadcastDavyJonesWarning } -In this example the router receives the ``Broadcast`` message, extracts its payload -(``"Watch out for Davy Jones' locker"``), and then sends the payload on to all of the router's +In this example the router receives the `Broadcast` message, extracts its payload +(`"Watch out for Davy Jones' locker"`), and then sends the payload on to all of the router's routees. It is up to each routee actor to handle the received payload message. -.. note:: - Do not use :ref:`broadcast-messages-scala` when you use :ref:`balancing-pool-scala` for routers. - Routees on :ref:`balancing-pool-scala` shares the same mailbox instance, thus some routees can - possibly get the broadcast message multiple times, while other routees get no broadcast message. +@@@ note +Do not use [Broadcast Messages](#broadcast-messages-scala) when you use [BalancingPool](#balancing-pool-scala) for routers. +Routees on [BalancingPool](#balancing-pool-scala) shares the same mailbox instance, thus some routees can +possibly get the broadcast message multiple times, while other routees get no broadcast message. -PoisonPill Messages -------------------- +@@@ -A ``PoisonPill`` message has special handling for all actors, including for routers. When any actor -receives a ``PoisonPill`` message, that actor will be stopped. See the :ref:`poison-pill-scala` +### PoisonPill Messages + +A `PoisonPill` message has special handling for all actors, including for routers. When any actor +receives a `PoisonPill` message, that actor will be stopped. See the @ref:[PoisonPill](actors.md#poison-pill-scala) documentation for details. -.. includecode:: code/docs/routing/RouterDocSpec.scala#poisonPill +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #poisonPill } For a router, which normally passes on messages to routees, it is important to realise that -``PoisonPill`` messages are processed by the router only. ``PoisonPill`` messages sent to a router +`PoisonPill` messages are processed by the router only. `PoisonPill` messages sent to a router will *not* be sent on to routees. -However, a ``PoisonPill`` message sent to a router may still affect its routees, because it will +However, a `PoisonPill` message sent to a router may still affect its routees, because it will stop the router and when the router stops it also stops its children. Stopping children is normal actor behavior. The router will stop routees that it has created as children. Each child will process its current message and then stop. This may lead to some messages being unprocessed. -See the documentation on :ref:`stopping-actors-scala` for more information. +See the documentation on @ref:[Stopping actors](actors.md#stopping-actors-scala) for more information. If you wish to stop a router and its routees, but you would like the routees to first process all -the messages currently in their mailboxes, then you should not send a ``PoisonPill`` message to the -router. Instead you should wrap a ``PoisonPill`` message inside a ``Broadcast`` message so that each -routee will receive the ``PoisonPill`` message. Note that this will stop all routees, even if the +the messages currently in their mailboxes, then you should not send a `PoisonPill` message to the +router. Instead you should wrap a `PoisonPill` message inside a `Broadcast` message so that each +routee will receive the `PoisonPill` message. Note that this will stop all routees, even if the routees aren't children of the router, i.e. even routees programmatically provided to the router. -.. includecode:: code/docs/routing/RouterDocSpec.scala#broadcastPoisonPill +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #broadcastPoisonPill } -With the code shown above, each routee will receive a ``PoisonPill`` message. Each routee will -continue to process its messages as normal, eventually processing the ``PoisonPill``. This will -cause the routee to stop. After all routees have stopped the router will itself be :ref:`stopped -automatically ` unless it is a dynamic router, e.g. using +With the code shown above, each routee will receive a `PoisonPill` message. Each routee will +continue to process its messages as normal, eventually processing the `PoisonPill`. This will +cause the routee to stop. After all routees have stopped the router will itself be stopped +automatically unless it is a dynamic router, e.g. using a resizer. -.. note:: +@@@ note - Brendan W McAdams' excellent blog post `Distributing Akka Workloads - And Shutting Down Afterwards - `_ - discusses in more detail how ``PoisonPill`` messages can be used to shut down routers and routees. +Brendan W McAdams' excellent blog post [Distributing Akka Workloads - And Shutting Down Afterwards](http://bytes.codes/2013/01/17/Distributing_Akka_Workloads_And_Shutting_Down_After/) +discusses in more detail how `PoisonPill` messages can be used to shut down routers and routees. -Kill Messages -------------- +@@@ -``Kill`` messages are another type of message that has special handling. See -:ref:`killing-actors-scala` for general information about how actors handle ``Kill`` messages. +### Kill Messages -When a ``Kill`` message is sent to a router the router processes the message internally, and does -*not* send it on to its routees. The router will throw an ``ActorKilledException`` and fail. It +`Kill` messages are another type of message that has special handling. See +@ref:[Killing an Actor](actors.md#killing-actors-scala) for general information about how actors handle `Kill` messages. + +When a `Kill` message is sent to a router the router processes the message internally, and does +*not* send it on to its routees. The router will throw an `ActorKilledException` and fail. It will then be either resumed, restarted or terminated, depending how it is supervised. Routees that are children of the router will also be suspended, and will be affected by the supervision directive that is applied to the router. Routees that are not the routers children, i.e. those that were created externally to the router, will not be affected. -.. includecode:: code/docs/routing/RouterDocSpec.scala#kill +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #kill } -As with the ``PoisonPill`` message, there is a distinction between killing a router, which +As with the `PoisonPill` message, there is a distinction between killing a router, which indirectly kills its children (who happen to be routees), and killing routees directly (some of whom -may not be children.) To kill routees directly the router should be sent a ``Kill`` message wrapped -in a ``Broadcast`` message. +may not be children.) To kill routees directly the router should be sent a `Kill` message wrapped +in a `Broadcast` message. -.. includecode:: code/docs/routing/RouterDocSpec.scala#broadcastKill +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #broadcastKill } -Management Messages -------------------- +### Management Messages -* Sending ``akka.routing.GetRoutees`` to a router actor will make it send back its currently used routees - in a ``akka.routing.Routees`` message. -* Sending ``akka.routing.AddRoutee`` to a router actor will add that routee to its collection of routees. -* Sending ``akka.routing.RemoveRoutee`` to a router actor will remove that routee to its collection of routees. -* Sending ``akka.routing.AdjustPoolSize`` to a pool router actor will add or remove that number of routees to - its collection of routees. + * Sending `akka.routing.GetRoutees` to a router actor will make it send back its currently used routees +in a `akka.routing.Routees` message. + * Sending `akka.routing.AddRoutee` to a router actor will add that routee to its collection of routees. + * Sending `akka.routing.RemoveRoutee` to a router actor will remove that routee to its collection of routees. + * Sending `akka.routing.AdjustPoolSize` to a pool router actor will add or remove that number of routees to +its collection of routees. -These management messages may be handled after other messages, so if you send ``AddRoutee`` immediately followed by +These management messages may be handled after other messages, so if you send `AddRoutee` immediately followed by an ordinary message you are not guaranteed that the routees have been changed when the ordinary message -is routed. If you need to know when the change has been applied you can send ``AddRoutee`` followed by ``GetRoutees`` -and when you receive the ``Routees`` reply you know that the preceding change has been applied. +is routed. If you need to know when the change has been applied you can send `AddRoutee` followed by `GetRoutees` +and when you receive the `Routees` reply you know that the preceding change has been applied. -.. _resizable-routers-scala: - -Dynamically Resizable Pool -^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## Dynamically Resizable Pool Most pools can be used with a fixed number of routees or with a resize strategy to adjust the number of routees dynamically. -There are two types of resizers: the default ``Resizer`` and the ``OptimalSizeExploringResizer``. +There are two types of resizers: the default `Resizer` and the `OptimalSizeExploringResizer`. -Default Resizer ---------------- +### Default Resizer The default resizer ramps up and down pool size based on pressure, measured by the percentage of busy routees in the pool. It ramps up pool size if the pressure is higher than a certain threshold and backs off if the @@ -645,25 +614,23 @@ pressure is lower than certain threshold. Both thresholds are configurable. Pool with default resizer defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-resize-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-resize-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#resize-pool-1 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #resize-pool-1 } -Several more configuration options are available and described in ``akka.actor.deployment.default.resizer`` -section of the reference :ref:`configuration`. +Several more configuration options are available and described in `akka.actor.deployment.default.resizer` +section of the reference configuration. Pool with resizer defined in code: -.. includecode:: code/docs/routing/RouterDocSpec.scala#resize-pool-2 +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #resize-pool-2 } *It is also worth pointing out that if you define the ``router`` in the configuration file then this value will be used instead of any programmatically sent parameters.* +### Optimal Size Exploring Resizer -Optimal Size Exploring Resizer ------------------------------- - -The ``OptimalSizeExploringResizer`` resizes the pool to an optimal size that provides the most message throughput. +The `OptimalSizeExploringResizer` resizes the pool to an optimal size that provides the most message throughput. This resizer works best when you expect the pool size to performance function to be a convex function. For example, when you have a CPU bound tasks, the optimal size is bound to the number of CPU cores. @@ -673,9 +640,9 @@ e.g. a 4 node elastic search cluster may handle 4-8 concurrent requests at optim It achieves this by keeping track of message throughput at each pool size and performing the following three resizing operations (one at a time) periodically: -* Downsize if it hasn't seen all routees ever fully utilized for a period of time. -* Explore to a random nearby pool size to try and collect throughput metrics. -* Optimize to a nearby pool size with a better (than any other nearby sizes) throughput metrics. + * Downsize if it hasn't seen all routees ever fully utilized for a period of time. + * Explore to a random nearby pool size to try and collect throughput metrics. + * Optimize to a nearby pool size with a better (than any other nearby sizes) throughput metrics. When the pool is fully-utilized (i.e. all routees are busy), it randomly choose between exploring and optimizing. When the pool has not been fully-utilized for a period of time, it will downsize the pool to the last seen max @@ -684,33 +651,32 @@ utilization multiplied by a configurable ratio. By constantly exploring and optimizing, the resizer will eventually walk to the optimal size and remain nearby. When the optimal size changes it will start walking towards the new one. -It keeps a performance log so it's stateful as well as having a larger memory footprint than the default ``Resizer``. +It keeps a performance log so it's stateful as well as having a larger memory footprint than the default `Resizer`. The memory usage is O(n) where n is the number of sizes you allow, i.e. upperBound - lowerBound. -Pool with ``OptimalSizeExploringResizer`` defined in configuration: +Pool with `OptimalSizeExploringResizer` defined in configuration: -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-optimal-size-exploring-resize-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-optimal-size-exploring-resize-pool } -.. includecode:: code/docs/routing/RouterDocSpec.scala#optimal-size-exploring-resize-pool +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #optimal-size-exploring-resize-pool } -Several more configuration options are available and described in ``akka.actor.deployment.default.optimal-size-exploring-resizer`` -section of the reference :ref:`configuration`. +Several more configuration options are available and described in `akka.actor.deployment.default.optimal-size-exploring-resizer` +section of the reference configuration. -.. note:: +@@@ note - Resizing is triggered by sending messages to the actor pool, but it is not - completed synchronously; instead a message is sent to the “head” - ``RouterActor`` to perform the size change. Thus you cannot rely on resizing - to instantaneously create new workers when all others are busy, because the - message just sent will be queued to the mailbox of a busy actor. To remedy - this, configure the pool to use a balancing dispatcher, see `Configuring - Dispatchers`_ for more information. +Resizing is triggered by sending messages to the actor pool, but it is not +completed synchronously; instead a message is sent to the “head” +`RouterActor` to perform the size change. Thus you cannot rely on resizing +to instantaneously create new workers when all others are busy, because the +message just sent will be queued to the mailbox of a busy actor. To remedy +this, configure the pool to use a balancing dispatcher, see [Configuring +Dispatchers](#configuring-dispatchers) for more information. +@@@ -.. _router-design-scala: - -How Routing is Designed within Akka -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +## How Routing is Designed within Akka On the surface routers look like normal actors, but they are actually implemented differently. Routers are designed to be extremely efficient at receiving messages and passing them quickly on to @@ -719,8 +685,8 @@ routees. A normal actor can be used for routing messages, but an actor's single-threaded processing can become a bottleneck. Routers can achieve much higher throughput with an optimization to the usual message-processing pipeline that allows concurrent routing. This is achieved by embedding routers' -routing logic directly in their ``ActorRef`` rather than in the router actor. Messages sent to -a router's ``ActorRef`` can be immediately routed to the routee, bypassing the single-threaded +routing logic directly in their `ActorRef` rather than in the router actor. Messages sent to +a router's `ActorRef` can be immediately routed to the routee, bypassing the single-threaded router actor entirely. The cost to this is, of course, that the internals of routing code are more complicated than if @@ -728,17 +694,15 @@ routers were implemented with normal actors. Fortunately all of this complexity consumers of the routing API. However, it is something to be aware of when implementing your own routers. -.. _custom-router-scala: - -Custom Router -^^^^^^^^^^^^^ + +## Custom Router You can create your own router should you not find any of the ones provided by Akka sufficient for your needs. In order to roll your own router you have to fulfill certain criteria which are explained in this section. Before creating your own router you should consider whether a normal actor with router-like behavior might do the job just as well as a full-blown router. As explained -:ref:`above `, the primary benefit of routers over normal actors is their +[above](#router-design-scala), the primary benefit of routers over normal actors is their higher performance. But they are somewhat more complicated to write than normal actors. Therefore if lower maximum throughput is acceptable in your application you may wish to stick with traditional actors. This section, however, assumes that you wish to get maximum performance and so demonstrates @@ -748,76 +712,79 @@ The router created in this example is replicating each message to a few destinat Start with the routing logic: -.. includecode:: code/docs/routing/CustomRouterDocSpec.scala#routing-logic +@@snip [CustomRouterDocSpec.scala](code/docs/routing/CustomRouterDocSpec.scala) { #routing-logic } -``select`` will be called for each message and in this example pick a few destinations by round-robin, -by reusing the existing ``RoundRobinRoutingLogic`` and wrap the result in a ``SeveralRoutees`` -instance. ``SeveralRoutees`` will send the message to all of the supplied routes. +`select` will be called for each message and in this example pick a few destinations by round-robin, +by reusing the existing `RoundRobinRoutingLogic` and wrap the result in a `SeveralRoutees` +instance. `SeveralRoutees` will send the message to all of the supplied routes. The implementation of the routing logic must be thread safe, since it might be used outside of actors. A unit test of the routing logic: -.. includecode:: code/docs/routing/CustomRouterDocSpec.scala#unit-test-logic +@@snip [CustomRouterDocSpec.scala](code/docs/routing/CustomRouterDocSpec.scala) { #unit-test-logic } -You could stop here and use the ``RedundancyRoutingLogic`` with a ``akka.routing.Router`` -as described in :ref:`simple-router-scala`. +You could stop here and use the `RedundancyRoutingLogic` with a `akka.routing.Router` +as described in [A Simple Router](#simple-router-scala). Let us continue and make this into a self contained, configurable, router actor. -Create a class that extends ``Pool``, ``Group`` or ``CustomRouterConfig``. That class is a factory -for the routing logic and holds the configuration for the router. Here we make it a ``Group``. +Create a class that extends `Pool`, `Group` or `CustomRouterConfig`. That class is a factory +for the routing logic and holds the configuration for the router. Here we make it a `Group`. -.. includecode:: code/docs/routing/CustomRouterDocSpec.scala#group +@@snip [CustomRouterDocSpec.scala](code/docs/routing/CustomRouterDocSpec.scala) { #group } This can be used exactly as the router actors provided by Akka. -.. includecode:: code/docs/routing/CustomRouterDocSpec.scala#usage-1 +@@snip [CustomRouterDocSpec.scala](code/docs/routing/CustomRouterDocSpec.scala) { #usage-1 } -Note that we added a constructor in ``RedundancyGroup`` that takes a ``Config`` parameter. +Note that we added a constructor in `RedundancyGroup` that takes a `Config` parameter. That makes it possible to define it in configuration. -.. includecode:: code/docs/routing/CustomRouterDocSpec.scala#config +@@snip [CustomRouterDocSpec.scala](code/docs/routing/CustomRouterDocSpec.scala) { #config } -Note the fully qualified class name in the ``router`` property. The router class must extend -``akka.routing.RouterConfig`` (``Pool``, ``Group`` or ``CustomRouterConfig``) and have -constructor with one ``com.typesafe.config.Config`` parameter. +Note the fully qualified class name in the `router` property. The router class must extend +`akka.routing.RouterConfig` (`Pool`, `Group` or `CustomRouterConfig`) and have +constructor with one `com.typesafe.config.Config` parameter. The deployment section of the configuration is passed to the constructor. -.. includecode:: code/docs/routing/CustomRouterDocSpec.scala#usage-2 - -Configuring Dispatchers -^^^^^^^^^^^^^^^^^^^^^^^ +@@snip [CustomRouterDocSpec.scala](code/docs/routing/CustomRouterDocSpec.scala) { #usage-2 } + +## Configuring Dispatchers The dispatcher for created children of the pool will be taken from -``Props`` as described in :ref:`dispatchers-scala`. +`Props` as described in @ref:[Dispatchers](dispatchers.md). To make it easy to define the dispatcher of the routees of the pool you can define the dispatcher inline in the deployment section of the config. -.. includecode:: code/docs/routing/RouterDocSpec.scala#config-pool-dispatcher +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #config-pool-dispatcher } That is the only thing you need to do enable a dedicated dispatcher for a pool. -.. note:: +@@@ note - If you use a group of actors and route to their paths, then they will still use the same dispatcher - that was configured for them in their ``Props``, it is not possible to change an actors dispatcher - after it has been created. +If you use a group of actors and route to their paths, then they will still use the same dispatcher +that was configured for them in their `Props`, it is not possible to change an actors dispatcher +after it has been created. + +@@@ The “head” router cannot always run on the same dispatcher, because it does not process the same type of messages, hence this special actor does -not use the dispatcher configured in ``Props``, but takes the -``routerDispatcher`` from the :class:`RouterConfig` instead, which defaults to +not use the dispatcher configured in `Props`, but takes the +`routerDispatcher` from the `RouterConfig` instead, which defaults to the actor system’s default dispatcher. All standard routers allow setting this property in their constructor or factory method, custom routers have to implement the method in a suitable way. -.. includecode:: code/docs/routing/RouterDocSpec.scala#dispatchers +@@snip [RouterDocSpec.scala](code/docs/routing/RouterDocSpec.scala) { #dispatchers } -.. note:: +@@@ note - It is not allowed to configure the ``routerDispatcher`` to be a - :class:`akka.dispatch.BalancingDispatcherConfigurator` since the messages meant - for the special router actor cannot be processed by any other actor. +It is not allowed to configure the `routerDispatcher` to be a +`akka.dispatch.BalancingDispatcherConfigurator` since the messages meant +for the special router actor cannot be processed by any other actor. + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/scheduler.md b/akka-docs/src/main/paradox/scala/scheduler.md index 6f9c32e9dd..bc0b1f2a71 100644 --- a/akka-docs/src/main/paradox/scala/scheduler.md +++ b/akka-docs/src/main/paradox/scala/scheduler.md @@ -1,100 +1,90 @@ - -.. _scheduler-scala: - -Scheduler -######### +# Scheduler Sometimes the need for making things happen in the future arises, and where do -you go look then? Look no further than ``ActorSystem``! There you find the -:meth:`scheduler` method that returns an instance of -:class:`akka.actor.Scheduler`, this instance is unique per ActorSystem and is +you go look then? Look no further than `ActorSystem`! There you find the +`scheduler` method that returns an instance of +`akka.actor.Scheduler`, this instance is unique per ActorSystem and is used internally for scheduling things to happen at specific points in time. You can schedule sending of messages to actors and execution of tasks -(functions or Runnable). You will get a ``Cancellable`` back that you can call -:meth:`cancel` on to cancel the execution of the scheduled operation. +(functions or Runnable). You will get a `Cancellable` back that you can call +`cancel` on to cancel the execution of the scheduled operation. The scheduler in Akka is designed for high-throughput of thousands up to millions of triggers. The prime use-case being triggering Actor receive timeouts, Future timeouts, circuit breakers and other time dependent events which happen all-the-time and in many instances at the same time. The implementation is based on a Hashed Wheel Timer, which is -a known datastructure and algorithm for handling such use cases, refer to the `Hashed and Hierarchical Timing Wheels`_ +a known datastructure and algorithm for handling such use cases, refer to the [Hashed and Hierarchical Timing Wheels](http://www.cs.columbia.edu/~nahum/w6998/papers/sosp87-timing-wheels.pdf) whitepaper by Varghese and Lauck if you'd like to understand its inner workings. -The Akka scheduler is **not** designed for long-term scheduling (see `akka-quartz-scheduler`_ +The Akka scheduler is **not** designed for long-term scheduling (see [akka-quartz-scheduler](https://github.com/enragedginger/akka-quartz-scheduler) instead for this use case) nor is it to be used for higly precise firing of the events. The maximum amount of time into the future you can schedule an event to trigger is around 8 months, which in practice is too much to be useful since this would assume the system never went down during that period. If you need long-term scheduling we highly recommend looking into alternative schedulers, as this is not the use-case the Akka scheduler is implemented for. -.. warning:: +@@@ warning - The default implementation of ``Scheduler`` used by Akka is based on job - buckets which are emptied according to a fixed schedule. It does not - execute tasks at the exact time, but on every tick, it will run everything - that is (over)due. The accuracy of the default Scheduler can be modified - by the ``akka.scheduler.tick-duration`` configuration property. +The default implementation of `Scheduler` used by Akka is based on job +buckets which are emptied according to a fixed schedule. It does not +execute tasks at the exact time, but on every tick, it will run everything +that is (over)due. The accuracy of the default Scheduler can be modified +by the `akka.scheduler.tick-duration` configuration property. -.. _akka-quartz-scheduler: https://github.com/enragedginger/akka-quartz-scheduler -.. _Hashed and Hierarchical Timing Wheels: http://www.cs.columbia.edu/~nahum/w6998/papers/sosp87-timing-wheels.pdf +@@@ -Some examples -------------- +## Some examples -.. includecode:: code/docs/actor/SchedulerDocSpec.scala - :include: imports1,schedule-one-off-message +@@snip [SchedulerDocSpec.scala](code/docs/actor/SchedulerDocSpec.scala) { #imports1 #schedule-one-off-message } -.. includecode:: code/docs/actor/SchedulerDocSpec.scala - :include: schedule-one-off-thunk +@@snip [SchedulerDocSpec.scala](code/docs/actor/SchedulerDocSpec.scala) { #schedule-one-off-thunk } -.. includecode:: code/docs/actor/SchedulerDocSpec.scala - :include: schedule-recurring +@@snip [SchedulerDocSpec.scala](code/docs/actor/SchedulerDocSpec.scala) { #schedule-recurring } -.. warning:: +@@@ warning - If you schedule functions or Runnable instances you should be extra careful - to not close over unstable references. In practice this means not using ``this`` - inside the closure in the scope of an Actor instance, not accessing ``sender()`` directly - and not calling the methods of the Actor instance directly. If you need to - schedule an invocation schedule a message to ``self`` instead (containing the - necessary parameters) and then call the method when the message is received. +If you schedule functions or Runnable instances you should be extra careful +to not close over unstable references. In practice this means not using `this` +inside the closure in the scope of an Actor instance, not accessing `sender()` directly +and not calling the methods of the Actor instance directly. If you need to +schedule an invocation schedule a message to `self` instead (containing the +necessary parameters) and then call the method when the message is received. -From ``akka.actor.ActorSystem`` -------------------------------- +@@@ -.. includecode:: ../../../akka-actor/src/main/scala/akka/actor/ActorSystem.scala - :include: scheduler +## From `akka.actor.ActorSystem` -.. warning:: +@@snip [ActorSystem.scala]../../../../../akka-actor/src/main/scala/akka/actor/ActorSystem.scala) { #scheduler } - All scheduled task will be executed when the ``ActorSystem`` is terminated, i.e. - the task may execute before its timeout. +@@@ warning -The Scheduler interface ------------------------ +All scheduled task will be executed when the `ActorSystem` is terminated, i.e. +the task may execute before its timeout. + +@@@ + +## The Scheduler interface The actual scheduler implementation is loaded reflectively upon -:class:`ActorSystem` start-up, which means that it is possible to provide a -different one using the ``akka.scheduler.implementation`` configuration +`ActorSystem` start-up, which means that it is possible to provide a +different one using the `akka.scheduler.implementation` configuration property. The referenced class must implement the following interface: -.. includecode:: ../../../akka-actor/src/main/scala/akka/actor/Scheduler.scala - :include: scheduler +@@snip [Scheduler.scala]../../../../../akka-actor/src/main/scala/akka/actor/Scheduler.scala) { #scheduler } -The Cancellable interface -------------------------- +## The Cancellable interface -Scheduling a task will result in a :class:`Cancellable` (or throw an -:class:`IllegalStateException` if attempted after the scheduler’s shutdown). +Scheduling a task will result in a `Cancellable` (or throw an +`IllegalStateException` if attempted after the scheduler’s shutdown). This allows you to cancel something that has been scheduled for execution. -.. warning:: +@@@ warning - This does not abort the execution of the task, if it had already been - started. Check the return value of ``cancel`` to detect whether the - scheduled task was canceled or will (eventually) have run. +This does not abort the execution of the task, if it had already been +started. Check the return value of `cancel` to detect whether the +scheduled task was canceled or will (eventually) have run. -.. includecode:: ../../../akka-actor/src/main/scala/akka/actor/Scheduler.scala - :include: cancellable +@@@ +@@snip [Scheduler.scala]../../../../../akka-actor/src/main/scala/akka/actor/Scheduler.scala) { #cancellable } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/serialization.md b/akka-docs/src/main/paradox/scala/serialization.md index b556c42d9d..d205e2b4fb 100644 --- a/akka-docs/src/main/paradox/scala/serialization.md +++ b/akka-docs/src/main/paradox/scala/serialization.md @@ -1,151 +1,139 @@ - -.. _serialization-scala: - -Serialization -############# +# Serialization The messages that Akka actors send to each other are JVM objects (e.g. instances of Scala case classes). Message passing between actors that live on the same JVM is straightforward. It is simply done via reference passing. However, messages that have to escape the JVM to reach an actor running on a different host have to undergo some form of serialization (i.e. the objects have to be converted to and from byte arrays). Akka itself uses Protocol Buffers to serialize internal messages (i.e. cluster gossip messages). However, the serialization mechanism in Akka allows you to write custom serializers and to define which serializer to use for what. -Usage -===== +## Usage -Configuration -------------- +### Configuration -For Akka to know which ``Serializer`` to use for what, you need edit your :ref:`configuration`, -in the "akka.actor.serializers"-section you bind names to implementations of the ``akka.serialization.Serializer`` +For Akka to know which `Serializer` to use for what, you need edit your [Configuration](), +in the "akka.actor.serializers"-section you bind names to implementations of the `akka.serialization.Serializer` you wish to use, like this: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala#serialize-serializers-config +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #serialize-serializers-config } -After you've bound names to different implementations of ``Serializer`` you need to wire which classes -should be serialized using which ``Serializer``, this is done in the "akka.actor.serialization-bindings"-section: +After you've bound names to different implementations of `Serializer` you need to wire which classes +should be serialized using which `Serializer`, this is done in the "akka.actor.serialization-bindings"-section: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala#serialization-bindings-config +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #serialization-bindings-config } You only need to specify the name of an interface or abstract base class of the messages. In case of ambiguity, i.e. the message implements several of the configured classes, the most specific configured class will be used, i.e. the one of which all other candidates are superclasses. If this condition cannot be -met, because e.g. ``java.io.Serializable`` and ``MyOwnSerializable`` both apply +met, because e.g. `java.io.Serializable` and `MyOwnSerializable` both apply and neither is a subtype of the other, a warning will be issued -.. note:: +@@@ note - If your messages are contained inside of a Scala object, then in order to - reference those messages, you will need use the fully qualified Java class name. For a message - named ``Message`` contained inside the object named ``Wrapper`` - you would need to reference it as ``Wrapper$Message`` instead of ``Wrapper.Message``. +If your messages are contained inside of a Scala object, then in order to +reference those messages, you will need use the fully qualified Java class name. For a message +named `Message` contained inside the object named `Wrapper` +you would need to reference it as `Wrapper$Message` instead of `Wrapper.Message`. -Akka provides serializers for :class:`java.io.Serializable` and `protobuf -`_ -:class:`com.google.protobuf.GeneratedMessage` by default (the latter only if +@@@ + +Akka provides serializers for `java.io.Serializable` and [protobuf](http://code.google.com/p/protobuf/) +`com.google.protobuf.GeneratedMessage` by default (the latter only if depending on the akka-remote module), so normally you don't need to add -configuration for that; since :class:`com.google.protobuf.GeneratedMessage` -implements :class:`java.io.Serializable`, protobuf messages will always be +configuration for that; since `com.google.protobuf.GeneratedMessage` +implements `java.io.Serializable`, protobuf messages will always be serialized using the protobuf protocol unless specifically overridden. In order -to disable a default serializer, map its marker type to “none”:: +to disable a default serializer, map its marker type to “none”: - akka.actor.serialization-bindings { - "java.io.Serializable" = none - } +``` +akka.actor.serialization-bindings { + "java.io.Serializable" = none +} +``` -Verification ------------- +### Verification Normally, messages sent between local actors (i.e. same JVM) do not undergo serialization. For testing, sometimes, it may be desirable to force serialization on all messages (both remote and local). If you want to do this in order to verify that your messages are serializable you can enable the following config option: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala#serialize-messages-config +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #serialize-messages-config } -If you want to verify that your ``Props`` are serializable you can enable the following config option: +If you want to verify that your `Props` are serializable you can enable the following config option: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala#serialize-creators-config +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #serialize-creators-config } -.. warning:: +@@@ warning - We recommend having these config options turned on **only** when you're running tests. Turning these options on in production is pointless, as it would negatively impact the performance of local message passing without giving any gain. +We recommend having these config options turned on **only** when you're running tests. Turning these options on in production is pointless, as it would negatively impact the performance of local message passing without giving any gain. -Programmatic ------------- +@@@ + +### Programmatic If you want to programmatically serialize/deserialize using Akka Serialization, here's some examples: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala - :include: imports,programmatic +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #imports #programmatic } -For more information, have a look at the ``ScalaDoc`` for ``akka.serialization._`` +For more information, have a look at the `ScalaDoc` for `akka.serialization._` -Customization -============= +## Customization -The first code snippet on this page contains a configuration file that references a custom serializer ``docs.serialization.MyOwnSerializer``. How would we go about creating such a custom serializer? +The first code snippet on this page contains a configuration file that references a custom serializer `docs.serialization.MyOwnSerializer`. How would we go about creating such a custom serializer? -Creating new Serializers ------------------------- +### Creating new Serializers -A custom ``Serializer`` has to inherit from ``akka.serialization.Serializer`` and can be defined like the following: +A custom `Serializer` has to inherit from `akka.serialization.Serializer` and can be defined like the following: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala - :include: imports,my-own-serializer - :exclude: ... +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #imports #my-own-serializer } The manifest is a type hint so that the same serializer can be used for different -classes. The manifest parameter in ``fromBinary`` is the class of the object that -was serialized. In ``fromBinary`` you can match on the class and deserialize the +classes. The manifest parameter in `fromBinary` is the class of the object that +was serialized. In `fromBinary` you can match on the class and deserialize the bytes to different objects. -Then you only need to fill in the blanks, bind it to a name in your :ref:`configuration` and then +Then you only need to fill in the blanks, bind it to a name in your [Configuration]() and then list which classes that should be serialized using it. -.. _string-manifest-serializer-scala: + +### Serializer with String Manifest -Serializer with String Manifest -------------------------------- +The `Serializer` illustrated above supports a class based manifest (type hint). +For serialization of data that need to evolve over time the `SerializerWithStringManifest` +is recommended instead of `Serializer` because the manifest (type hint) is a `String` +instead of a `Class`. That means that the class can be moved/removed and the serializer +can still deserialize old data by matching on the `String`. This is especially useful +for @ref:[Persistence](persistence.md). -The ``Serializer`` illustrated above supports a class based manifest (type hint). -For serialization of data that need to evolve over time the ``SerializerWithStringManifest`` -is recommended instead of ``Serializer`` because the manifest (type hint) is a ``String`` -instead of a ``Class``. That means that the class can be moved/removed and the serializer -can still deserialize old data by matching on the ``String``. This is especially useful -for :ref:`persistence-scala`. - -The manifest string can also encode a version number that can be used in ``fromBinary`` to +The manifest string can also encode a version number that can be used in `fromBinary` to deserialize in different ways to migrate old data to new domain objects. -If the data was originally serialized with ``Serializer`` and in a later version of the -system you change to ``SerializerWithStringManifest`` the manifest string will be the full -class name if you used ``includeManifest=true``, otherwise it will be the empty string. +If the data was originally serialized with `Serializer` and in a later version of the +system you change to `SerializerWithStringManifest` the manifest string will be the full +class name if you used `includeManifest=true`, otherwise it will be the empty string. -This is how a ``SerializerWithStringManifest`` looks like: +This is how a `SerializerWithStringManifest` looks like: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala#my-own-serializer2 +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #my-own-serializer2 } -You must also bind it to a name in your :ref:`configuration` and then list which classes +You must also bind it to a name in your [Configuration]() and then list which classes that should be serialized using it. -It's recommended to throw ``java.io.NotSerializableException`` in ``fromBinary`` +It's recommended to throw `java.io.NotSerializableException` in `fromBinary` if the manifest is unknown. This makes it possible to introduce new message types and send them to nodes that don't know about them. This is typically needed when performing rolling upgrades, i.e. running a cluster with mixed versions for while. -``NotSerializableException`` is treated as a transient problem in the TCP based remoting +`NotSerializableException` is treated as a transient problem in the TCP based remoting layer. The problem will be logged and message is dropped. Other exceptions will tear down the TCP connection because it can be an indication of corrupt bytes from the underlying transport. -Serializing ActorRefs ---------------------- +### Serializing ActorRefs All ActorRefs are serializable using JavaSerializer, but in case you are writing your own serializer, you might want to know how to serialize and deserialize them properly. In the general case, the local address to be used depends on the type of remote address which shall be the recipient of the serialized information. Use -:meth:`Serialization.serializedActorPath(actorRef)` like this: +`Serialization.serializedActorPath(actorRef)` like this: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala - :include: imports,actorref-serializer +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #imports #actorref-serializer } This assumes that serialization happens in the context of sending a message through the remote transport. There are other uses of serialization, though, @@ -157,74 +145,65 @@ in the same logical context, but it is not enough when deserializing it on a different network host: for that it would need to include the system’s remote transport address. An actor system is not limited to having just one remote transport per se, which makes this question a bit more interesting. To find out -the appropriate address to use when sending to ``remoteAddr`` you can use -:meth:`ActorRefProvider.getExternalAddressFor(remoteAddr)` like this: +the appropriate address to use when sending to `remoteAddr` you can use +`ActorRefProvider.getExternalAddressFor(remoteAddr)` like this: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala - :include: external-address +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #external-address } -.. note:: - - ``ActorPath.toSerializationFormatWithAddress`` differs from ``toString`` if the - address does not already have ``host`` and ``port`` components, i.e. it only - inserts address information for local addresses. +@@@ note - ``toSerializationFormatWithAddress`` also adds the unique id of the actor, which will - change when the actor is stopped and then created again with the same name. - Sending messages to a reference pointing the old actor will not be delivered - to the new actor. If you don't want this behavior, e.g. in case of long term - storage of the reference, you can use ``toStringWithAddress``, which doesn't - include the unique id. +`ActorPath.toSerializationFormatWithAddress` differs from `toString` if the +address does not already have `host` and `port` components, i.e. it only +inserts address information for local addresses. +`toSerializationFormatWithAddress` also adds the unique id of the actor, which will +change when the actor is stopped and then created again with the same name. +Sending messages to a reference pointing the old actor will not be delivered +to the new actor. If you don't want this behavior, e.g. in case of long term +storage of the reference, you can use `toStringWithAddress`, which doesn't +include the unique id. + +@@@ This requires that you know at least which type of address will be supported by the system which will deserialize the resulting actor reference; if you have no concrete address handy you can create a dummy one for the right protocol using -``Address(protocol, "", "", 0)`` (assuming that the actual transport used is as +`Address(protocol, "", "", 0)` (assuming that the actual transport used is as lenient as Akka’s RemoteActorRefProvider). There is also a default remote address which is the one used by cluster support (and typical systems have just this one); you can get it like this: -.. includecode:: code/docs/serialization/SerializationDocSpec.scala - :include: external-address-default +@@snip [SerializationDocSpec.scala](code/docs/serialization/SerializationDocSpec.scala) { #external-address-default } -Deep serialization of Actors ----------------------------- +### Deep serialization of Actors -The recommended approach to do deep serialization of internal actor state is to use Akka :ref:`persistence-scala`. +The recommended approach to do deep serialization of internal actor state is to use Akka @ref:[Persistence](persistence.md). -A Word About Java Serialization -=============================== +## A Word About Java Serialization -When using Java serialization without employing the :class:`JavaSerializer` for -the task, you must make sure to supply a valid :class:`ExtendedActorSystem` in -the dynamic variable ``JavaSerializer.currentSystem``. This is used when -reading in the representation of an :class:`ActorRef` for turning the string -representation into a real reference. :class:`DynamicVariable` is a +When using Java serialization without employing the `JavaSerializer` for +the task, you must make sure to supply a valid `ExtendedActorSystem` in +the dynamic variable `JavaSerializer.currentSystem`. This is used when +reading in the representation of an `ActorRef` for turning the string +representation into a real reference. `DynamicVariable` is a thread-local variable, so be sure to have it set while deserializing anything which might contain actor references. - -Serialization compatibility -=========================== +## Serialization compatibility It is not safe to mix major Scala versions when using the Java serialization as Scala does not guarantee compatibility and this could lead to very surprising errors. -If using the Akka Protobuf serializers (implicitly with ``akka.actor.allow-java-serialization = off`` or explicitly with -``enable-additional-serialization-bindings = true``) for the internal Akka messages those will not require the same major +If using the Akka Protobuf serializers (implicitly with `akka.actor.allow-java-serialization = off` or explicitly with +`enable-additional-serialization-bindings = true`) for the internal Akka messages those will not require the same major Scala version however you must also ensure the serializers used for your own types does not introduce the same incompatibility as Java serialization does. +## External Akka Serializers -External Akka Serializers -========================= +[Akka-quickser by Roman Levenstein](https://github.com/romix/akka-quickser-serialization) -`Akka-quickser by Roman Levenstein `_ +[Akka-kryo by Roman Levenstein](https://github.com/romix/akka-kryo-serialization) - -`Akka-kryo by Roman Levenstein `_ - - -`Twitter Chill Scala extensions for Kryo (based on Akka Version 2.3.x but due to backwards compatibility of the Serializer Interface this extension also works with 2.4.x) `_ +[Twitter Chill Scala extensions for Kryo (based on Akka Version 2.3.x but due to backwards compatibility of the Serializer Interface this extension also works with 2.4.x)](https://github.com/twitter/chill) \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/index.md b/akka-docs/src/main/paradox/scala/stream/index.md index a7d1f143de..c47a6edee6 100644 --- a/akka-docs/src/main/paradox/scala/stream/index.md +++ b/akka-docs/src/main/paradox/scala/stream/index.md @@ -1,25 +1,25 @@ -.. _streams-scala: +# Streams -Streams -======= +@@toc { depth=2 } -.. toctree:: - :maxdepth: 2 +@@@ index - stream-introduction - stream-quickstart - ../../general/stream/stream-design - stream-flows-and-basics - stream-graphs - stream-composition - stream-rate - stream-dynamic - stream-customize - stream-integrations - stream-error - stream-io - stream-parallelism - stream-testkit - stages-overview - stream-cookbook - ../../general/stream/stream-configuration +* [stream-introduction](stream-introduction.md) +* [stream-quickstart](stream-quickstart.md) +* [../../general/stream/stream-design](../../general/stream/stream-design.md) +* [stream-flows-and-basics](stream-flows-and-basics.md) +* [stream-graphs](stream-graphs.md) +* [stream-composition](stream-composition.md) +* [stream-rate](stream-rate.md) +* [stream-dynamic](stream-dynamic.md) +* [stream-customize](stream-customize.md) +* [stream-integrations](stream-integrations.md) +* [stream-error](stream-error.md) +* [stream-io](stream-io.md) +* [stream-parallelism](stream-parallelism.md) +* [stream-testkit](stream-testkit.md) +* [stages-overview](stages-overview.md) +* [stream-cookbook](stream-cookbook.md) +* [../../general/stream/stream-configuration](../../general/stream/stream-configuration.md) + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stages-overview.md b/akka-docs/src/main/paradox/scala/stream/stages-overview.md index 026d246b6b..b51a82bf5b 100644 --- a/akka-docs/src/main/paradox/scala/stream/stages-overview.md +++ b/akka-docs/src/main/paradox/scala/stream/stages-overview.md @@ -1,18 +1,12 @@ -.. _stages-overview_scala: +# Overview of built-in stages and their semantics -Overview of built-in stages and their semantics -=============================================== +## Source stages +These built-in sources are available from `akka.stream.scaladsl.Source`: -Source stages -------------- -These built-in sources are available from ``akka.stream.scaladsl.Source``: +### fromIterator - - -fromIterator -^^^^^^^^^^^^ -Stream the values from an ``Iterator``, requesting the next value when there is demand. The iterator will be created anew +Stream the values from an `Iterator`, requesting the next value when there is demand. The iterator will be created anew for each materialization, which is the reason the method takes a function rather than an iterator directly. If the iterator perform blocking operations, make sure to run it on a separate dispatcher. @@ -21,33 +15,32 @@ If the iterator perform blocking operations, make sure to run it on a separate d **completes** when the iterator reaches its end -apply -^^^^^ -Stream the values of an ``immutable.Seq``. +### apply + +Stream the values of an `immutable.Seq`. **emits** the next value of the seq **completes** when the last element of the seq has been emitted +### single -single -^^^^^^ Stream a single object **emits** the value once **completes** when the single value has been emitted -repeat -^^^^^^ +### repeat + Stream a single object repeatedly **emits** the same value repeatedly when there is demand **completes** never -cycle -^^^^^ +### cycle + Stream iterator in cycled manner. Internally new iterator is being created to cycle the one provided via argument meaning when original iterator runs out of elements process will start all over again from the beginning of the iterator provided by the evaluation of provided parameter. If method argument provides empty iterator stream will be terminated with @@ -57,8 +50,8 @@ exception. **completes** never -tick -^^^^ +### tick + A periodical repetition of an arbitrary object. Delay of first tick is specified separately from interval of the following ticks. @@ -66,67 +59,67 @@ separately from interval of the following ticks. **completes** never -fromFuture -^^^^^^^^^^ -Send the single value of the ``Future`` when it completes and there is demand. +### fromFuture + +Send the single value of the `Future` when it completes and there is demand. If the future fails the stream is failed with that exception. **emits** the future completes **completes** after the future has completed -fromCompletionStage -^^^^^^^^^^^^^^^^^^^ -Send the single value of the Java ``CompletionStage`` when it completes and there is demand. +### fromCompletionStage + +Send the single value of the Java `CompletionStage` when it completes and there is demand. If the future fails the stream is failed with that exception. **emits** the future completes **completes** after the future has completed -fromFutureSource -^^^^^^^^^^^^^^^^ +### fromFutureSource + Streams the elements of the given future source once it successfully completes. If the future fails the stream is failed. -**emits** the next value from the `future` source, once it has completed +**emits** the next value from the *future* source, once it has completed -**completes** after the `future` source completes +**completes** after the *future* source completes -fromSourceCompletionStage -^^^^^^^^^^^^^^^^^^^^^^^^^ -Streams the elements of an asynchronous source once its given `completion` stage completes. -If the `completion` fails the stream is failed with that exception. +### fromSourceCompletionStage -**emits** the next value from the asynchronous source, once its `completion stage` has completed +Streams the elements of an asynchronous source once its given *completion* stage completes. +If the *completion* fails the stream is failed with that exception. + +**emits** the next value from the asynchronous source, once its *completion stage* has completed **completes** after the asynchronous source completes -unfold -^^^^^^ -Stream the result of a function as long as it returns a ``Some``, the value inside the option -consists of a tuple where the first value is a state passed back into the next call to the function allowing -to pass a state. The first invocation of the provided fold function will receive the ``zero`` state. +### unfold -Can be used to implement many stateful sources without having to touch the more low level ``GraphStage`` API. +Stream the result of a function as long as it returns a `Some`, the value inside the option +consists of a tuple where the first value is a state passed back into the next call to the function allowing +to pass a state. The first invocation of the provided fold function will receive the `zero` state. + +Can be used to implement many stateful sources without having to touch the more low level `GraphStage` API. **emits** when there is demand and the unfold function over the previous state returns non empty value **completes** when the unfold function returns an empty value -unfoldAsync -^^^^^^^^^^^ -Just like ``unfold`` but the fold function returns a ``Future`` which will cause the source to +### unfoldAsync + +Just like `unfold` but the fold function returns a `Future` which will cause the source to complete or emit when it completes. -Can be used to implement many stateful sources without having to touch the more low level ``GraphStage`` API. +Can be used to implement many stateful sources without having to touch the more low level `GraphStage` API. **emits** when there is demand and unfold state returned future completes with some value **completes** when the future returned by the unfold function completes with an empty value -empty -^^^^^ +### empty + Complete right away without ever emitting any elements. Useful when you have to provide a source to an API but there are no elements to emit. @@ -134,157 +127,152 @@ an API but there are no elements to emit. **completes** directly -maybe -^^^^^ -Materialize a ``Promise[Option[T]]`` that if completed with a ``Some[T]`` will emit that `T` and then complete -the stream, or if completed with ``None`` complete the stream right away. +### maybe + +Materialize a `Promise[Option[T]]` that if completed with a `Some[T]` will emit that *T* and then complete +the stream, or if completed with `None` complete the stream right away. **emits** when the returned promise is completed with some value **completes** after emitting some value, or directly if the promise is completed with no value -failed -^^^^^^ +### failed + Fail directly with a user specified exception. **emits** never **completes** fails the stream directly with the given exception -lazily -~~~~~~ -Defers creation and materialization of a ``Source`` until there is demand. +#### lazily -**emits** depends on the wrapped ``Source`` +Defers creation and materialization of a `Source` until there is demand. -**completes** depends on the wrapped ``Source`` +**emits** depends on the wrapped `Source` -actorPublisher -^^^^^^^^^^^^^^ -Wrap an actor extending ``ActorPublisher`` as a source. +**completes** depends on the wrapped `Source` + +### actorPublisher + +Wrap an actor extending `ActorPublisher` as a source. **emits** depends on the actor implementation **completes** when the actor stops -actorRef -^^^^^^^^ -Materialize an ``ActorRef``, sending messages to it will emit them on the stream. The actor contain +### actorRef + +Materialize an `ActorRef`, sending messages to it will emit them on the stream. The actor contain a buffer but since communication is one way, there is no back pressure. Handling overflow is done by either dropping elements or failing the stream, the strategy is chosen by the user. **emits** when there is demand and there are messages in the buffer or a message is sent to the actorref -**completes** when the actorref is sent ``akka.actor.Status.Success`` or ``PoisonPill`` +**completes** when the actorref is sent `akka.actor.Status.Success` or `PoisonPill` + +### combine -combine -^^^^^^^ Combine several sources, using a given strategy such as merge or concat, into one source. **emits** when there is demand, but depending on the strategy **completes** when all sources has completed -unfoldResource -^^^^^^^^^^^^^^ +### unfoldResource + Wrap any resource that can be opened, queried for next element (in a blocking way) and closed using three distinct functions into a source. **emits** when there is demand and read function returns value -**completes** when read function returns ``None`` +**completes** when read function returns `None` + +### unfoldResourceAsync -unfoldResourceAsync -^^^^^^^^^^^^^^^^^^^ Wrap any resource that can be opened, queried for next element (in a blocking way) and closed using three distinct functions into a source. -Functions return ``Future`` to achieve asynchronous processing +Functions return `Future` to achieve asynchronous processing -**emits** when there is demand and ``Future`` from read function returns value +**emits** when there is demand and `Future` from read function returns value -**completes** when ``Future`` from read function returns ``None`` +**completes** when `Future` from read function returns `None` -queue -^^^^^ -Materialize a ``SourceQueue`` onto which elements can be pushed for emitting from the source. The queue contains +### queue + +Materialize a `SourceQueue` onto which elements can be pushed for emitting from the source. The queue contains a buffer, if elements are pushed onto the queue faster than the source is consumed the overflow will be handled with a strategy specified by the user. Functionality for tracking when an element has been emitted is available through -``SourceQueue.offer``. +`SourceQueue.offer`. **emits** when there is demand and the queue contains elements **completes** when downstream completes -asSubscriber -^^^^^^^^^^^^ -Integration with Reactive Streams, materializes into a ``org.reactivestreams.Subscriber``. +### asSubscriber +Integration with Reactive Streams, materializes into a `org.reactivestreams.Subscriber`. -fromPublisher -^^^^^^^^^^^^^ -Integration with Reactive Streams, subscribes to a ``org.reactivestreams.Publisher``. +### fromPublisher + +Integration with Reactive Streams, subscribes to a `org.reactivestreams.Publisher`. + +### zipN -zipN -^^^^ Combine the elements of multiple streams into a stream of sequences. **emits** when all of the inputs has an element available **completes** when any upstream completes -zipWithN -^^^^^^^^ +### zipWithN + Combine the elements of multiple streams into a stream of sequences using a combiner function. **emits** when all of the inputs has an element available **completes** when any upstream completes +## Sink stages +These built-in sinks are available from `akka.stream.scaladsl.Sink`: +### head -Sink stages ------------ -These built-in sinks are available from ``akka.stream.scaladsl.Sink``: - - -head -^^^^ -Materializes into a ``Future`` which completes with the first value arriving, +Materializes into a `Future` which completes with the first value arriving, after this the stream is canceled. If no element is emitted, the future is be failed. **cancels** after receiving one element **backpressures** never -headOption -^^^^^^^^^^ -Materializes into a ``Future[Option[T]]`` which completes with the first value arriving wrapped in a ``Some``, -or a ``None`` if the stream completes without any elements emitted. +### headOption + +Materializes into a `Future[Option[T]]` which completes with the first value arriving wrapped in a `Some`, +or a `None` if the stream completes without any elements emitted. **cancels** after receiving one element **backpressures** never -last -^^^^ -Materializes into a ``Future`` which will complete with the last value emitted when the stream +### last + +Materializes into a `Future` which will complete with the last value emitted when the stream completes. If the stream completes with no elements the future is failed. **cancels** never **backpressures** never -lastOption -^^^^^^^^^^ -Materialize a ``Future[Option[T]]`` which completes with the last value -emitted wrapped in an ``Some`` when the stream completes. if the stream completes with no elements the future is -completed with ``None``. +### lastOption + +Materialize a `Future[Option[T]]` which completes with the last value +emitted wrapped in an `Some` when the stream completes. if the stream completes with no elements the future is +completed with `None`. **cancels** never **backpressures** never -ignore -^^^^^^ +### ignore + Consume all elements but discards them. Useful when a stream has to be consumed but there is no use to actually do anything with the elements. @@ -292,25 +280,25 @@ do anything with the elements. **backpressures** never -cancelled -^^^^^^^^^ +### cancelled + Immediately cancel the stream **cancels** immediately -seq -^^^ -Collect values emitted from the stream into a collection, the collection is available through a ``Future`` or -which completes when the stream completes. Note that the collection is bounded to ``Int.MaxValue``, +### seq + +Collect values emitted from the stream into a collection, the collection is available through a `Future` or +which completes when the stream completes. Note that the collection is bounded to `Int.MaxValue`, if more element are emitted the sink will cancel the stream **cancels** If too many values are collected -foreach -^^^^^^^ +### foreach + Invoke a given procedure for each element received. Note that it is not safe to mutate shared state from the procedure. -The sink materializes into a ``Future[Option[Done]]`` which completes when the +The sink materializes into a `Future[Option[Done]]` which completes when the stream completes, or fails if the stream fails. Note that it is not safe to mutate state from the procedure. @@ -319,46 +307,44 @@ Note that it is not safe to mutate state from the procedure. **backpressures** when the previous procedure invocation has not yet completed +### foreachParallel -foreachParallel -^^^^^^^^^^^^^^^ -Like ``foreach`` but allows up to ``parallellism`` procedure calls to happen in parallel. +Like `foreach` but allows up to `parallellism` procedure calls to happen in parallel. **cancels** never **backpressures** when the previous parallel procedure invocations has not yet completed +### onComplete -onComplete -^^^^^^^^^^ Invoke a callback when the stream has completed or failed. **cancels** never **backpressures** never -lazyInit -^^^^^^^^ -Invoke sinkFactory function to create a real sink upon receiving the first element. Internal ``Sink`` will not be created if there are no elements, -because of completion or error. `fallback` will be invoked if there was no elements and completed is received from upstream. +### lazyInit + +Invoke sinkFactory function to create a real sink upon receiving the first element. Internal `Sink` will not be created if there are no elements, +because of completion or error. *fallback* will be invoked if there was no elements and completed is received from upstream. **cancels** never **backpressures** when initialized and when created sink backpressures -queue -^^^^^ -Materialize a ``SinkQueue`` that can be pulled to trigger demand through the sink. The queue contains +### queue + +Materialize a `SinkQueue` that can be pulled to trigger demand through the sink. The queue contains a buffer in case stream emitting elements faster than queue pulling them. -**cancels** when ``SinkQueue.cancel`` is called +**cancels** when `SinkQueue.cancel` is called **backpressures** when buffer has some space -fold -^^^^ +### fold + Fold over emitted element with a function, where each invocation will get the new element and the result from the -previous fold invocation. The first invocation will be provided the ``zero`` value. +previous fold invocation. The first invocation will be provided the `zero` value. Materializes into a future that will complete with the last state when the stream has completed. @@ -369,8 +355,8 @@ between invocations. **backpressures** when the previous fold function invocation has not yet completed -reduce -^^^^^^ +### reduce + Apply a reduction function on the incoming elements and pass the result to the next invocation. The first invocation receives the two first elements of the flow. @@ -380,163 +366,151 @@ Materializes into a future that will be completed by the last result of the redu **backpressures** when the previous reduction function invocation has not yet completed +### combine -combine -^^^^^^^ Combine several sinks into one using a user specified strategy **cancels** depends on the strategy **backpressures** depends on the strategy +### actorRef -actorRef -^^^^^^^^ -Send the elements from the stream to an ``ActorRef``. No backpressure so care must be taken to not overflow the inbox. +Send the elements from the stream to an `ActorRef`. No backpressure so care must be taken to not overflow the inbox. **cancels** when the actor terminates **backpressures** never +### actorRefWithAck -actorRefWithAck -^^^^^^^^^^^^^^^ -Send the elements from the stream to an ``ActorRef`` which must then acknowledge reception after completing a message, +Send the elements from the stream to an `ActorRef` which must then acknowledge reception after completing a message, to provide back pressure onto the sink. **cancels** when the actor terminates **backpressures** when the actor acknowledgement has not arrived +### actorSubscriber -actorSubscriber -^^^^^^^^^^^^^^^ -Create an actor from a ``Props`` upon materialization, where the actor implements ``ActorSubscriber``, which will +Create an actor from a `Props` upon materialization, where the actor implements `ActorSubscriber`, which will receive the elements from the stream. -Materializes into an ``ActorRef`` to the created actor. +Materializes into an `ActorRef` to the created actor. **cancels** when the actor terminates **backpressures** depends on the actor implementation +### asPublisher -asPublisher -^^^^^^^^^^^ -Integration with Reactive Streams, materializes into a ``org.reactivestreams.Publisher``. +Integration with Reactive Streams, materializes into a `org.reactivestreams.Publisher`. +### fromSubscriber -fromSubscriber -^^^^^^^^^^^^^^ -Integration with Reactive Streams, wraps a ``org.reactivestreams.Subscriber`` as a sink +Integration with Reactive Streams, wraps a `org.reactivestreams.Subscriber` as a sink +## Additional Sink and Source converters +Sources and sinks for integrating with `java.io.InputStream` and `java.io.OutputStream` can be found on +`StreamConverters`. As they are blocking APIs the implementations of these stages are run on a separate +dispatcher configured through the `akka.stream.blocking-io-dispatcher`. +### fromOutputStream -Additional Sink and Source converters -------------------------------------- -Sources and sinks for integrating with ``java.io.InputStream`` and ``java.io.OutputStream`` can be found on -``StreamConverters``. As they are blocking APIs the implementations of these stages are run on a separate -dispatcher configured through the ``akka.stream.blocking-io-dispatcher``. +Create a sink that wraps an `OutputStream`. Takes a function that produces an `OutputStream`, when the sink is +materialized the function will be called and bytes sent to the sink will be written to the returned `OutputStream`. -fromOutputStream -^^^^^^^^^^^^^^^^ -Create a sink that wraps an ``OutputStream``. Takes a function that produces an ``OutputStream``, when the sink is -materialized the function will be called and bytes sent to the sink will be written to the returned ``OutputStream``. - -Materializes into a ``Future`` which will complete with a ``IOResult`` when the stream +Materializes into a `Future` which will complete with a `IOResult` when the stream completes. -Note that a flow can be materialized multiple times, so the function producing the ``OutputStream`` must be able +Note that a flow can be materialized multiple times, so the function producing the `OutputStream` must be able to handle multiple invocations. -The ``OutputStream`` will be closed when the stream that flows into the ``Sink`` is completed, and the ``Sink`` -will cancel its inflow when the ``OutputStream`` is no longer writable. +The `OutputStream` will be closed when the stream that flows into the `Sink` is completed, and the `Sink` +will cancel its inflow when the `OutputStream` is no longer writable. -asInputStream -^^^^^^^^^^^^^ -Create a sink which materializes into an ``InputStream`` that can be read to trigger demand through the sink. -Bytes emitted through the stream will be available for reading through the ``InputStream`` +### asInputStream -The ``InputStream`` will be ended when the stream flowing into this ``Sink`` completes, and the closing the -``InputStream`` will cancel the inflow of this ``Sink``. +Create a sink which materializes into an `InputStream` that can be read to trigger demand through the sink. +Bytes emitted through the stream will be available for reading through the `InputStream` -fromInputStream -^^^^^^^^^^^^^^^ -Create a source that wraps an ``InputStream``. Takes a function that produces an ``InputStream``, when the source is -materialized the function will be called and bytes from the ``InputStream`` will be emitted into the stream. +The `InputStream` will be ended when the stream flowing into this `Sink` completes, and the closing the +`InputStream` will cancel the inflow of this `Sink`. -Materializes into a ``Future`` which will complete with a ``IOResult`` when the stream +### fromInputStream + +Create a source that wraps an `InputStream`. Takes a function that produces an `InputStream`, when the source is +materialized the function will be called and bytes from the `InputStream` will be emitted into the stream. + +Materializes into a `Future` which will complete with a `IOResult` when the stream completes. -Note that a flow can be materialized multiple times, so the function producing the ``InputStream`` must be able +Note that a flow can be materialized multiple times, so the function producing the `InputStream` must be able to handle multiple invocations. -The ``InputStream`` will be closed when the ``Source`` is canceled from its downstream, and reaching the end of the -``InputStream`` will complete the ``Source``. +The `InputStream` will be closed when the `Source` is canceled from its downstream, and reaching the end of the +`InputStream` will complete the `Source`. -asOutputStream -^^^^^^^^^^^^^^ -Create a source that materializes into an ``OutputStream``. When bytes are written to the ``OutputStream`` they +### asOutputStream + +Create a source that materializes into an `OutputStream`. When bytes are written to the `OutputStream` they are emitted from the source. -The ``OutputStream`` will no longer be writable when the ``Source`` has been canceled from its downstream, and -closing the ``OutputStream`` will complete the ``Source``. +The `OutputStream` will no longer be writable when the `Source` has been canceled from its downstream, and +closing the `OutputStream` will complete the `Source`. -asJavaStream -^^^^^^^^^^^^ -Create a sink which materializes into Java 8 ``Stream`` that can be run to trigger demand through the sink. -Elements emitted through the stream will be available for reading through the Java 8 ``Stream``. +### asJavaStream -The Java 8 ``Stream`` will be ended when the stream flowing into this ``Sink`` completes, and closing the Java -``Stream`` will cancel the inflow of this ``Sink``. Java ``Stream`` throws exception in case reactive stream failed. +Create a sink which materializes into Java 8 `Stream` that can be run to trigger demand through the sink. +Elements emitted through the stream will be available for reading through the Java 8 `Stream`. -Be aware that Java ``Stream`` blocks current thread while waiting on next element from downstream. +The Java 8 `Stream` will be ended when the stream flowing into this `Sink` completes, and closing the Java +`Stream` will cancel the inflow of this `Sink`. Java `Stream` throws exception in case reactive stream failed. -fromJavaStream -^^^^^^^^^^^^^^ -Create a source that wraps a Java 8 ``Stream``. ``Source`` uses a stream iterator to get all its elements and send them +Be aware that Java `Stream` blocks current thread while waiting on next element from downstream. + +### fromJavaStream + +Create a source that wraps a Java 8 `Stream`. `Source` uses a stream iterator to get all its elements and send them downstream on demand. -javaCollector -^^^^^^^^^^^^^ -Create a sink which materializes into a ``Future`` which will be completed with a result of the Java 8 ``Collector`` -transformation and reduction operations. This allows usage of Java 8 streams transformations for reactive streams. -The ``Collector`` will trigger demand downstream. Elements emitted through the stream will be accumulated into a mutable -result container, optionally transformed into a final representation after all input elements have been processed. -The ``Collector`` can also do reduction at the end. Reduction processing is performed sequentially +### javaCollector -Note that a flow can be materialized multiple times, so the function producing the ``Collector`` must be able +Create a sink which materializes into a `Future` which will be completed with a result of the Java 8 `Collector` +transformation and reduction operations. This allows usage of Java 8 streams transformations for reactive streams. +The `Collector` will trigger demand downstream. Elements emitted through the stream will be accumulated into a mutable +result container, optionally transformed into a final representation after all input elements have been processed. +The `Collector` can also do reduction at the end. Reduction processing is performed sequentially + +Note that a flow can be materialized multiple times, so the function producing the `Collector` must be able to handle multiple invocations. -javaCollectorParallelUnordered -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Create a sink which materializes into a ``Future`` which will be completed with a result of the Java 8 ``Collector`` -transformation and reduction operations. This allows usage of Java 8 streams transformations for reactive streams. -The ``Collector`` is triggering demand downstream. Elements emitted through the stream will be accumulated into a mutable -result container, optionally transformed into a final representation after all input elements have been processed. -The ``Collector`` can also do reduction at the end. Reduction processing is performed in parallel based on graph ``Balance``. +### javaCollectorParallelUnordered -Note that a flow can be materialized multiple times, so the function producing the ``Collector`` must be able +Create a sink which materializes into a `Future` which will be completed with a result of the Java 8 `Collector` +transformation and reduction operations. This allows usage of Java 8 streams transformations for reactive streams. +The `Collector` is triggering demand downstream. Elements emitted through the stream will be accumulated into a mutable +result container, optionally transformed into a final representation after all input elements have been processed. +The `Collector` can also do reduction at the end. Reduction processing is performed in parallel based on graph `Balance`. + +Note that a flow can be materialized multiple times, so the function producing the `Collector` must be able to handle multiple invocations. -File IO Sinks and Sources -------------------------- -Sources and sinks for reading and writing files can be found on ``FileIO``. +## File IO Sinks and Sources -fromPath -^^^^^^^^ -Emit the contents of a file, as ``ByteString`` s, materializes into a ``Future`` which will be completed with -a ``IOResult`` upon reaching the end of the file or if there is a failure. +Sources and sinks for reading and writing files can be found on `FileIO`. -toPath -^^^^^^ -Create a sink which will write incoming ``ByteString`` s to a given file path. +### fromPath +Emit the contents of a file, as `ByteString` s, materializes into a `Future` which will be completed with +a `IOResult` upon reaching the end of the file or if there is a failure. +### toPath -Flow stages ------------ +Create a sink which will write incoming `ByteString` s to a given file path. + +## Flow stages All flows by default backpressure if the computation they encapsulate is not fast enough to keep up with the rate of incoming elements from the preceding stage. There are differences though how the different stages handle when some of @@ -547,20 +521,18 @@ This happens to ensure reliable teardown of streams and cleanup when failures ha be to model unrecoverable conditions, therefore they are always eagerly propagated. For in-band error handling of normal errors (dropping elements if a map fails for example) you should use the supervision support, or explicitly wrap your element types in a proper container that can express error or success -states (for example ``Try`` in Scala). +states (for example `Try` in Scala). - -Simple processing stages ------------------------- +## Simple processing stages These stages can transform the rate of incoming elements since there are stages that emit multiple elements for a -single input (e.g. `mapConcat') or consume multiple elements before emitting one output (e.g. ``filter``). +single input (e.g. `mapConcat') or consume multiple elements before emitting one output (e.g. `filter`). However, these rate transformations are data-driven, i.e. it is the incoming elements that define how the -rate is affected. This is in contrast with :ref:`detached-stages-overview_scala` which can change their processing behavior +rate is affected. This is in contrast with [detached-stages-overview_scala](#detached-stages-overview-scala) which can change their processing behavior depending on being backpressured by downstream or not. -map -^^^ +### map + Transform each element in the stream by calling a mapping function with it and passing the returned value downstream. **emits** when the mapping function returns an element @@ -569,8 +541,8 @@ Transform each element in the stream by calling a mapping function with it and p **completes** when upstream completes -mapConcat -^^^^^^^^^ +### mapConcat + Transform each element into zero or more elements that are individually passed downstream. **emits** when the mapping function returns an element or there are still remaining elements from the previously calculated collection @@ -579,9 +551,9 @@ Transform each element into zero or more elements that are individually passed d **completes** when upstream completes and all remaining elements has been emitted -statefulMapConcat -^^^^^^^^^^^^^^^^^ -Transform each element into zero or more elements that are individually passed downstream. The difference to ``mapConcat`` is that +### statefulMapConcat + +Transform each element into zero or more elements that are individually passed downstream. The difference to `mapConcat` is that the transformation function is created from a factory for every materialization of the flow. **emits** when the mapping function returns an element or there are still remaining elements from the previously calculated collection @@ -590,8 +562,8 @@ the transformation function is created from a factory for every materialization **completes** when upstream completes and all remaining elements has been emitted -filter -^^^^^^ +### filter + Filter the incoming elements using a predicate. If the predicate returns true the element is passed downstream, if it returns false the element is discarded. @@ -601,8 +573,8 @@ it returns false the element is discarded. **completes** when upstream completes -filterNot -^^^^^^^^^ +### filterNot + Filter the incoming elements using a predicate. If the predicate returns false the element is passed downstream, if it returns true the element is discarded. @@ -612,10 +584,10 @@ it returns true the element is discarded. **completes** when upstream completes -collect -^^^^^^^ +### collect + Apply a partial function to each incoming element, if the partial function is defined for a value the returned -value is passed downstream. Can often replace ``filter`` followed by ``map`` to achieve the same in one single stage. +value is passed downstream. Can often replace `filter` followed by `map` to achieve the same in one single stage. **emits** when the provided partial function is defined for the element @@ -623,8 +595,8 @@ value is passed downstream. Can often replace ``filter`` followed by ``map`` to **completes** when upstream completes -grouped -^^^^^^^ +### grouped + Accumulate incoming events until the specified number of elements have been accumulated and then pass the collection of elements downstream. @@ -634,8 +606,8 @@ elements downstream. **completes** when upstream completes -sliding -^^^^^^^ +### sliding + Provide a sliding window over the incoming stream and pass the windows as groups of elements downstream. Note: the last window might be smaller than the requested size due to end of stream. @@ -646,10 +618,9 @@ Note: the last window might be smaller than the requested size due to end of str **completes** when upstream completes +### scan -scan -^^^^ -Emit its current value which starts at ``zero`` and then applies the current and next value to the given function +Emit its current value which starts at `zero` and then applies the current and next value to the given function emitting the next current value. Note that this means that scan emits one element downstream before and upstream elements will not be requested until @@ -661,19 +632,19 @@ the second element is required from downstream. **completes** when upstream completes -scanAsync -^^^^^^^^^ -Just like ``scan`` but receiving a function that results in a ``Future`` to the next value. +### scanAsync -**emits** when the ``Future`` resulting from the function scanning the element resolves to the next value +Just like `scan` but receiving a function that results in a `Future` to the next value. + +**emits** when the `Future` resulting from the function scanning the element resolves to the next value **backpressures** when downstream backpressures -**completes** when upstream completes and the last ``Future`` is resolved +**completes** when upstream completes and the last `Future` is resolved -fold -^^^^ -Start with current value ``zero`` and then apply the current and next value to the given function, when upstream +### fold + +Start with current value `zero` and then apply the current and next value to the given function, when upstream complete the current value is emitted downstream. **emits** when upstream completes @@ -682,20 +653,20 @@ complete the current value is emitted downstream. **completes** when upstream completes -foldAsync -^^^^^^^^^ -Just like ``fold`` but receiving a function that results in a ``Future`` to the next value. +### foldAsync -**emits** when upstream completes and the last ``Future`` is resolved +Just like `fold` but receiving a function that results in a `Future` to the next value. + +**emits** when upstream completes and the last `Future` is resolved **backpressures** when downstream backpressures -**completes** when upstream completes and the last ``Future`` is resolved +**completes** when upstream completes and the last `Future` is resolved + +### reduce -reduce -^^^^^^ Start with first element and then apply the current and next value to the given function, when upstream -complete the current value is emitted downstream. Similar to ``fold``. +complete the current value is emitted downstream. Similar to `fold`. **emits** when upstream completes @@ -703,9 +674,9 @@ complete the current value is emitted downstream. Similar to ``fold``. **completes** when upstream completes -drop -^^^^ -Drop ``n`` elements and then pass any subsequent element downstream. +### drop + +Drop `n` elements and then pass any subsequent element downstream. **emits** when the specified number of elements has been dropped already @@ -713,9 +684,9 @@ Drop ``n`` elements and then pass any subsequent element downstream. **completes** when upstream completes -take -^^^^ -Pass ``n`` incoming elements downstream and then complete +### take + +Pass `n` incoming elements downstream and then complete **emits** while the specified number of elements to take has not yet been reached @@ -723,9 +694,8 @@ Pass ``n`` incoming elements downstream and then complete **completes** when the defined number of elements has been taken or upstream completes +### takeWhile -takeWhile -^^^^^^^^^ Pass elements downstream as long as a predicate function return true for the element include the element when the predicate first return false and then complete. @@ -735,8 +705,8 @@ when the predicate first return false and then complete. **completes** when predicate returned false or upstream completes -dropWhile -^^^^^^^^^ +### dropWhile + Drop elements as long as a predicate function return true for the element **emits** when the predicate returned false and for all following stream elements @@ -745,11 +715,11 @@ Drop elements as long as a predicate function return true for the element **completes** when upstream completes -recover -^^^^^^^ +### recover + Allow sending of one last element downstream when a failure has happened upstream. -Throwing an exception inside ``recover`` _will_ be logged on ERROR level automatically. +Throwing an exception inside `recover` _will_ be logged on ERROR level automatically. **emits** when the element is available from the upstream or upstream is failed and pf returns an element @@ -757,11 +727,11 @@ Throwing an exception inside ``recover`` _will_ be logged on ERROR level automat **completes** when upstream completes or upstream failed with exception pf can handle -recoverWith -^^^^^^^^^^^ +### recoverWith + Allow switching to alternative Source when a failure has happened upstream. -Throwing an exception inside ``recoverWith`` _will_ be logged on ERROR level automatically. +Throwing an exception inside `recoverWith` _will_ be logged on ERROR level automatically. **emits** the element is available from the upstream or upstream is failed and pf returns alternative Source @@ -769,12 +739,12 @@ Throwing an exception inside ``recoverWith`` _will_ be logged on ERROR level aut **completes** upstream completes or upstream failed with exception pf can handle -recoverWithRetries -^^^^^^^^^^^^^^^^^^ +### recoverWithRetries + RecoverWithRetries allows to switch to alternative Source on flow failure. It will stay in effect after -a failure has been recovered up to `attempts` number of times so that each time there is a failure -it is fed into the `pf` and a new Source may be materialized. Note that if you pass in 0, this won't -attempt to recover at all. Passing -1 will behave exactly the same as `recoverWith`. +a failure has been recovered up to *attempts* number of times so that each time there is a failure +it is fed into the *pf* and a new Source may be materialized. Note that if you pass in 0, this won't +attempt to recover at all. Passing -1 will behave exactly the same as *recoverWith*. Since the underlying failure signal onError arrives out-of-band, it might jump over existing elements. This stage can recover the failure signal, but not the skipped elements, which will be dropped. @@ -785,23 +755,23 @@ This stage can recover the failure signal, but not the skipped elements, which w **completes** when upstream completes or upstream failed with exception pf can handle -mapError -^^^^^^^^ -While similar to ``recover`` this stage can be used to transform an error signal to a different one *without* logging -it as an error in the process. So in that sense it is NOT exactly equivalent to ``recover(t => throw t2)`` since recover -would log the ``t2`` error. +### mapError + +While similar to `recover` this stage can be used to transform an error signal to a different one *without* logging +it as an error in the process. So in that sense it is NOT exactly equivalent to `recover(t => throw t2)` since recover +would log the `t2` error. Since the underlying failure signal onError arrives out-of-band, it might jump over existing elements. This stage can recover the failure signal, but not the skipped elements, which will be dropped. -Similarily to ``recover`` throwing an exception inside ``mapError`` _will_ be logged on ERROR level automatically. +Similarily to `recover` throwing an exception inside `mapError` _will_ be logged on ERROR level automatically. **emits** when element is available from the upstream or upstream is failed and pf returns an element **backpressures** when downstream backpressures **completes** when upstream completes or upstream failed with exception pf can handle -detach -^^^^^^ +### detach + Detach upstream demand from downstream demand without detaching the stream rates. **emits** when the upstream stage has emitted and there is demand @@ -810,9 +780,8 @@ Detach upstream demand from downstream demand without detaching the stream rates **completes** when upstream completes +### throttle -throttle -^^^^^^^^ Limit the throughput to a specific number of elements per time unit, or a specific total cost per time unit, where a function has to be provided to calculate the individual cost of each element. @@ -822,19 +791,19 @@ a function has to be provided to calculate the individual cost of each element. **completes** when upstream completes -intersperse -^^^^^^^^^^^ -Intersperse stream with provided element similar to ``List.mkString``. It can inject start and end marker elements to stream. +### intersperse -**emits** when upstream emits an element or before with the `start` element if provided +Intersperse stream with provided element similar to `List.mkString`. It can inject start and end marker elements to stream. + +**emits** when upstream emits an element or before with the *start* element if provided **backpressures** when downstream backpressures **completes** when upstream completes -limit -^^^^^ -Limit number of element from upstream to given ``max`` number. +### limit + +Limit number of element from upstream to given `max` number. **emits** when upstream emits and the number of emitted elements has not reached max @@ -842,8 +811,8 @@ Limit number of element from upstream to given ``max`` number. **completes** when upstream completes and the number of emitted elements has not reached max -limitWeighted -^^^^^^^^^^^^^ +### limitWeighted + Ensure stream boundedness by evaluating the cost of incoming elements using a cost function. Evaluated cost of each element defines how many elements will be allowed to travel downstream. @@ -853,11 +822,11 @@ Evaluated cost of each element defines how many elements will be allowed to trav **completes** when upstream completes and the number of emitted elements has not reached max -log -^^^ +### log + Log elements flowing through the stream as well as completion and erroring. By default element and completion signals are logged on debug level, and errors are logged on Error level. -This can be changed by calling ``Attributes.logLevels(...)`` on the given Flow. +This can be changed by calling `Attributes.logLevels(...)` on the given Flow. **emits** when upstream emits @@ -865,9 +834,9 @@ This can be changed by calling ``Attributes.logLevels(...)`` on the given Flow. **completes** when upstream completes -recoverWithRetries -^^^^^^^^^^^^^^^^^^ -Switch to alternative Source on flow failure. It stays in effect after a failure has been recovered up to ``attempts`` +### recoverWithRetries + +Switch to alternative Source on flow failure. It stays in effect after a failure has been recovered up to `attempts` number of times. Each time a failure is fed into the partial function and a new Source may be materialized. **emits** when element is available from the upstream or upstream is failed and element is available from alternative Source @@ -876,28 +845,25 @@ number of times. Each time a failure is fed into the partial function and a new **completes** when upstream completes or upstream failed with exception provided partial function can handle -Flow stages composed of Sinks and Sources ------------------------------------------ +## Flow stages composed of Sinks and Sources -Flow.fromSinkAndSource -^^^^^^^^^^^^^^^^^^^^^^ +### Flow.fromSinkAndSource -Creates a ``Flow`` from a ``Sink`` and a ``Source`` where the Flow's input will be sent to the ``Sink`` -and the ``Flow`` 's output will come from the Source. +Creates a `Flow` from a `Sink` and a `Source` where the Flow's input will be sent to the `Sink` +and the `Flow` 's output will come from the Source. Note that termination events, like completion and cancelation is not automatically propagated through to the "other-side" -of the such-composed Flow. Use ``CoupledTerminationFlow`` if you want to couple termination of both of the ends, +of the such-composed Flow. Use `CoupledTerminationFlow` if you want to couple termination of both of the ends, for example most useful in handling websocket connections. -CoupledTerminationFlow.fromSinkAndSource -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### CoupledTerminationFlow.fromSinkAndSource Allows coupling termination (cancellation, completion, erroring) of Sinks and Sources while creating a Flow them them. -Similar to ``Flow.fromSinkAndSource`` however that API does not connect the completion signals of the wrapped stages. +Similar to `Flow.fromSinkAndSource` however that API does not connect the completion signals of the wrapped stages. -Similar to ``Flow.fromSinkAndSource`` however couples the termination of these two stages. +Similar to `Flow.fromSinkAndSource` however couples the termination of these two stages. -E.g. if the emitted ``Flow`` gets a cancellation, the ``Source`` of course is cancelled, +E.g. if the emitted `Flow` gets a cancellation, the `Source` of course is cancelled, however the Sink will also be completed. The table below illustrates the effects in detail: +=================================================+=============================+=================================+ @@ -916,20 +882,18 @@ however the Sink will also be completed. The table below illustrates the effects | effect: cancels upstream, completes downstream | cause: cancels | effect: receives cancel | +=================================================+=============================+=================================+ -The order in which the `in` and `out` sides receive their respective completion signals is not defined, do not rely on its ordering. +The order in which the *in* and *out* sides receive their respective completion signals is not defined, do not rely on its ordering. -Asynchronous processing stages ------------------------------- +## Asynchronous processing stages These stages encapsulate an asynchronous computation, properly handling backpressure while taking care of the asynchronous operation at the same time (usually handling the completion of a Future). +### mapAsync -mapAsync -^^^^^^^^ -Pass incoming elements to a function that return a ``Future`` result. When the future arrives the result is passed -downstream. Up to ``n`` elements can be processed concurrently, but regardless of their completion time the incoming -order will be kept when results complete. For use cases where order does not mather ``mapAsyncUnordered`` can be used. +Pass incoming elements to a function that return a `Future` result. When the future arrives the result is passed +downstream. Up to `n` elements can be processed concurrently, but regardless of their completion time the incoming +order will be kept when results complete. For use cases where order does not mather `mapAsyncUnordered` can be used. If a Future fails, the stream also fails (unless a different supervision strategy is applied) @@ -939,9 +903,9 @@ If a Future fails, the stream also fails (unless a different supervision strateg **completes** when upstream completes and all futures has been completed and all elements has been emitted -mapAsyncUnordered -^^^^^^^^^^^^^^^^^ -Like ``mapAsync`` but ``Future`` results are passed downstream as they arrive regardless of the order of the elements +### mapAsyncUnordered + +Like `mapAsync` but `Future` results are passed downstream as they arrive regardless of the order of the elements that triggered them. If a Future fails, the stream also fails (unless a different supervision strategy is applied) @@ -952,14 +916,12 @@ If a Future fails, the stream also fails (unless a different supervision strateg **completes** upstream completes and all futures has been completed and all elements has been emitted - -Timer driven stages -------------------- +## Timer driven stages These stages process elements using timers, delaying, dropping or grouping elements for certain time durations. -takeWithin -^^^^^^^^^^ +### takeWithin + Pass elements downstream within a timeout and then complete. **emits** when an upstream element arrives @@ -968,9 +930,8 @@ Pass elements downstream within a timeout and then complete. **completes** upstream completes or timer fires +### dropWithin -dropWithin -^^^^^^^^^^ Drop elements until a timeout has fired **emits** after the timer fired and a new upstream element arrives @@ -979,8 +940,8 @@ Drop elements until a timeout has fired **completes** upstream completes -groupedWithin -^^^^^^^^^^^^^ +### groupedWithin + Chunk up this stream into groups of elements received within a time window, or limited by the number of the elements, whatever happens first. Empty groups will not be emitted if no elements are received from upstream. The last group before end-of-stream will contain the buffered elements since the previously emitted group. @@ -988,7 +949,7 @@ The last group before end-of-stream will contain the buffered elements since the **emits** when the configured time elapses since the last group has been emitted, but not if no elements has been grouped (i.e: no empty groups), or when limit has been reached. -**backpressures** downstream backpressures, and there are `n+1` buffered elements +**backpressures** downstream backpressures, and there are *n+1* buffered elements **completes** when upstream completes @@ -1001,13 +962,12 @@ The last group before end-of-stream will contain the buffered elements since the **emits** when the configured time elapses since the last group has been emitted, but not if no elements has been grouped (i.e: no empty groups), or when weight limit has been reached. -**backpressures** downstream backpressures, and buffered group (+ pending element) weighs more than `maxWeight` +**backpressures** downstream backpressures, and buffered group (+ pending element) weighs more than *maxWeight* **completes** when upstream completes +### initialDelay -initialDelay -^^^^^^^^^^^^ Delay the initial element by a user specified duration from stream materialization. **emits** upstream emits an element if the initial delay already elapsed @@ -1016,31 +976,26 @@ Delay the initial element by a user specified duration from stream materializati **completes** when upstream completes +### delay -delay -^^^^^ Delay every element passed through with a specific duration. **emits** there is a pending element in the buffer and configured time for this element elapsed -**backpressures** differs, depends on ``OverflowStrategy`` set +**backpressures** differs, depends on `OverflowStrategy` set **completes** when upstream completes and buffered elements has been drained - - -.. _detached-stages-overview_scala: - -Backpressure aware stages -------------------------- + +## Backpressure aware stages These stages are aware of the backpressure provided by their downstreams and able to adapt their behavior to that signal. -conflate -^^^^^^^^ +### conflate + Allow for a slower downstream by passing incoming elements and a summary into an aggregate function as long as there is backpressure. The summary value must be of the same type as the incoming elements, for example the sum or -average of incoming numbers, if aggregation should lead to a different type ``conflateWithSeed`` can be used: +average of incoming numbers, if aggregation should lead to a different type `conflateWithSeed` can be used: **emits** when downstream stops backpressuring and there is a conflated element available @@ -1048,10 +1003,10 @@ average of incoming numbers, if aggregation should lead to a different type ``co **completes** when upstream completes -conflateWithSeed -^^^^^^^^^^^^^^^^ +### conflateWithSeed + Allow for a slower downstream by passing incoming elements and a summary into an aggregate function as long as there -is backpressure. When backpressure starts or there is no backpressure element is passed into a ``seed`` function to +is backpressure. When backpressure starts or there is no backpressure element is passed into a `seed` function to transform it to the summary type. **emits** when downstream stops backpressuring and there is a conflated element available @@ -1060,13 +1015,13 @@ transform it to the summary type. **completes** when upstream completes -batch -^^^^^ +### batch + Allow for a slower downstream by passing incoming elements and a summary into an aggregate function as long as there is backpressure and a maximum number of batched elements is not yet reached. When the maximum number is reached and downstream still backpressures batch will also backpressure. -When backpressure starts or there is no backpressure element is passed into a ``seed`` function to transform it +When backpressure starts or there is no backpressure element is passed into a `seed` function to transform it to the summary type. Will eagerly pull elements, this behavior may result in a single pending (i.e. buffered) element which cannot be @@ -1078,12 +1033,11 @@ aggregated to the batched value. **completes** when upstream completes and a "possibly pending" element was drained +### batchWeighted -batchWeighted -^^^^^^^^^^^^^ Allow for a slower downstream by passing incoming elements and a summary into an aggregate function as long as there is backpressure and a maximum weight batched elements is not yet reached. The weight of each element is determined by -applying ``costFn``. When the maximum total weight is reached and downstream still backpressures batch will also +applying `costFn`. When the maximum total weight is reached and downstream still backpressures batch will also backpressure. Will eagerly pull elements, this behavior may result in a single pending (i.e. buffered) element which cannot be @@ -1095,10 +1049,10 @@ aggregated to the batched value. **completes** upstream completes and a "possibly pending" element was drained -expand -^^^^^^ -Allow for a faster downstream by expanding the last incoming element to an ``Iterator``. For example -``Iterator.continually(element)`` to keep repeating the last incoming element. +### expand + +Allow for a faster downstream by expanding the last incoming element to an `Iterator`. For example +`Iterator.continually(element)` to keep repeating the last incoming element. **emits** when downstream stops backpressuring @@ -1106,9 +1060,9 @@ Allow for a faster downstream by expanding the last incoming element to an ``Ite **completes** when upstream completes -buffer (Backpressure) -^^^^^^^^^^^^^^^^^^^^^ -Allow for a temporarily faster upstream events by buffering ``size`` elements. When the buffer is full backpressure +### buffer (Backpressure) + +Allow for a temporarily faster upstream events by buffering `size` elements. When the buffer is full backpressure is applied. **emits** when downstream stops backpressuring and there is a pending element in the buffer @@ -1117,15 +1071,15 @@ is applied. **completes** when upstream completes and buffered elements has been drained -buffer (Drop) -^^^^^^^^^^^^^ -Allow for a temporarily faster upstream events by buffering ``size`` elements. When the buffer is full elements are -dropped according to the specified ``OverflowStrategy``: +### buffer (Drop) -* ``dropHead`` drops the oldest element in the buffer to make space for the new element -* ``dropTail`` drops the youngest element in the buffer to make space for the new element -* ``dropBuffer`` drops the entire buffer and buffers the new element -* ``dropNew`` drops the new element +Allow for a temporarily faster upstream events by buffering `size` elements. When the buffer is full elements are +dropped according to the specified `OverflowStrategy`: + + * `dropHead` drops the oldest element in the buffer to make space for the new element + * `dropTail` drops the youngest element in the buffer to make space for the new element + * `dropBuffer` drops the entire buffer and buffers the new element + * `dropNew` drops the new element **emits** when downstream stops backpressuring and there is a pending element in the buffer @@ -1133,10 +1087,10 @@ dropped according to the specified ``OverflowStrategy``: **completes** upstream completes and buffered elements has been drained -buffer (Fail) -^^^^^^^^^^^^^ -Allow for a temporarily faster upstream events by buffering ``size`` elements. When the buffer is full the stage fails -the flow with a ``BufferOverflowException``. +### buffer (Fail) + +Allow for a temporarily faster upstream events by buffering `size` elements. When the buffer is full the stage fails +the flow with a `BufferOverflowException`. **emits** when downstream stops backpressuring and there is a pending element in the buffer @@ -1144,16 +1098,14 @@ the flow with a ``BufferOverflowException``. **completes** when upstream completes and buffered elements has been drained - -Nesting and flattening stages ------------------------------ +## Nesting and flattening stages These stages either take a stream and turn it into a stream of streams (nesting) or they take a stream that contains nested streams and turn them into a stream of elements instead (flattening). -prefixAndTail -^^^^^^^^^^^^^ -Take up to `n` elements from the stream (less than `n` only if the upstream completes before emitting `n` elements) +### prefixAndTail + +Take up to *n* elements from the stream (less than *n* only if the upstream completes before emitting *n* elements) and returns a pair containing a strict sequence of the taken element and a stream representing the remaining elements. **emits** when the configured number of prefix elements are available. Emits this prefix, and the rest as a substream @@ -1162,9 +1114,8 @@ and returns a pair containing a strict sequence of the taken element and a strea **completes** when prefix elements has been consumed and substream has been consumed +### groupBy -groupBy -^^^^^^^ Demultiplex the incoming stream into separate output streams. **emits** an element for which the grouping function returns a group that has not yet been created. Emits the new group @@ -1172,9 +1123,9 @@ there is an element pending for a group whose substream backpressures **completes** when upstream completes (Until the end of stream it is not possible to know whether new substreams will be needed or not) -splitWhen -^^^^^^^^^ -Split off elements into a new substream whenever a predicate function return ``true``. +### splitWhen + +Split off elements into a new substream whenever a predicate function return `true`. **emits** an element for which the provided predicate is true, opening and emitting a new substream for subsequent elements @@ -1182,9 +1133,9 @@ Split off elements into a new substream whenever a predicate function return ``t **completes** when upstream completes (Until the end of stream it is not possible to know whether new substreams will be needed or not) -splitAfter -^^^^^^^^^^ -End the current substream whenever a predicate returns ``true``, starting a new substream for the next element. +### splitAfter + +End the current substream whenever a predicate returns `true`, starting a new substream for the next element. **emits** when an element passes through. When the provided predicate is true it emits the element * and opens a new substream for subsequent element @@ -1192,9 +1143,9 @@ End the current substream whenever a predicate returns ``true``, starting a new **completes** when upstream completes (Until the end of stream it is not possible to know whether new substreams will be needed or not) -flatMapConcat -^^^^^^^^^^^^^ -Transform each input element into a ``Source`` whose elements are then flattened into the output stream through +### flatMapConcat + +Transform each input element into a `Source` whose elements are then flattened into the output stream through concatenation. This means each source is fully consumed before consumption of the next source starts. **emits** when the current consumed substream has an element available @@ -1203,10 +1154,9 @@ concatenation. This means each source is fully consumed before consumption of th **completes** when upstream completes and all consumed substreams complete +### flatMapMerge -flatMapMerge -^^^^^^^^^^^^ -Transform each input element into a ``Source`` whose elements are then flattened into the output stream through +Transform each input element into a `Source` whose elements are then flattened into the output stream through merging. The maximum number of merged sources has to be specified. **emits** when one of the currently consumed substreams has an element available @@ -1215,16 +1165,14 @@ merging. The maximum number of merged sources has to be specified. **completes** when upstream completes and all consumed substreams complete - -Time aware stages ------------------ +## Time aware stages Those stages operate taking time into consideration. -initialTimeout -^^^^^^^^^^^^^^ +### initialTimeout + If the first element has not passed through this stage before the provided timeout, the stream is failed -with a ``TimeoutException``. +with a `TimeoutException`. **emits** when upstream emits an element @@ -1234,10 +1182,10 @@ with a ``TimeoutException``. **cancels** when downstream cancels -completionTimeout -^^^^^^^^^^^^^^^^^ +### completionTimeout + If the completion of the stream does not happen until the provided timeout, the stream is failed -with a ``TimeoutException``. +with a `TimeoutException`. **emits** when upstream emits an element @@ -1247,10 +1195,10 @@ with a ``TimeoutException``. **cancels** when downstream cancels -idleTimeout -^^^^^^^^^^^ +### idleTimeout + If the time between two processed elements exceeds the provided timeout, the stream is failed -with a ``TimeoutException``. The timeout is checked periodically, so the resolution of the +with a `TimeoutException`. The timeout is checked periodically, so the resolution of the check is one period (equals to timeout value). **emits** when upstream emits an element @@ -1261,10 +1209,10 @@ check is one period (equals to timeout value). **cancels** when downstream cancels -backpressureTimeout -^^^^^^^^^^^^^^^^^^^ +### backpressureTimeout + If the time between the emission of an element and the following downstream demand exceeds the provided timeout, -the stream is failed with a ``TimeoutException``. The timeout is checked periodically, so the resolution of the +the stream is failed with a `TimeoutException`. The timeout is checked periodically, so the resolution of the check is one period (equals to timeout value). **emits** when upstream emits an element @@ -1275,8 +1223,8 @@ check is one period (equals to timeout value). **cancels** when downstream cancels -keepAlive -^^^^^^^^^ +### keepAlive + Injects additional (configured) elements if upstream does not emit for a configured amount of time. **emits** when upstream emits an element or if the upstream was idle for the configured period @@ -1287,8 +1235,8 @@ Injects additional (configured) elements if upstream does not emit for a configu **cancels** when downstream cancels -initialDelay -^^^^^^^^^^^^ +### initialDelay + Delays the initial element by the specified duration. **emits** when upstream emits an element if the initial delay is already elapsed @@ -1299,25 +1247,23 @@ Delays the initial element by the specified duration. **cancels** when downstream cancels - -Fan-in stages -------------- +## Fan-in stages These stages take multiple streams as their input and provide a single output combining the elements from all of the inputs in different ways. -merge -^^^^^ +### merge + Merge multiple sources. Picks elements randomly if all sources has elements ready. **emits** when one of the inputs has an element available **backpressures** when downstream backpressures -**completes** when all upstreams complete (This behavior is changeable to completing when any upstream completes by setting ``eagerComplete=true``.) +**completes** when all upstreams complete (This behavior is changeable to completing when any upstream completes by setting `eagerComplete=true`.) + +### mergeSorted -mergeSorted -^^^^^^^^^^^ Merge multiple sources. Waits for one element to be ready from each input stream and emits the smallest element. @@ -1327,18 +1273,18 @@ smallest element. **completes** when all upstreams complete -mergePreferred -^^^^^^^^^^^^^^ +### mergePreferred + Merge multiple sources. Prefer one source if all sources has elements ready. **emits** when one of the inputs has an element available, preferring a defined input if multiple have elements available **backpressures** when downstream backpressures -**completes** when all upstreams complete (This behavior is changeable to completing when any upstream completes by setting ``eagerComplete=true``.) +**completes** when all upstreams complete (This behavior is changeable to completing when any upstream completes by setting `eagerComplete=true`.) + +### zip -zip -^^^ Combines elements from each of multiple sources into tuples and passes the tuples downstream. **emits** when all of the inputs have an element available @@ -1347,9 +1293,9 @@ Combines elements from each of multiple sources into tuples and passes the tuple **completes** when any upstream completes -zipWith -^^^^^^^ -Combines elements from multiple sources through a ``combine`` function and passes the +### zipWith + +Combines elements from multiple sources through a `combine` function and passes the returned value downstream. **emits** when all of the inputs have an element available @@ -1358,8 +1304,8 @@ returned value downstream. **completes** when any upstream completes -zipWithIndex -^^^^^^^^^^^^ +### zipWithIndex + Zips elements of current flow with its indices. **emits** upstream emits an element and is paired with their index @@ -1368,8 +1314,8 @@ Zips elements of current flow with its indices. **completes** when upstream completes -concat -^^^^^^ +### concat + After completion of the original upstream the elements of the given source will be emitted. **emits** when the current stream has an element available; if the current input completes, it tries the next one @@ -1378,8 +1324,8 @@ After completion of the original upstream the elements of the given source will **completes** when all upstreams complete -++ -^^ +### ++ + Just a shorthand for concat **emits** when the current stream has an element available; if the current input completes, it tries the next one @@ -1388,11 +1334,11 @@ Just a shorthand for concat **completes** when all upstreams complete -prepend -^^^^^^^ +### prepend + Prepends the given source to the flow, consuming it until completion before the original source is consumed. -If materialized values needs to be collected ``prependMat`` is available. +If materialized values needs to be collected `prependMat` is available. **emits** when the given stream has an element available; if the given input completes, it tries the current one @@ -1400,8 +1346,8 @@ If materialized values needs to be collected ``prependMat`` is available. **completes** when all upstreams complete -orElse -^^^^^^ +### orElse + If the primary source completes without emitting any elements, the elements from the secondary source are emitted. If the primary source emits any elements the secondary source is cancelled. @@ -1418,8 +1364,8 @@ is available from the second stream **completes** the primary stream completes after emitting at least one element, when the primary stream completes without emitting and the secondary stream already has completed or when the secondary stream completes -interleave -^^^^^^^^^^ +### interleave + Emits a specifiable number of elements from the original source, then from the provided source and repeats. If one source completes the rest of the other stream will be emitted. @@ -1429,14 +1375,13 @@ source completes the rest of the other stream will be emitted. **completes** when both upstreams have completed -Fan-out stages --------------- +## Fan-out stages These have one input and multiple outputs. They might route the elements between different outputs, or emit elements on multiple outputs at the same time. -unzip -^^^^^ +### unzip + Takes a stream of two element tuples and unzips the two elements ino two different downstreams. **emits** when all of the outputs stops backpressuring and there is an input element available @@ -1445,8 +1390,8 @@ Takes a stream of two element tuples and unzips the two elements ino two differe **completes** when upstream completes -unzipWith -^^^^^^^^^ +### unzipWith + Splits each element of input into multiple downstreams using a function **emits** when all of the outputs stops backpressuring and there is an input element available @@ -1455,9 +1400,9 @@ Splits each element of input into multiple downstreams using a function **completes** when upstream completes -broadcast -^^^^^^^^^ -Emit each incoming element each of ``n`` outputs. +### broadcast + +Emit each incoming element each of `n` outputs. **emits** when all of the outputs stops backpressuring and there is an input element available @@ -1465,8 +1410,8 @@ Emit each incoming element each of ``n`` outputs. **completes** when upstream completes -balance -^^^^^^^ +### balance + Fan-out the stream to several streams. Each upstream element is emitted to the first available downstream consumer. **emits** when any of the outputs stops backpressuring; emits the element to the first available output @@ -1475,8 +1420,8 @@ Fan-out the stream to several streams. Each upstream element is emitted to the f **completes** when upstream completes -partition -^^^^^^^^^ +### partition + Fan-out the stream to several streams. Each upstream element is emitted to one downstream consumer according to the partitioner function applied to the element. @@ -1486,13 +1431,11 @@ partitioner function applied to the element. **completes** when upstream completes and no output is pending +## Watching status stages -Watching status stages ----------------------- +### watchTermination -watchTermination -^^^^^^^^^^^^^^^^ -Materializes to a ``Future`` that will be completed with Done or failed depending whether the upstream of the stage has been completed or failed. +Materializes to a `Future` that will be completed with Done or failed depending whether the upstream of the stage has been completed or failed. The stage otherwise passes through elements unchanged. **emits** when input has an element available @@ -1501,15 +1444,14 @@ The stage otherwise passes through elements unchanged. **completes** when upstream completes -monitor -^^^^^^^ -Materializes to a ``FlowMonitor`` that monitors messages flowing through or completion of the stage. The stage otherwise -passes through elements unchanged. Note that the ``FlowMonitor`` inserts a memory barrier every time it processes an +### monitor + +Materializes to a `FlowMonitor` that monitors messages flowing through or completion of the stage. The stage otherwise +passes through elements unchanged. Note that the `FlowMonitor` inserts a memory barrier every time it processes an event, and may therefore affect performance. **emits** when upstream emits an element **backpressures** when downstream **backpressures** -**completes** when upstream completes - +**completes** when upstream completes \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-composition.md b/akka-docs/src/main/paradox/scala/stream/stream-composition.md index 7565bc970e..0cd55f4373 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-composition.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-composition.md @@ -1,100 +1,93 @@ -.. _composition-scala: - -Modularity, Composition and Hierarchy -===================================== +# Modularity, Composition and Hierarchy Akka Streams provide a uniform model of stream processing graphs, which allows flexible composition of reusable components. In this chapter we show how these look like from the conceptual and API perspective, demonstrating the modularity aspects of the library. -Basics of composition and modularity ------------------------------------- +## Basics of composition and modularity Every processing stage used in Akka Streams can be imagined as a "box" with input and output ports where elements to -be processed arrive and leave the stage. In this view, a :class:`Source` is nothing else than a "box" with a single -output port, or, a :class:`BidiFlow` is a "box" with exactly two input and two output ports. In the figure below +be processed arrive and leave the stage. In this view, a `Source` is nothing else than a "box" with a single +output port, or, a `BidiFlow` is a "box" with exactly two input and two output ports. In the figure below we illustrate the most common used stages viewed as "boxes". | -.. image:: ../../images/compose_shapes.png - :align: center +![compose_shapes.png](../../images/compose_shapes.png) | -The *linear* stages are :class:`Source`, :class:`Sink` -and :class:`Flow`, as these can be used to compose strict chains of processing stages. +The *linear* stages are `Source`, `Sink` +and `Flow`, as these can be used to compose strict chains of processing stages. Fan-in and Fan-out stages have usually multiple input or multiple output ports, therefore they allow to build -more complex graph layouts, not just chains. :class:`BidiFlow` stages are usually useful in IO related tasks, where -there are input and output channels to be handled. Due to the specific shape of :class:`BidiFlow` it is easy to -stack them on top of each other to build a layered protocol for example. The ``TLS`` support in Akka is for example -implemented as a :class:`BidiFlow`. +more complex graph layouts, not just chains. `BidiFlow` stages are usually useful in IO related tasks, where +there are input and output channels to be handled. Due to the specific shape of `BidiFlow` it is easy to +stack them on top of each other to build a layered protocol for example. The `TLS` support in Akka is for example +implemented as a `BidiFlow`. These reusable components already allow the creation of complex processing networks. What we have seen so far does not implement modularity though. It is desirable for example to package up a larger graph entity into a reusable component which hides its internals only exposing the ports that are meant to the users of the module -to interact with. One good example is the ``Http`` server component, which is encoded internally as a -:class:`BidiFlow` which interfaces with the client TCP connection using an input-output port pair accepting and sending -:class:`ByteString` s, while its upper ports emit and receive :class:`HttpRequest` and :class:`HttpResponse` instances. +to interact with. One good example is the `Http` server component, which is encoded internally as a +`BidiFlow` which interfaces with the client TCP connection using an input-output port pair accepting and sending +`ByteString` s, while its upper ports emit and receive `HttpRequest` and `HttpResponse` instances. The following figure demonstrates various composite stages, that contain various other type of stages internally, but -hiding them behind a *shape* that looks like a :class:`Source`, :class:`Flow`, etc. +hiding them behind a *shape* that looks like a `Source`, `Flow`, etc. | -.. image:: ../../images/compose_composites.png - :align: center +![compose_composites.png](../../images/compose_composites.png) | -One interesting example above is a :class:`Flow` which is composed of a disconnected :class:`Sink` and :class:`Source`. -This can be achieved by using the ``fromSinkAndSource()`` constructor method on :class:`Flow` which takes the two parts as +One interesting example above is a `Flow` which is composed of a disconnected `Sink` and `Source`. +This can be achieved by using the `fromSinkAndSource()` constructor method on `Flow` which takes the two parts as parameters. -Please note that when combining a :class:`Flow` using that method, the termination signals are not carried -"through" as the :class:`Sink` and :class:`Source` are assumed to be fully independent. If however you want to construct -a :class:`Flow` like this but need the termination events to trigger "the other side" of the composite flow, you can use -``CoupledTerminationFlow.fromSinkAndSource`` which does just that. For example the cancelation of the composite flows -source-side will then lead to completion of its sink-side. Read :class:`CoupledTerminationFlow`'s scaladoc for a +Please note that when combining a `Flow` using that method, the termination signals are not carried +"through" as the `Sink` and `Source` are assumed to be fully independent. If however you want to construct +a `Flow` like this but need the termination events to trigger "the other side" of the composite flow, you can use +`CoupledTerminationFlow.fromSinkAndSource` which does just that. For example the cancelation of the composite flows +source-side will then lead to completion of its sink-side. Read `CoupledTerminationFlow`'s scaladoc for a detailed explanation how this works. -The example :class:`BidiFlow` demonstrates that internally a module can be of arbitrary complexity, and the exposed +The example `BidiFlow` demonstrates that internally a module can be of arbitrary complexity, and the exposed ports can be wired in flexible ways. The only constraint is that all the ports of enclosed modules must be either connected to each other, or exposed as interface ports, and the number of such ports needs to match the requirement -of the shape, for example a :class:`Source` allows only one exposed output port, the rest of the internal ports must +of the shape, for example a `Source` allows only one exposed output port, the rest of the internal ports must be properly connected. -These mechanics allow arbitrary nesting of modules. For example the following figure demonstrates a :class:`RunnableGraph` -that is built from a composite :class:`Source` and a composite :class:`Sink` (which in turn contains a composite -:class:`Flow`). +These mechanics allow arbitrary nesting of modules. For example the following figure demonstrates a `RunnableGraph` +that is built from a composite `Source` and a composite `Sink` (which in turn contains a composite +`Flow`). | -.. image:: ../../images/compose_nested_flow.png - :align: center +![compose_nested_flow.png](../../images/compose_nested_flow.png) | -The above diagram contains one more shape that we have not seen yet, which is called :class:`RunnableGraph`. It turns +The above diagram contains one more shape that we have not seen yet, which is called `RunnableGraph`. It turns out, that if we wire all exposed ports together, so that no more open ports remain, we get a module that is *closed*. -This is what the :class:`RunnableGraph` class represents. This is the shape that a :class:`Materializer` can take -and turn into a network of running entities that perform the task described. In fact, a :class:`RunnableGraph` is a +This is what the `RunnableGraph` class represents. This is the shape that a `Materializer` can take +and turn into a network of running entities that perform the task described. In fact, a `RunnableGraph` is a module itself, and (maybe somewhat surprisingly) it can be used as part of larger graphs. It is rarely useful to embed a closed graph shape in a larger graph (since it becomes an isolated island as there are no open port for communication with the rest of the graph), but this demonstrates the uniform underlying model. If we try to build a code snippet that corresponds to the above diagram, our first try might look like this: -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#non-nested-flow +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #non-nested-flow } It is clear however that there is no nesting present in our first attempt, since the library cannot figure out where we intended to put composite module boundaries, it is our responsibility to do that. If we are using the -DSL provided by the :class:`Flow`, :class:`Source`, :class:`Sink` classes then nesting can be achieved by calling one of the -methods ``withAttributes()`` or ``named()`` (where the latter is just a shorthand for adding a name attribute). +DSL provided by the `Flow`, `Source`, `Sink` classes then nesting can be achieved by calling one of the +methods `withAttributes()` or `named()` (where the latter is just a shorthand for adding a name attribute). The following code demonstrates how to achieve the desired nesting: -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#nested-flow +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #nested-flow } Once we have hidden the internals of our components, they act like any other built-in component of similar shape. If we hide some of the internals of our composites, the result looks just like if any other predefine component has been @@ -102,22 +95,20 @@ used: | -.. image:: ../../images/compose_nested_flow_opaque.png - :align: center +![compose_nested_flow_opaque.png](../../images/compose_nested_flow_opaque.png) | If we look at usage of built-in components, and our custom components, there is no difference in usage as the code snippet below demonstrates. -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#reuse +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #reuse } -Composing complex systems -------------------------- +## Composing complex systems In the previous section we explored the possibility of composition, and hierarchy, but we stayed away from non-linear, generalized graph components. There is nothing in Akka Streams though that enforces that stream processing layouts -can only be linear. The DSL for :class:`Source` and friends is optimized for creating such linear chains, as they are +can only be linear. The DSL for `Source` and friends is optimized for creating such linear chains, as they are the most common in practice. There is a more advanced DSL for building complex graphs, that can be used if more flexibility is needed. We will see that the difference between the two DSLs is only on the surface: the concepts they operate on are uniform across all DSLs and fit together nicely. @@ -126,193 +117,190 @@ As a first example, let's look at a more complex layout: | -.. image:: ../../images/compose_graph.png - :align: center +![compose_graph.png](../../images/compose_graph.png) | -The diagram shows a :class:`RunnableGraph` (remember, if there are no unwired ports, the graph is closed, and therefore +The diagram shows a `RunnableGraph` (remember, if there are no unwired ports, the graph is closed, and therefore can be materialized) that encapsulates a non-trivial stream processing network. It contains fan-in, fan-out stages, -directed and non-directed cycles. The ``runnable()`` method of the :class:`GraphDSL` object allows the creation of a +directed and non-directed cycles. The `runnable()` method of the `GraphDSL` object allows the creation of a general, closed, and runnable graph. For example the network on the diagram can be realized like this: -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#complex-graph +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #complex-graph } In the code above we used the implicit port numbering feature (to make the graph more readable and similar to the diagram) -and we imported :class:`Source` s, :class:`Sink` s and :class:`Flow` s explicitly. It is possible to refer to the ports -explicitly, and it is not necessary to import our linear stages via ``add()``, so another version might look like this: +and we imported `Source` s, `Sink` s and `Flow` s explicitly. It is possible to refer to the ports +explicitly, and it is not necessary to import our linear stages via `add()`, so another version might look like this: -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#complex-graph-alt +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #complex-graph-alt } | Similar to the case in the first section, so far we have not considered modularity. We created a complex graph, but the layout is flat, not modularized. We will modify our example, and create a reusable component with the graph DSL. -The way to do it is to use the ``create()`` factory method on :class:`GraphDSL`. If we remove the sources and sinks +The way to do it is to use the `create()` factory method on `GraphDSL`. If we remove the sources and sinks from the previous example, what remains is a partial graph: | -.. image:: ../../images/compose_graph_partial.png - :align: center +![compose_graph_partial.png](../../images/compose_graph_partial.png) | We can recreate a similar graph in code, using the DSL in a similar way than before: -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#partial-graph +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #partial-graph } -The only new addition is the return value of the builder block, which is a :class:`Shape`. All graphs (including -:class:`Source`, :class:`BidiFlow`, etc) have a shape, which encodes the *typed* ports of the module. In our example -there is exactly one input and output port left, so we can declare it to have a :class:`FlowShape` by returning an -instance of it. While it is possible to create new :class:`Shape` types, it is usually recommended to use one of the +The only new addition is the return value of the builder block, which is a `Shape`. All graphs (including +`Source`, `BidiFlow`, etc) have a shape, which encodes the *typed* ports of the module. In our example +there is exactly one input and output port left, so we can declare it to have a `FlowShape` by returning an +instance of it. While it is possible to create new `Shape` types, it is usually recommended to use one of the matching built-in ones. -The resulting graph is already a properly wrapped module, so there is no need to call `named()` to encapsulate the graph, but +The resulting graph is already a properly wrapped module, so there is no need to call *named()* to encapsulate the graph, but it is a good practice to give names to modules to help debugging. | -.. image:: ../../images/compose_graph_shape.png - :align: center +![compose_graph_shape.png](../../images/compose_graph_shape.png) | Since our partial graph has the right shape, it can be already used in the simpler, linear DSL: -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#partial-use +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #partial-use } -It is not possible to use it as a :class:`Flow` yet, though (i.e. we cannot call ``.filter()`` on it), but :class:`Flow` -has a ``fromGraph()`` method that just adds the DSL to a :class:`FlowShape`. There are similar methods on :class:`Source`, -:class:`Sink` and :class:`BidiShape`, so it is easy to get back to the simpler DSL if a graph has the right shape. +It is not possible to use it as a `Flow` yet, though (i.e. we cannot call `.filter()` on it), but `Flow` +has a `fromGraph()` method that just adds the DSL to a `FlowShape`. There are similar methods on `Source`, +`Sink` and `BidiShape`, so it is easy to get back to the simpler DSL if a graph has the right shape. For convenience, it is also possible to skip the partial graph creation, and use one of the convenience creator methods. To demonstrate this, we will create the following graph: | -.. image:: ../../images/compose_graph_flow.png - :align: center +![compose_graph_flow.png](../../images/compose_graph_flow.png) | The code version of the above closed graph might look like this: -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#partial-flow-dsl +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #partial-flow-dsl } -.. note:: - All graph builder sections check if the resulting graph has all ports connected except the exposed ones and will - throw an exception if this is violated. +@@@ note -We are still in debt of demonstrating that :class:`RunnableGraph` is a component just like any other, which can +All graph builder sections check if the resulting graph has all ports connected except the exposed ones and will +throw an exception if this is violated. + +@@@ + +We are still in debt of demonstrating that `RunnableGraph` is a component just like any other, which can be embedded in graphs. In the following snippet we embed one closed graph in another: -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#embed-closed +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #embed-closed } -The type of the imported module indicates that the imported module has a :class:`ClosedShape`, and so we are not +The type of the imported module indicates that the imported module has a `ClosedShape`, and so we are not able to wire it to anything else inside the enclosing closed graph. Nevertheless, this "island" is embedded properly, and will be materialized just like any other module that is part of the graph. As we have demonstrated, the two DSLs are fully interoperable, as they encode a similar nested structure of "boxes with ports", it is only the DSLs that differ to be as much powerful as possible on the given abstraction level. It is possible -to embed complex graphs in the fluid DSL, and it is just as easy to import and embed a :class:`Flow`, etc, in a larger, +to embed complex graphs in the fluid DSL, and it is just as easy to import and embed a `Flow`, etc, in a larger, complex structure. -We have also seen, that every module has a :class:`Shape` (for example a :class:`Sink` has a :class:`SinkShape`) +We have also seen, that every module has a `Shape` (for example a `Sink` has a `SinkShape`) independently which DSL was used to create it. This uniform representation enables the rich composability of various stream processing entities in a convenient way. -Materialized values -------------------- +## Materialized values -After realizing that :class:`RunnableGraph` is nothing more than a module with no unused ports (it is an island), it becomes clear that +After realizing that `RunnableGraph` is nothing more than a module with no unused ports (it is an island), it becomes clear that after materialization the only way to communicate with the running stream processing logic is via some side-channel. -This side channel is represented as a *materialized value*. The situation is similar to :class:`Actor` s, where the -:class:`Props` instance describes the actor logic, but it is the call to ``actorOf()`` that creates an actually running -actor, and returns an :class:`ActorRef` that can be used to communicate with the running actor itself. Since the -:class:`Props` can be reused, each call will return a different reference. +This side channel is represented as a *materialized value*. The situation is similar to `Actor` s, where the +`Props` instance describes the actor logic, but it is the call to `actorOf()` that creates an actually running +actor, and returns an `ActorRef` that can be used to communicate with the running actor itself. Since the +`Props` can be reused, each call will return a different reference. When it comes to streams, each materialization creates a new running network corresponding to the blueprint that was -encoded in the provided :class:`RunnableGraph`. To be able to interact with the running network, each materialization +encoded in the provided `RunnableGraph`. To be able to interact with the running network, each materialization needs to return a different object that provides the necessary interaction capabilities. In other words, the -:class:`RunnableGraph` can be seen as a factory, which creates: +`RunnableGraph` can be seen as a factory, which creates: - * a network of running processing entities, inaccessible from the outside - * a materialized value, optionally providing a controlled interaction capability with the network +> + * a network of running processing entities, inaccessible from the outside + * a materialized value, optionally providing a controlled interaction capability with the network Unlike actors though, each of the processing stages might provide a materialized value, so when we compose multiple stages or modules, we need to combine the materialized value as well (there are default rules which make this easier, -for example `to()` and `via()` takes care of the most common case of taking the materialized value to the left. -See :ref:`flow-combine-mat-scala` for details). We demonstrate how this works by a code example and a diagram which +for example *to()* and *via()* takes care of the most common case of taking the materialized value to the left. +See @ref:[Combining materialized values](stream-flows-and-basics.md#flow-combine-mat-scala) for details). We demonstrate how this works by a code example and a diagram which graphically demonstrates what is happening. The propagation of the individual materialized values from the enclosed modules towards the top will look like this: | -.. image:: ../../images/compose_mat.png - :align: center +![compose_mat.png](../../images/compose_mat.png) | -To implement the above, first, we create a composite :class:`Source`, where the enclosed :class:`Source` have a -materialized type of :class:`Promise[[Option[Int]]`. By using the combiner function ``Keep.left``, the resulting materialized +To implement the above, first, we create a composite `Source`, where the enclosed `Source` have a +materialized type of `Promise[[Option[Int]]`. By using the combiner function `Keep.left`, the resulting materialized type is of the nested module (indicated by the color *red* on the diagram): -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#mat-combine-1 +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #mat-combine-1 } -Next, we create a composite :class:`Flow` from two smaller components. Here, the second enclosed :class:`Flow` has a -materialized type of :class:`Future[OutgoingConnection]`, and we propagate this to the parent by using ``Keep.right`` +Next, we create a composite `Flow` from two smaller components. Here, the second enclosed `Flow` has a +materialized type of `Future[OutgoingConnection]`, and we propagate this to the parent by using `Keep.right` as the combiner function (indicated by the color *yellow* on the diagram): -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#mat-combine-2 +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #mat-combine-2 } -As a third step, we create a composite :class:`Sink`, using our ``nestedFlow`` as a building block. In this snippet, both -the enclosed :class:`Flow` and the folding :class:`Sink` has a materialized value that is interesting for us, so -we use ``Keep.both`` to get a :class:`Pair` of them as the materialized type of ``nestedSink`` (indicated by the color +As a third step, we create a composite `Sink`, using our `nestedFlow` as a building block. In this snippet, both +the enclosed `Flow` and the folding `Sink` has a materialized value that is interesting for us, so +we use `Keep.both` to get a `Pair` of them as the materialized type of `nestedSink` (indicated by the color *blue* on the diagram) -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#mat-combine-3 +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #mat-combine-3 } -As the last example, we wire together ``nestedSource`` and ``nestedSink`` and we use a custom combiner function to -create a yet another materialized type of the resulting :class:`RunnableGraph`. This combiner function just ignores -the :class:`Future[Sink]` part, and wraps the other two values in a custom case class :class:`MyClass` +As the last example, we wire together `nestedSource` and `nestedSink` and we use a custom combiner function to +create a yet another materialized type of the resulting `RunnableGraph`. This combiner function just ignores +the `Future[Sink]` part, and wraps the other two values in a custom case class `MyClass` (indicated by color *purple* on the diagram): -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#mat-combine-4 +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #mat-combine-4 } +@@@ note +The nested structure in the above example is not necessary for combining the materialized values, it just +demonstrates how the two features work together. See @ref:[Combining materialized values](stream-flows-and-basics.md#flow-combine-mat-scala) for further examples +of combining materialized values without nesting and hierarchy involved. -.. note:: - The nested structure in the above example is not necessary for combining the materialized values, it just - demonstrates how the two features work together. See :ref:`flow-combine-mat-scala` for further examples - of combining materialized values without nesting and hierarchy involved. +@@@ -Attributes ----------- +## Attributes -We have seen that we can use ``named()`` to introduce a nesting level in the fluid DSL (and also explicit nesting by using -``create()`` from :class:`GraphDSL`). Apart from having the effect of adding a nesting level, ``named()`` is actually -a shorthand for calling ``withAttributes(Attributes.name("someName"))``. Attributes provide a way to fine-tune certain +We have seen that we can use `named()` to introduce a nesting level in the fluid DSL (and also explicit nesting by using +`create()` from `GraphDSL`). Apart from having the effect of adding a nesting level, `named()` is actually +a shorthand for calling `withAttributes(Attributes.name("someName"))`. Attributes provide a way to fine-tune certain aspects of the materialized running entity. For example buffer sizes for asynchronous stages can be controlled via -attributes (see :ref:`async-stream-buffers-scala`). When it comes to hierarchic composition, attributes are inherited +attributes (see @ref:[Buffers for asynchronous stages](stream-rate.md#async-stream-buffers-scala)). When it comes to hierarchic composition, attributes are inherited by nested modules, unless they override them with a custom value. -The code below, a modification of an earlier example sets the ``inputBuffer`` attribute on certain modules, but not +The code below, a modification of an earlier example sets the `inputBuffer` attribute on certain modules, but not on others: -.. includecode:: ../code/docs/stream/CompositionDocSpec.scala#attributes-inheritance +@@snip [CompositionDocSpec.scala](../code/docs/stream/CompositionDocSpec.scala) { #attributes-inheritance } -The effect is, that each module inherits the ``inputBuffer`` attribute from its enclosing parent, unless it has -the same attribute explicitly set. ``nestedSource`` gets the default attributes from the materializer itself. ``nestedSink`` -on the other hand has this attribute set, so it will be used by all nested modules. ``nestedFlow`` will inherit from ``nestedSink`` -except the ``map`` stage which has again an explicitly provided attribute overriding the inherited one. +The effect is, that each module inherits the `inputBuffer` attribute from its enclosing parent, unless it has +the same attribute explicitly set. `nestedSource` gets the default attributes from the materializer itself. `nestedSink` +on the other hand has this attribute set, so it will be used by all nested modules. `nestedFlow` will inherit from `nestedSink` +except the `map` stage which has again an explicitly provided attribute overriding the inherited one. | -.. image:: ../../images/compose_attributes.png - :align: center +![compose_attributes.png](../../images/compose_attributes.png) | This diagram illustrates the inheritance process for the example code (representing the materializer default attributes -as the color *red*, the attributes set on ``nestedSink`` as *blue* and the attributes set on ``nestedFlow`` as *green*). +as the color *red*, the attributes set on `nestedSink` as *blue* and the attributes set on `nestedFlow` as *green*). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-cookbook.md b/akka-docs/src/main/paradox/scala/stream/stream-cookbook.md index 26eb32bb11..57980ad4e6 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-cookbook.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-cookbook.md @@ -1,11 +1,6 @@ -.. _stream-cookbook-scala: +# Streams Cookbook -################ -Streams Cookbook -################ - -Introduction -============ +## Introduction This is a collection of patterns to demonstrate various usage of the Akka Streams API by solving small targeted problems in the format of "recipes". The purpose of this page is to give inspiration and ideas how to approach @@ -16,375 +11,356 @@ This part also serves as supplementary material for the main body of documentati open while reading the manual and look for examples demonstrating various streaming concepts as they appear in the main body of documentation. -If you need a quick reference of the available processing stages used in the recipes see :ref:`stages-overview_scala`. +If you need a quick reference of the available processing stages used in the recipes see @ref:[stages-overview_scala](stages-overview.md). -Working with Flows -================== +## Working with Flows In this collection we show simple recipes that involve linear flows. The recipes in this section are rather -general, more targeted recipes are available as separate sections (:ref:`stream-rate-scala`, :ref:`stream-io-scala`). +general, more targeted recipes are available as separate sections (@ref:[Buffers and working with rate](stream-rate.md), @ref:[Working with streaming IO](stream-io.md)). -Logging elements of a stream ----------------------------- +### Logging elements of a stream **Situation:** During development it is sometimes helpful to see what happens in a particular section of a stream. -The simplest solution is to simply use a ``map`` operation and use ``println`` to print the elements received to the console. +The simplest solution is to simply use a `map` operation and use `println` to print the elements received to the console. While this recipe is rather simplistic, it is often suitable for a quick debug session. -.. includecode:: ../code/docs/stream/cookbook/RecipeLoggingElements.scala#println-debug +@@snip [RecipeLoggingElements.scala](../code/docs/stream/cookbook/RecipeLoggingElements.scala) { #println-debug } -Another approach to logging is to use ``log()`` operation which allows configuring logging for elements flowing through +Another approach to logging is to use `log()` operation which allows configuring logging for elements flowing through the stream as well as completion and erroring. -.. includecode:: ../code/docs/stream/cookbook/RecipeLoggingElements.scala#log-custom +@@snip [RecipeLoggingElements.scala](../code/docs/stream/cookbook/RecipeLoggingElements.scala) { #log-custom } -Flattening a stream of sequences --------------------------------- +### Flattening a stream of sequences **Situation:** A stream is given as a stream of sequence of elements, but a stream of elements needed instead, streaming all the nested elements inside the sequences separately. -The ``mapConcat`` operation can be used to implement a one-to-many transformation of elements using a mapper function -in the form of ``In => immutable.Seq[Out]``. In this case we want to map a ``Seq`` of elements to the elements in the -collection itself, so we can just call ``mapConcat(identity)``. +The `mapConcat` operation can be used to implement a one-to-many transformation of elements using a mapper function +in the form of `In => immutable.Seq[Out]`. In this case we want to map a `Seq` of elements to the elements in the +collection itself, so we can just call `mapConcat(identity)`. -.. includecode:: ../code/docs/stream/cookbook/RecipeFlattenSeq.scala#flattening-seqs +@@snip [RecipeFlattenSeq.scala](../code/docs/stream/cookbook/RecipeFlattenSeq.scala) { #flattening-seqs } -Draining a stream to a strict collection ----------------------------------------- +### Draining a stream to a strict collection **Situation:** A possibly unbounded sequence of elements is given as a stream, which needs to be collected into a Scala collection while ensuring boundedness A common situation when working with streams is one where we need to collect incoming elements into a Scala collection. -This operation is supported via ``Sink.seq`` which materializes into a ``Future[Seq[T]]``. +This operation is supported via `Sink.seq` which materializes into a `Future[Seq[T]]`. -The function ``limit`` or ``take`` should always be used in conjunction in order to guarantee stream boundedness, thus preventing the program from running out of memory. +The function `limit` or `take` should always be used in conjunction in order to guarantee stream boundedness, thus preventing the program from running out of memory. For example, this is best avoided: -.. includecode:: ../code/docs/stream/cookbook/RecipeSeq.scala#draining-to-seq-unsafe +@@snip [RecipeSeq.scala](../code/docs/stream/cookbook/RecipeSeq.scala) { #draining-to-seq-unsafe } -Rather, use ``limit`` or ``take`` to ensure that the resulting ``Seq`` will contain only up to ``max`` elements: +Rather, use `limit` or `take` to ensure that the resulting `Seq` will contain only up to `max` elements: -.. includecode:: ../code/docs/stream/cookbook/RecipeSeq.scala#draining-to-seq-safe +@@snip [RecipeSeq.scala](../code/docs/stream/cookbook/RecipeSeq.scala) { #draining-to-seq-safe } -Calculating the digest of a ByteString stream ---------------------------------------------- +### Calculating the digest of a ByteString stream -**Situation:** A stream of bytes is given as a stream of ``ByteString`` s and we want to calculate the cryptographic digest +**Situation:** A stream of bytes is given as a stream of `ByteString` s and we want to calculate the cryptographic digest of the stream. -This recipe uses a :class:`GraphStage` to host a mutable :class:`MessageDigest` class (part of the Java Cryptography -API) and update it with the bytes arriving from the stream. When the stream starts, the ``onPull`` handler of the -stage is called, which just bubbles up the ``pull`` event to its upstream. As a response to this pull, a ByteString -chunk will arrive (``onPush``) which we use to update the digest, then it will pull for the next chunk. +This recipe uses a `GraphStage` to host a mutable `MessageDigest` class (part of the Java Cryptography +API) and update it with the bytes arriving from the stream. When the stream starts, the `onPull` handler of the +stage is called, which just bubbles up the `pull` event to its upstream. As a response to this pull, a ByteString +chunk will arrive (`onPush`) which we use to update the digest, then it will pull for the next chunk. -Eventually the stream of ``ByteString`` s depletes and we get a notification about this event via ``onUpstreamFinish``. -At this point we want to emit the digest value, but we cannot do it with ``push`` in this handler directly since there may -be no downstream demand. Instead we call ``emit`` which will temporarily replace the handlers, emit the provided value when +Eventually the stream of `ByteString` s depletes and we get a notification about this event via `onUpstreamFinish`. +At this point we want to emit the digest value, but we cannot do it with `push` in this handler directly since there may +be no downstream demand. Instead we call `emit` which will temporarily replace the handlers, emit the provided value when demand comes in and then reset the stage state. It will then complete the stage. -.. includecode:: ../code/docs/stream/cookbook/RecipeDigest.scala#calculating-digest +@@snip [RecipeDigest.scala](../code/docs/stream/cookbook/RecipeDigest.scala) { #calculating-digest } -.. _cookbook-parse-lines-scala: + +### Parsing lines from a stream of ByteStrings -Parsing lines from a stream of ByteStrings ------------------------------------------- - -**Situation:** A stream of bytes is given as a stream of ``ByteString`` s containing lines terminated by line ending +**Situation:** A stream of bytes is given as a stream of `ByteString` s containing lines terminated by line ending characters (or, alternatively, containing binary frames delimited by a special delimiter byte sequence) which needs to be parsed. -The :class:`Framing` helper object contains a convenience method to parse messages from a stream of ``ByteString`` s: +The `Framing` helper object contains a convenience method to parse messages from a stream of `ByteString` s: -.. includecode:: ../code/docs/stream/cookbook/RecipeParseLines.scala#parse-lines +@@snip [RecipeParseLines.scala](../code/docs/stream/cookbook/RecipeParseLines.scala) { #parse-lines } -Dealing with compressed data streams ------------------------------------- +### Dealing with compressed data streams -**Situation:** A gzipped stream of bytes is given as a stream of ``ByteString`` s, for example from a ``FileIO`` source. +**Situation:** A gzipped stream of bytes is given as a stream of `ByteString` s, for example from a `FileIO` source. -The :class:`Compression` helper object contains convenience methods for decompressing data streams compressed with +The `Compression` helper object contains convenience methods for decompressing data streams compressed with Gzip or Deflate. -.. includecode:: ../code/docs/stream/cookbook/RecipeDecompress.scala#decompress-gzip +@@snip [RecipeDecompress.scala](../code/docs/stream/cookbook/RecipeDecompress.scala) { #decompress-gzip } -Implementing reduce-by-key --------------------------- +### Implementing reduce-by-key **Situation:** Given a stream of elements, we want to calculate some aggregated value on different subgroups of the elements. The "hello world" of reduce-by-key style operations is *wordcount* which we demonstrate below. Given a stream of words -we first create a new stream that groups the words according to the ``identity`` function, i.e. now +we first create a new stream that groups the words according to the `identity` function, i.e. now we have a stream of streams, where every substream will serve identical words. To count the words, we need to process the stream of streams (the actual groups -containing identical words). ``groupBy`` returns a :class:`SubFlow`, which +containing identical words). `groupBy` returns a `SubFlow`, which means that we transform the resulting substreams directly. In this case we use -the ``reduce`` combinator to aggregate the word itself and the number of its -occurrences within a tuple :class:`(String, Integer)`. Each substream will then +the `reduce` combinator to aggregate the word itself and the number of its +occurrences within a tuple `(String, Integer)`. Each substream will then emit one final value—precisely such a pair—when the overall input completes. As a last step we merge back these values from the substreams into one single output stream. -One noteworthy detail pertains to the ``MaximumDistinctWords`` parameter: this +One noteworthy detail pertains to the `MaximumDistinctWords` parameter: this defines the breadth of the groupBy and merge operations. Akka Streams is focused on bounded resource consumption and the number of concurrently open inputs to the merge operator describes the amount of resources needed by the merge itself. Therefore only a finite number of substreams can be active at -any given time. If the ``groupBy`` operator encounters more keys than this +any given time. If the `groupBy` operator encounters more keys than this number then the stream cannot continue without violating its resource bound, in -this case ``groupBy`` will terminate with a failure. +this case `groupBy` will terminate with a failure. -.. includecode:: ../code/docs/stream/cookbook/RecipeReduceByKey.scala#word-count +@@snip [RecipeReduceByKey.scala](../code/docs/stream/cookbook/RecipeReduceByKey.scala) { #word-count } By extracting the parts specific to *wordcount* into -* a ``groupKey`` function that defines the groups -* a ``map`` map each element to value that is used by the reduce on the substream -* a ``reduce`` function that does the actual reduction + * a `groupKey` function that defines the groups + * a `map` map each element to value that is used by the reduce on the substream + * a `reduce` function that does the actual reduction we get a generalized version below: -.. includecode:: ../code/docs/stream/cookbook/RecipeReduceByKey.scala#reduce-by-key-general +@@snip [RecipeReduceByKey.scala](../code/docs/stream/cookbook/RecipeReduceByKey.scala) { #reduce-by-key-general } -.. note:: - Please note that the reduce-by-key version we discussed above is sequential - in reading the overall input stream, in other words it is **NOT** a - parallelization pattern like MapReduce and similar frameworks. +@@@ note -Sorting elements to multiple groups with groupBy ------------------------------------------------- +Please note that the reduce-by-key version we discussed above is sequential +in reading the overall input stream, in other words it is **NOT** a +parallelization pattern like MapReduce and similar frameworks. -**Situation:** The ``groupBy`` operation strictly partitions incoming elements, each element belongs to exactly one group. +@@@ + +### Sorting elements to multiple groups with groupBy + +**Situation:** The `groupBy` operation strictly partitions incoming elements, each element belongs to exactly one group. Sometimes we want to map elements into multiple groups simultaneously. To achieve the desired result, we attack the problem in two steps: -* first, using a function ``topicMapper`` that gives a list of topics (groups) a message belongs to, we transform our - stream of ``Message`` to a stream of ``(Message, Topic)`` where for each topic the message belongs to a separate pair - will be emitted. This is achieved by using ``mapConcat`` -* Then we take this new stream of message topic pairs (containing a separate pair for each topic a given message - belongs to) and feed it into groupBy, using the topic as the group key. + * first, using a function `topicMapper` that gives a list of topics (groups) a message belongs to, we transform our +stream of `Message` to a stream of `(Message, Topic)` where for each topic the message belongs to a separate pair +will be emitted. This is achieved by using `mapConcat` + * Then we take this new stream of message topic pairs (containing a separate pair for each topic a given message +belongs to) and feed it into groupBy, using the topic as the group key. -.. includecode:: ../code/docs/stream/cookbook/RecipeMultiGroupBy.scala#multi-groupby +@@snip [RecipeMultiGroupBy.scala](../code/docs/stream/cookbook/RecipeMultiGroupBy.scala) { #multi-groupby } -Working with Graphs -=================== +## Working with Graphs In this collection we show recipes that use stream graph elements to achieve various goals. -Triggering the flow of elements programmatically ------------------------------------------------- +### Triggering the flow of elements programmatically **Situation:** Given a stream of elements we want to control the emission of those elements according to a trigger signal. In other words, even if the stream would be able to flow (not being backpressured) we want to hold back elements until a trigger signal arrives. -This recipe solves the problem by simply zipping the stream of ``Message`` elements with the stream of ``Trigger`` -signals. Since ``Zip`` produces pairs, we simply map the output stream selecting the first element of the pair. +This recipe solves the problem by simply zipping the stream of `Message` elements with the stream of `Trigger` +signals. Since `Zip` produces pairs, we simply map the output stream selecting the first element of the pair. -.. includecode:: ../code/docs/stream/cookbook/RecipeManualTrigger.scala#manually-triggered-stream +@@snip [RecipeManualTrigger.scala](../code/docs/stream/cookbook/RecipeManualTrigger.scala) { #manually-triggered-stream } -Alternatively, instead of using a ``Zip``, and then using ``map`` to get the first element of the pairs, we can avoid -creating the pairs in the first place by using ``ZipWith`` which takes a two argument function to produce the output -element. If this function would return a pair of the two argument it would be exactly the behavior of ``Zip`` so -``ZipWith`` is a generalization of zipping. +Alternatively, instead of using a `Zip`, and then using `map` to get the first element of the pairs, we can avoid +creating the pairs in the first place by using `ZipWith` which takes a two argument function to produce the output +element. If this function would return a pair of the two argument it would be exactly the behavior of `Zip` so +`ZipWith` is a generalization of zipping. -.. includecode:: ../code/docs/stream/cookbook/RecipeManualTrigger.scala#manually-triggered-stream-zipwith +@@snip [RecipeManualTrigger.scala](../code/docs/stream/cookbook/RecipeManualTrigger.scala) { #manually-triggered-stream-zipwith } -.. _cookbook-balance-scala: + +### Balancing jobs to a fixed pool of workers -Balancing jobs to a fixed pool of workers ------------------------------------------ - -**Situation:** Given a stream of jobs and a worker process expressed as a :class:`Flow` create a pool of workers +**Situation:** Given a stream of jobs and a worker process expressed as a `Flow` create a pool of workers that automatically balances incoming jobs to available workers, then merges the results. We will express our solution as a function that takes a worker flow and the number of workers to be allocated and gives -a flow that internally contains a pool of these workers. To achieve the desired result we will create a :class:`Flow` +a flow that internally contains a pool of these workers. To achieve the desired result we will create a `Flow` from a graph. -The graph consists of a ``Balance`` node which is a special fan-out operation that tries to route elements to available -downstream consumers. In a ``for`` loop we wire all of our desired workers as outputs of this balancer element, then -we wire the outputs of these workers to a ``Merge`` element that will collect the results from the workers. +The graph consists of a `Balance` node which is a special fan-out operation that tries to route elements to available +downstream consumers. In a `for` loop we wire all of our desired workers as outputs of this balancer element, then +we wire the outputs of these workers to a `Merge` element that will collect the results from the workers. -To make the worker stages run in parallel we mark them as asynchronous with `async`. +To make the worker stages run in parallel we mark them as asynchronous with *async*. -.. includecode:: ../code/docs/stream/cookbook/RecipeWorkerPool.scala#worker-pool +@@snip [RecipeWorkerPool.scala](../code/docs/stream/cookbook/RecipeWorkerPool.scala) { #worker-pool } -Working with rate -================= +## Working with rate This collection of recipes demonstrate various patterns where rate differences between upstream and downstream needs to be handled by other strategies than simple backpressure. -Dropping elements ------------------ +### Dropping elements **Situation:** Given a fast producer and a slow consumer, we want to drop elements if necessary to not slow down the producer too much. -This can be solved by using a versatile rate-transforming operation, ``conflate``. Conflate can be thought as -a special ``reduce`` operation that collapses multiple upstream elements into one aggregate element if needed to keep +This can be solved by using a versatile rate-transforming operation, `conflate`. Conflate can be thought as +a special `reduce` operation that collapses multiple upstream elements into one aggregate element if needed to keep the speed of the upstream unaffected by the downstream. -When the upstream is faster, the reducing process of the ``conflate`` starts. Our reducer function simply takes +When the upstream is faster, the reducing process of the `conflate` starts. Our reducer function simply takes the freshest element. This in a simple dropping operation. -.. includecode:: ../code/docs/stream/cookbook/RecipeSimpleDrop.scala#simple-drop +@@snip [RecipeSimpleDrop.scala](../code/docs/stream/cookbook/RecipeSimpleDrop.scala) { #simple-drop } -There is a more general version of ``conflate`` named ``conflateWithSeed`` that allows to express more complex aggregations, more -similar to a ``fold``. +There is a more general version of `conflate` named `conflateWithSeed` that allows to express more complex aggregations, more +similar to a `fold`. -Dropping broadcast ------------------- +### Dropping broadcast -**Situation:** The default ``Broadcast`` graph element is properly backpressured, but that means that a slow downstream +**Situation:** The default `Broadcast` graph element is properly backpressured, but that means that a slow downstream consumer can hold back the other downstream consumers resulting in lowered throughput. In other words the rate of -``Broadcast`` is the rate of its slowest downstream consumer. In certain cases it is desirable to allow faster consumers +`Broadcast` is the rate of its slowest downstream consumer. In certain cases it is desirable to allow faster consumers to progress independently of their slower siblings by dropping elements if necessary. -One solution to this problem is to append a ``buffer`` element in front of all of the downstream consumers -defining a dropping strategy instead of the default ``Backpressure``. This allows small temporary rate differences +One solution to this problem is to append a `buffer` element in front of all of the downstream consumers +defining a dropping strategy instead of the default `Backpressure`. This allows small temporary rate differences between the different consumers (the buffer smooths out small rate variances), but also allows faster consumers to progress by dropping from the buffer of the slow consumers if necessary. -.. includecode:: ../code/docs/stream/cookbook/RecipeDroppyBroadcast.scala#droppy-bcast +@@snip [RecipeDroppyBroadcast.scala](../code/docs/stream/cookbook/RecipeDroppyBroadcast.scala) { #droppy-bcast } -Collecting missed ticks ------------------------ +### Collecting missed ticks **Situation:** Given a regular (stream) source of ticks, instead of trying to backpressure the producer of the ticks we want to keep a counter of the missed ticks instead and pass it down when possible. -We will use ``conflateWithSeed`` to solve the problem. The seed version of conflate takes two functions: +We will use `conflateWithSeed` to solve the problem. The seed version of conflate takes two functions: -* A seed function that produces the zero element for the folding process that happens when the upstream is faster than - the downstream. In our case the seed function is a constant function that returns 0 since there were no missed ticks - at that point. -* A fold function that is invoked when multiple upstream messages needs to be collapsed to an aggregate value due - to the insufficient processing rate of the downstream. Our folding function simply increments the currently stored - count of the missed ticks so far. + * A seed function that produces the zero element for the folding process that happens when the upstream is faster than +the downstream. In our case the seed function is a constant function that returns 0 since there were no missed ticks +at that point. + * A fold function that is invoked when multiple upstream messages needs to be collapsed to an aggregate value due +to the insufficient processing rate of the downstream. Our folding function simply increments the currently stored +count of the missed ticks so far. -As a result, we have a flow of ``Int`` where the number represents the missed ticks. A number 0 means that we were +As a result, we have a flow of `Int` where the number represents the missed ticks. A number 0 means that we were able to consume the tick fast enough (i.e. zero means: 1 non-missed tick + 0 missed ticks) -.. includecode:: ../code/docs/stream/cookbook/RecipeMissedTicks.scala#missed-ticks +@@snip [RecipeMissedTicks.scala](../code/docs/stream/cookbook/RecipeMissedTicks.scala) { #missed-ticks } -Create a stream processor that repeats the last element seen ------------------------------------------------------------- +### Create a stream processor that repeats the last element seen **Situation:** Given a producer and consumer, where the rate of neither is known in advance, we want to ensure that none of them is slowing down the other by dropping earlier unconsumed elements from the upstream if necessary, and repeating the last value for the downstream if necessary. -We have two options to implement this feature. In both cases we will use :class:`GraphStage` to build our custom -element. In the first version we will use a provided initial value ``initial`` that will be used -to feed the downstream if no upstream element is ready yet. In the ``onPush()`` handler we just overwrite the -``currentValue`` variable and immediately relieve the upstream by calling ``pull()``. The downstream ``onPull`` handler -is very similar, we immediately relieve the downstream by emitting ``currentValue``. +We have two options to implement this feature. In both cases we will use `GraphStage` to build our custom +element. In the first version we will use a provided initial value `initial` that will be used +to feed the downstream if no upstream element is ready yet. In the `onPush()` handler we just overwrite the +`currentValue` variable and immediately relieve the upstream by calling `pull()`. The downstream `onPull` handler +is very similar, we immediately relieve the downstream by emitting `currentValue`. -.. includecode:: ../code/docs/stream/cookbook/RecipeHold.scala#hold-version-1 +@@snip [RecipeHold.scala](../code/docs/stream/cookbook/RecipeHold.scala) { #hold-version-1 } While it is relatively simple, the drawback of the first version is that it needs an arbitrary initial element which is not always possible to provide. Hence, we create a second version where the downstream might need to wait in one single case: if the very first element is not yet available. -We introduce a boolean variable ``waitingFirstValue`` to denote whether the first element has been provided or not -(alternatively an :class:`Option` can be used for ``currentValue`` or if the element type is a subclass of AnyRef -a null can be used with the same purpose). In the downstream ``onPull()`` handler the difference from the previous +We introduce a boolean variable `waitingFirstValue` to denote whether the first element has been provided or not +(alternatively an `Option` can be used for `currentValue` or if the element type is a subclass of AnyRef +a null can be used with the same purpose). In the downstream `onPull()` handler the difference from the previous version is that we check if we have received the first value and only emit if we have. This leads to that when the first element comes in we must check if there possibly already was demand from downstream so that we in that case can push the element directly. -.. includecode:: ../code/docs/stream/cookbook/RecipeHold.scala#hold-version-2 +@@snip [RecipeHold.scala](../code/docs/stream/cookbook/RecipeHold.scala) { #hold-version-2 } -Globally limiting the rate of a set of streams ----------------------------------------------- +### Globally limiting the rate of a set of streams **Situation:** Given a set of independent streams that we cannot merge, we want to globally limit the aggregate throughput of the set of streams. One possible solution uses a shared actor as the global limiter combined with mapAsync to create a reusable -:class:`Flow` that can be plugged into a stream to limit its rate. +`Flow` that can be plugged into a stream to limit its rate. As the first step we define an actor that will do the accounting for the global rate limit. The actor maintains a timer, a counter for pending permit tokens and a queue for possibly waiting participants. The actor has -an ``open`` and ``closed`` state. The actor is in the ``open`` state while it has still pending permits. Whenever a -request for permit arrives as a ``WantToPass`` message to the actor the number of available permits is decremented -and we notify the sender that it can pass by answering with a ``MayPass`` message. If the amount of permits reaches -zero, the actor transitions to the ``closed`` state. In this state requests are not immediately answered, instead the reference -of the sender is added to a queue. Once the timer for replenishing the pending permits fires by sending a ``ReplenishTokens`` +an `open` and `closed` state. The actor is in the `open` state while it has still pending permits. Whenever a +request for permit arrives as a `WantToPass` message to the actor the number of available permits is decremented +and we notify the sender that it can pass by answering with a `MayPass` message. If the amount of permits reaches +zero, the actor transitions to the `closed` state. In this state requests are not immediately answered, instead the reference +of the sender is added to a queue. Once the timer for replenishing the pending permits fires by sending a `ReplenishTokens` message, we increment the pending permits counter and send a reply to each of the waiting senders. If there are more -waiting senders than permits available we will stay in the ``closed`` state. +waiting senders than permits available we will stay in the `closed` state. -.. includecode:: ../code/docs/stream/cookbook/RecipeGlobalRateLimit.scala#global-limiter-actor +@@snip [RecipeGlobalRateLimit.scala](../code/docs/stream/cookbook/RecipeGlobalRateLimit.scala) { #global-limiter-actor } -To create a Flow that uses this global limiter actor we use the ``mapAsync`` function with the combination of the ``ask`` +To create a Flow that uses this global limiter actor we use the `mapAsync` function with the combination of the `ask` pattern. We also define a timeout, so if a reply is not received during the configured maximum wait period the returned -future from ``ask`` will fail, which will fail the corresponding stream as well. +future from `ask` will fail, which will fail the corresponding stream as well. -.. includecode:: ../code/docs/stream/cookbook/RecipeGlobalRateLimit.scala#global-limiter-flow +@@snip [RecipeGlobalRateLimit.scala](../code/docs/stream/cookbook/RecipeGlobalRateLimit.scala) { #global-limiter-flow } -.. note:: - The global actor used for limiting introduces a global bottleneck. You might want to assign a dedicated dispatcher - for this actor. +@@@ note -Working with IO -=============== +The global actor used for limiting introduces a global bottleneck. You might want to assign a dedicated dispatcher +for this actor. -Chunking up a stream of ByteStrings into limited size ByteStrings ------------------------------------------------------------------ +@@@ -**Situation:** Given a stream of ``ByteString`` s we want to produce a stream of ``ByteString`` s containing the same bytes in -the same sequence, but capping the size of ``ByteString`` s. In other words we want to slice up ``ByteString`` s into smaller +## Working with IO + +### Chunking up a stream of ByteStrings into limited size ByteStrings + +**Situation:** Given a stream of `ByteString` s we want to produce a stream of `ByteString` s containing the same bytes in +the same sequence, but capping the size of `ByteString` s. In other words we want to slice up `ByteString` s into smaller chunks if they exceed a size threshold. -This can be achieved with a single :class:`GraphStage`. The main logic of our stage is in ``emitChunk()`` +This can be achieved with a single `GraphStage`. The main logic of our stage is in `emitChunk()` which implements the following logic: -* if the buffer is empty, and upstream is not closed we pull for more bytes, if it is closed we complete -* if the buffer is nonEmpty, we split it according to the ``chunkSize``. This will give a next chunk that we will emit, - and an empty or nonempty remaining buffer. + * if the buffer is empty, and upstream is not closed we pull for more bytes, if it is closed we complete + * if the buffer is nonEmpty, we split it according to the `chunkSize`. This will give a next chunk that we will emit, +and an empty or nonempty remaining buffer. -Both ``onPush()`` and ``onPull()`` calls ``emitChunk()`` the only difference is that the push handler also stores +Both `onPush()` and `onPull()` calls `emitChunk()` the only difference is that the push handler also stores the incoming chunk by appending to the end of the buffer. -.. includecode:: ../code/docs/stream/cookbook/RecipeByteStrings.scala#bytestring-chunker +@@snip [RecipeByteStrings.scala](../code/docs/stream/cookbook/RecipeByteStrings.scala) { #bytestring-chunker } -Limit the number of bytes passing through a stream of ByteStrings ------------------------------------------------------------------ +### Limit the number of bytes passing through a stream of ByteStrings -**Situation:** Given a stream of ``ByteString`` s we want to fail the stream if more than a given maximum of bytes has been +**Situation:** Given a stream of `ByteString` s we want to fail the stream if more than a given maximum of bytes has been consumed. -This recipe uses a :class:`GraphStage` to implement the desired feature. In the only handler we override, -``onPush()`` we just update a counter and see if it gets larger than ``maximumBytes``. If a violation happens +This recipe uses a `GraphStage` to implement the desired feature. In the only handler we override, +`onPush()` we just update a counter and see if it gets larger than `maximumBytes`. If a violation happens we signal failure, otherwise we forward the chunk we have received. -.. includecode:: ../code/docs/stream/cookbook/RecipeByteStrings.scala#bytes-limiter +@@snip [RecipeByteStrings.scala](../code/docs/stream/cookbook/RecipeByteStrings.scala) { #bytes-limiter } -Compact ByteStrings in a stream of ByteStrings ----------------------------------------------- +### Compact ByteStrings in a stream of ByteStrings -**Situation:** After a long stream of transformations, due to their immutable, structural sharing nature ``ByteString`` s may +**Situation:** After a long stream of transformations, due to their immutable, structural sharing nature `ByteString` s may refer to multiple original ByteString instances unnecessarily retaining memory. As the final step of a transformation -chain we want to have clean copies that are no longer referencing the original ``ByteString`` s. +chain we want to have clean copies that are no longer referencing the original `ByteString` s. -The recipe is a simple use of map, calling the ``compact()`` method of the :class:`ByteString` elements. This does +The recipe is a simple use of map, calling the `compact()` method of the `ByteString` elements. This does copying of the underlying arrays, so this should be the last element of a long chain if used. -.. includecode:: ../code/docs/stream/cookbook/RecipeByteStrings.scala#compacting-bytestrings +@@snip [RecipeByteStrings.scala](../code/docs/stream/cookbook/RecipeByteStrings.scala) { #compacting-bytestrings } -Injecting keep-alive messages into a stream of ByteStrings ----------------------------------------------------------- +### Injecting keep-alive messages into a stream of ByteStrings -**Situation:** Given a communication channel expressed as a stream of ``ByteString`` s we want to inject keep-alive messages +**Situation:** Given a communication channel expressed as a stream of `ByteString` s we want to inject keep-alive messages but only if this does not interfere with normal traffic. There is a built-in operation that allows to do this directly: -.. includecode:: ../code/docs/stream/cookbook/RecipeKeepAlive.scala#inject-keepalive +@@snip [RecipeKeepAlive.scala](../code/docs/stream/cookbook/RecipeKeepAlive.scala) { #inject-keepalive } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-customize.md b/akka-docs/src/main/paradox/scala/stream/stream-customize.md index 8e0839c5bf..1a750109b0 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-customize.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-customize.md @@ -1,98 +1,94 @@ -.. _stream-customize-scala: +# Custom stream processing -######################## -Custom stream processing -######################## - -While the processing vocabulary of Akka Streams is quite rich (see the :ref:`stream-cookbook-scala` for examples) it +While the processing vocabulary of Akka Streams is quite rich (see the @ref:[Streams Cookbook](stream-cookbook.md) for examples) it is sometimes necessary to define new transformation stages either because some functionality is missing from the stock operations, or for performance reasons. In this part we show how to build custom processing stages and graph junctions of various kinds. -.. note:: - A custom graph stage should not be the first tool you reach for, defining graphs using flows - and the graph DSL is in general easier and does to a larger extent protect you from mistakes that - might be easy to make with a custom :class:`GraphStage` +@@@ note +A custom graph stage should not be the first tool you reach for, defining graphs using flows +and the graph DSL is in general easier and does to a larger extent protect you from mistakes that +might be easy to make with a custom `GraphStage` -.. _graphstage-scala: +@@@ -Custom processing with GraphStage -================================= + +## Custom processing with GraphStage -The :class:`GraphStage` abstraction can be used to create arbitrary graph processing stages with any number of input -or output ports. It is a counterpart of the ``GraphDSL.create()`` method which creates new stream processing -stages by composing others. Where :class:`GraphStage` differs is that it creates a stage that is itself not divisible into +The `GraphStage` abstraction can be used to create arbitrary graph processing stages with any number of input +or output ports. It is a counterpart of the `GraphDSL.create()` method which creates new stream processing +stages by composing others. Where `GraphStage` differs is that it creates a stage that is itself not divisible into smaller ones, and allows state to be maintained inside it in a safe way. -As a first motivating example, we will build a new :class:`Source` that will simply emit numbers from 1 until it is +As a first motivating example, we will build a new `Source` that will simply emit numbers from 1 until it is cancelled. To start, we need to define the "interface" of our stage, which is called *shape* in Akka Streams terminology -(this is explained in more detail in the section :ref:`composition-scala`). This is how this looks like: +(this is explained in more detail in the section @ref:[Modularity, Composition and Hierarchy](stream-composition.md)). This is how this looks like: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#boilerplate-example +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #boilerplate-example } -As you see, in itself the :class:`GraphStage` only defines the ports of this stage and a shape that contains the ports. -It also has, a currently unimplemented method called ``createLogic``. If you recall, stages are reusable in multiple -materializations, each resulting in a different executing entity. In the case of :class:`GraphStage` the actual running -logic is modeled as an instance of a :class:`GraphStageLogic` which will be created by the materializer by calling -the ``createLogic`` method. In other words, all we need to do is to create a suitable logic that will emit the +As you see, in itself the `GraphStage` only defines the ports of this stage and a shape that contains the ports. +It also has, a currently unimplemented method called `createLogic`. If you recall, stages are reusable in multiple +materializations, each resulting in a different executing entity. In the case of `GraphStage` the actual running +logic is modeled as an instance of a `GraphStageLogic` which will be created by the materializer by calling +the `createLogic` method. In other words, all we need to do is to create a suitable logic that will emit the numbers we want. -.. note:: +@@@ note - It is very important to keep the GraphStage object itself immutable and reusable. All mutable state needs to be - confined to the GraphStageLogic that is created for every materialization. +It is very important to keep the GraphStage object itself immutable and reusable. All mutable state needs to be +confined to the GraphStageLogic that is created for every materialization. -In order to emit from a :class:`Source` in a backpressured stream one needs first to have demand from downstream. -To receive the necessary events one needs to register a subclass of :class:`OutHandler` with the output port -(:class:`Outlet`). This handler will receive events related to the lifecycle of the port. In our case we need to -override ``onPull()`` which indicates that we are free to emit a single element. There is another callback, -``onDownstreamFinish()`` which is called if the downstream cancelled. Since the default behavior of that callback is -to stop the stage, we don't need to override it. In the ``onPull`` callback we will simply emit the next number. This +@@@ + +In order to emit from a `Source` in a backpressured stream one needs first to have demand from downstream. +To receive the necessary events one needs to register a subclass of `OutHandler` with the output port +(`Outlet`). This handler will receive events related to the lifecycle of the port. In our case we need to +override `onPull()` which indicates that we are free to emit a single element. There is another callback, +`onDownstreamFinish()` which is called if the downstream cancelled. Since the default behavior of that callback is +to stop the stage, we don't need to override it. In the `onPull` callback we will simply emit the next number. This is how it looks like in the end: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#custom-source-example +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #custom-source-example } -Instances of the above :class:`GraphStage` are subclasses of ``Graph[SourceShape[Int],NotUsed]`` which means +Instances of the above `GraphStage` are subclasses of `Graph[SourceShape[Int],NotUsed]` which means that they are already usable in many situations, but do not provide the DSL methods we usually have for other -:class:`Source` s. In order to convert this :class:`Graph` to a proper :class:`Source` we need to wrap it using -``Source.fromGraph`` (see :ref:`composition-scala` for more details about graphs and DSLs). Now we can use the +`Source` s. In order to convert this `Graph` to a proper `Source` we need to wrap it using +`Source.fromGraph` (see @ref:[Modularity, Composition and Hierarchy](stream-composition.md) for more details about graphs and DSLs). Now we can use the source as any other built-in one: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#simple-source-usage +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #simple-source-usage } -Similarly, to create a custom :class:`Sink` one can register a subclass :class:`InHandler` with the stage :class:`Inlet`. -The ``onPush()`` callback is used to signal the handler a new element has been pushed to the stage, -and can hence be grabbed and used. ``onPush()`` can be overridden to provide custom behaviour. +Similarly, to create a custom `Sink` one can register a subclass `InHandler` with the stage `Inlet`. +The `onPush()` callback is used to signal the handler a new element has been pushed to the stage, +and can hence be grabbed and used. `onPush()` can be overridden to provide custom behaviour. Please note, most Sinks would need to request upstream elements as soon as they are created: this can be -done by calling ``pull(inlet)`` in the ``preStart()`` callback. +done by calling `pull(inlet)` in the `preStart()` callback. -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#custom-sink-example +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #custom-sink-example } -Port states, InHandler and OutHandler -------------------------------------- +### Port states, InHandler and OutHandler -In order to interact with a port (:class:`Inlet` or :class:`Outlet`) of the stage we need to be able to receive events -and generate new events belonging to the port. From the :class:`GraphStageLogic` the following operations are available +In order to interact with a port (`Inlet` or `Outlet`) of the stage we need to be able to receive events +and generate new events belonging to the port. From the `GraphStageLogic` the following operations are available on an output port: -* ``push(out,elem)`` pushes an element to the output port. Only possible after the port has been pulled by downstream. -* ``complete(out)`` closes the output port normally. -* ``fail(out,exception)`` closes the port with a failure signal. + * `push(out,elem)` pushes an element to the output port. Only possible after the port has been pulled by downstream. + * `complete(out)` closes the output port normally. + * `fail(out,exception)` closes the port with a failure signal. +The events corresponding to an *output* port can be received in an `OutHandler` instance registered to the +output port using `setHandler(out,handler)`. This handler has two callbacks: -The events corresponding to an *output* port can be received in an :class:`OutHandler` instance registered to the -output port using ``setHandler(out,handler)``. This handler has two callbacks: - -* ``onPull()`` is called when the output port is ready to emit the next element, ``push(out, elem)`` is now allowed - to be called on this port. -* ``onDownstreamFinish()`` is called once the downstream has cancelled and no longer allows messages to be pushed to it. - No more ``onPull()`` will arrive after this event. If not overridden this will default to stopping the stage. + * `onPull()` is called when the output port is ready to emit the next element, `push(out, elem)` is now allowed +to be called on this port. + * `onDownstreamFinish()` is called once the downstream has cancelled and no longer allows messages to be pushed to it. +No more `onPull()` will arrive after this event. If not overridden this will default to stopping the stage. Also, there are two query methods available for output ports: -* ``isAvailable(out)`` returns true if the port can be pushed -* ``isClosed(out)`` returns true if the port is closed. At this point the port can not be pushed and will not be pulled anymore. + * `isAvailable(out)` returns true if the port can be pushed + * `isClosed(out)` returns true if the port is closed. At this point the port can not be pushed and will not be pulled anymore. The relationship of the above operations, events and queries are summarized in the state machine below. Green shows the initial state while orange indicates the end state. If an operation is not listed for a state, then it is invalid @@ -101,34 +97,33 @@ in that state. | -.. image:: ../../images/outport_transitions.png - :align: center +![outport_transitions.png](../../images/outport_transitions.png) | The following operations are available for *input* ports: -* ``pull(in)`` requests a new element from an input port. This is only possible after the port has been pushed by upstream. -* ``grab(in)`` acquires the element that has been received during an ``onPush()``. It cannot be called again until the - port is pushed again by the upstream. -* ``cancel(in)`` closes the input port. + * `pull(in)` requests a new element from an input port. This is only possible after the port has been pushed by upstream. + * `grab(in)` acquires the element that has been received during an `onPush()`. It cannot be called again until the +port is pushed again by the upstream. + * `cancel(in)` closes the input port. -The events corresponding to an *input* port can be received in an :class:`InHandler` instance registered to the -input port using ``setHandler(in, handler)``. This handler has three callbacks: +The events corresponding to an *input* port can be received in an `InHandler` instance registered to the +input port using `setHandler(in, handler)`. This handler has three callbacks: -* ``onPush()`` is called when the input port has now a new element. Now it is possible to acquire this element using - ``grab(in)`` and/or call ``pull(in)`` on the port to request the next element. It is not mandatory to grab the - element, but if it is pulled while the element has not been grabbed it will drop the buffered element. -* ``onUpstreamFinish()`` is called once the upstream has completed and no longer can be pulled for new elements. - No more ``onPush()`` will arrive after this event. If not overridden this will default to stopping the stage. -* ``onUpstreamFailure()`` is called if the upstream failed with an exception and no longer can be pulled for new elements. - No more ``onPush()`` will arrive after this event. If not overridden this will default to failing the stage. + * `onPush()` is called when the input port has now a new element. Now it is possible to acquire this element using +`grab(in)` and/or call `pull(in)` on the port to request the next element. It is not mandatory to grab the +element, but if it is pulled while the element has not been grabbed it will drop the buffered element. + * `onUpstreamFinish()` is called once the upstream has completed and no longer can be pulled for new elements. +No more `onPush()` will arrive after this event. If not overridden this will default to stopping the stage. + * `onUpstreamFailure()` is called if the upstream failed with an exception and no longer can be pulled for new elements. +No more `onPush()` will arrive after this event. If not overridden this will default to failing the stage. Also, there are three query methods available for input ports: -* ``isAvailable(in)`` returns true if the port can be grabbed. -* ``hasBeenPulled(in)`` returns true if the port has been already pulled. Calling ``pull(in)`` in this state is illegal. -* ``isClosed(in)`` returns true if the port is closed. At this point the port can not be pulled and will not be pushed anymore. + * `isAvailable(in)` returns true if the port can be grabbed. + * `hasBeenPulled(in)` returns true if the port has been already pulled. Calling `pull(in)` in this state is illegal. + * `isClosed(in)` returns true if the port is closed. At this point the port can not be pulled and will not be pushed anymore. The relationship of the above operations, events and queries are summarized in the state machine below. Green shows the initial state while orange indicates the end state. If an operation is not listed for a state, then it is invalid @@ -137,16 +132,14 @@ in that state. | -.. image:: ../../images/inport_transitions.png - :align: center +![inport_transitions.png](../../images/inport_transitions.png) | Finally, there are two methods available for convenience to complete the stage and all of its ports: -* ``completeStage()`` is equivalent to closing all output ports and cancelling all input ports. -* ``failStage(exception)`` is equivalent to failing all output ports and cancelling all input ports. - + * `completeStage()` is equivalent to closing all output ports and cancelling all input ports. + * `failStage(exception)` is equivalent to failing all output ports and cancelling all input ports. In some cases it is inconvenient and error prone to react on the regular state machine events with the signal based API described above. For those cases there is an API which allows for a more declarative sequencing @@ -154,28 +147,26 @@ of actions which will greatly simplify some use cases at the cost of some extra between the two APIs could be described as that the first one is signal driven from the outside, while this API is more active and drives its surroundings. -The operations of this part of the :class:``GraphStage`` API are: +The operations of this part of the :class:`GraphStage` API are: -* ``emit(out, elem)`` and ``emitMultiple(out, Iterable(elem1, elem2))`` replaces the ``OutHandler`` with a handler that emits - one or more elements when there is demand, and then reinstalls the current handlers -* ``read(in)(andThen)`` and ``readN(in, n)(andThen)`` replaces the ``InHandler`` with a handler that reads one or - more elements as they are pushed and allows the handler to react once the requested number of elements has been read. -* ``abortEmitting()`` and ``abortReading()`` which will cancel an ongoing emit or read + * `emit(out, elem)` and `emitMultiple(out, Iterable(elem1, elem2))` replaces the `OutHandler` with a handler that emits +one or more elements when there is demand, and then reinstalls the current handlers + * `read(in)(andThen)` and `readN(in, n)(andThen)` replaces the `InHandler` with a handler that reads one or +more elements as they are pushed and allows the handler to react once the requested number of elements has been read. + * `abortEmitting()` and `abortReading()` which will cancel an ongoing emit or read Note that since the above methods are implemented by temporarily replacing the handlers of the stage you should never -call ``setHandler`` while they are running ``emit`` or ``read`` as that interferes with how they are implemented. -The following methods are safe to call after invoking ``emit`` and ``read`` (and will lead to actually running the -operation when those are done): ``complete(out)``, ``completeStage()``, ``emit``, ``emitMultiple``, ``abortEmitting()`` -and ``abortReading()`` +call `setHandler` while they are running `emit` or `read` as that interferes with how they are implemented. +The following methods are safe to call after invoking `emit` and `read` (and will lead to actually running the +operation when those are done): `complete(out)`, `completeStage()`, `emit`, `emitMultiple`, `abortEmitting()` +and `abortReading()` -An example of how this API simplifies a stage can be found below in the second version of the :class:``Duplicator``. +An example of how this API simplifies a stage can be found below in the second version of the :class:`Duplicator`. - -Custom linear processing stages using GraphStage ------------------------------------------------- +### Custom linear processing stages using GraphStage Graph stages allows for custom linear processing stages through letting them -have one input and one output and using :class:`FlowShape` as their shape. +have one input and one output and using `FlowShape` as their shape. Such a stage can be illustrated as a box with two flows as it is seen in the illustration below. Demand flowing upstream leading to elements @@ -183,58 +174,48 @@ flowing downstream. | -.. image:: ../../images/graph_stage_conceptual.png - :align: center - :width: 500 +![graph_stage_conceptual.png](../../images/graph_stage_conceptual.png) | - -To illustrate these concepts we create a small :class:`GraphStage` that implements the ``map`` transformation. +To illustrate these concepts we create a small `GraphStage` that implements the `map` transformation. | -.. image:: ../../images/graph_stage_map.png - :align: center - :width: 300 +![graph_stage_map.png](../../images/graph_stage_map.png) | -Map calls ``push(out)`` from the ``onPush()`` handler and it also calls ``pull()`` from the ``onPull`` handler resulting in the +Map calls `push(out)` from the `onPush()` handler and it also calls `pull()` from the `onPull` handler resulting in the conceptual wiring above, and fully expressed in code below: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#one-to-one +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #one-to-one } Map is a typical example of a one-to-one transformation of a stream where demand is passed along upstream elements passed on downstream. To demonstrate a many-to-one stage we will implement -filter. The conceptual wiring of ``Filter`` looks like this: +filter. The conceptual wiring of `Filter` looks like this: | -.. image:: ../../images/graph_stage_filter.png - :align: center - :width: 300 +![graph_stage_filter.png](../../images/graph_stage_filter.png) | - As we see above, if the given predicate matches the current element we are propagating it downwards, otherwise we return the “ball” to our upstream so that we get the new element. This is achieved by modifying the map -example by adding a conditional in the ``onPush`` handler and decide between a ``pull(in)`` or ``push(out)`` call -(and of course not having a mapping ``f`` function). +example by adding a conditional in the `onPush` handler and decide between a `pull(in)` or `push(out)` call +(and of course not having a mapping `f` function). -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#many-to-one +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #many-to-one } To complete the picture we define a one-to-many transformation as the next step. We chose a straightforward example stage that emits every upstream element twice downstream. The conceptual wiring of this stage looks like this: | -.. image:: ../../images/graph_stage_duplicate.png - :align: center - :width: 300 +![graph_stage_duplicate.png](../../images/graph_stage_duplicate.png) | @@ -242,189 +223,177 @@ This is a stage that has state: an option with the last element it has seen indi has duplicated this last element already or not. We must also make sure to emit the extra element if the upstream completes. -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#one-to-many +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #one-to-many } In this case a pull from downstream might be consumed by the stage itself rather than passed along upstream as the stage might contain an element it wants to push. Note that we also need to handle the case where the upstream closes while the stage still has elements it wants to push downstream. This is done by -overriding `onUpstreamFinish` in the `InHandler` and provide custom logic +overriding *onUpstreamFinish* in the *InHandler* and provide custom logic that should happen when the upstream has been finished. This example can be simplified by replacing the usage of a mutable state with calls to -``emitMultiple`` which will replace the handlers, emit each of multiple elements and then +`emitMultiple` which will replace the handlers, emit each of multiple elements and then reinstate the original handlers: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#simpler-one-to-many - +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #simpler-one-to-many } Finally, to demonstrate all of the stages above, we put them together into a processing chain, which conceptually would correspond to the following structure: +| + +![graph_stage_chain.png](../../images/graph_stage_chain.png) | -.. image:: ../../images/graph_stage_chain.png - :align: center - :width: 700 +In code this is only a few lines, using the `via` use our custom stages in a stream: -| - -In code this is only a few lines, using the ``via`` use our custom stages in a stream: - -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#graph-stage-chain +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #graph-stage-chain } If we attempt to draw the sequence of events, it shows that there is one "event token" in circulation in a potential chain of stages, just like our conceptual "railroad tracks" representation predicts. +| + +![graph_stage_tracks_1.png](../../images/graph_stage_tracks_1.png) | -.. image:: ../../images/graph_stage_tracks_1.png - :align: center - :width: 700 - -| - -Completion ----------- +### Completion Completion handling usually (but not exclusively) comes into the picture when processing stages need to emit a few more elements after their upstream source has been completed. We have seen an example of this in our -first :class:`Duplicator` implementation where the last element needs to be doubled even after the upstream neighbor -stage has been completed. This can be done by overriding the ``onUpstreamFinish`` method in ``InHandler``. +first `Duplicator` implementation where the last element needs to be doubled even after the upstream neighbor +stage has been completed. This can be done by overriding the `onUpstreamFinish` method in `InHandler`. Stages by default automatically stop once all of their ports (input and output) have been closed externally or internally. -It is possible to opt out from this behavior by invoking ``setKeepGoing(true)`` (which is not supported from the stage’s -constructor and usually done in ``preStart``). In this case the stage **must** be explicitly closed by calling ``completeStage()`` -or ``failStage(exception)``. This feature carries the risk of leaking streams and actors, therefore it should be used +It is possible to opt out from this behavior by invoking `setKeepGoing(true)` (which is not supported from the stage’s +constructor and usually done in `preStart`). In this case the stage **must** be explicitly closed by calling `completeStage()` +or `failStage(exception)`. This feature carries the risk of leaking streams and actors, therefore it should be used with care. -Logging inside GraphStages --------------------------- +### Logging inside GraphStages Logging debug or other important information in your stages is often a very good idea, especially when developing more advances stages which may need to be debugged at some point. -The helper trait ``akka.stream.stage.StageLogging`` is provided to enable you to easily obtain a ``LoggingAdapter`` -inside of a ``GraphStage`` as long as the ``Materializer`` you're using is able to provide you with a logger. -In that sense, it serves a very similar purpose as ``ActorLogging`` does for Actors. +The helper trait `akka.stream.stage.StageLogging` is provided to enable you to easily obtain a `LoggingAdapter` +inside of a `GraphStage` as long as the `Materializer` you're using is able to provide you with a logger. +In that sense, it serves a very similar purpose as `ActorLogging` does for Actors. -.. note:: - Please note that you can always simply use a logging library directly inside a Stage. - Make sure to use an asynchronous appender however, to not accidentally block the stage when writing to files etc. - See :ref:`slf4j-directly-scala` for more details on setting up async appenders in SLF4J. +@@@ note -The stage then gets access to the ``log`` field which it can safely use from any ``GraphStage`` callbacks: +Please note that you can always simply use a logging library directly inside a Stage. +Make sure to use an asynchronous appender however, to not accidentally block the stage when writing to files etc. +See @ref:[Using the SLF4J API directly](../logging.md#slf4j-directly-scala) for more details on setting up async appenders in SLF4J. -.. includecode:: ../code/docs/stream/GraphStageLoggingDocSpec.scala#stage-with-logging +@@@ -.. note:: - **SPI Note:** If you're implementing a Materializer, you can add this ability to your materializer by implementing - ``MaterializerLoggingProvider`` in your ``Materializer``. +The stage then gets access to the `log` field which it can safely use from any `GraphStage` callbacks: +@@snip [GraphStageLoggingDocSpec.scala](../code/docs/stream/GraphStageLoggingDocSpec.scala) { #stage-with-logging } -Using timers ------------- +@@@ note -It is possible to use timers in :class:`GraphStages` by using :class:`TimerGraphStageLogic` as the base class for -the returned logic. Timers can be scheduled by calling one of ``scheduleOnce(key,delay)``, ``schedulePeriodically(key,period)`` or -``schedulePeriodicallyWithInitialDelay(key,delay,period)`` and passing an object as a key for that timer (can be any object, for example -a :class:`String`). The ``onTimer(key)`` method needs to be overridden and it will be called once the timer of ``key`` -fires. It is possible to cancel a timer using ``cancelTimer(key)`` and check the status of a timer with -``isTimerActive(key)``. Timers will be automatically cleaned up when the stage completes. +**SPI Note:** If you're implementing a Materializer, you can add this ability to your materializer by implementing +`MaterializerLoggingProvider` in your `Materializer`. + +@@@ + +### Using timers + +It is possible to use timers in `GraphStages` by using `TimerGraphStageLogic` as the base class for +the returned logic. Timers can be scheduled by calling one of `scheduleOnce(key,delay)`, `schedulePeriodically(key,period)` or +`schedulePeriodicallyWithInitialDelay(key,delay,period)` and passing an object as a key for that timer (can be any object, for example +a `String`). The `onTimer(key)` method needs to be overridden and it will be called once the timer of `key` +fires. It is possible to cancel a timer using `cancelTimer(key)` and check the status of a timer with +`isTimerActive(key)`. Timers will be automatically cleaned up when the stage completes. Timers can not be scheduled from the constructor of the logic, but it is possible to schedule them from the -``preStart()`` lifecycle hook. +`preStart()` lifecycle hook. In this sample the stage toggles between open and closed, where open means no elements are passed through. The stage starts out as closed but as soon as an element is pushed downstream the gate becomes open for a duration of time during which it will consume and drop upstream messages: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#timed +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #timed } +### Using asynchronous side-channels -Using asynchronous side-channels --------------------------------- In order to receive asynchronous events that are not arriving as stream elements (for example a completion of a future -or a callback from a 3rd party API) one must acquire a :class:`AsyncCallback` by calling ``getAsyncCallback()`` from the -stage logic. The method ``getAsyncCallback`` takes as a parameter a callback that will be called once the asynchronous +or a callback from a 3rd party API) one must acquire a `AsyncCallback` by calling `getAsyncCallback()` from the +stage logic. The method `getAsyncCallback` takes as a parameter a callback that will be called once the asynchronous event fires. It is important to **not call the callback directly**, instead, the external API must call the -``invoke(event)`` method on the returned :class:`AsyncCallback`. The execution engine will take care of calling the -provided callback in a thread-safe way. The callback can safely access the state of the :class:`GraphStageLogic` +`invoke(event)` method on the returned `AsyncCallback`. The execution engine will take care of calling the +provided callback in a thread-safe way. The callback can safely access the state of the `GraphStageLogic` implementation. Sharing the AsyncCallback from the constructor risks race conditions, therefore it is recommended to use the -``preStart()`` lifecycle hook instead. - +`preStart()` lifecycle hook instead. This example shows an asynchronous side channel graph stage that starts dropping elements when a future completes: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#async-side-channel +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #async-side-channel } -Integration with actors ------------------------ +### Integration with actors **This section is a stub and will be extended in the next release** **This is a :ref:`may change ` feature*** It is possible to acquire an ActorRef that can be addressed from the outside of the stage, similarly how -:class:`AsyncCallback` allows injecting asynchronous events into a stage logic. This reference can be obtained -by calling ``getStageActorRef(receive)`` passing in a function that takes a :class:`Pair` of the sender -:class:`ActorRef` and the received message. This reference can be used to watch other actors by calling its ``watch(ref)`` -or ``unwatch(ref)`` methods. The reference can be also watched by external actors. The current limitations of this -:class:`ActorRef` are: +`AsyncCallback` allows injecting asynchronous events into a stage logic. This reference can be obtained +by calling `getStageActorRef(receive)` passing in a function that takes a `Pair` of the sender +`ActorRef` and the received message. This reference can be used to watch other actors by calling its `watch(ref)` +or `unwatch(ref)` methods. The reference can be also watched by external actors. The current limitations of this +`ActorRef` are: - - they are not location transparent, they cannot be accessed via remoting. - - they cannot be returned as materialized values. - - they cannot be accessed from the constructor of the :class:`GraphStageLogic`, but they can be accessed from the - ``preStart()`` method. +> + * they are not location transparent, they cannot be accessed via remoting. + * they cannot be returned as materialized values. + * they cannot be accessed from the constructor of the `GraphStageLogic`, but they can be accessed from the +`preStart()` method. +### Custom materialized values -Custom materialized values --------------------------- - -Custom stages can return materialized values instead of ``NotUsed`` by inheriting from :class:`GraphStageWithMaterializedValue` -instead of the simpler :class:`GraphStage`. The difference is that in this case the method -``createLogicAndMaterializedValue(inheritedAttributes)`` needs to be overridden, and in addition to the +Custom stages can return materialized values instead of `NotUsed` by inheriting from `GraphStageWithMaterializedValue` +instead of the simpler `GraphStage`. The difference is that in this case the method +`createLogicAndMaterializedValue(inheritedAttributes)` needs to be overridden, and in addition to the stage logic the materialized value must be provided -.. warning:: - There is no built-in synchronization of accessing this value from both of the thread where the logic runs and - the thread that got hold of the materialized value. It is the responsibility of the programmer to add the - necessary (non-blocking) synchronization and visibility guarantees to this shared object. +@@@ warning + +There is no built-in synchronization of accessing this value from both of the thread where the logic runs and +the thread that got hold of the materialized value. It is the responsibility of the programmer to add the +necessary (non-blocking) synchronization and visibility guarantees to this shared object. + +@@@ In this sample the materialized value is a future containing the first element to go through the stream: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#materialized +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #materialized } - -Using attributes to affect the behavior of a stage --------------------------------------------------- +### Using attributes to affect the behavior of a stage **This section is a stub and will be extended in the next release** -Stages can access the :class:`Attributes` object created by the materializer. This contains all the applied (inherited) +Stages can access the `Attributes` object created by the materializer. This contains all the applied (inherited) attributes applying to the stage, ordered from least specific (outermost) towards the most specific (innermost) attribute. It is the responsibility of the stage to decide how to reconcile this inheritance chain to a final effective decision. -See :ref:`composition-scala` for an explanation on how attributes work. +See @ref:[Modularity, Composition and Hierarchy](stream-composition.md) for an explanation on how attributes work. - - - -Rate decoupled graph stages ---------------------------- +### Rate decoupled graph stages Sometimes it is desirable to *decouple* the rate of the upstream and downstream of a stage, synchronizing only when needed. -This is achieved in the model by representing a :class:`GraphStage` as a *boundary* between two regions where the +This is achieved in the model by representing a `GraphStage` as a *boundary* between two regions where the demand sent upstream is decoupled from the demand that arrives from downstream. One immediate consequence of this -difference is that an ``onPush`` call does not always lead to calling ``push`` and an ``onPull`` call does not always -lead to calling ``pull``. +difference is that an `onPush` call does not always lead to calling `push` and an `onPull` call does not always +lead to calling `pull`. One of the important use-case for this is to build buffer-like entities, that allow independent progress of upstream and downstream stages when the buffer is not full or empty, and slowing down the appropriate side if the @@ -436,91 +405,86 @@ is seen from downstream. | -.. image:: ../../images/graph_stage_detached_tracks_1.png - :align: center - :width: 500 +![graph_stage_detached_tracks_1.png](../../images/graph_stage_detached_tracks_1.png) | Another scenario would be where the demand from downstream starts coming in before any element is pushed into the buffer stage. +| + +![graph_stage_detached_tracks_2.png](../../images/graph_stage_detached_tracks_2.png) | -.. image:: ../../images/graph_stage_detached_tracks_2.png - :align: center - :width: 500 - -| - - -The first difference we can notice is that our ``Buffer`` stage is automatically pulling its upstream on +The first difference we can notice is that our `Buffer` stage is automatically pulling its upstream on initialization. The buffer has demand for up to two elements without any downstream demand. The following code example demonstrates a buffer class corresponding to the message sequence chart above. -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#detached +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #detached } - -Thread safety of custom processing stages -========================================= +## Thread safety of custom processing stages All of the above custom stages (linear or graph) provide a few simple guarantees that implementors can rely on. - - The callbacks exposed by all of these classes are never called concurrently. - - The state encapsulated by these classes can be safely modified from the provided callbacks, without any further - synchronization. +: + * The callbacks exposed by all of these classes are never called concurrently. + * The state encapsulated by these classes can be safely modified from the provided callbacks, without any further +synchronization. -In essence, the above guarantees are similar to what :class:`Actor` s provide, if one thinks of the state of a custom -stage as state of an actor, and the callbacks as the ``receive`` block of the actor. -.. warning:: - It is **not safe** to access the state of any custom stage outside of the callbacks that it provides, just like it - is unsafe to access the state of an actor from the outside. This means that Future callbacks should **not close over** - internal state of custom stages because such access can be concurrent with the provided callbacks, leading to undefined - behavior. +In essence, the above guarantees are similar to what `Actor` s provide, if one thinks of the state of a custom +stage as state of an actor, and the callbacks as the `receive` block of the actor. -Resources and the stage lifecycle -================================= +@@@ warning + +It is **not safe** to access the state of any custom stage outside of the callbacks that it provides, just like it +is unsafe to access the state of an actor from the outside. This means that Future callbacks should **not close over** +internal state of custom stages because such access can be concurrent with the provided callbacks, leading to undefined +behavior. + +@@@ + +## Resources and the stage lifecycle If a stage manages a resource with a lifecycle, for example objects that need to be shutdown when they are not used anymore it is important to make sure this will happen in all circumstances when the stage shuts down. -Cleaning up resources should be done in ``GraphStageLogic.postStop`` and not in the ``InHandler`` and ``OutHandler`` +Cleaning up resources should be done in `GraphStageLogic.postStop` and not in the `InHandler` and `OutHandler` callbacks. The reason for this is that when the stage itself completes or is failed there is no signal from the upstreams or the downstreams. Even for stages that do not complete or fail in this manner, this can happen when the -``Materializer`` is shutdown or the ``ActorSystem`` is terminated while a stream is still running, what is called an +`Materializer` is shutdown or the `ActorSystem` is terminated while a stream is still running, what is called an "abrupt termination". -Extending Flow Combinators with Custom Operators -================================================ +## Extending Flow Combinators with Custom Operators -The most general way of extending any :class:`Source`, :class:`Flow` or :class:`SubFlow` (e.g. from ``groupBy``) is -demonstrated above: create a graph of flow-shape like the :class:`Duplicator` example given above and use the ``.via(...)`` -combinator to integrate it into your stream topology. This works with all :class:`FlowOps` sub-types, including the +The most general way of extending any `Source`, `Flow` or `SubFlow` (e.g. from `groupBy`) is +demonstrated above: create a graph of flow-shape like the `Duplicator` example given above and use the `.via(...)` +combinator to integrate it into your stream topology. This works with all `FlowOps` sub-types, including the ports that you connect with the graph DSL. -Advanced Scala users may wonder whether it is possible to write extension methods that enrich :class:`FlowOps` to +Advanced Scala users may wonder whether it is possible to write extension methods that enrich `FlowOps` to allow nicer syntax. The short answer is that Scala 2 does not support this in a fully generic fashion, the problem is -that it is impossible to abstract over the kind of stream that is being extended because :class:`Source`, :class:`Flow` -and :class:`SubFlow` differ in the number and kind of their type parameters. While it would be possible to write +that it is impossible to abstract over the kind of stream that is being extended because `Source`, `Flow` +and `SubFlow` differ in the number and kind of their type parameters. While it would be possible to write an implicit class that enriches them generically, this class would require explicit instantiation with all type -parameters due to `SI-2712 `_. For a partial workaround that unifies -extensions to :class:`Source` and :class:`Flow` see `this sketch by R. Kuhn `_. +parameters due to [SI-2712](https://issues.scala-lang.org/browse/SI-2712). For a partial workaround that unifies +extensions to `Source` and `Flow` see [this sketch by R. Kuhn](https://gist.github.com/rkuhn/2870fcee4937dda2cad5). -A lot simpler is the task of just adding an extension method to :class:`Source` as shown below: +A lot simpler is the task of just adding an extension method to `Source` as shown below: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#extending-source +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #extending-source } -The analog works for :class:`Flow` as well: +The analog works for `Flow` as well: -.. includecode:: ../code/docs/stream/GraphStageDocSpec.scala#extending-flow +@@snip [GraphStageDocSpec.scala](../code/docs/stream/GraphStageDocSpec.scala) { #extending-flow } -If you try to write this for :class:`SubFlow`, though, you will run into the same issue as when trying to unify +If you try to write this for `SubFlow`, though, you will run into the same issue as when trying to unify the two solutions above, only on a higher level (the type constructors needed for that unification would have rank two, meaning that some of their type arguments are type constructors themselves—when trying to extend the solution shown in the linked sketch the author encountered such a density of compiler StackOverflowErrors and IDE failures that he gave up). -It is interesting to note that a simplified form of this problem has found its way into the `dotty test suite `_. -Dotty is the development version of Scala on its way to Scala 3. +It is interesting to note that a simplified form of this problem has found its way into the [dotty test suite](https://github.com/lampepfl/dotty/pull/1186/files). +Dotty is the development version of Scala on its way to Scala 3. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-dynamic.md b/akka-docs/src/main/paradox/scala/stream/stream-dynamic.md index 357d0e4c0b..a46844d637 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-dynamic.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-dynamic.md @@ -1,140 +1,127 @@ -.. _stream-dynamic-scala: +# Dynamic stream handling -####################### -Dynamic stream handling -####################### + +## Controlling graph completion with KillSwitch -.. _kill-switch-scala: +A `KillSwitch` allows the completion of graphs of `FlowShape` from the outside. It consists of a flow element that +can be linked to a graph of `FlowShape` needing completion control. +The `KillSwitch` trait allows to complete or fail the graph(s). -Controlling graph completion with KillSwitch --------------------------------------------- +@@snip [KillSwitch.scala]../../../../../../akka-stream/src/main/scala/akka/stream/KillSwitch.scala) { #kill-switch } -A ``KillSwitch`` allows the completion of graphs of ``FlowShape`` from the outside. It consists of a flow element that -can be linked to a graph of ``FlowShape`` needing completion control. -The ``KillSwitch`` trait allows to complete or fail the graph(s). - -.. includecode:: ../../../../akka-stream/src/main/scala/akka/stream/KillSwitch.scala - :include: kill-switch - -After the first call to either ``shutdown`` or ``abort``, all subsequent calls to any of these methods will be ignored. +After the first call to either `shutdown` or `abort`, all subsequent calls to any of these methods will be ignored. Graph completion is performed by both -* completing its downstream -* cancelling (in case of ``shutdown``) or failing (in case of ``abort``) its upstream. + * completing its downstream + * cancelling (in case of `shutdown`) or failing (in case of `abort`) its upstream. -A ``KillSwitch`` can control the completion of one or multiple streams, and therefore comes in two different flavours. +A `KillSwitch` can control the completion of one or multiple streams, and therefore comes in two different flavours. -.. _unique-kill-switch-scala: + +### UniqueKillSwitch -UniqueKillSwitch -^^^^^^^^^^^^^^^^ - -``UniqueKillSwitch`` allows to control the completion of **one** materialized ``Graph`` of ``FlowShape``. Refer to the +`UniqueKillSwitch` allows to control the completion of **one** materialized `Graph` of `FlowShape`. Refer to the below for usage examples. -* **Shutdown** + * **Shutdown** -.. includecode:: ../code/docs/stream/KillSwitchDocSpec.scala#unique-shutdown +@@snip [KillSwitchDocSpec.scala](../code/docs/stream/KillSwitchDocSpec.scala) { #unique-shutdown } -* **Abort** + * **Abort** -.. includecode:: ../code/docs/stream/KillSwitchDocSpec.scala#unique-abort +@@snip [KillSwitchDocSpec.scala](../code/docs/stream/KillSwitchDocSpec.scala) { #unique-abort } -.. _shared-kill-switch-scala: + +### SharedKillSwitch -SharedKillSwitch -^^^^^^^^^^^^^^^^ - -A ``SharedKillSwitch`` allows to control the completion of an arbitrary number graphs of ``FlowShape``. It can be -materialized multiple times via its ``flow`` method, and all materialized graphs linked to it are controlled by the switch. +A `SharedKillSwitch` allows to control the completion of an arbitrary number graphs of `FlowShape`. It can be +materialized multiple times via its `flow` method, and all materialized graphs linked to it are controlled by the switch. Refer to the below for usage examples. -* **Shutdown** + * **Shutdown** -.. includecode:: ../code/docs/stream/KillSwitchDocSpec.scala#shared-shutdown +@@snip [KillSwitchDocSpec.scala](../code/docs/stream/KillSwitchDocSpec.scala) { #shared-shutdown } -* **Abort** + * **Abort** -.. includecode:: ../code/docs/stream/KillSwitchDocSpec.scala#shared-abort +@@snip [KillSwitchDocSpec.scala](../code/docs/stream/KillSwitchDocSpec.scala) { #shared-abort } -.. note:: - A ``UniqueKillSwitch`` is always a result of a materialization, whilst ``SharedKillSwitch`` needs to be constructed - before any materialization takes place. +@@@ note -Dynamic fan-in and fan-out with MergeHub and BroadcastHub ---------------------------------------------------------- +A `UniqueKillSwitch` is always a result of a materialization, whilst `SharedKillSwitch` needs to be constructed +before any materialization takes place. + +@@@ + +## Dynamic fan-in and fan-out with MergeHub and BroadcastHub There are many cases when consumers or producers of a certain service (represented as a Sink, Source, or possibly Flow) are dynamic and not known in advance. The Graph DSL does not allow to represent this, all connections of the graph must be known in advance and must be connected upfront. To allow dynamic fan-in and fan-out streaming, the Hubs -should be used. They provide means to construct :class:`Sink` and :class:`Source` pairs that are "attached" to each +should be used. They provide means to construct `Sink` and `Source` pairs that are "attached" to each other, but one of them can be materialized multiple times to implement dynamic fan-in or fan-out. -Using the MergeHub -^^^^^^^^^^^^^^^^^^ +### Using the MergeHub -A :class:`MergeHub` allows to implement a dynamic fan-in junction point in a graph where elements coming from +A `MergeHub` allows to implement a dynamic fan-in junction point in a graph where elements coming from different producers are emitted in a First-Comes-First-Served fashion. If the consumer cannot keep up then *all* of the -producers are backpressured. The hub itself comes as a :class:`Source` to which the single consumer can be attached. -It is not possible to attach any producers until this :class:`Source` has been materialized (started). This is ensured -by the fact that we only get the corresponding :class:`Sink` as a materialized value. Usage might look like this: +producers are backpressured. The hub itself comes as a `Source` to which the single consumer can be attached. +It is not possible to attach any producers until this `Source` has been materialized (started). This is ensured +by the fact that we only get the corresponding `Sink` as a materialized value. Usage might look like this: -.. includecode:: ../code/docs/stream/HubsDocSpec.scala#merge-hub +@@snip [HubsDocSpec.scala](../code/docs/stream/HubsDocSpec.scala) { #merge-hub } -This sequence, while might look odd at first, ensures proper startup order. Once we get the :class:`Sink`, +This sequence, while might look odd at first, ensures proper startup order. Once we get the `Sink`, we can use it as many times as wanted. Everything that is fed to it will be delivered to the consumer we attached previously until it cancels. -Using the BroadcastHub -^^^^^^^^^^^^^^^^^^^^^^ +### Using the BroadcastHub -A :class:`BroadcastHub` can be used to consume elements from a common producer by a dynamic set of consumers. The -rate of the producer will be automatically adapted to the slowest consumer. In this case, the hub is a :class:`Sink` -to which the single producer must be attached first. Consumers can only be attached once the :class:`Sink` has -been materialized (i.e. the producer has been started). One example of using the :class:`BroadcastHub`: +A `BroadcastHub` can be used to consume elements from a common producer by a dynamic set of consumers. The +rate of the producer will be automatically adapted to the slowest consumer. In this case, the hub is a `Sink` +to which the single producer must be attached first. Consumers can only be attached once the `Sink` has +been materialized (i.e. the producer has been started). One example of using the `BroadcastHub`: -.. includecode:: ../code/docs/stream/HubsDocSpec.scala#broadcast-hub +@@snip [HubsDocSpec.scala](../code/docs/stream/HubsDocSpec.scala) { #broadcast-hub } -The resulting :class:`Source` can be materialized any number of times, each materialization effectively attaching +The resulting `Source` can be materialized any number of times, each materialization effectively attaching a new subscriber. If there are no subscribers attached to this hub then it will not drop any elements but instead backpressure the upstream producer until subscribers arrive. This behavior can be tweaked by using the combinators -``.buffer`` for example with a drop strategy, or just attaching a subscriber that drops all messages. If there +`.buffer` for example with a drop strategy, or just attaching a subscriber that drops all messages. If there are no other subscribers, this will ensure that the producer is kept drained (dropping all elements) and once a new subscriber arrives it will adaptively slow down, ensuring no more messages are dropped. -Combining dynamic stages to build a simple Publish-Subscribe service -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Combining dynamic stages to build a simple Publish-Subscribe service The features provided by the Hub implementations are limited by default. This is by design, as various combinations can be used to express additional features like unsubscribing producers or consumers externally. We show here -an example that builds a :class:`Flow` representing a publish-subscribe channel. The input of the :class:`Flow` is +an example that builds a `Flow` representing a publish-subscribe channel. The input of the `Flow` is published to all subscribers while the output streams all the elements published. -First, we connect a :class:`MergeHub` and a :class:`BroadcastHub` together to form a publish-subscribe channel. Once -we materialize this small stream, we get back a pair of :class:`Source` and :class:`Sink` that together define +First, we connect a `MergeHub` and a `BroadcastHub` together to form a publish-subscribe channel. Once +we materialize this small stream, we get back a pair of `Source` and `Sink` that together define the publish and subscribe sides of our channel. -.. includecode:: ../code/docs/stream/HubsDocSpec.scala#pub-sub-1 +@@snip [HubsDocSpec.scala](../code/docs/stream/HubsDocSpec.scala) { #pub-sub-1 } -We now use a few tricks to add more features. First of all, we attach a ``Sink.ignore`` +We now use a few tricks to add more features. First of all, we attach a `Sink.ignore` at the broadcast side of the channel to keep it drained when there are no subscribers. If this behavior is not the desired one this line can be simply dropped. -.. includecode:: ../code/docs/stream/HubsDocSpec.scala#pub-sub-2 +@@snip [HubsDocSpec.scala](../code/docs/stream/HubsDocSpec.scala) { #pub-sub-2 } -We now wrap the :class:`Sink` and :class:`Source` in a :class:`Flow` using ``Flow.fromSinkAndSource``. This bundles +We now wrap the `Sink` and `Source` in a `Flow` using `Flow.fromSinkAndSource`. This bundles up the two sides of the channel into one and forces users of it to always define a publisher and subscriber side -(even if the subscriber side is just dropping). It also allows us to very simply attach a :class:`KillSwitch` as -a :class:`BidiStage` which in turn makes it possible to close both the original :class:`Sink` and :class:`Source` at the +(even if the subscriber side is just dropping). It also allows us to very simply attach a `KillSwitch` as +a `BidiStage` which in turn makes it possible to close both the original `Sink` and `Source` at the same time. -Finally, we add ``backpressureTimeout`` on the consumer side to ensure that subscribers that block the channel for more +Finally, we add `backpressureTimeout` on the consumer side to ensure that subscribers that block the channel for more than 3 seconds are forcefully removed (and their stream failed). -.. includecode:: ../code/docs/stream/HubsDocSpec.scala#pub-sub-3 +@@snip [HubsDocSpec.scala](../code/docs/stream/HubsDocSpec.scala) { #pub-sub-3 } -The resulting Flow now has a type of ``Flow[String, String, UniqueKillSwitch]`` representing a publish-subscribe +The resulting Flow now has a type of `Flow[String, String, UniqueKillSwitch]` representing a publish-subscribe channel which can be used any number of times to attach new producers or consumers. In addition, it materializes -to a :class:`UniqueKillSwitch` (see :ref:`unique-kill-switch-scala`) that can be used to deregister a single user externally: +to a `UniqueKillSwitch` (see [UniqueKillSwitch](#unique-kill-switch-scala)) that can be used to deregister a single user externally: - -.. includecode:: ../code/docs/stream/HubsDocSpec.scala#pub-sub-4 +@@snip [HubsDocSpec.scala](../code/docs/stream/HubsDocSpec.scala) { #pub-sub-4 } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-error.md b/akka-docs/src/main/paradox/scala/stream/stream-error.md index 6333d447dc..30a259837b 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-error.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-error.md @@ -1,79 +1,76 @@ -.. _stream-error-scala: - -############## -Error Handling -############## +# Error Handling Strategies for how to handle exceptions from processing stream elements can be defined when materializing the stream. The error handling strategies are inspired by actor supervision strategies, but the semantics have been adapted to the domain of stream processing. -.. warning:: +@@@ warning - *ZipWith*, *GraphStage* junction, *ActorPublisher* source and *ActorSubscriber* sink - components do not honour the supervision strategy attribute yet. +*ZipWith*, *GraphStage* junction, *ActorPublisher* source and *ActorSubscriber* sink +components do not honour the supervision strategy attribute yet. -Supervision Strategies -====================== +@@@ + +## Supervision Strategies There are three ways to handle exceptions from application code: -* ``Stop`` - The stream is completed with failure. -* ``Resume`` - The element is dropped and the stream continues. -* ``Restart`` - The element is dropped and the stream continues after restarting the stage. - Restarting a stage means that any accumulated state is cleared. This is typically - performed by creating a new instance of the stage. - + * `Stop` - The stream is completed with failure. + * `Resume` - The element is dropped and the stream continues. + * `Restart` - The element is dropped and the stream continues after restarting the stage. +Restarting a stage means that any accumulated state is cleared. This is typically +performed by creating a new instance of the stage. By default the stopping strategy is used for all exceptions, i.e. the stream will be completed with failure when an exception is thrown. -.. includecode:: ../code/docs/stream/FlowErrorDocSpec.scala#stop +@@snip [FlowErrorDocSpec.scala](../code/docs/stream/FlowErrorDocSpec.scala) { #stop } The default supervision strategy for a stream can be defined on the settings of the materializer. -.. includecode:: ../code/docs/stream/FlowErrorDocSpec.scala#resume +@@snip [FlowErrorDocSpec.scala](../code/docs/stream/FlowErrorDocSpec.scala) { #resume } -Here you can see that all ``ArithmeticException`` will resume the processing, i.e. the +Here you can see that all `ArithmeticException` will resume the processing, i.e. the elements that cause the division by zero are effectively dropped. -.. note:: +@@@ note - Be aware that dropping elements may result in deadlocks in graphs with - cycles, as explained in :ref:`graph-cycles-scala`. +Be aware that dropping elements may result in deadlocks in graphs with +cycles, as explained in @ref:[Graph cycles, liveness and deadlocks](stream-graphs.md#graph-cycles-scala). + +@@@ The supervision strategy can also be defined for all operators of a flow. -.. includecode:: ../code/docs/stream/FlowErrorDocSpec.scala#resume-section +@@snip [FlowErrorDocSpec.scala](../code/docs/stream/FlowErrorDocSpec.scala) { #resume-section } -``Restart`` works in a similar way as ``Resume`` with the addition that accumulated state, +`Restart` works in a similar way as `Resume` with the addition that accumulated state, if any, of the failing processing stage will be reset. -.. includecode:: ../code/docs/stream/FlowErrorDocSpec.scala#restart-section +@@snip [FlowErrorDocSpec.scala](../code/docs/stream/FlowErrorDocSpec.scala) { #restart-section } -Errors from mapAsync -==================== +## Errors from mapAsync -Stream supervision can also be applied to the futures of ``mapAsync``. +Stream supervision can also be applied to the futures of `mapAsync`. Let's say that we use an external service to lookup email addresses and we would like to discard those that cannot be found. We start with the tweet stream of authors: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#tweet-authors +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #tweet-authors } Assume that we can lookup their email address using: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#email-address-lookup2 +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #email-address-lookup2 } -The ``Future`` is completed with ``Failure`` if the email is not found. +The `Future` is completed with `Failure` if the email is not found. -Transforming the stream of authors to a stream of email addresses by using the ``lookupEmail`` -service can be done with ``mapAsync`` and we use ``Supervision.resumingDecider`` to drop +Transforming the stream of authors to a stream of email addresses by using the `lookupEmail` +service can be done with `mapAsync` and we use `Supervision.resumingDecider` to drop unknown email addresses: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#email-addresses-mapAsync-supervision +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #email-addresses-mapAsync-supervision } -If we would not use ``Resume`` the default stopping strategy would complete the stream -with failure on the first ``Future`` that was completed with ``Failure``. +If we would not use `Resume` the default stopping strategy would complete the stream +with failure on the first `Future` that was completed with `Failure`. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-flows-and-basics.md b/akka-docs/src/main/paradox/scala/stream/stream-flows-and-basics.md index 7b70cfc1d8..8fc5c9076d 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-flows-and-basics.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-flows-and-basics.md @@ -1,13 +1,7 @@ -.. _stream-flow-scala: +# Basics and working with Flows -############################# -Basics and working with Flows -############################# - -.. _core-concepts-scala: - -Core concepts -============= + +## Core concepts Akka Streams is a library to process and transfer a sequence of elements using bounded buffer space. This latter property is what we refer to as *boundedness* and it is the defining feature of Akka Streams. Translated to @@ -20,24 +14,30 @@ do not drop. Before we move on, let's define some basic terminology which will be used throughout the entire documentation: Stream - An active process that involves moving and transforming data. +: An active process that involves moving and transforming data. + Element - An element is the processing unit of streams. All operations transform and transfer elements from upstream to - downstream. Buffer sizes are always expressed as number of elements independently from the actual size of the elements. +: An element is the processing unit of streams. All operations transform and transfer elements from upstream to +downstream. Buffer sizes are always expressed as number of elements independently from the actual size of the elements. + Back-pressure - A means of flow-control, a way for consumers of data to notify a producer about their current availability, effectively - slowing down the upstream producer to match their consumption speeds. - In the context of Akka Streams back-pressure is always understood as *non-blocking* and *asynchronous*. +: A means of flow-control, a way for consumers of data to notify a producer about their current availability, effectively +slowing down the upstream producer to match their consumption speeds. +In the context of Akka Streams back-pressure is always understood as *non-blocking* and *asynchronous*. + Non-Blocking - Means that a certain operation does not hinder the progress of the calling thread, even if it takes long time to - finish the requested operation. +: Means that a certain operation does not hinder the progress of the calling thread, even if it takes long time to +finish the requested operation. + Graph - A description of a stream processing topology, defining the pathways through which elements shall flow when the stream - is running. +: A description of a stream processing topology, defining the pathways through which elements shall flow when the stream +is running. + Processing Stage - The common name for all building blocks that build up a Graph. - Examples of a processing stage would be operations like ``map()``, ``filter()``, custom ``GraphStage`` s and graph - junctions like ``Merge`` or ``Broadcast``. For the full list of built-in processing stages see :ref:`stages-overview_scala` +: The common name for all building blocks that build up a Graph. +Examples of a processing stage would be operations like `map()`, `filter()`, custom `GraphStage` s and graph +junctions like `Merge` or `Broadcast`. For the full list of built-in processing stages see @ref:[stages-overview_scala](stages-overview.md) + When we talk about *asynchronous, non-blocking backpressure* we mean that the processing stages available in Akka Streams will not use blocking calls but asynchronous message passing to exchange messages between each other, and they @@ -45,123 +45,124 @@ will use asynchronous means to slow down a fast producer, without blocking its t design, since entities that need to wait (a fast producer waiting on a slow consumer) will not block the thread but can hand it back for further use to an underlying thread-pool. -.. _defining-and-running-streams-scala: - -Defining and running streams -============================ + +## Defining and running streams Linear processing pipelines can be expressed in Akka Streams using the following core abstractions: Source - A processing stage with *exactly one output*, emitting data elements whenever downstream processing stages are - ready to receive them. +: A processing stage with *exactly one output*, emitting data elements whenever downstream processing stages are +ready to receive them. + Sink - A processing stage with *exactly one input*, requesting and accepting data elements possibly slowing down the upstream - producer of elements +: A processing stage with *exactly one input*, requesting and accepting data elements possibly slowing down the upstream +producer of elements + Flow - A processing stage which has *exactly one input and output*, which connects its up- and downstreams by - transforming the data elements flowing through it. +: A processing stage which has *exactly one input and output*, which connects its up- and downstreams by +transforming the data elements flowing through it. + RunnableGraph - A Flow that has both ends "attached" to a Source and Sink respectively, and is ready to be ``run()``. +: A Flow that has both ends "attached" to a Source and Sink respectively, and is ready to be `run()`. -It is possible to attach a ``Flow`` to a ``Source`` resulting in a composite source, and it is also possible to prepend -a ``Flow`` to a ``Sink`` to get a new sink. After a stream is properly terminated by having both a source and a sink, -it will be represented by the ``RunnableGraph`` type, indicating that it is ready to be executed. -It is important to remember that even after constructing the ``RunnableGraph`` by connecting all the source, sink and +It is possible to attach a `Flow` to a `Source` resulting in a composite source, and it is also possible to prepend +a `Flow` to a `Sink` to get a new sink. After a stream is properly terminated by having both a source and a sink, +it will be represented by the `RunnableGraph` type, indicating that it is ready to be executed. + +It is important to remember that even after constructing the `RunnableGraph` by connecting all the source, sink and different processing stages, no data will flow through it until it is materialized. Materialization is the process of allocating all resources needed to run the computation described by a Graph (in Akka Streams this will often involve starting up Actors). Thanks to Flows being simply a description of the processing pipeline they are *immutable, thread-safe, and freely shareable*, which means that it is for example safe to share and send them between actors, to have one actor prepare the work, and then have it be materialized at some completely different place in the code. -.. includecode:: ../code/docs/stream/FlowDocSpec.scala#materialization-in-steps +@@snip [FlowDocSpec.scala](../code/docs/stream/FlowDocSpec.scala) { #materialization-in-steps } -After running (materializing) the ``RunnableGraph[T]`` we get back the materialized value of type T. Every stream processing +After running (materializing) the `RunnableGraph[T]` we get back the materialized value of type T. Every stream processing stage can produce a materialized value, and it is the responsibility of the user to combine them to a new type. -In the above example we used ``toMat`` to indicate that we want to transform the materialized value of the source and -sink, and we used the convenience function ``Keep.right`` to say that we are only interested in the materialized value +In the above example we used `toMat` to indicate that we want to transform the materialized value of the source and +sink, and we used the convenience function `Keep.right` to say that we are only interested in the materialized value of the sink. -In our example the ``FoldSink`` materializes a value of type ``Future`` which will represent the result +In our example the `FoldSink` materializes a value of type `Future` which will represent the result of the folding process over the stream. In general, a stream can expose multiple materialized values, but it is quite common to be interested in only the value of the Source or the Sink in the stream. For this reason -there is a convenience method called ``runWith()`` available for ``Sink``, ``Source`` or ``Flow`` requiring, respectively, -a supplied ``Source`` (in order to run a ``Sink``), a ``Sink`` (in order to run a ``Source``) or -both a ``Source`` and a ``Sink`` (in order to run a ``Flow``, since it has neither attached yet). +there is a convenience method called `runWith()` available for `Sink`, `Source` or `Flow` requiring, respectively, +a supplied `Source` (in order to run a `Sink`), a `Sink` (in order to run a `Source`) or +both a `Source` and a `Sink` (in order to run a `Flow`, since it has neither attached yet). -.. includecode:: ../code/docs/stream/FlowDocSpec.scala#materialization-runWith +@@snip [FlowDocSpec.scala](../code/docs/stream/FlowDocSpec.scala) { #materialization-runWith } It is worth pointing out that since processing stages are *immutable*, connecting them returns a new processing stage, instead of modifying the existing instance, so while constructing long flows, remember to assign the new value to a variable or run it: -.. includecode:: ../code/docs/stream/FlowDocSpec.scala#source-immutable +@@snip [FlowDocSpec.scala](../code/docs/stream/FlowDocSpec.scala) { #source-immutable } -.. note:: - By default Akka Streams elements support **exactly one** downstream processing stage. - Making fan-out (supporting multiple downstream processing stages) an explicit opt-in feature allows default stream elements to - be less complex and more efficient. Also it allows for greater flexibility on *how exactly* to handle the multicast scenarios, - by providing named fan-out elements such as broadcast (signals all down-stream elements) or balance (signals one of available down-stream elements). +@@@ note -In the above example we used the ``runWith`` method, which both materializes the stream and returns the materialized value +By default Akka Streams elements support **exactly one** downstream processing stage. +Making fan-out (supporting multiple downstream processing stages) an explicit opt-in feature allows default stream elements to +be less complex and more efficient. Also it allows for greater flexibility on *how exactly* to handle the multicast scenarios, +by providing named fan-out elements such as broadcast (signals all down-stream elements) or balance (signals one of available down-stream elements). + +@@@ + +In the above example we used the `runWith` method, which both materializes the stream and returns the materialized value of the given sink or source. Since a stream can be materialized multiple times, the materialized value will also be calculated anew for each such materialization, usually leading to different values being returned each time. -In the example below we create two running materialized instance of the stream that we described in the ``runnable`` -variable, and both materializations give us a different ``Future`` from the map even though we used the same ``sink`` +In the example below we create two running materialized instance of the stream that we described in the `runnable` +variable, and both materializations give us a different `Future` from the map even though we used the same `sink` to refer to the future: -.. includecode:: ../code/docs/stream/FlowDocSpec.scala#stream-reuse +@@snip [FlowDocSpec.scala](../code/docs/stream/FlowDocSpec.scala) { #stream-reuse } -Defining sources, sinks and flows ---------------------------------- +### Defining sources, sinks and flows -The objects :class:`Source` and :class:`Sink` define various ways to create sources and sinks of elements. The following +The objects `Source` and `Sink` define various ways to create sources and sinks of elements. The following examples show some of the most useful constructs (refer to the API documentation for more details): -.. includecode:: ../code/docs/stream/FlowDocSpec.scala#source-sink +@@snip [FlowDocSpec.scala](../code/docs/stream/FlowDocSpec.scala) { #source-sink } There are various ways to wire up different parts of a stream, the following examples show some of the available options: -.. includecode:: ../code/docs/stream/FlowDocSpec.scala#flow-connecting +@@snip [FlowDocSpec.scala](../code/docs/stream/FlowDocSpec.scala) { #flow-connecting } -Illegal stream elements ------------------------ +### Illegal stream elements -In accordance to the Reactive Streams specification (`Rule 2.13 `_) -Akka Streams do not allow ``null`` to be passed through the stream as an element. In case you want to model the concept -of absence of a value we recommend using ``scala.Option`` or ``scala.util.Either``. +In accordance to the Reactive Streams specification ([Rule 2.13](https://github.com/reactive-streams/reactive-streams-jvm#2.13)) +Akka Streams do not allow `null` to be passed through the stream as an element. In case you want to model the concept +of absence of a value we recommend using `scala.Option` or `scala.util.Either`. -.. _back-pressure-explained-scala: + +## Back-pressure explained -Back-pressure explained -======================= - -Akka Streams implement an asynchronous non-blocking back-pressure protocol standardised by the `Reactive Streams`_ +Akka Streams implement an asynchronous non-blocking back-pressure protocol standardised by the [Reactive Streams](http://reactive-streams.org/) specification, which Akka is a founding member of. -.. _Reactive Streams: http://reactive-streams.org/ - The user of the library does not have to write any explicit back-pressure handling code — it is built in and dealt with automatically by all of the provided Akka Streams processing stages. It is possible however to add explicit buffer stages with overflow strategies that can influence the behaviour of the stream. This is especially important in complex processing graphs which may even contain loops (which *must* be treated with very special -care, as explained in :ref:`graph-cycles-scala`). +care, as explained in @ref:[Graph cycles, liveness and deadlocks](stream-graphs.md#graph-cycles-scala)). -The back pressure protocol is defined in terms of the number of elements a downstream ``Subscriber`` is able to receive -and buffer, referred to as ``demand``. -The source of data, referred to as ``Publisher`` in Reactive Streams terminology and implemented as ``Source`` in Akka -Streams, guarantees that it will never emit more elements than the received total demand for any given ``Subscriber``. +The back pressure protocol is defined in terms of the number of elements a downstream `Subscriber` is able to receive +and buffer, referred to as `demand`. +The source of data, referred to as `Publisher` in Reactive Streams terminology and implemented as `Source` in Akka +Streams, guarantees that it will never emit more elements than the received total demand for any given `Subscriber`. -.. note:: +@@@ note - The Reactive Streams specification defines its protocol in terms of ``Publisher`` and ``Subscriber``. - These types are **not** meant to be user facing API, instead they serve as the low level building blocks for - different Reactive Streams implementations. +The Reactive Streams specification defines its protocol in terms of `Publisher` and `Subscriber`. +These types are **not** meant to be user facing API, instead they serve as the low level building blocks for +different Reactive Streams implementations. - Akka Streams implements these concepts as ``Source``, ``Flow`` (referred to as ``Processor`` in Reactive Streams) - and ``Sink`` without exposing the Reactive Streams interfaces directly. - If you need to integrate with other Reactive Stream libraries read :ref:`reactive-streams-integration-scala`. +Akka Streams implements these concepts as `Source`, `Flow` (referred to as `Processor` in Reactive Streams) +and `Sink` without exposing the Reactive Streams interfaces directly. +If you need to integrate with other Reactive Stream libraries read @ref:[Integrating with Reactive Streams](stream-integrations.md#reactive-streams-integration-scala). + +@@@ The mode in which Reactive Streams back-pressure works can be colloquially described as "dynamic push / pull mode", since it will switch between push and pull based back-pressure models depending on the downstream being able to cope @@ -169,8 +170,7 @@ with the upstream production rate or not. To illustrate this further let us consider both problem situations and how the back-pressure protocol handles them: -Slow Publisher, fast Subscriber -------------------------------- +### Slow Publisher, fast Subscriber This is the happy case of course – we do not need to slow down the Publisher in this case. However signalling rates are rarely constant and could change at any point in time, suddenly ending up in a situation where the Subscriber is now @@ -178,7 +178,7 @@ slower than the Publisher. In order to safeguard from these situations, the back during such situations, however we do not want to pay a high penalty for this safety net being enabled. The Reactive Streams protocol solves this by asynchronously signalling from the Subscriber to the Publisher -``Request(n:Int)`` signals. The protocol guarantees that the Publisher will never signal *more* elements than the +`Request(n:Int)` signals. The protocol guarantees that the Publisher will never signal *more* elements than the signalled demand. Since the Subscriber however is currently faster, it will be signalling these Request messages at a higher rate (and possibly also batching together the demand - requesting multiple elements in one Request signal). This means that the Publisher should not ever have to wait (be back-pressured) with publishing its incoming elements. @@ -186,64 +186,63 @@ that the Publisher should not ever have to wait (be back-pressured) with publish As we can see, in this scenario we effectively operate in so called push-mode since the Publisher can continue producing elements as fast as it can, since the pending demand will be recovered just-in-time while it is emitting elements. -Fast Publisher, slow Subscriber -------------------------------- +### Fast Publisher, slow Subscriber -This is the case when back-pressuring the ``Publisher`` is required, because the ``Subscriber`` is not able to cope with +This is the case when back-pressuring the `Publisher` is required, because the `Subscriber` is not able to cope with the rate at which its upstream would like to emit data elements. -Since the ``Publisher`` is not allowed to signal more elements than the pending demand signalled by the ``Subscriber``, +Since the `Publisher` is not allowed to signal more elements than the pending demand signalled by the `Subscriber`, it will have to abide to this back-pressure by applying one of the below strategies: -- not generate elements, if it is able to control their production rate, -- try buffering the elements in a *bounded* manner until more demand is signalled, -- drop elements until more demand is signalled, -- tear down the stream if unable to apply any of the above strategies. + * not generate elements, if it is able to control their production rate, + * try buffering the elements in a *bounded* manner until more demand is signalled, + * drop elements until more demand is signalled, + * tear down the stream if unable to apply any of the above strategies. -As we can see, this scenario effectively means that the ``Subscriber`` will *pull* the elements from the Publisher – +As we can see, this scenario effectively means that the `Subscriber` will *pull* the elements from the Publisher – this mode of operation is referred to as pull-based back-pressure. -.. _stream-materialization-scala: - -Stream Materialization -====================== + +## Stream Materialization When constructing flows and graphs in Akka Streams think of them as preparing a blueprint, an execution plan. Stream materialization is the process of taking a stream description (the graph) and allocating all the necessary resources it needs in order to run. In the case of Akka Streams this often means starting up Actors which power the processing, but is not restricted to that—it could also mean opening files or socket connections etc.—depending on what the stream needs. -Materialization is triggered at so called "terminal operations". Most notably this includes the various forms of the ``run()`` -and ``runWith()`` methods defined on :class:`Source` and :class:`Flow` elements as well as a small number of special syntactic sugars for running with -well-known sinks, such as ``runForeach(el => ...)`` (being an alias to ``runWith(Sink.foreach(el => ...))``. +Materialization is triggered at so called "terminal operations". Most notably this includes the various forms of the `run()` +and `runWith()` methods defined on `Source` and `Flow` elements as well as a small number of special syntactic sugars for running with +well-known sinks, such as `runForeach(el => ...)` (being an alias to `runWith(Sink.foreach(el => ...))`. Materialization is currently performed synchronously on the materializing thread. The actual stream processing is handled by actors started up during the streams materialization, which will be running on the thread pools they have been configured to run on - which defaults to the dispatcher set in -:class:`MaterializationSettings` while constructing the :class:`ActorMaterializer`. +`MaterializationSettings` while constructing the `ActorMaterializer`. -.. note:: - Reusing *instances* of linear computation stages (Source, Sink, Flow) inside composite Graphs is legal, - yet will materialize that stage multiple times. +@@@ note -.. _operator-fusion-scala: +Reusing *instances* of linear computation stages (Source, Sink, Flow) inside composite Graphs is legal, +yet will materialize that stage multiple times. -Operator Fusion ---------------- +@@@ + + +### Operator Fusion By default Akka Streams will fuse the stream operators. This means that the processing steps of a flow or stream graph can be executed within the same Actor and has two consequences: - * passing elements from one processing stage to the next is a lot faster between fused - stages due to avoiding the asynchronous messaging overhead - * fused stream processing stages does not run in parallel to each other, meaning that - only up to one CPU core is used for each fused part +> + * passing elements from one processing stage to the next is a lot faster between fused +stages due to avoiding the asynchronous messaging overhead + * fused stream processing stages does not run in parallel to each other, meaning that +only up to one CPU core is used for each fused part To allow for parallel processing you will have to insert asynchronous boundaries manually into your flows and -graphs by way of adding ``Attributes.asyncBoundary`` using the method ``async`` on ``Source``, ``Sink`` and ``Flow`` +graphs by way of adding `Attributes.asyncBoundary` using the method `async` on `Source`, `Sink` and `Flow` to pieces that shall communicate with the rest of the graph in an asynchronous fashion. -.. includecode:: ../code/docs/stream/FlowDocSpec.scala#flow-async +@@snip [FlowDocSpec.scala](../code/docs/stream/FlowDocSpec.scala) { #flow-async } In this example we create two regions within the flow which will be executed in one Actor each—assuming that adding and multiplying integers is an extremely costly operation this will lead to a performance gain since two CPUs can @@ -253,9 +252,7 @@ by adding information to the flow graph that has been constructed up to this poi | -.. image:: ../../images/asyncBoundary.png - :align: center - :width: 700 +![asyncBoundary.png](../../images/asyncBoundary.png) | @@ -263,50 +260,52 @@ This means that everything that is inside the red bubble will be executed by one by another. This scheme can be applied successively, always having one such boundary enclose the previous ones plus all processing stages that have been added since them. -.. warning:: +@@@ warning - Without fusing (i.e. up to version 2.0-M2) each stream processing stage had an implicit input buffer - that holds a few elements for efficiency reasons. If your flow graphs contain cycles then these buffers - may have been crucial in order to avoid deadlocks. With fusing these implicit buffers are no longer - there, data elements are passed without buffering between fused stages. In those cases where buffering - is needed in order to allow the stream to run at all, you will have to insert explicit buffers with the - ``.buffer()`` combinator—typically a buffer of size 2 is enough to allow a feedback loop to function. +Without fusing (i.e. up to version 2.0-M2) each stream processing stage had an implicit input buffer +that holds a few elements for efficiency reasons. If your flow graphs contain cycles then these buffers +may have been crucial in order to avoid deadlocks. With fusing these implicit buffers are no longer +there, data elements are passed without buffering between fused stages. In those cases where buffering +is needed in order to allow the stream to run at all, you will have to insert explicit buffers with the +`.buffer()` combinator—typically a buffer of size 2 is enough to allow a feedback loop to function. -The new fusing behavior can be disabled by setting the configuration parameter ``akka.stream.materializer.auto-fusing=off``. +@@@ + +The new fusing behavior can be disabled by setting the configuration parameter `akka.stream.materializer.auto-fusing=off`. In that case you can still manually fuse those graphs which shall run on less Actors. With the exception of the -:class:`SslTlsStage` and the ``groupBy`` operator all built-in processing stages can be fused. +`SslTlsStage` and the `groupBy` operator all built-in processing stages can be fused. -.. _flow-combine-mat-scala: - -Combining materialized values ------------------------------ + +### Combining materialized values Since every processing stage in Akka Streams can provide a materialized value after being materialized, it is necessary to somehow express how these values should be composed to a final value when we plug these stages together. For this, many combinator methods have variants that take an additional argument, a function, that will be used to combine the resulting values. Some examples of using these combiners are illustrated in the example below. -.. includecode:: ../code/docs/stream/FlowDocSpec.scala#flow-mat-combine +@@snip [FlowDocSpec.scala](../code/docs/stream/FlowDocSpec.scala) { #flow-mat-combine } -.. note:: +@@@ note - In Graphs it is possible to access the materialized value from inside the stream processing graph. For details see :ref:`graph-matvalue-scala`. +In Graphs it is possible to access the materialized value from inside the stream processing graph. For details see @ref:[Accessing the materialized value inside the Graph](stream-graphs.md#graph-matvalue-scala). -Stream ordering -=============== -In Akka Streams almost all computation stages *preserve input order* of elements. This means that if inputs ``{IA1,IA2,...,IAn}`` -"cause" outputs ``{OA1,OA2,...,OAk}`` and inputs ``{IB1,IB2,...,IBm}`` "cause" outputs ``{OB1,OB2,...,OBl}`` and all of -``IAi`` happened before all ``IBi`` then ``OAi`` happens before ``OBi``. +@@@ -This property is even uphold by async operations such as ``mapAsync``, however an unordered version exists -called ``mapAsyncUnordered`` which does not preserve this ordering. +## Stream ordering -However, in the case of Junctions which handle multiple input streams (e.g. :class:`Merge`) the output order is, -in general, *not defined* for elements arriving on different input ports. That is a merge-like operation may emit ``Ai`` -before emitting ``Bi``, and it is up to its internal logic to decide the order of emitted elements. Specialized elements -such as ``Zip`` however *do guarantee* their outputs order, as each output element depends on all upstream elements having +In Akka Streams almost all computation stages *preserve input order* of elements. This means that if inputs `{IA1,IA2,...,IAn}` +"cause" outputs `{OA1,OA2,...,OAk}` and inputs `{IB1,IB2,...,IBm}` "cause" outputs `{OB1,OB2,...,OBl}` and all of +`IAi` happened before all `IBi` then `OAi` happens before `OBi`. + +This property is even uphold by async operations such as `mapAsync`, however an unordered version exists +called `mapAsyncUnordered` which does not preserve this ordering. + +However, in the case of Junctions which handle multiple input streams (e.g. `Merge`) the output order is, +in general, *not defined* for elements arriving on different input ports. That is a merge-like operation may emit `Ai` +before emitting `Bi`, and it is up to its internal logic to decide the order of emitted elements. Specialized elements +such as `Zip` however *do guarantee* their outputs order, as each output element depends on all upstream elements having been signalled already – thus the ordering in the case of zipping is defined by this property. If you find yourself in need of fine grained control over order of emitted elements in fan-in -scenarios consider using :class:`MergePreferred` or :class:`GraphStage` – which gives you full control over how the -merge is performed. +scenarios consider using `MergePreferred` or `GraphStage` – which gives you full control over how the +merge is performed. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-graphs.md b/akka-docs/src/main/paradox/scala/stream/stream-graphs.md index 8bfa9611c4..e4983cbf5b 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-graphs.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-graphs.md @@ -1,8 +1,4 @@ -.. _stream-graph-scala: - -################### -Working with Graphs -################### +# Working with Graphs In Akka Streams computation graphs are not expressed using a fluent DSL like linear computations are, instead they are written in a more graph-resembling DSL which aims to make translating graph drawings (e.g. from notes taken @@ -11,89 +7,90 @@ dive into the multiple ways of constructing and re-using graphs, as well as expl Graphs are needed whenever you want to perform any kind of fan-in ("multiple inputs") or fan-out ("multiple outputs") operations. Considering linear Flows to be like roads, we can picture graph operations as junctions: multiple flows being connected at a single point. -Some graph operations which are common enough and fit the linear style of Flows, such as ``concat`` (which concatenates two +Some graph operations which are common enough and fit the linear style of Flows, such as `concat` (which concatenates two streams, such that the second one is consumed after the first one has completed), may have shorthand methods defined on -:class:`Flow` or :class:`Source` themselves, however you should keep in mind that those are also implemented as graph junctions. +`Flow` or `Source` themselves, however you should keep in mind that those are also implemented as graph junctions. -.. _graph-dsl-scala: - -Constructing Graphs -------------------- + +## Constructing Graphs Graphs are built from simple Flows which serve as the linear connections within the graphs as well as junctions which serve as fan-in and fan-out points for Flows. Thanks to the junctions having meaningful types based on their behaviour and making them explicit elements these elements should be rather straightforward to use. -Akka Streams currently provide these junctions (for a detailed list see :ref:`stages-overview_scala`): +Akka Streams currently provide these junctions (for a detailed list see @ref:[stages-overview_scala](stages-overview.md)): -* **Fan-out** + * **Fan-out** - - ``Broadcast[T]`` – *(1 input, N outputs)* given an input element emits to each output - - ``Balance[T]`` – *(1 input, N outputs)* given an input element emits to one of its output ports - - ``UnzipWith[In,A,B,...]`` – *(1 input, N outputs)* takes a function of 1 input that given a value for each input emits N output elements (where N <= 20) - - ``UnZip[A,B]`` – *(1 input, 2 outputs)* splits a stream of ``(A,B)`` tuples into two streams, one of type ``A`` and one of type ``B`` +> + * `Broadcast[T]` – *(1 input, N outputs)* given an input element emits to each output + * `Balance[T]` – *(1 input, N outputs)* given an input element emits to one of its output ports + * `UnzipWith[In,A,B,...]` – *(1 input, N outputs)* takes a function of 1 input that given a value for each input emits N output elements (where N <= 20) + * `UnZip[A,B]` – *(1 input, 2 outputs)* splits a stream of `(A,B)` tuples into two streams, one of type `A` and one of type `B` -* **Fan-in** + * **Fan-in** - - ``Merge[In]`` – *(N inputs , 1 output)* picks randomly from inputs pushing them one by one to its output - - ``MergePreferred[In]`` – like :class:`Merge` but if elements are available on ``preferred`` port, it picks from it, otherwise randomly from ``others`` - - ``ZipWith[A,B,...,Out]`` – *(N inputs, 1 output)* which takes a function of N inputs that given a value for each input emits 1 output element - - ``Zip[A,B]`` – *(2 inputs, 1 output)* is a :class:`ZipWith` specialised to zipping input streams of ``A`` and ``B`` into an ``(A,B)`` tuple stream - - ``Concat[A]`` – *(2 inputs, 1 output)* concatenates two streams (first consume one, then the second one) +> + * `Merge[In]` – *(N inputs , 1 output)* picks randomly from inputs pushing them one by one to its output + * `MergePreferred[In]` – like `Merge` but if elements are available on `preferred` port, it picks from it, otherwise randomly from `others` + * `ZipWith[A,B,...,Out]` – *(N inputs, 1 output)* which takes a function of N inputs that given a value for each input emits 1 output element + * `Zip[A,B]` – *(2 inputs, 1 output)* is a `ZipWith` specialised to zipping input streams of `A` and `B` into an `(A,B)` tuple stream + * `Concat[A]` – *(2 inputs, 1 output)* concatenates two streams (first consume one, then the second one) One of the goals of the GraphDSL DSL is to look similar to how one would draw a graph on a whiteboard, so that it is simple to translate a design from whiteboard to code and be able to relate those two. Let's illustrate this by translating the below hand drawn graph into Akka Streams: -.. image:: ../../images/simple-graph-example.png +![simple-graph-example.png](../../images/simple-graph-example.png) -Such graph is simple to translate to the Graph DSL since each linear element corresponds to a :class:`Flow`, -and each circle corresponds to either a :class:`Junction` or a :class:`Source` or :class:`Sink` if it is beginning -or ending a :class:`Flow`. Junctions must always be created with defined type parameters, as otherwise the ``Nothing`` type +Such graph is simple to translate to the Graph DSL since each linear element corresponds to a `Flow`, +and each circle corresponds to either a `Junction` or a `Source` or `Sink` if it is beginning +or ending a `Flow`. Junctions must always be created with defined type parameters, as otherwise the `Nothing` type will be inferred. -.. includecode:: ../code/docs/stream/GraphDSLDocSpec.scala#simple-graph-dsl +@@snip [GraphDSLDocSpec.scala](../code/docs/stream/GraphDSLDocSpec.scala) { #simple-graph-dsl } -.. note:: - Junction *reference equality* defines *graph node equality* (i.e. the same merge *instance* used in a GraphDSL - refers to the same location in the resulting graph). +@@@ note -Notice the ``import GraphDSL.Implicits._`` which brings into scope the ``~>`` operator (read as "edge", "via" or "to") -and its inverted counterpart ``<~`` (for noting down flows in the opposite direction where appropriate). +Junction *reference equality* defines *graph node equality* (i.e. the same merge *instance* used in a GraphDSL +refers to the same location in the resulting graph). -By looking at the snippets above, it should be apparent that the :class:`GraphDSL.Builder` object is *mutable*. -It is used (implicitly) by the ``~>`` operator, also making it a mutable operation as well. +@@@ + +Notice the `import GraphDSL.Implicits._` which brings into scope the `~>` operator (read as "edge", "via" or "to") +and its inverted counterpart `<~` (for noting down flows in the opposite direction where appropriate). + +By looking at the snippets above, it should be apparent that the `GraphDSL.Builder` object is *mutable*. +It is used (implicitly) by the `~>` operator, also making it a mutable operation as well. The reason for this design choice is to enable simpler creation of complex graphs, which may even contain cycles. -Once the GraphDSL has been constructed though, the :class:`GraphDSL` instance *is immutable, thread-safe, and freely shareable*. +Once the GraphDSL has been constructed though, the `GraphDSL` instance *is immutable, thread-safe, and freely shareable*. The same is true of all graph pieces—sources, sinks, and flows—once they are constructed. This means that you can safely re-use one given Flow or junction in multiple places in a processing graph. We have seen examples of such re-use already above: the merge and broadcast junctions were imported -into the graph using ``builder.add(...)``, an operation that will make a copy of the blueprint that +into the graph using `builder.add(...)`, an operation that will make a copy of the blueprint that is passed to it and return the inlets and outlets of the resulting copy so that they can be wired up. Another alternative is to pass existing graphs—of any shape—into the factory method that produces a -new graph. The difference between these approaches is that importing using ``builder.add(...)`` ignores the +new graph. The difference between these approaches is that importing using `builder.add(...)` ignores the materialized value of the imported graph while importing via the factory method allows its inclusion; -for more details see :ref:`stream-materialization-scala`. +for more details see @ref:[Stream Materialization](stream-flows-and-basics.md#stream-materialization-scala). In the example below we prepare a graph that consists of two parallel streams, -in which we re-use the same instance of :class:`Flow`, yet it will properly be +in which we re-use the same instance of `Flow`, yet it will properly be materialized as two connections between the corresponding Sources and Sinks: -.. includecode:: ../code/docs/stream/GraphDSLDocSpec.scala#graph-dsl-reusing-a-flow +@@snip [GraphDSLDocSpec.scala](../code/docs/stream/GraphDSLDocSpec.scala) { #graph-dsl-reusing-a-flow } -.. _partial-graph-dsl-scala: - -Constructing and combining Partial Graphs ------------------------------------------ + +## Constructing and combining Partial Graphs Sometimes it is not possible (or needed) to construct the entire computation graph in one place, but instead construct all of its different phases in different places and in the end connect them all into a complete graph and run it. -This can be achieved by returning a different ``Shape`` than ``ClosedShape``, for example ``FlowShape(in, out)``, from the -function given to ``GraphDSL.create``. See :ref:`predefined-shapes`) for a list of such predefined shapes. +This can be achieved by returning a different `Shape` than `ClosedShape`, for example `FlowShape(in, out)`, from the +function given to `GraphDSL.create`. See [Predefined shapes](#predefined-shapes)) for a list of such predefined shapes. -Making a ``Graph`` a :class:`RunnableGraph` requires all ports to be connected, and if they are not +Making a `Graph` a `RunnableGraph` requires all ports to be connected, and if they are not it will throw an exception at construction time, which helps to avoid simple wiring errors while working with graphs. A partial graph however allows you to return the set of yet to be connected ports from the code block that @@ -103,120 +100,115 @@ Let's imagine we want to provide users with a specialized element that given 3 i the greatest int value of each zipped triple. We'll want to expose 3 input ports (unconnected sources) and one output port (unconnected sink). -.. includecode:: ../code/docs/stream/StreamPartialGraphDSLDocSpec.scala#simple-partial-graph-dsl +@@snip [StreamPartialGraphDSLDocSpec.scala](../code/docs/stream/StreamPartialGraphDSLDocSpec.scala) { #simple-partial-graph-dsl } As you can see, first we construct the partial graph that contains all the zipping and comparing of stream -elements. This partial graph will have three inputs and one output, wherefore we use the :class:`UniformFanInShape`. +elements. This partial graph will have three inputs and one output, wherefore we use the `UniformFanInShape`. Then we import it (all of its nodes and connections) explicitly into the closed graph built in the second step in which all the undefined elements are rewired to real sources and sinks. The graph can then be run and yields the expected result. -.. warning:: +@@@ warning - Please note that :class:`GraphDSL` is not able to provide compile time type-safety about whether or not all - elements have been properly connected—this validation is performed as a runtime check during the graph's instantiation. +Please note that `GraphDSL` is not able to provide compile time type-safety about whether or not all +elements have been properly connected—this validation is performed as a runtime check during the graph's instantiation. - A partial graph also verifies that all ports are either connected or part of the returned :class:`Shape`. +A partial graph also verifies that all ports are either connected or part of the returned `Shape`. -.. _constructing-sources-sinks-flows-from-partial-graphs-scala: +@@@ -Constructing Sources, Sinks and Flows from Partial Graphs ---------------------------------------------------------- + +## Constructing Sources, Sinks and Flows from Partial Graphs Instead of treating a partial graph as simply a collection of flows and junctions which may not yet all be connected it is sometimes useful to expose such a complex graph as a simpler structure, -such as a :class:`Source`, :class:`Sink` or :class:`Flow`. +such as a `Source`, `Sink` or `Flow`. In fact, these concepts can be easily expressed as special cases of a partially connected graph: -* :class:`Source` is a partial graph with *exactly one* output, that is it returns a :class:`SourceShape`. -* :class:`Sink` is a partial graph with *exactly one* input, that is it returns a :class:`SinkShape`. -* :class:`Flow` is a partial graph with *exactly one* input and *exactly one* output, that is it returns a :class:`FlowShape`. + * `Source` is a partial graph with *exactly one* output, that is it returns a `SourceShape`. + * `Sink` is a partial graph with *exactly one* input, that is it returns a `SinkShape`. + * `Flow` is a partial graph with *exactly one* input and *exactly one* output, that is it returns a `FlowShape`. Being able to hide complex graphs inside of simple elements such as Sink / Source / Flow enables you to easily create one complex element and from there on treat it as simple compound stage for linear computations. -In order to create a Source from a graph the method ``Source.fromGraph`` is used, to use it we must have a -``Graph[SourceShape, T]``. This is constructed using ``GraphDSL.create`` and returning a ``SourceShape`` -from the function passed in . The single outlet must be provided to the ``SourceShape.of`` method and will become +In order to create a Source from a graph the method `Source.fromGraph` is used, to use it we must have a +`Graph[SourceShape, T]`. This is constructed using `GraphDSL.create` and returning a `SourceShape` +from the function passed in . The single outlet must be provided to the `SourceShape.of` method and will become “the sink that must be attached before this Source can run”. Refer to the example below, in which we create a Source that zips together two numbers, to see this graph construction in action: -.. includecode:: ../code/docs/stream/StreamPartialGraphDSLDocSpec.scala#source-from-partial-graph-dsl +@@snip [StreamPartialGraphDSLDocSpec.scala](../code/docs/stream/StreamPartialGraphDSLDocSpec.scala) { #source-from-partial-graph-dsl } -Similarly the same can be done for a ``Sink[T]``, using ``SinkShape.of`` in which case the provided value -must be an ``Inlet[T]``. For defining a ``Flow[T]`` we need to expose both an inlet and an outlet: +Similarly the same can be done for a `Sink[T]`, using `SinkShape.of` in which case the provided value +must be an `Inlet[T]`. For defining a `Flow[T]` we need to expose both an inlet and an outlet: -.. includecode:: ../code/docs/stream/StreamPartialGraphDSLDocSpec.scala#flow-from-partial-graph-dsl +@@snip [StreamPartialGraphDSLDocSpec.scala](../code/docs/stream/StreamPartialGraphDSLDocSpec.scala) { #flow-from-partial-graph-dsl } -Combining Sources and Sinks with simplified API ------------------------------------------------ +## Combining Sources and Sinks with simplified API -There is a simplified API you can use to combine sources and sinks with junctions like: ``Broadcast[T]``, ``Balance[T]``, -``Merge[In]`` and ``Concat[A]`` without the need for using the Graph DSL. The combine method takes care of constructing +There is a simplified API you can use to combine sources and sinks with junctions like: `Broadcast[T]`, `Balance[T]`, +`Merge[In]` and `Concat[A]` without the need for using the Graph DSL. The combine method takes care of constructing the necessary graph underneath. In following example we combine two sources into one (fan-in): -.. includecode:: ../code/docs/stream/StreamPartialGraphDSLDocSpec.scala#source-combine +@@snip [StreamPartialGraphDSLDocSpec.scala](../code/docs/stream/StreamPartialGraphDSLDocSpec.scala) { #source-combine } -The same can be done for a ``Sink[T]`` but in this case it will be fan-out: +The same can be done for a `Sink[T]` but in this case it will be fan-out: -.. includecode:: ../code/docs/stream/StreamPartialGraphDSLDocSpec.scala#sink-combine +@@snip [StreamPartialGraphDSLDocSpec.scala](../code/docs/stream/StreamPartialGraphDSLDocSpec.scala) { #sink-combine } -Building reusable Graph components ----------------------------------- +## Building reusable Graph components It is possible to build reusable, encapsulated components of arbitrary input and output ports using the graph DSL. As an example, we will build a graph junction that represents a pool of workers, where a worker is expressed -as a ``Flow[I,O,_]``, i.e. a simple transformation of jobs of type ``I`` to results of type ``O`` (as you have seen +as a `Flow[I,O,_]`, i.e. a simple transformation of jobs of type `I` to results of type `O` (as you have seen already, this flow can actually contain a complex graph inside). Our reusable worker pool junction will -not preserve the order of the incoming jobs (they are assumed to have a proper ID field) and it will use a ``Balance`` +not preserve the order of the incoming jobs (they are assumed to have a proper ID field) and it will use a `Balance` junction to schedule jobs to available workers. On top of this, our junction will feature a "fastlane", a dedicated port where jobs of higher priority can be sent. -Altogether, our junction will have two input ports of type ``I`` (for the normal and priority jobs) and an output port -of type ``O``. To represent this interface, we need to define a custom :class:`Shape`. The following lines show how to do that. +Altogether, our junction will have two input ports of type `I` (for the normal and priority jobs) and an output port +of type `O`. To represent this interface, we need to define a custom `Shape`. The following lines show how to do that. -.. includecode:: ../code/docs/stream/GraphDSLDocSpec.scala#graph-dsl-components-shape +@@snip [GraphDSLDocSpec.scala](../code/docs/stream/GraphDSLDocSpec.scala) { #graph-dsl-components-shape } -.. _predefined-shapes: + +## Predefined shapes -Predefined shapes ------------------ - -In general a custom :class:`Shape` needs to be able to provide all its input and output ports, be able to copy itself, and also be +In general a custom `Shape` needs to be able to provide all its input and output ports, be able to copy itself, and also be able to create a new instance from given ports. There are some predefined shapes provided to avoid unnecessary boilerplate: - * :class:`SourceShape`, :class:`SinkShape`, :class:`FlowShape` for simpler shapes, - * :class:`UniformFanInShape` and :class:`UniformFanOutShape` for junctions with multiple input (or output) ports - of the same type, - * :class:`FanInShape1`, :class:`FanInShape2`, ..., :class:`FanOutShape1`, :class:`FanOutShape2`, ... for junctions - with multiple input (or output) ports of different types. +> + * `SourceShape`, `SinkShape`, `FlowShape` for simpler shapes, + * `UniformFanInShape` and `UniformFanOutShape` for junctions with multiple input (or output) ports +of the same type, + * `FanInShape1`, `FanInShape2`, ..., `FanOutShape1`, `FanOutShape2`, ... for junctions +with multiple input (or output) ports of different types. -Since our shape has two input ports and one output port, we can just use the :class:`FanInShape` DSL to define +Since our shape has two input ports and one output port, we can just use the `FanInShape` DSL to define our custom shape: -.. includecode:: ../code/docs/stream/GraphDSLDocSpec.scala#graph-dsl-components-shape2 +@@snip [GraphDSLDocSpec.scala](../code/docs/stream/GraphDSLDocSpec.scala) { #graph-dsl-components-shape2 } -Now that we have a :class:`Shape` we can wire up a Graph that represents -our worker pool. First, we will merge incoming normal and priority jobs using ``MergePreferred``, then we will send the jobs -to a ``Balance`` junction which will fan-out to a configurable number of workers (flows), finally we merge all these +Now that we have a `Shape` we can wire up a Graph that represents +our worker pool. First, we will merge incoming normal and priority jobs using `MergePreferred`, then we will send the jobs +to a `Balance` junction which will fan-out to a configurable number of workers (flows), finally we merge all these results together and send them out through our only output port. This is expressed by the following code: -.. includecode:: ../code/docs/stream/GraphDSLDocSpec.scala#graph-dsl-components-create +@@snip [GraphDSLDocSpec.scala](../code/docs/stream/GraphDSLDocSpec.scala) { #graph-dsl-components-create } All we need to do now is to use our custom junction in a graph. The following code simulates some simple workers and jobs using plain strings and prints out the results. Actually we used *two* instances of our worker pool junction -using ``add()`` twice. +using `add()` twice. -.. includecode:: ../code/docs/stream/GraphDSLDocSpec.scala#graph-dsl-components-use +@@snip [GraphDSLDocSpec.scala](../code/docs/stream/GraphDSLDocSpec.scala) { #graph-dsl-components-use } -.. _bidi-flow-scala: - -Bidirectional Flows -------------------- + +## Bidirectional Flows A graph topology that is often useful is that of two flows going in opposite directions. Take for example a codec stage that serializes outgoing messages @@ -224,152 +216,149 @@ and deserializes incoming octet streams. Another such stage could add a framing protocol that attaches a length header to outgoing data and parses incoming frames back into the original octet stream chunks. These two stages are meant to be composed, applying one atop the other as part of a protocol stack. For -this purpose exists the special type :class:`BidiFlow` which is a graph that +this purpose exists the special type `BidiFlow` which is a graph that has exactly two open inlets and two open outlets. The corresponding shape is -called :class:`BidiShape` and is defined like this: +called `BidiShape` and is defined like this: -.. includecode:: ../../../../akka-stream/src/main/scala/akka/stream/Shape.scala - :include: bidi-shape - :exclude: implementation-details-elided +@@snip [Shape.scala]../../../../../../akka-stream/src/main/scala/akka/stream/Shape.scala) { #bidi-shape } -A bidirectional flow is defined just like a unidirectional :class:`Flow` as +A bidirectional flow is defined just like a unidirectional `Flow` as demonstrated for the codec mentioned above: -.. includecode:: ../code/docs/stream/BidiFlowDocSpec.scala - :include: codec - :exclude: implementation-details-elided +@@snip [BidiFlowDocSpec.scala](../code/docs/stream/BidiFlowDocSpec.scala) { #codec } The first version resembles the partial graph constructor, while for the simple case of a functional 1:1 transformation there is a concise convenience method as shown on the last line. The implementation of the two functions is not difficult either: -.. includecode:: ../code/docs/stream/BidiFlowDocSpec.scala#codec-impl +@@snip [BidiFlowDocSpec.scala](../code/docs/stream/BidiFlowDocSpec.scala) { #codec-impl } In this way you could easily integrate any other serialization library that turns an object into a sequence of bytes. The other stage that we talked about is a little more involved since reversing a framing protocol means that any received chunk of bytes may correspond to -zero or more messages. This is best implemented using a :class:`GraphStage` -(see also :ref:`graphstage-scala`). +zero or more messages. This is best implemented using a `GraphStage` +(see also @ref:[Custom processing with GraphStage](stream-customize.md#graphstage-scala)). -.. includecode:: ../code/docs/stream/BidiFlowDocSpec.scala#framing +@@snip [BidiFlowDocSpec.scala](../code/docs/stream/BidiFlowDocSpec.scala) { #framing } With these implementations we can build a protocol stack and test it: -.. includecode:: ../code/docs/stream/BidiFlowDocSpec.scala#compose +@@snip [BidiFlowDocSpec.scala](../code/docs/stream/BidiFlowDocSpec.scala) { #compose } -This example demonstrates how :class:`BidiFlow` subgraphs can be hooked -together and also turned around with the ``.reversed`` method. The test +This example demonstrates how `BidiFlow` subgraphs can be hooked +together and also turned around with the `.reversed` method. The test simulates both parties of a network communication protocol without actually having to open a network connection—the flows can just be connected directly. -.. _graph-matvalue-scala: - -Accessing the materialized value inside the Graph -------------------------------------------------- + +## Accessing the materialized value inside the Graph In certain cases it might be necessary to feed back the materialized value of a Graph (partial, closed or backing a -Source, Sink, Flow or BidiFlow). This is possible by using ``builder.materializedValue`` which gives an ``Outlet`` that +Source, Sink, Flow or BidiFlow). This is possible by using `builder.materializedValue` which gives an `Outlet` that can be used in the graph as an ordinary source or outlet, and which will eventually emit the materialized value. -If the materialized value is needed at more than one place, it is possible to call ``materializedValue`` any number of +If the materialized value is needed at more than one place, it is possible to call `materializedValue` any number of times to acquire the necessary number of outlets. -.. includecode:: ../code/docs/stream/GraphDSLDocSpec.scala#graph-dsl-matvalue +@@snip [GraphDSLDocSpec.scala](../code/docs/stream/GraphDSLDocSpec.scala) { #graph-dsl-matvalue } Be careful not to introduce a cycle where the materialized value actually contributes to the materialized value. -The following example demonstrates a case where the materialized ``Future`` of a fold is fed back to the fold itself. +The following example demonstrates a case where the materialized `Future` of a fold is fed back to the fold itself. -.. includecode:: ../code/docs/stream/GraphDSLDocSpec.scala#graph-dsl-matvalue-cycle +@@snip [GraphDSLDocSpec.scala](../code/docs/stream/GraphDSLDocSpec.scala) { #graph-dsl-matvalue-cycle } -.. _graph-cycles-scala: - -Graph cycles, liveness and deadlocks ------------------------------------- + +## Graph cycles, liveness and deadlocks Cycles in bounded stream topologies need special considerations to avoid potential deadlocks and other liveness issues. This section shows several examples of problems that can arise from the presence of feedback arcs in stream processing graphs. In the following examples runnable graphs are created but do not run because each have some issue and will deadlock after start. -``Source`` variable is not defined as the nature and number of element does not matter for described problems. +`Source` variable is not defined as the nature and number of element does not matter for described problems. The first example demonstrates a graph that contains a naïve cycle. The graph takes elements from the source, prints them, then broadcasts those elements -to a consumer (we just used ``Sink.ignore`` for now) and to a feedback arc that is merged back into the main stream via -a ``Merge`` junction. +to a consumer (we just used `Sink.ignore` for now) and to a feedback arc that is merged back into the main stream via +a `Merge` junction. -.. note:: +@@@ note - The graph DSL allows the connection arrows to be reversed, which is particularly handy when writing cycles—as we will - see there are cases where this is very helpful. +The graph DSL allows the connection arrows to be reversed, which is particularly handy when writing cycles—as we will +see there are cases where this is very helpful. -.. includecode:: ../code/docs/stream/GraphCyclesSpec.scala#deadlocked +@@@ + +@@snip [GraphCyclesSpec.scala](../code/docs/stream/GraphCyclesSpec.scala) { #deadlocked } Running this we observe that after a few numbers have been printed, no more elements are logged to the console - all processing stops after some time. After some investigation we observe that: -* through merging from ``source`` we increase the number of elements flowing in the cycle -* by broadcasting back to the cycle we do not decrease the number of elements in the cycle + * through merging from `source` we increase the number of elements flowing in the cycle + * by broadcasting back to the cycle we do not decrease the number of elements in the cycle Since Akka Streams (and Reactive Streams in general) guarantee bounded processing (see the "Buffering" section for more details) it means that only a bounded number of elements are buffered over any time span. Since our cycle gains more and -more elements, eventually all of its internal buffers become full, backpressuring ``source`` forever. To be able -to process more elements from ``source`` elements would need to leave the cycle somehow. +more elements, eventually all of its internal buffers become full, backpressuring `source` forever. To be able +to process more elements from `source` elements would need to leave the cycle somehow. -If we modify our feedback loop by replacing the ``Merge`` junction with a ``MergePreferred`` we can avoid the deadlock. -``MergePreferred`` is unfair as it always tries to consume from a preferred input port if there are elements available +If we modify our feedback loop by replacing the `Merge` junction with a `MergePreferred` we can avoid the deadlock. +`MergePreferred` is unfair as it always tries to consume from a preferred input port if there are elements available before trying the other lower priority input ports. Since we feed back through the preferred port it is always guaranteed that the elements in the cycles can flow. -.. includecode:: ../code/docs/stream/GraphCyclesSpec.scala#unfair +@@snip [GraphCyclesSpec.scala](../code/docs/stream/GraphCyclesSpec.scala) { #unfair } If we run the example we see that the same sequence of numbers are printed -over and over again, but the processing does not stop. Hence, we avoided the deadlock, but ``source`` is still +over and over again, but the processing does not stop. Hence, we avoided the deadlock, but `source` is still back-pressured forever, because buffer space is never recovered: the only action we see is the circulation of a couple -of initial elements from ``source``. +of initial elements from `source`. -.. note:: - What we see here is that in certain cases we need to choose between boundedness and liveness. Our first example would - not deadlock if there would be an infinite buffer in the loop, or vice versa, if the elements in the cycle would - be balanced (as many elements are removed as many are injected) then there would be no deadlock. +@@@ note + +What we see here is that in certain cases we need to choose between boundedness and liveness. Our first example would +not deadlock if there would be an infinite buffer in the loop, or vice versa, if the elements in the cycle would +be balanced (as many elements are removed as many are injected) then there would be no deadlock. + +@@@ To make our cycle both live (not deadlocking) and fair we can introduce a dropping element on the feedback arc. In this -case we chose the ``buffer()`` operation giving it a dropping strategy ``OverflowStrategy.dropHead``. +case we chose the `buffer()` operation giving it a dropping strategy `OverflowStrategy.dropHead`. -.. includecode:: ../code/docs/stream/GraphCyclesSpec.scala#dropping +@@snip [GraphCyclesSpec.scala](../code/docs/stream/GraphCyclesSpec.scala) { #dropping } If we run this example we see that -* The flow of elements does not stop, there are always elements printed -* We see that some of the numbers are printed several times over time (due to the feedback loop) but on average - the numbers are increasing in the long term + * The flow of elements does not stop, there are always elements printed + * We see that some of the numbers are printed several times over time (due to the feedback loop) but on average +the numbers are increasing in the long term This example highlights that one solution to avoid deadlocks in the presence of potentially unbalanced cycles (cycles where the number of circulating elements are unbounded) is to drop elements. An alternative would be to -define a larger buffer with ``OverflowStrategy.fail`` which would fail the stream instead of deadlocking it after +define a larger buffer with `OverflowStrategy.fail` which would fail the stream instead of deadlocking it after all buffer space has been consumed. As we discovered in the previous examples, the core problem was the unbalanced nature of the feedback loop. We circumvented this issue by adding a dropping element, but now we want to build a cycle that is balanced from -the beginning instead. To achieve this we modify our first graph by replacing the ``Merge`` junction with a ``ZipWith``. -Since ``ZipWith`` takes one element from ``source`` *and* from the feedback arc to inject one element into the cycle, +the beginning instead. To achieve this we modify our first graph by replacing the `Merge` junction with a `ZipWith`. +Since `ZipWith` takes one element from `source` *and* from the feedback arc to inject one element into the cycle, we maintain the balance of elements. -.. includecode:: ../code/docs/stream/GraphCyclesSpec.scala#zipping-dead +@@snip [GraphCyclesSpec.scala](../code/docs/stream/GraphCyclesSpec.scala) { #zipping-dead } Still, when we try to run the example it turns out that no element is printed at all! After some investigation we realize that: -* In order to get the first element from ``source`` into the cycle we need an already existing element in the cycle -* In order to get an initial element in the cycle we need an element from ``source`` + * In order to get the first element from `source` into the cycle we need an already existing element in the cycle + * In order to get an initial element in the cycle we need an element from `source` These two conditions are a typical "chicken-and-egg" problem. The solution is to inject an initial -element into the cycle that is independent from ``source``. We do this by using a ``Concat`` junction on the backwards -arc that injects a single element using ``Source.single``. +element into the cycle that is independent from `source`. We do this by using a `Concat` junction on the backwards +arc that injects a single element using `Source.single`. -.. includecode:: ../code/docs/stream/GraphCyclesSpec.scala#zipping-live +@@snip [GraphCyclesSpec.scala](../code/docs/stream/GraphCyclesSpec.scala) { #zipping-live } When we run the above example we see that processing starts and never stops. The important takeaway from this example -is that balanced cycles often need an initial "kick-off" element to be injected into the cycle. +is that balanced cycles often need an initial "kick-off" element to be injected into the cycle. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-integrations.md b/akka-docs/src/main/paradox/scala/stream/stream-integrations.md index 5aafbeb201..ac636693c8 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-integrations.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-integrations.md @@ -1,478 +1,464 @@ -.. _stream-integrations-scala: +# Integration -########### -Integration -########### - -Integrating with Actors -======================= +## Integrating with Actors For piping the elements of a stream as messages to an ordinary actor you can use -``ask`` in a ``mapAsync`` or use ``Sink.actorRefWithAck``. +`ask` in a `mapAsync` or use `Sink.actorRefWithAck`. -Messages can be sent to a stream with ``Source.queue`` or via the :class:`ActorRef` that is -materialized by ``Source.actorRef``. +Messages can be sent to a stream with `Source.queue` or via the `ActorRef` that is +materialized by `Source.actorRef`. -mapAsync + ask -^^^^^^^^^^^^^^ +### mapAsync + ask A nice way to delegate some processing of elements in a stream to an actor is to -use ``ask`` in ``mapAsync``. The back-pressure of the stream is maintained by -the ``Future`` of the ``ask`` and the mailbox of the actor will not be filled with -more messages than the given ``parallelism`` of the ``mapAsync`` stage. +use `ask` in `mapAsync`. The back-pressure of the stream is maintained by +the `Future` of the `ask` and the mailbox of the actor will not be filled with +more messages than the given `parallelism` of the `mapAsync` stage. -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#mapAsync-ask +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #mapAsync-ask } Note that the messages received in the actor will be in the same order as -the stream elements, i.e. the ``parallelism`` does not change the ordering +the stream elements, i.e. the `parallelism` does not change the ordering of the messages. There is a performance advantage of using parallelism > 1 even though the actor will only process one message at a time because then there is already a message in the mailbox when the actor has completed previous message. -The actor must reply to the ``sender()`` for each message from the stream. That -reply will complete the ``Future`` of the ``ask`` and it will be the element that -is emitted downstreams from ``mapAsync``. +The actor must reply to the `sender()` for each message from the stream. That +reply will complete the `Future` of the `ask` and it will be the element that +is emitted downstreams from `mapAsync`. -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#ask-actor +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #ask-actor } -The stream can be completed with failure by sending ``akka.actor.Status.Failure`` +The stream can be completed with failure by sending `akka.actor.Status.Failure` as reply from the actor. -If the ``ask`` fails due to timeout the stream will be completed with -``TimeoutException`` failure. If that is not desired outcome you can use ``recover`` -on the ``ask`` :class:`Future`. +If the `ask` fails due to timeout the stream will be completed with +`TimeoutException` failure. If that is not desired outcome you can use `recover` +on the `ask` `Future`. If you don't care about the reply values and only use them as back-pressure signals you -can use ``Sink.ignore`` after the ``mapAsync`` stage and then actor is effectively a sink +can use `Sink.ignore` after the `mapAsync` stage and then actor is effectively a sink of the stream. -The same pattern can be used with :ref:`Actor routers `. Then you -can use ``mapAsyncUnordered`` for better efficiency if you don't care about the +The same pattern can be used with @ref:[Actor routers](../routing.md). Then you +can use `mapAsyncUnordered` for better efficiency if you don't care about the order of the emitted downstream elements (the replies). -Sink.actorRefWithAck -^^^^^^^^^^^^^^^^^^^^ +### Sink.actorRefWithAck -The sink sends the elements of the stream to the given :class:`ActorRef` that sends back back-pressure signal. -First element is always `onInitMessage`, then stream is waiting for the given acknowledgement message +The sink sends the elements of the stream to the given `ActorRef` that sends back back-pressure signal. +First element is always *onInitMessage*, then stream is waiting for the given acknowledgement message from the given actor which means that it is ready to process elements. It also requires the given acknowledgement message after each stream element to make back-pressure work. If the target actor terminates the stream will be cancelled. When the stream is completed successfully the -given ``onCompleteMessage`` will be sent to the destination actor. When the stream is completed with -failure a ``akka.actor.Status.Failure`` message will be sent to the destination actor. +given `onCompleteMessage` will be sent to the destination actor. When the stream is completed with +failure a `akka.actor.Status.Failure` message will be sent to the destination actor. -.. note:: +@@@ note - Using ``Sink.actorRef`` or ordinary ``tell`` from a ``map`` or ``foreach`` stage means that there is - no back-pressure signal from the destination actor, i.e. if the actor is not consuming the messages - fast enough the mailbox of the actor will grow, unless you use a bounded mailbox with zero - `mailbox-push-timeout-time` or use a rate limiting stage in front. It's often better to - use ``Sink.actorRefWithAck`` or ``ask`` in ``mapAsync``, though. +Using `Sink.actorRef` or ordinary `tell` from a `map` or `foreach` stage means that there is +no back-pressure signal from the destination actor, i.e. if the actor is not consuming the messages +fast enough the mailbox of the actor will grow, unless you use a bounded mailbox with zero +*mailbox-push-timeout-time* or use a rate limiting stage in front. It's often better to +use `Sink.actorRefWithAck` or `ask` in `mapAsync`, though. -Source.queue -^^^^^^^^^^^^ +@@@ -``Source.queue`` can be used for emitting elements to a stream from an actor (or from anything running outside -the stream). The elements will be buffered until the stream can process them. You can ``offer`` elements to +### Source.queue + +`Source.queue` can be used for emitting elements to a stream from an actor (or from anything running outside +the stream). The elements will be buffered until the stream can process them. You can `offer` elements to the queue and they will be emitted to the stream if there is demand from downstream, otherwise they will be buffered until request for demand is received. -Use overflow strategy ``akka.stream.OverflowStrategy.backpressure`` to avoid dropping of elements if the +Use overflow strategy `akka.stream.OverflowStrategy.backpressure` to avoid dropping of elements if the buffer is full. -``SourceQueue.offer`` returns ``Future[QueueOfferResult]`` which completes with ``QueueOfferResult.Enqueued`` -if element was added to buffer or sent downstream. It completes with ``QueueOfferResult.Dropped`` if element -was dropped. Can also complete with ``QueueOfferResult.Failure`` - when stream failed or -``QueueOfferResult.QueueClosed`` when downstream is completed. +`SourceQueue.offer` returns `Future[QueueOfferResult]` which completes with `QueueOfferResult.Enqueued` +if element was added to buffer or sent downstream. It completes with `QueueOfferResult.Dropped` if element +was dropped. Can also complete with `QueueOfferResult.Failure` - when stream failed or +`QueueOfferResult.QueueClosed` when downstream is completed. -When used from an actor you typically ``pipe`` the result of the ``Future`` back to the actor to +When used from an actor you typically `pipe` the result of the `Future` back to the actor to continue processing. -Source.actorRef -^^^^^^^^^^^^^^^ +### Source.actorRef -Messages sent to the actor that is materialized by ``Source.actorRef`` will be emitted to the +Messages sent to the actor that is materialized by `Source.actorRef` will be emitted to the stream if there is demand from downstream, otherwise they will be buffered until request for demand is received. -Depending on the defined :class:`OverflowStrategy` it might drop elements if there is no space -available in the buffer. The strategy ``OverflowStrategy.backpressure`` is not supported +Depending on the defined `OverflowStrategy` it might drop elements if there is no space +available in the buffer. The strategy `OverflowStrategy.backpressure` is not supported for this Source type, i.e. elements will be dropped if the buffer is filled by sending -at a rate that is faster than the stream can consume. You should consider using ``Source.queue`` +at a rate that is faster than the stream can consume. You should consider using `Source.queue` if you want a backpressured actor interface. -The stream can be completed successfully by sending ``akka.actor.PoisonPill`` or -``akka.actor.Status.Success`` to the actor reference. +The stream can be completed successfully by sending `akka.actor.PoisonPill` or +`akka.actor.Status.Success` to the actor reference. -The stream can be completed with failure by sending ``akka.actor.Status.Failure`` to the +The stream can be completed with failure by sending `akka.actor.Status.Failure` to the actor reference. The actor will be stopped when the stream is completed, failed or cancelled from downstream, i.e. you can watch it to get notified when that happens. -Integrating with External Services -================================== +## Integrating with External Services Stream transformations and side effects involving external non-stream based services can be -performed with ``mapAsync`` or ``mapAsyncUnordered``. +performed with `mapAsync` or `mapAsyncUnordered`. For example, sending emails to the authors of selected tweets using an external email service: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#email-server-send +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #email-server-send } We start with the tweet stream of authors: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#tweet-authors +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #tweet-authors } Assume that we can lookup their email address using: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#email-address-lookup +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #email-address-lookup } -Transforming the stream of authors to a stream of email addresses by using the ``lookupEmail`` -service can be done with ``mapAsync``: +Transforming the stream of authors to a stream of email addresses by using the `lookupEmail` +service can be done with `mapAsync`: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#email-addresses-mapAsync +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #email-addresses-mapAsync } Finally, sending the emails: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#send-emails +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #send-emails } -``mapAsync`` is applying the given function that is calling out to the external service to -each of the elements as they pass through this processing step. The function returns a :class:`Future` +`mapAsync` is applying the given function that is calling out to the external service to +each of the elements as they pass through this processing step. The function returns a `Future` and the value of that future will be emitted downstreams. The number of Futures -that shall run in parallel is given as the first argument to ``mapAsync``. +that shall run in parallel is given as the first argument to `mapAsync`. These Futures may complete in any order, but the elements that are emitted downstream are in the same order as received from upstream. -That means that back-pressure works as expected. For example if the ``emailServer.send`` +That means that back-pressure works as expected. For example if the `emailServer.send` is the bottleneck it will limit the rate at which incoming tweets are retrieved and email addresses looked up. The final piece of this pipeline is to generate the demand that pulls the tweet -authors information through the emailing pipeline: we attach a ``Sink.ignore`` +authors information through the emailing pipeline: we attach a `Sink.ignore` which makes it all run. If our email process would return some interesting data for further transformation then we would of course not ignore it but send that result stream onwards for further processing or storage. -Note that ``mapAsync`` preserves the order of the stream elements. In this example the order -is not important and then we can use the more efficient ``mapAsyncUnordered``: +Note that `mapAsync` preserves the order of the stream elements. In this example the order +is not important and then we can use the more efficient `mapAsyncUnordered`: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#external-service-mapAsyncUnordered +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #external-service-mapAsyncUnordered } -In the above example the services conveniently returned a :class:`Future` of the result. -If that is not the case you need to wrap the call in a :class:`Future`. If the service call +In the above example the services conveniently returned a `Future` of the result. +If that is not the case you need to wrap the call in a `Future`. If the service call involves blocking you must also make sure that you run it on a dedicated execution context, to avoid starvation and disturbance of other tasks in the system. -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#blocking-mapAsync +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #blocking-mapAsync } -The configuration of the ``"blocking-dispatcher"`` may look something like: +The configuration of the `"blocking-dispatcher"` may look something like: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#blocking-dispatcher-config +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #blocking-dispatcher-config } -An alternative for blocking calls is to perform them in a ``map`` operation, still using a +An alternative for blocking calls is to perform them in a `map` operation, still using a dedicated dispatcher for that operation. -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#blocking-map +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #blocking-map } -However, that is not exactly the same as ``mapAsync``, since the ``mapAsync`` may run -several calls concurrently, but ``map`` performs them one at a time. +However, that is not exactly the same as `mapAsync`, since the `mapAsync` may run +several calls concurrently, but `map` performs them one at a time. For a service that is exposed as an actor, or if an actor is used as a gateway in front of an -external service, you can use ``ask``: +external service, you can use `ask`: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#save-tweets +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #save-tweets } -Note that if the ``ask`` is not completed within the given timeout the stream is completed with failure. -If that is not desired outcome you can use ``recover`` on the ``ask`` :class:`Future`. +Note that if the `ask` is not completed within the given timeout the stream is completed with failure. +If that is not desired outcome you can use `recover` on the `ask` `Future`. -Illustrating ordering and parallelism -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Illustrating ordering and parallelism Let us look at another example to get a better understanding of the ordering -and parallelism characteristics of ``mapAsync`` and ``mapAsyncUnordered``. +and parallelism characteristics of `mapAsync` and `mapAsyncUnordered`. -Several ``mapAsync`` and ``mapAsyncUnordered`` futures may run concurrently. +Several `mapAsync` and `mapAsyncUnordered` futures may run concurrently. The number of concurrent futures are limited by the downstream demand. For example, if 5 elements have been requested by downstream there will be at most 5 futures in progress. -``mapAsync`` emits the future results in the same order as the input elements +`mapAsync` emits the future results in the same order as the input elements were received. That means that completed results are only emitted downstream when earlier results have been completed and emitted. One slow call will thereby delay the results of all successive calls, even though they are completed before the slow call. -``mapAsyncUnordered`` emits the future results as soon as they are completed, i.e. +`mapAsyncUnordered` emits the future results as soon as they are completed, i.e. it is possible that the elements are not emitted downstream in the same order as received from upstream. One slow call will thereby not delay the results of faster successive calls as long as there is downstream demand of several elements. Here is a fictive service that we can use to illustrate these aspects. -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#sometimes-slow-service +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #sometimes-slow-service } Elements starting with a lower case character are simulated to take longer time to process. -Here is how we can use it with ``mapAsync``: +Here is how we can use it with `mapAsync`: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#sometimes-slow-mapAsync +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #sometimes-slow-mapAsync } The output may look like this: -:: +``` +before: a +before: B +before: C +before: D +running: a (1) +running: B (2) +before: e +running: C (3) +before: F +running: D (4) +before: g +before: H +completed: C (3) +completed: B (2) +completed: D (1) +completed: a (0) +after: A +after: B +running: e (1) +after: C +after: D +running: F (2) +before: i +before: J +running: g (3) +running: H (4) +completed: H (2) +completed: F (3) +completed: e (1) +completed: g (0) +after: E +after: F +running: i (1) +after: G +after: H +running: J (2) +completed: J (1) +completed: i (0) +after: I +after: J +``` - before: a - before: B - before: C - before: D - running: a (1) - running: B (2) - before: e - running: C (3) - before: F - running: D (4) - before: g - before: H - completed: C (3) - completed: B (2) - completed: D (1) - completed: a (0) - after: A - after: B - running: e (1) - after: C - after: D - running: F (2) - before: i - before: J - running: g (3) - running: H (4) - completed: H (2) - completed: F (3) - completed: e (1) - completed: g (0) - after: E - after: F - running: i (1) - after: G - after: H - running: J (2) - completed: J (1) - completed: i (0) - after: I - after: J - -Note that ``after`` lines are in the same order as the ``before`` lines even -though elements are ``completed`` in a different order. For example ``H`` -is ``completed`` before ``g``, but still emitted afterwards. +Note that `after` lines are in the same order as the `before` lines even +though elements are `completed` in a different order. For example `H` +is `completed` before `g`, but still emitted afterwards. The numbers in parenthesis illustrates how many calls that are in progress at the same time. Here the downstream demand and thereby the number of concurrent -calls are limited by the buffer size (4) of the :class:`ActorMaterializerSettings`. +calls are limited by the buffer size (4) of the `ActorMaterializerSettings`. -Here is how we can use the same service with ``mapAsyncUnordered``: +Here is how we can use the same service with `mapAsyncUnordered`: -.. includecode:: ../code/docs/stream/IntegrationDocSpec.scala#sometimes-slow-mapAsyncUnordered +@@snip [IntegrationDocSpec.scala](../code/docs/stream/IntegrationDocSpec.scala) { #sometimes-slow-mapAsyncUnordered } The output may look like this: -:: +``` +before: a +before: B +before: C +before: D +running: a (1) +running: B (2) +before: e +running: C (3) +before: F +running: D (4) +before: g +before: H +completed: B (3) +completed: C (1) +completed: D (2) +after: B +after: D +running: e (2) +after: C +running: F (3) +before: i +before: J +completed: F (2) +after: F +running: g (3) +running: H (4) +completed: H (3) +after: H +completed: a (2) +after: A +running: i (3) +running: J (4) +completed: J (3) +after: J +completed: e (2) +after: E +completed: g (1) +after: G +completed: i (0) +after: I +``` - before: a - before: B - before: C - before: D - running: a (1) - running: B (2) - before: e - running: C (3) - before: F - running: D (4) - before: g - before: H - completed: B (3) - completed: C (1) - completed: D (2) - after: B - after: D - running: e (2) - after: C - running: F (3) - before: i - before: J - completed: F (2) - after: F - running: g (3) - running: H (4) - completed: H (3) - after: H - completed: a (2) - after: A - running: i (3) - running: J (4) - completed: J (3) - after: J - completed: e (2) - after: E - completed: g (1) - after: G - completed: i (0) - after: I - -Note that ``after`` lines are not in the same order as the ``before`` lines. For example -``H`` overtakes the slow ``G``. +Note that `after` lines are not in the same order as the `before` lines. For example +`H` overtakes the slow `G`. The numbers in parenthesis illustrates how many calls that are in progress at the same time. Here the downstream demand and thereby the number of concurrent -calls are limited by the buffer size (4) of the :class:`ActorMaterializerSettings`. +calls are limited by the buffer size (4) of the `ActorMaterializerSettings`. -.. _reactive-streams-integration-scala: + +## Integrating with Reactive Streams -Integrating with Reactive Streams -================================= - -`Reactive Streams`_ defines a standard for asynchronous stream processing with non-blocking +[Reactive Streams](http://reactive-streams.org/) defines a standard for asynchronous stream processing with non-blocking back pressure. It makes it possible to plug together stream libraries that adhere to the standard. Akka Streams is one such library. An incomplete list of other implementations: -* `Reactor (1.1+)`_ -* `RxJava`_ -* `Ratpack`_ -* `Slick`_ + * [Reactor (1.1+)](http://github.com/reactor/reactor) + * [RxJava](https://github.com/ReactiveX/RxJavaReactiveStreams) + * [Ratpack](http://www.ratpack.io/manual/current/streams.html) + * [Slick](http://slick.lightbend.com) -.. _Reactive Streams: http://reactive-streams.org/ -.. _Reactor (1.1+): http://github.com/reactor/reactor -.. _RxJava: https://github.com/ReactiveX/RxJavaReactiveStreams -.. _Ratpack: http://www.ratpack.io/manual/current/streams.html -.. _Slick: http://slick.lightbend.com +The two most important interfaces in Reactive Streams are the `Publisher` and `Subscriber`. -The two most important interfaces in Reactive Streams are the :class:`Publisher` and :class:`Subscriber`. - -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala#imports +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #imports } Let us assume that a library provides a publisher of tweets: -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala#tweets-publisher +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #tweets-publisher } and another library knows how to store author handles in a database: -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala#author-storage-subscriber +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #author-storage-subscriber } -Using an Akka Streams :class:`Flow` we can transform the stream and connect those: +Using an Akka Streams `Flow` we can transform the stream and connect those: -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala - :include: authors,connect-all +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #authors #connect-all } -The :class:`Publisher` is used as an input :class:`Source` to the flow and the -:class:`Subscriber` is used as an output :class:`Sink`. +The `Publisher` is used as an input `Source` to the flow and the +`Subscriber` is used as an output `Sink`. -A :class:`Flow` can also be also converted to a :class:`RunnableGraph[Processor[In, Out]]` which -materializes to a :class:`Processor` when ``run()`` is called. ``run()`` itself can be called multiple -times, resulting in a new :class:`Processor` instance each time. +A `Flow` can also be also converted to a `RunnableGraph[Processor[In, Out]]` which +materializes to a `Processor` when `run()` is called. `run()` itself can be called multiple +times, resulting in a new `Processor` instance each time. -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala#flow-publisher-subscriber +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #flow-publisher-subscriber } -A publisher can be connected to a subscriber with the ``subscribe`` method. +A publisher can be connected to a subscriber with the `subscribe` method. -It is also possible to expose a :class:`Source` as a :class:`Publisher` -by using the Publisher-:class:`Sink`: +It is also possible to expose a `Source` as a `Publisher` +by using the Publisher-`Sink`: -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala#source-publisher +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #source-publisher } -A publisher that is created with ``Sink.asPublisher(fanout = false)`` supports only a single subscription. -Additional subscription attempts will be rejected with an :class:`IllegalStateException`. +A publisher that is created with `Sink.asPublisher(fanout = false)` supports only a single subscription. +Additional subscription attempts will be rejected with an `IllegalStateException`. A publisher that supports multiple subscribers using fan-out/broadcasting is created as follows: -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala - :include: author-alert-subscriber,author-storage-subscriber +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #author-alert-subscriber #author-storage-subscriber } -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala#source-fanoutPublisher +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #source-fanoutPublisher } The input buffer size of the stage controls how far apart the slowest subscriber can be from the fastest subscriber before slowing down the stream. -To make the picture complete, it is also possible to expose a :class:`Sink` as a :class:`Subscriber` -by using the Subscriber-:class:`Source`: +To make the picture complete, it is also possible to expose a `Sink` as a `Subscriber` +by using the Subscriber-`Source`: -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala#sink-subscriber +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #sink-subscriber } -It is also possible to use re-wrap :class:`Processor` instances as a :class:`Flow` by -passing a factory function that will create the :class:`Processor` instances: +It is also possible to use re-wrap `Processor` instances as a `Flow` by +passing a factory function that will create the `Processor` instances: -.. includecode:: ../code/docs/stream/ReactiveStreamsDocSpec.scala#use-processor +@@snip [ReactiveStreamsDocSpec.scala](../code/docs/stream/ReactiveStreamsDocSpec.scala) { #use-processor } -Please note that a factory is necessary to achieve reusability of the resulting :class:`Flow`. +Please note that a factory is necessary to achieve reusability of the resulting `Flow`. -Implementing Reactive Streams Publisher or Subscriber -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Implementing Reactive Streams Publisher or Subscriber -As described above any Akka Streams ``Source`` can be exposed as a Reactive Streams ``Publisher`` and -any ``Sink`` can be exposed as a Reactive Streams ``Subscriber``. Therefore we recommend that you -implement Reactive Streams integrations with built-in stages or :ref:`custom stages `. +As described above any Akka Streams `Source` can be exposed as a Reactive Streams `Publisher` and +any `Sink` can be exposed as a Reactive Streams `Subscriber`. Therefore we recommend that you +implement Reactive Streams integrations with built-in stages or @ref:[custom stages](stream-customize.md). -For historical reasons the :class:`ActorPublisher` and :class:`ActorSubscriber` traits are -provided to support implementing Reactive Streams :class:`Publisher` and :class:`Subscriber` with -an :class:`Actor`. +For historical reasons the `ActorPublisher` and `ActorSubscriber` traits are +provided to support implementing Reactive Streams `Publisher` and `Subscriber` with +an `Actor`. -These can be consumed by other Reactive Stream libraries or used as an Akka Streams :class:`Source` or :class:`Sink`. +These can be consumed by other Reactive Stream libraries or used as an Akka Streams `Source` or `Sink`. -.. warning:: +@@@ warning - :class:`ActorPublisher` and :class:`ActorSubscriber` will probably be deprecated in future versions of Akka. +`ActorPublisher` and `ActorSubscriber` will probably be deprecated in future versions of Akka. -.. warning:: +@@@ - :class:`ActorPublisher` and :class:`ActorSubscriber` cannot be used with remote actors, - because if signals of the Reactive Streams protocol (e.g. ``request``) are lost the - the stream may deadlock. +@@@ warning -ActorPublisher --------------- +`ActorPublisher` and `ActorSubscriber` cannot be used with remote actors, +because if signals of the Reactive Streams protocol (e.g. `request`) are lost the +the stream may deadlock. -.. warning:: - **Deprecation warning:** ``ActorPublisher`` is deprecated in favour of the vastly more - type-safe and safe to implement :class:`akka.stream.stage.GraphStage`. It can also - expose a "stage actor ref" is needed to be addressed as-if an Actor. - Custom stages implemented using ``GraphStage`` are also automatically fusable. - - To learn more about implementing custom stages using it refer to :ref:`graphstage-scala`. +@@@ -Extend/mixin :class:`akka.stream.actor.ActorPublisher` in your :class:`Actor` to make it a +#### ActorPublisher + +@@@ warning + +**Deprecation warning:** `ActorPublisher` is deprecated in favour of the vastly more +type-safe and safe to implement `akka.stream.stage.GraphStage`. It can also +expose a "stage actor ref" is needed to be addressed as-if an Actor. +Custom stages implemented using `GraphStage` are also automatically fusable. + +To learn more about implementing custom stages using it refer to @ref:[Custom processing with GraphStage](stream-customize.md#graphstage-scala). + +@@@ + +Extend/mixin `akka.stream.actor.ActorPublisher` in your `Actor` to make it a stream publisher that keeps track of the subscription life cycle and requested elements. Here is an example of such an actor. It dispatches incoming jobs to the attached subscriber: -.. includecode:: ../code/docs/stream/ActorPublisherDocSpec.scala#job-manager +@@snip [ActorPublisherDocSpec.scala](../code/docs/stream/ActorPublisherDocSpec.scala) { #job-manager } -You send elements to the stream by calling ``onNext``. You are allowed to send as many +You send elements to the stream by calling `onNext`. You are allowed to send as many elements as have been requested by the stream subscriber. This amount can be inquired with -``totalDemand``. It is only allowed to use ``onNext`` when ``isActive`` and ``totalDemand>0``, -otherwise ``onNext`` will throw ``IllegalStateException``. +`totalDemand`. It is only allowed to use `onNext` when `isActive` and `totalDemand>0`, +otherwise `onNext` will throw `IllegalStateException`. -When the stream subscriber requests more elements the ``ActorPublisherMessage.Request`` message -is delivered to this actor, and you can act on that event. The ``totalDemand`` +When the stream subscriber requests more elements the `ActorPublisherMessage.Request` message +is delivered to this actor, and you can act on that event. The `totalDemand` is updated automatically. -When the stream subscriber cancels the subscription the ``ActorPublisherMessage.Cancel`` message -is delivered to this actor. After that subsequent calls to ``onNext`` will be ignored. +When the stream subscriber cancels the subscription the `ActorPublisherMessage.Cancel` message +is delivered to this actor. After that subsequent calls to `onNext` will be ignored. -You can complete the stream by calling ``onComplete``. After that you are not allowed to -call ``onNext``, ``onError`` and ``onComplete``. +You can complete the stream by calling `onComplete`. After that you are not allowed to +call `onNext`, `onError` and `onComplete`. -You can terminate the stream with failure by calling ``onError``. After that you are not allowed to -call ``onNext``, ``onError`` and ``onComplete``. +You can terminate the stream with failure by calling `onError`. After that you are not allowed to +call `onNext`, `onError` and `onComplete`. -If you suspect that this ``ActorPublisher`` may never get subscribed to, you can override the ``subscriptionTimeout`` +If you suspect that this `ActorPublisher` may never get subscribed to, you can override the `subscriptionTimeout` method to provide a timeout after which this Publisher should be considered canceled. The actor will be notified when -the timeout triggers via an ``ActorPublisherMessage.SubscriptionTimeoutExceeded`` message and MUST then perform +the timeout triggers via an `ActorPublisherMessage.SubscriptionTimeoutExceeded` message and MUST then perform cleanup and stop itself. If the actor is stopped the stream will be completed, unless it was not already terminated with @@ -480,47 +466,49 @@ failure, completed or canceled. More detailed information can be found in the API documentation. -This is how it can be used as input :class:`Source` to a :class:`Flow`: +This is how it can be used as input `Source` to a `Flow`: -.. includecode:: ../code/docs/stream/ActorPublisherDocSpec.scala#actor-publisher-usage +@@snip [ActorPublisherDocSpec.scala](../code/docs/stream/ActorPublisherDocSpec.scala) { #actor-publisher-usage } -A publisher that is created with ``Sink.asPublisher`` supports a specified number of subscribers. Additional -subscription attempts will be rejected with an :class:`IllegalStateException`. +A publisher that is created with `Sink.asPublisher` supports a specified number of subscribers. Additional +subscription attempts will be rejected with an `IllegalStateException`. -ActorSubscriber ---------------- +#### ActorSubscriber -.. warning:: - **Deprecation warning:** ``ActorSubscriber`` is deprecated in favour of the vastly more - type-safe and safe to implement :class:`akka.stream.stage.GraphStage`. It can also - expose a "stage actor ref" is needed to be addressed as-if an Actor. - Custom stages implemented using ``GraphStage`` are also automatically fusable. - - To learn more about implementing custom stages using it refer to :ref:`graphstage-scala`. +@@@ warning -Extend/mixin :class:`akka.stream.actor.ActorSubscriber` in your :class:`Actor` to make it a +**Deprecation warning:** `ActorSubscriber` is deprecated in favour of the vastly more +type-safe and safe to implement `akka.stream.stage.GraphStage`. It can also +expose a "stage actor ref" is needed to be addressed as-if an Actor. +Custom stages implemented using `GraphStage` are also automatically fusable. + +To learn more about implementing custom stages using it refer to @ref:[Custom processing with GraphStage](stream-customize.md#graphstage-scala). + +@@@ + +Extend/mixin `akka.stream.actor.ActorSubscriber` in your `Actor` to make it a stream subscriber with full control of stream back pressure. It will receive -``ActorSubscriberMessage.OnNext``, ``ActorSubscriberMessage.OnComplete`` and ``ActorSubscriberMessage.OnError`` +`ActorSubscriberMessage.OnNext`, `ActorSubscriberMessage.OnComplete` and `ActorSubscriberMessage.OnError` messages from the stream. It can also receive other, non-stream messages, in the same way as any actor. Here is an example of such an actor. It dispatches incoming jobs to child worker actors: -.. includecode:: ../code/docs/stream/ActorSubscriberDocSpec.scala#worker-pool +@@snip [ActorSubscriberDocSpec.scala](../code/docs/stream/ActorSubscriberDocSpec.scala) { #worker-pool } -Subclass must define the ``RequestStrategy`` to control stream back pressure. -After each incoming message the ``ActorSubscriber`` will automatically invoke -the ``RequestStrategy.requestDemand`` and propagate the returned demand to the stream. +Subclass must define the `RequestStrategy` to control stream back pressure. +After each incoming message the `ActorSubscriber` will automatically invoke +the `RequestStrategy.requestDemand` and propagate the returned demand to the stream. -* The provided ``WatermarkRequestStrategy`` is a good strategy if the actor performs work itself. -* The provided ``MaxInFlightRequestStrategy`` is useful if messages are queued internally or - delegated to other actors. -* You can also implement a custom ``RequestStrategy`` or call ``request`` manually together with - ``ZeroRequestStrategy`` or some other strategy. In that case - you must also call ``request`` when the actor is started or when it is ready, otherwise - it will not receive any elements. + * The provided `WatermarkRequestStrategy` is a good strategy if the actor performs work itself. + * The provided `MaxInFlightRequestStrategy` is useful if messages are queued internally or +delegated to other actors. + * You can also implement a custom `RequestStrategy` or call `request` manually together with +`ZeroRequestStrategy` or some other strategy. In that case +you must also call `request` when the actor is started or when it is ready, otherwise +it will not receive any elements. More detailed information can be found in the API documentation. -This is how it can be used as output :class:`Sink` to a :class:`Flow`: +This is how it can be used as output `Sink` to a `Flow`: -.. includecode:: ../code/docs/stream/ActorSubscriberDocSpec.scala#actor-subscriber-usage +@@snip [ActorSubscriberDocSpec.scala](../code/docs/stream/ActorSubscriberDocSpec.scala) { #actor-subscriber-usage } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-introduction.md b/akka-docs/src/main/paradox/scala/stream/stream-introduction.md index 177cc294aa..77788b0ad5 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-introduction.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-introduction.md @@ -1,11 +1,6 @@ -.. _stream-introduction-scala: +# Introduction -############ -Introduction -############ - -Motivation -========== +## Motivation The way we consume services from the Internet today includes many instances of streaming data, both downloading from a service as well as uploading to it or @@ -35,7 +30,7 @@ efficiently and with bounded resource usage—no more OutOfMemoryErrors. In orde to achieve this our streams need to be able to limit the buffering that they employ, they need to be able to slow down producers if the consumers cannot keep up. This feature is called back-pressure and is at the core of the -`Reactive Streams`_ initiative of which Akka is a +[Reactive Streams](http://reactive-streams.org/) initiative of which Akka is a founding member. For you this means that the hard problem of propagating and reacting to back-pressure has been incorporated in the design of Akka Streams already, so you have one less thing to worry about; it also means that Akka @@ -43,10 +38,7 @@ Streams interoperate seamlessly with all other Reactive Streams implementations (where Reactive Streams interfaces define the interoperability SPI while implementations like Akka Streams offer a nice user API). -.. _Reactive Streams: http://reactive-streams.org/ - -Relationship with Reactive Streams ----------------------------------- +### Relationship with Reactive Streams The Akka Streams API is completely decoupled from the Reactive Streams interfaces. While Akka Streams focus on the formulation of transformations on @@ -64,22 +56,20 @@ define interfaces such that different streaming implementation can interoperate; it is not the purpose of Reactive Streams to describe an end-user API. -How to read these docs -====================== +## How to read these docs Stream processing is a different paradigm to the Actor Model or to Future composition, therefore it may take some careful study of this subject until you feel familiar with the tools and techniques. The documentation is here to help and for best results we recommend the following approach: -* Read the :ref:`stream-quickstart-scala` to get a feel for how streams - look like and what they can do. -* The top-down learners may want to peruse the :ref:`stream-design` at this - point. -* The bottom-up learners may feel more at home rummaging through the - :ref:`stream-cookbook-scala`. -* For a complete overview of the built-in processing stages you can look at the - table in :ref:`stages-overview_scala` -* The other sections can be read sequentially or as needed during the previous - steps, each digging deeper into specific topics. - + * Read the @ref:[Quick Start Guide](stream-quickstart.md#stream-quickstart-scala) to get a feel for how streams +look like and what they can do. + * The top-down learners may want to peruse the @ref:[Design Principles behind Akka Streams](../../general/stream/stream-design.md) at this +point. + * The bottom-up learners may feel more at home rummaging through the +@ref:[Streams Cookbook](stream-cookbook.md). + * For a complete overview of the built-in processing stages you can look at the +table in @ref:[stages-overview_scala](stages-overview.md) + * The other sections can be read sequentially or as needed during the previous +steps, each digging deeper into specific topics. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-io.md b/akka-docs/src/main/paradox/scala/stream/stream-io.md index d58eb2fee8..175206b290 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-io.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-io.md @@ -1,64 +1,59 @@ -.. _stream-io-scala: - -######################### -Working with streaming IO -######################### +# Working with streaming IO Akka Streams provides a way of handling File IO and TCP connections with Streams. -While the general approach is very similar to the :ref:`Actor based TCP handling ` using Akka IO, +While the general approach is very similar to the @ref:[Actor based TCP handling](../io-tcp.md) using Akka IO, by using Akka Streams you are freed of having to manually react to back-pressure signals, as the library does it transparently for you. -Streaming TCP -============= +## Streaming TCP -Accepting connections: Echo Server -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -In order to implement a simple EchoServer we ``bind`` to a given address, which returns a ``Source[IncomingConnection, Future[ServerBinding]]``, -which will emit an :class:`IncomingConnection` element for each new connection that the Server should handle: +### Accepting connections: Echo Server -.. includecode:: ../code/docs/stream/io/StreamTcpDocSpec.scala#echo-server-simple-bind +In order to implement a simple EchoServer we `bind` to a given address, which returns a `Source[IncomingConnection, Future[ServerBinding]]`, +which will emit an `IncomingConnection` element for each new connection that the Server should handle: -.. image:: ../../images/tcp-stream-bind.png +@@snip [StreamTcpDocSpec.scala](../code/docs/stream/io/StreamTcpDocSpec.scala) { #echo-server-simple-bind } -Next, we simply handle *each* incoming connection using a :class:`Flow` which will be used as the processing stage -to handle and emit ``ByteString`` s from and to the TCP Socket. Since one :class:`ByteString` does not have to necessarily -correspond to exactly one line of text (the client might be sending the line in chunks) we use the ``Framing.delimiter`` +![tcp-stream-bind.png](../../images/tcp-stream-bind.png) + +Next, we simply handle *each* incoming connection using a `Flow` which will be used as the processing stage +to handle and emit `ByteString` s from and to the TCP Socket. Since one `ByteString` does not have to necessarily +correspond to exactly one line of text (the client might be sending the line in chunks) we use the `Framing.delimiter` helper Flow to chunk the inputs up into actual lines of text. The last boolean argument indicates that we require an explicit line ending even for the last message before the connection is closed. In this example we simply add exclamation marks to each incoming text message and push it through the flow: -.. includecode:: ../code/docs/stream/io/StreamTcpDocSpec.scala#echo-server-simple-handle +@@snip [StreamTcpDocSpec.scala](../code/docs/stream/io/StreamTcpDocSpec.scala) { #echo-server-simple-handle } -.. image:: ../../images/tcp-stream-run.png +![tcp-stream-run.png](../../images/tcp-stream-run.png) Notice that while most building blocks in Akka Streams are reusable and freely shareable, this is *not* the case for the incoming connection Flow, since it directly corresponds to an existing, already accepted connection its handling can only ever be materialized *once*. -Closing connections is possible by cancelling the *incoming connection* :class:`Flow` from your server logic (e.g. by -connecting its downstream to a :class:`Sink.cancelled` and its upstream to a :class:`Source.empty`). -It is also possible to shut down the server's socket by cancelling the :class:`IncomingConnection` source ``connections``. +Closing connections is possible by cancelling the *incoming connection* `Flow` from your server logic (e.g. by +connecting its downstream to a `Sink.cancelled` and its upstream to a `Source.empty`). +It is also possible to shut down the server's socket by cancelling the `IncomingConnection` source `connections`. -We can then test the TCP server by sending data to the TCP Socket using ``netcat``: +We can then test the TCP server by sending data to the TCP Socket using `netcat`: -:: +``` +$ echo -n "Hello World" | netcat 127.0.0.1 8888 +Hello World!!! +``` - $ echo -n "Hello World" | netcat 127.0.0.1 8888 - Hello World!!! +### Connecting: REPL Client -Connecting: REPL Client -^^^^^^^^^^^^^^^^^^^^^^^ In this example we implement a rather naive Read Evaluate Print Loop client over TCP. Let's say we know a server has exposed a simple command line interface over TCP, and would like to interact with it using Akka Streams over TCP. To open an outgoing connection socket we use -the ``outgoingConnection`` method: +the `outgoingConnection` method: -.. includecode:: ../code/docs/stream/io/StreamTcpDocSpec.scala#repl-client +@@snip [StreamTcpDocSpec.scala](../code/docs/stream/io/StreamTcpDocSpec.scala) { #repl-client } -The ``repl`` flow we use to handle the server interaction first prints the servers response, then awaits on input from +The `repl` flow we use to handle the server interaction first prints the servers response, then awaits on input from the command line (this blocking call is used here just for the sake of simplicity) and converts it to a -:class:`ByteString` which is then sent over the wire to the server. Then we simply connect the TCP pipeline to this +`ByteString` which is then sent over the wire to the server. Then we simply connect the TCP pipeline to this processing stage–at this point it will be materialized and start processing data once the server responds with an *initial message*. @@ -66,19 +61,22 @@ A resilient REPL client would be more sophisticated than this, for example it sh a separate mapAsync step and have a way to let the server write more data than one ByteString chunk at any given time, these improvements however are left as exercise for the reader. -Avoiding deadlocks and liveness issues in back-pressured cycles -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Avoiding deadlocks and liveness issues in back-pressured cycles + When writing such end-to-end back-pressured systems you may sometimes end up in a situation of a loop, in which *either side is waiting for the other one to start the conversation*. One does not need to look far to find examples of such back-pressure loops. In the two examples shown previously, we always assumed that the side we are connecting to would start the conversation, which effectively means both sides are back-pressured and can not get -the conversation started. There are multiple ways of dealing with this which are explained in depth in :ref:`graph-cycles-scala`, +the conversation started. There are multiple ways of dealing with this which are explained in depth in @ref:[Graph cycles, liveness and deadlocks](stream-graphs.md#graph-cycles-scala), however in client-server scenarios it is often the simplest to make either side simply send an initial message. -.. note:: - In case of back-pressured cycles (which can occur even between different systems) sometimes you have to decide - which of the sides has start the conversation in order to kick it off. This can be often done by injecting an - initial message from one of the sides–a conversation starter. +@@@ note + +In case of back-pressured cycles (which can occur even between different systems) sometimes you have to decide +which of the sides has start the conversation in order to kick it off. This can be often done by injecting an +initial message from one of the sides–a conversation starter. + +@@@ To break this back-pressure cycle we need to inject some initial message, a "conversation starter". First, we need to decide which side of the connection should remain passive and which active. @@ -86,31 +84,29 @@ Thankfully in most situations finding the right spot to start the conversation i to the protocol we are trying to implement using Streams. In chat-like applications, which our examples resemble, it makes sense to make the Server initiate the conversation by emitting a "hello" message: -.. includecode:: ../code/docs/stream/io/StreamTcpDocSpec.scala#welcome-banner-chat-server +@@snip [StreamTcpDocSpec.scala](../code/docs/stream/io/StreamTcpDocSpec.scala) { #welcome-banner-chat-server } -To emit the initial message we merge a ``Source`` with a single element, after the command processing but before the -framing and transformation to ``ByteString`` s this way we do not have to repeat such logic. +To emit the initial message we merge a `Source` with a single element, after the command processing but before the +framing and transformation to `ByteString` s this way we do not have to repeat such logic. -In this example both client and server may need to close the stream based on a parsed command - ``BYE`` in the case -of the server, and ``q`` in the case of the client. This is implemented by taking from the stream until ``q`` and -and concatenating a ``Source`` with a single ``BYE`` element which will then be sent after the original source completed. +In this example both client and server may need to close the stream based on a parsed command - `BYE` in the case +of the server, and `q` in the case of the client. This is implemented by taking from the stream until `q` and +and concatenating a `Source` with a single `BYE` element which will then be sent after the original source completed. -Streaming File IO -================= +## Streaming File IO -Akka Streams provide simple Sources and Sinks that can work with :class:`ByteString` instances to perform IO operations +Akka Streams provide simple Sources and Sinks that can work with `ByteString` instances to perform IO operations on files. +Streaming data from a file is as easy as creating a *FileIO.fromPath* given a target path, and an optional +`chunkSize` which determines the buffer size determined as one "element" in such stream: -Streaming data from a file is as easy as creating a `FileIO.fromPath` given a target path, and an optional -``chunkSize`` which determines the buffer size determined as one "element" in such stream: - -.. includecode:: ../code/docs/stream/io/StreamFileDocSpec.scala#file-source +@@snip [StreamFileDocSpec.scala](../code/docs/stream/io/StreamFileDocSpec.scala) { #file-source } Please note that these processing stages are backed by Actors and by default are configured to run on a pre-configured threadpool-backed dispatcher dedicated for File IO. This is very important as it isolates the blocking file IO operations from the rest of the ActorSystem allowing each dispatcher to be utilised in the most efficient way. If you want to configure a custom -dispatcher for file IO operations globally, you can do so by changing the ``akka.stream.blocking-io-dispatcher``, +dispatcher for file IO operations globally, you can do so by changing the `akka.stream.blocking-io-dispatcher`, or for a specific stage by specifying a custom Dispatcher in code, like this: -.. includecode:: ../code/docs/stream/io/StreamFileDocSpec.scala#custom-dispatcher-code +@@snip [StreamFileDocSpec.scala](../code/docs/stream/io/StreamFileDocSpec.scala) { #custom-dispatcher-code } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-parallelism.md b/akka-docs/src/main/paradox/scala/stream/stream-parallelism.md index 57d46533b0..2631c1cc1a 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-parallelism.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-parallelism.md @@ -1,15 +1,11 @@ -.. _stream-parallelism-scala: - -########################## -Pipelining and Parallelism -########################## +# Pipelining and Parallelism Akka Streams processing stages (be it simple operators on Flows and Sources or graph junctions) are "fused" together and executed sequentially by default. This avoids the overhead of events crossing asynchronous boundaries but limits the flow to execute at most one stage at any given time. In many cases it is useful to be able to concurrently execute the stages of a flow, this is done by explicitly marking -them as asynchronous using the ``async`` method. Each processing stage marked as asynchronous will run in a +them as asynchronous using the `async` method. Each processing stage marked as asynchronous will run in a dedicated actor internally, while all stages not marked asynchronous will run in one single actor. We will illustrate through the example of pancake cooking how streams can be used for various processing patterns, @@ -18,8 +14,7 @@ like to make pancakes, but they need to produce sufficient amount in a cooking s happy. To increase their pancake production throughput they use two frying pans. How they organize their pancake processing is markedly different. -Pipelining ----------- +## Pipelining Roland uses the two frying pans in an asymmetric fashion. The first pan is only used to fry one side of the pancake then the half-finished pancake is flipped into the second pan for the finishing fry on the other side. @@ -28,49 +23,51 @@ are two pancakes being cooked at the same time, one being cooked on its first si completion. This is how this setup would look like implemented as a stream: -.. includecode:: ../code/docs/stream/FlowParallelismDocSpec.scala#pipelining +@@snip [FlowParallelismDocSpec.scala](../code/docs/stream/FlowParallelismDocSpec.scala) { #pipelining } -The two ``map`` stages in sequence (encapsulated in the "frying pan" flows) will be executed in a pipelined way, +The two `map` stages in sequence (encapsulated in the "frying pan" flows) will be executed in a pipelined way, basically doing the same as Roland with his frying pans: - 1. A ``ScoopOfBatter`` enters ``fryingPan1`` - 2. ``fryingPan1`` emits a HalfCookedPancake once ``fryingPan2`` becomes available - 3. ``fryingPan2`` takes the HalfCookedPancake +> + 1. A `ScoopOfBatter` enters `fryingPan1` + 2. `fryingPan1` emits a HalfCookedPancake once `fryingPan2` becomes available + 3. `fryingPan2` takes the HalfCookedPancake 4. at this point fryingPan1 already takes the next scoop, without waiting for fryingPan2 to finish The benefit of pipelining is that it can be applied to any sequence of processing steps that are otherwise not parallelisable (for example because the result of a processing step depends on all the information from the previous step). One drawback is that if the processing times of the stages are very different then some of the stages will not be able to operate at full throughput because they will wait on a previous or subsequent stage most of the time. In the -pancake example frying the second half of the pancake is usually faster than frying the first half, ``fryingPan2`` will -not be able to operate at full capacity [#]_. +pancake example frying the second half of the pancake is usually faster than frying the first half, `fryingPan2` will +not be able to operate at full capacity [1]. -.. note:: - Asynchronous stream processing stages have internal buffers to make communication between them more efficient. - For more details about the behavior of these and how to add additional buffers refer to :ref:`stream-rate-scala`. +@@@ note -Parallel processing -------------------- +Asynchronous stream processing stages have internal buffers to make communication between them more efficient. +For more details about the behavior of these and how to add additional buffers refer to @ref:[Buffers and working with rate](stream-rate.md). + +@@@ + +## Parallel processing Patrik uses the two frying pans symmetrically. He uses both pans to fully fry a pancake on both sides, then puts the results on a shared plate. Whenever a pan becomes empty, he takes the next scoop from the shared bowl of batter. In essence he parallelizes the same process over multiple pans. This is how this setup will look like if implemented using streams: -.. includecode:: ../code/docs/stream/FlowParallelismDocSpec.scala#parallelism +@@snip [FlowParallelismDocSpec.scala](../code/docs/stream/FlowParallelismDocSpec.scala) { #parallelism } The benefit of parallelizing is that it is easy to scale. In the pancake example it is easy to add a third frying pan with Patrik's method, but Roland cannot add a third frying pan, since that would require a third processing step, which is not practically possible in the case of frying pancakes. One drawback of the example code above that it does not preserve the ordering of pancakes. This might be a problem -if children like to track their "own" pancakes. In those cases the ``Balance`` and ``Merge`` stages should be replaced +if children like to track their "own" pancakes. In those cases the `Balance` and `Merge` stages should be replaced by strict-round robing balancing and merging stages that put in and take out pancakes in a strict order. -A more detailed example of creating a worker pool can be found in the cookbook: :ref:`cookbook-balance-scala` +A more detailed example of creating a worker pool can be found in the cookbook: @ref:[Balancing jobs to a fixed pool of workers](stream-cookbook.md#cookbook-balance-scala) -Combining pipelining and parallel processing --------------------------------------------- +## Combining pipelining and parallel processing The two concurrency patterns that we demonstrated as means to increase throughput are not exclusive. In fact, it is rather simple to combine the two approaches and streams provide @@ -80,7 +77,7 @@ First, let's look at how we can parallelize pipelined processing stages. In the will employ two chefs, each working using Roland's pipelining method, but we use the two chefs in parallel, just like Patrik used the two frying pans. This is how it looks like if expressed as streams: -.. includecode:: ../code/docs/stream/FlowParallelismDocSpec.scala#parallel-pipeline +@@snip [FlowParallelismDocSpec.scala](../code/docs/stream/FlowParallelismDocSpec.scala) { #parallel-pipeline } The above pattern works well if there are many independent jobs that do not depend on the results of each other, but the jobs themselves need multiple processing steps where each step builds on the result of @@ -90,14 +87,15 @@ in sequence. It is also possible to organize parallelized stages into pipelines. This would mean employing four chefs: - - the first two chefs prepare half-cooked pancakes from batter, in parallel, then putting those on a large enough - flat surface. - - the second two chefs take these and fry their other side in their own pans, then they put the pancakes on a shared - plate. +> + * the first two chefs prepare half-cooked pancakes from batter, in parallel, then putting those on a large enough +flat surface. + * the second two chefs take these and fry their other side in their own pans, then they put the pancakes on a shared +plate. This is again straightforward to implement with the streams API: -.. includecode:: ../code/docs/stream/FlowParallelismDocSpec.scala#pipelined-parallel +@@snip [FlowParallelismDocSpec.scala](../code/docs/stream/FlowParallelismDocSpec.scala) { #pipelined-parallel } This usage pattern is less common but might be usable if a certain step in the pipeline might take wildly different times to finish different jobs. The reason is that there are more balance-merge steps in this pattern @@ -105,5 +103,5 @@ compared to the parallel pipelines. This pattern rebalances after each step, whi at the entry point of the pipeline. This only matters however if the processing time distribution has a large deviation. -.. [#] Roland's reason for this seemingly suboptimal procedure is that he prefers the temperature of the second pan - to be slightly lower than the first in order to achieve a more homogeneous result. \ No newline at end of file +> [1] Roland's reason for this seemingly suboptimal procedure is that he prefers the temperature of the second pan +to be slightly lower than the first in order to achieve a more homogeneous result. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-quickstart.md b/akka-docs/src/main/paradox/scala/stream/stream-quickstart.md index f4278125f4..fe9ca0c447 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-quickstart.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-quickstart.md @@ -1,40 +1,39 @@ -.. _stream-quickstart-scala: -Quick Start Guide -================= + +# Quick Start Guide Create a project and add the akka-streams dependency to the build tool of your -choice as described in :ref:`build-tool`. +choice as described in @ref:[Using a build tool](../../intro/getting-started.md#build-tool). A stream usually begins at a source, so this is also how we start an Akka Stream. Before we create one, we import the full complement of streaming tools: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#stream-imports +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #stream-imports } If you want to execute the code samples while you read through the quick start guide, you will also need the following imports: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#other-imports +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #other-imports } And an object to hold your code, for example: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#main-app +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #main-app } Now we will start with a rather simple source, emitting the integers 1 to 100: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#create-source +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #create-source } -The :class:`Source` type is parameterized with two types: the first one is the +The `Source` type is parameterized with two types: the first one is the type of element that this source emits and the second one may signal that running the source produces some auxiliary value (e.g. a network source may provide information about the bound port or the peer’s address). Where no -auxiliary information is produced, the type ``akka.NotUsed`` is used—and a +auxiliary information is produced, the type `akka.NotUsed` is used—and a simple range of integers surely falls into this category. Having created this source means that we have a description of how to emit the first 100 natural numbers, but this source is not yet active. In order to get those numbers out we have to run it: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#run-source +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #run-source } This line will complement the source with a consumer function—in this example we simply print out the numbers to the console—and pass this little stream @@ -42,96 +41,94 @@ setup to an Actor that runs it. This activation is signaled by having “run” part of the method name; there are other methods that run Akka Streams, and they all follow this pattern. -When running this source in a :class:`scala.App` you might notice it does not -terminate, because the :class:`ActorSystem` is never terminated. Luckily -``runForeach`` returns a :class:`Future[Done]` which resolves when the stream finishes: +When running this source in a `scala.App` you might notice it does not +terminate, because the `ActorSystem` is never terminated. Luckily +`runForeach` returns a `Future[Done]` which resolves when the stream finishes: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#run-source-and-terminate +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #run-source-and-terminate } You may wonder where the Actor gets created that runs the stream, and you are -probably also asking yourself what this ``materializer`` means. In order to get +probably also asking yourself what this `materializer` means. In order to get this value we first need to create an Actor system: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#create-materializer +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #create-materializer } There are other ways to create a materializer, e.g. from an -:class:`ActorContext` when using streams from within Actors. The -:class:`Materializer` is a factory for stream execution engines, it is the +`ActorContext` when using streams from within Actors. The +`Materializer` is a factory for stream execution engines, it is the thing that makes streams run—you don’t need to worry about any of the details -just now apart from that you need one for calling any of the ``run`` methods on -a :class:`Source`. The materializer is picked up implicitly if it is omitted -from the ``run`` method call arguments, which we will do in the following. +just now apart from that you need one for calling any of the `run` methods on +a `Source`. The materializer is picked up implicitly if it is omitted +from the `run` method call arguments, which we will do in the following. -The nice thing about Akka Streams is that the :class:`Source` is just a +The nice thing about Akka Streams is that the `Source` is just a description of what you want to run, and like an architect’s blueprint it can be reused, incorporated into a larger design. We may choose to transform the source of integers and write it to a file instead: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#transform-source +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #transform-source } -First we use the ``scan`` combinator to run a computation over the whole -stream: starting with the number 1 (``BigInt(1)``) we multiple by each of +First we use the `scan` combinator to run a computation over the whole +stream: starting with the number 1 (`BigInt(1)`) we multiple by each of the incoming numbers, one after the other; the scan operation emits the initial value and then every calculation result. This yields the series of factorial -numbers which we stash away as a :class:`Source` for later reuse—it is +numbers which we stash away as a `Source` for later reuse—it is important to keep in mind that nothing is actually computed yet, this is just a description of what we want to have computed once we run the stream. Then we -convert the resulting series of numbers into a stream of :class:`ByteString` +convert the resulting series of numbers into a stream of `ByteString` objects describing lines in a text file. This stream is then run by attaching a file as the receiver of the data. In the terminology of Akka Streams this is -called a :class:`Sink`. :class:`IOResult` is a type that IO operations return in +called a `Sink`. `IOResult` is a type that IO operations return in Akka Streams in order to tell you how many bytes or elements were consumed and whether the stream terminated normally or exceptionally. -Reusable Pieces ---------------- +## Reusable Pieces One of the nice parts of Akka Streams—and something that other stream libraries do not offer—is that not only sources can be reused like blueprints, all other -elements can be as well. We can take the file-writing :class:`Sink`, prepend -the processing steps necessary to get the :class:`ByteString` elements from +elements can be as well. We can take the file-writing `Sink`, prepend +the processing steps necessary to get the `ByteString` elements from incoming strings and package that up as a reusable piece as well. Since the language for writing these streams always flows from left to right (just like plain English), we need a starting point that is like a source but with an -“open” input. In Akka Streams this is called a :class:`Flow`: +“open” input. In Akka Streams this is called a `Flow`: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#transform-sink +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #transform-sink } -Starting from a flow of strings we convert each to :class:`ByteString` and then -feed to the already known file-writing :class:`Sink`. The resulting blueprint -is a :class:`Sink[String, Future[IOResult]]`, which means that it +Starting from a flow of strings we convert each to `ByteString` and then +feed to the already known file-writing `Sink`. The resulting blueprint +is a `Sink[String, Future[IOResult]]`, which means that it accepts strings as its input and when materialized it will create auxiliary -information of type ``Future[IOResult]`` (when chaining operations on -a :class:`Source` or :class:`Flow` the type of the auxiliary information—called +information of type `Future[IOResult]` (when chaining operations on +a `Source` or `Flow` the type of the auxiliary information—called the “materialized value”—is given by the leftmost starting point; since we want -to retain what the ``FileIO.toPath`` sink has to offer, we need to say -``Keep.right``). +to retain what the `FileIO.toPath` sink has to offer, we need to say +`Keep.right`). -We can use the new and shiny :class:`Sink` we just created by -attaching it to our ``factorials`` source—after a small adaptation to turn the +We can use the new and shiny `Sink` we just created by +attaching it to our `factorials` source—after a small adaptation to turn the numbers into strings: -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#use-transformed-sink +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #use-transformed-sink } -Time-Based Processing ---------------------- +## Time-Based Processing Before we start looking at a more involved example we explore the streaming -nature of what Akka Streams can do. Starting from the ``factorials`` source +nature of what Akka Streams can do. Starting from the `factorials` source we transform the stream by zipping it together with another stream, -represented by a :class:`Source` that emits the number 0 to 100: the first -number emitted by the ``factorials`` source is the factorial of zero, the +represented by a `Source` that emits the number 0 to 100: the first +number emitted by the `factorials` source is the factorial of zero, the second is the factorial of one, and so on. We combine these two by forming -strings like ``"3! = 6"``. +strings like `"3! = 6"`. -.. includecode:: ../code/docs/stream/QuickStartDocSpec.scala#add-streams +@@snip [QuickStartDocSpec.scala](../code/docs/stream/QuickStartDocSpec.scala) { #add-streams } All operations so far have been time-independent and could have been performed in the same fashion on strict collections of elements. The next line demonstrates that we are in fact dealing with streams that can flow at a -certain speed: we use the ``throttle`` combinator to slow down the stream to 1 -element per second (the second ``1`` in the argument list is the maximum size -of a burst that we want to allow—passing ``1`` means that the first element +certain speed: we use the `throttle` combinator to slow down the stream to 1 +element per second (the second `1` in the argument list is the maximum size +of a burst that we want to allow—passing `1` means that the first element gets through immediately and the second then has to wait for one second and so on). @@ -140,7 +137,7 @@ that is not immediately visible deserves mention, though: if you try and set the streams to produce a billion numbers each then you will notice that your JVM does not crash with an OutOfMemoryError, even though you will also notice that running the streams happens in the background, asynchronously (this is the -reason for the auxiliary information to be provided as a :class:`Future`). The +reason for the auxiliary information to be provided as a `Future`). The secret that makes this work is that Akka Streams implicitly implement pervasive flow control, all combinators respect back-pressure. This allows the throttle combinator to signal to all its upstream sources of data that it can only @@ -149,10 +146,9 @@ second the throttle combinator will assert *back-pressure* upstream. This is basically all there is to Akka Streams in a nutshell—glossing over the fact that there are dozens of sources and sinks and many more stream -transformation combinators to choose from, see also :ref:`stages-overview_scala`. +transformation combinators to choose from, see also @ref:[stages-overview_scala](stages-overview.md). -Reactive Tweets -=============== +# Reactive Tweets A typical use case for stream processing is consuming a live stream of data that we want to extract or aggregate some other data from. In this example we'll consider consuming a stream of tweets and extracting information concerning Akka from them. @@ -166,137 +162,142 @@ allow to control what should happen in such scenarios. Here's the data model we'll be working with throughout the quickstart examples: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#model +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #model } -.. note:: - If you would like to get an overview of the used vocabulary first instead of diving head-first - into an actual example you can have a look at the :ref:`core-concepts-scala` and :ref:`defining-and-running-streams-scala` - sections of the docs, and then come back to this quickstart to see it all pieced together into a simple example application. +@@@ note + +If you would like to get an overview of the used vocabulary first instead of diving head-first +into an actual example you can have a look at the @ref:[Core concepts](stream-flows-and-basics.md#core-concepts-scala) and @ref:[Defining and running streams](stream-flows-and-basics.md#defining-and-running-streams-scala) +sections of the docs, and then come back to this quickstart to see it all pieced together into a simple example application. + +@@@ + +## Transforming and consuming simple streams -Transforming and consuming simple streams ------------------------------------------ The example application we will be looking at is a simple Twitter feed stream from which we'll want to extract certain information, -like for example finding all twitter handles of users who tweet about ``#akka``. +like for example finding all twitter handles of users who tweet about `#akka`. -In order to prepare our environment by creating an :class:`ActorSystem` and :class:`ActorMaterializer`, +In order to prepare our environment by creating an `ActorSystem` and `ActorMaterializer`, which will be responsible for materializing and running the streams we are about to create: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#materializer-setup +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #materializer-setup } -The :class:`ActorMaterializer` can optionally take :class:`ActorMaterializerSettings` which can be used to define -materialization properties, such as default buffer sizes (see also :ref:`async-stream-buffers-scala`), the dispatcher to -be used by the pipeline etc. These can be overridden with ``withAttributes`` on :class:`Flow`, :class:`Source`, :class:`Sink` and :class:`Graph`. +The `ActorMaterializer` can optionally take `ActorMaterializerSettings` which can be used to define +materialization properties, such as default buffer sizes (see also @ref:[Buffers for asynchronous stages](stream-rate.md#async-stream-buffers-scala)), the dispatcher to +be used by the pipeline etc. These can be overridden with `withAttributes` on `Flow`, `Source`, `Sink` and `Graph`. -Let's assume we have a stream of tweets readily available. In Akka this is expressed as a :class:`Source[Out, M]`: +Let's assume we have a stream of tweets readily available. In Akka this is expressed as a `Source[Out, M]`: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#tweet-source +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #tweet-source } -Streams always start flowing from a :class:`Source[Out,M1]` then can continue through :class:`Flow[In,Out,M2]` elements or -more advanced graph elements to finally be consumed by a :class:`Sink[In,M3]` (ignore the type parameters ``M1``, ``M2`` -and ``M3`` for now, they are not relevant to the types of the elements produced/consumed by these classes – they are -"materialized types", which we'll talk about :ref:`below `). +Streams always start flowing from a `Source[Out,M1]` then can continue through `Flow[In,Out,M2]` elements or +more advanced graph elements to finally be consumed by a `Sink[In,M3]` (ignore the type parameters `M1`, `M2` +and `M3` for now, they are not relevant to the types of the elements produced/consumed by these classes – they are +"materialized types", which we'll talk about [below](#materialized-values-quick-scala)). The operations should look familiar to anyone who has used the Scala Collections library, however they operate on streams and not collections of data (which is a very important distinction, as some operations only make sense in streaming and vice versa): -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#authors-filter-map +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #authors-filter-map } -Finally in order to :ref:`materialize ` and run the stream computation we need to attach -the Flow to a :class:`Sink` that will get the Flow running. The simplest way to do this is to call -``runWith(sink)`` on a ``Source``. For convenience a number of common Sinks are predefined and collected as methods on -the :class:`Sink` companion object. +Finally in order to @ref:[materialize](stream-flows-and-basics.md#stream-materialization-scala) and run the stream computation we need to attach +the Flow to a `Sink` that will get the Flow running. The simplest way to do this is to call +`runWith(sink)` on a `Source`. For convenience a number of common Sinks are predefined and collected as methods on +the `Sink` companion object. For now let's simply print each author: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#authors-foreachsink-println +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #authors-foreachsink-println } -or by using the shorthand version (which are defined only for the most popular Sinks such as ``Sink.fold`` and ``Sink.foreach``): +or by using the shorthand version (which are defined only for the most popular Sinks such as `Sink.fold` and `Sink.foreach`): -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#authors-foreach-println +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #authors-foreach-println } -Materializing and running a stream always requires a :class:`Materializer` to be in implicit scope (or passed in explicitly, -like this: ``.run(materializer)``). +Materializing and running a stream always requires a `Materializer` to be in implicit scope (or passed in explicitly, +like this: `.run(materializer)`). The complete snippet looks like this: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#first-sample +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #first-sample } + +## Flattening sequences in streams -Flattening sequences in streams -------------------------------- In the previous section we were working on 1:1 relationships of elements which is the most common case, but sometimes -we might want to map from one element to a number of elements and receive a "flattened" stream, similarly like ``flatMap`` -works on Scala Collections. In order to get a flattened stream of hashtags from our stream of tweets we can use the ``mapConcat`` +we might want to map from one element to a number of elements and receive a "flattened" stream, similarly like `flatMap` +works on Scala Collections. In order to get a flattened stream of hashtags from our stream of tweets we can use the `mapConcat` combinator: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#hashtags-mapConcat +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #hashtags-mapConcat } -.. note:: - The name ``flatMap`` was consciously avoided due to its proximity with for-comprehensions and monadic composition. - It is problematic for two reasons: first, flattening by concatenation is often undesirable in bounded stream processing - due to the risk of deadlock (with merge being the preferred strategy), and second, the monad laws would not hold for - our implementation of flatMap (due to the liveness issues). +@@@ note - Please note that the ``mapConcat`` requires the supplied function to return an iterable (``f: Out => immutable.Iterable[T]``), - whereas ``flatMap`` would have to operate on streams all the way through. +The name `flatMap` was consciously avoided due to its proximity with for-comprehensions and monadic composition. +It is problematic for two reasons: first, flattening by concatenation is often undesirable in bounded stream processing +due to the risk of deadlock (with merge being the preferred strategy), and second, the monad laws would not hold for +our implementation of flatMap (due to the liveness issues). + +Please note that the `mapConcat` requires the supplied function to return an iterable (`f: Out => immutable.Iterable[T]`), +whereas `flatMap` would have to operate on streams all the way through. + +@@@ + +## Broadcasting a stream -Broadcasting a stream ---------------------- Now let's say we want to persist all hashtags, as well as all author names from this one live stream. For example we'd like to write all author handles into one file, and all hashtags into another file on disk. This means we have to split the source stream into two streams which will handle the writing to these different files. Elements that can be used to form such "fan-out" (or "fan-in") structures are referred to as "junctions" in Akka Streams. -One of these that we'll be using in this example is called :class:`Broadcast`, and it simply emits elements from its +One of these that we'll be using in this example is called `Broadcast`, and it simply emits elements from its input port to all of its output ports. Akka Streams intentionally separate the linear stream structures (Flows) from the non-linear, branching ones (Graphs) in order to offer the most convenient API for both of these cases. Graphs can express arbitrarily complex stream setups at the expense of not reading as familiarly as collection transformations. -Graphs are constructed using :class:`GraphDSL` like this: +Graphs are constructed using `GraphDSL` like this: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#graph-dsl-broadcast +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #graph-dsl-broadcast } -As you can see, inside the :class:`GraphDSL` we use an implicit graph builder ``b`` to mutably construct the graph -using the ``~>`` "edge operator" (also read as "connect" or "via" or "to"). The operator is provided implicitly -by importing ``GraphDSL.Implicits._``. +As you can see, inside the `GraphDSL` we use an implicit graph builder `b` to mutably construct the graph +using the `~>` "edge operator" (also read as "connect" or "via" or "to"). The operator is provided implicitly +by importing `GraphDSL.Implicits._`. -``GraphDSL.create`` returns a :class:`Graph`, in this example a :class:`Graph[ClosedShape, NotUsed]` where -:class:`ClosedShape` means that it is *a fully connected graph* or "closed" - there are no unconnected inputs or outputs. -Since it is closed it is possible to transform the graph into a :class:`RunnableGraph` using ``RunnableGraph.fromGraph``. -The runnable graph can then be ``run()`` to materialize a stream out of it. +`GraphDSL.create` returns a `Graph`, in this example a `Graph[ClosedShape, NotUsed]` where +`ClosedShape` means that it is *a fully connected graph* or "closed" - there are no unconnected inputs or outputs. +Since it is closed it is possible to transform the graph into a `RunnableGraph` using `RunnableGraph.fromGraph`. +The runnable graph can then be `run()` to materialize a stream out of it. -Both :class:`Graph` and :class:`RunnableGraph` are *immutable, thread-safe, and freely shareable*. +Both `Graph` and `RunnableGraph` are *immutable, thread-safe, and freely shareable*. A graph can also have one of several other shapes, with one or more unconnected ports. Having unconnected ports expresses a graph that is a *partial graph*. Concepts around composing and nesting graphs in large structures are -explained in detail in :ref:`composition-scala`. It is also possible to wrap complex computation graphs +explained in detail in @ref:[Modularity, Composition and Hierarchy](stream-composition.md). It is also possible to wrap complex computation graphs as Flows, Sinks or Sources, which will be explained in detail in -:ref:`constructing-sources-sinks-flows-from-partial-graphs-scala`. +@ref:[Constructing Sources, Sinks and Flows from Partial Graphs](stream-graphs.md#constructing-sources-sinks-flows-from-partial-graphs-scala). + +## Back-pressure in action -Back-pressure in action ------------------------ One of the main advantages of Akka Streams is that they *always* propagate back-pressure information from stream Sinks (Subscribers) to their Sources (Publishers). It is not an optional feature, and is enabled at all times. To learn more about the back-pressure protocol used by Akka Streams and all other Reactive Streams compatible implementations read -:ref:`back-pressure-explained-scala`. +@ref:[Back-pressure explained](stream-flows-and-basics.md#back-pressure-explained-scala). A typical problem applications (not using Akka Streams) like this often face is that they are unable to process the incoming data fast enough, either temporarily or by design, and will start buffering incoming data until there's no more space to buffer, resulting -in either ``OutOfMemoryError`` s or other severe degradations of service responsiveness. With Akka Streams buffering can +in either `OutOfMemoryError` s or other severe degradations of service responsiveness. With Akka Streams buffering can and must be handled explicitly. For example, if we are only interested in the "*most recent tweets, with a buffer of 10 -elements*" this can be expressed using the ``buffer`` element: +elements*" this can be expressed using the `buffer` element: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#tweets-slow-consumption-dropHead +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #tweets-slow-consumption-dropHead } -The ``buffer`` element takes an explicit and required ``OverflowStrategy``, which defines how the buffer should react -when it receives another element while it is full. Strategies provided include dropping the oldest element (``dropHead``), +The `buffer` element takes an explicit and required `OverflowStrategy`, which defines how the buffer should react +when it receives another element while it is full. Strategies provided include dropping the oldest element (`dropHead`), dropping the entire buffer, signalling errors etc. Be sure to pick and choose the strategy that fits your use case best. -.. _materialized-values-quick-scala: + +## Materialized values -Materialized values -------------------- So far we've been only processing data using Flows and consuming it into some kind of external Sink - be it by printing values or storing them in some external system. However sometimes we may be interested in some value that can be obtained from the materialized processing pipeline. For example, we want to know how many tweets we have processed. @@ -304,43 +305,46 @@ While this question is not as obvious to give an answer to in case of an infinit this question in a streaming setting would be to create a stream of counts described as "*up until now*, we've processed N tweets"), but in general it is possible to deal with finite streams and come up with a nice result such as a total count of elements. -First, let's write such an element counter using ``Sink.fold`` and see how the types look like: +First, let's write such an element counter using `Sink.fold` and see how the types look like: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#tweets-fold-count +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #tweets-fold-count } -First we prepare a reusable ``Flow`` that will change each incoming tweet into an integer of value ``1``. We'll use this in -order to combine those with a ``Sink.fold`` that will sum all ``Int`` elements of the stream and make its result available as -a ``Future[Int]``. Next we connect the ``tweets`` stream to ``count`` with ``via``. Finally we connect the Flow to the previously -prepared Sink using ``toMat``. +First we prepare a reusable `Flow` that will change each incoming tweet into an integer of value `1`. We'll use this in +order to combine those with a `Sink.fold` that will sum all `Int` elements of the stream and make its result available as +a `Future[Int]`. Next we connect the `tweets` stream to `count` with `via`. Finally we connect the Flow to the previously +prepared Sink using `toMat`. -Remember those mysterious ``Mat`` type parameters on ``Source[+Out, +Mat]``, ``Flow[-In, +Out, +Mat]`` and ``Sink[-In, +Mat]``? +Remember those mysterious `Mat` type parameters on ``Source[+Out, +Mat]``, `Flow[-In, +Out, +Mat]` and `Sink[-In, +Mat]`? They represent the type of values these processing parts return when materialized. When you chain these together, -you can explicitly combine their materialized values. In our example we used the ``Keep.right`` predefined function, +you can explicitly combine their materialized values. In our example we used the `Keep.right` predefined function, which tells the implementation to only care about the materialized type of the stage currently appended to the right. -The materialized type of ``sumSink`` is ``Future[Int]`` and because of using ``Keep.right``, the resulting :class:`RunnableGraph` -has also a type parameter of ``Future[Int]``. +The materialized type of `sumSink` is `Future[Int]` and because of using `Keep.right`, the resulting `RunnableGraph` +has also a type parameter of `Future[Int]`. This step does *not* yet materialize the processing pipeline, it merely prepares the description of the Flow, which is now connected to a Sink, and therefore can -be ``run()``, as indicated by its type: ``RunnableGraph[Future[Int]]``. Next we call ``run()`` which uses the implicit :class:`ActorMaterializer` -to materialize and run the Flow. The value returned by calling ``run()`` on a ``RunnableGraph[T]`` is of type ``T``. -In our case this type is ``Future[Int]`` which, when completed, will contain the total length of our ``tweets`` stream. +be `run()`, as indicated by its type: `RunnableGraph[Future[Int]]`. Next we call `run()` which uses the implicit `ActorMaterializer` +to materialize and run the Flow. The value returned by calling `run()` on a `RunnableGraph[T]` is of type `T`. +In our case this type is `Future[Int]` which, when completed, will contain the total length of our `tweets` stream. In case of the stream failing, this future would complete with a Failure. -A :class:`RunnableGraph` may be reused +A `RunnableGraph` may be reused and materialized multiple times, because it is just the "blueprint" of the stream. This means that if we materialize a stream, for example one that consumes a live stream of tweets within a minute, the materialized values for those two materializations will be different, as illustrated by this example: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#tweets-runnable-flow-materialized-twice +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #tweets-runnable-flow-materialized-twice } Many elements in Akka Streams provide materialized values which can be used for obtaining either results of computation or -steering these elements which will be discussed in detail in :ref:`stream-materialization-scala`. Summing up this section, now we know +steering these elements which will be discussed in detail in @ref:[Stream Materialization](stream-flows-and-basics.md#stream-materialization-scala). Summing up this section, now we know what happens behind the scenes when we run this one-liner, which is equivalent to the multi line version above: -.. includecode:: ../code/docs/stream/TwitterStreamQuickstartDocSpec.scala#tweets-fold-count-oneline +@@snip [TwitterStreamQuickstartDocSpec.scala](../code/docs/stream/TwitterStreamQuickstartDocSpec.scala) { #tweets-fold-count-oneline } -.. note:: - ``runWith()`` is a convenience method that automatically ignores the materialized value of any other stages except - those appended by the ``runWith()`` itself. In the above example it translates to using ``Keep.right`` as the combiner - for materialized values. +@@@ note + +`runWith()` is a convenience method that automatically ignores the materialized value of any other stages except +those appended by the `runWith()` itself. In the above example it translates to using `Keep.right` as the combiner +for materialized values. + +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-rate.md b/akka-docs/src/main/paradox/scala/stream/stream-rate.md index 9ea1495cfa..b142f16089 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-rate.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-rate.md @@ -1,46 +1,40 @@ -.. _stream-rate-scala: - -############################# -Buffers and working with rate -############################# +# Buffers and working with rate When upstream and downstream rates differ, especially when the throughput has spikes, it can be useful to introduce buffers in a stream. In this chapter we cover how buffers are used in Akka Streams. -.. _async-stream-buffers-scala: - -Buffers for asynchronous stages -=============================== + +## Buffers for asynchronous stages In this section we will discuss internal buffers that are introduced as an optimization when using asynchronous stages. -To run a stage asynchronously it has to be marked explicitly as such using the ``.async`` method. Being run +To run a stage asynchronously it has to be marked explicitly as such using the `.async` method. Being run asynchronously means that a stage, after handing out an element to its downstream consumer is able to immediately process the next message. To demonstrate what we mean by this, let's take a look at the following example: -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#pipelining +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #pipelining } Running the above example, one of the possible outputs looks like this: -:: +``` +A: 1 +A: 2 +B: 1 +A: 3 +B: 2 +C: 1 +B: 3 +C: 2 +C: 3 +``` - A: 1 - A: 2 - B: 1 - A: 3 - B: 2 - C: 1 - B: 3 - C: 2 - C: 3 - -Note that the order is *not* ``A:1, B:1, C:1, A:2, B:2, C:2,`` which would correspond to the normal fused synchronous +Note that the order is *not* `A:1, B:1, C:1, A:2, B:2, C:2,` which would correspond to the normal fused synchronous execution model of flows where an element completely passes through the processing pipeline before the next element enters the flow. The next element is processed by an asynchronous stage as soon as it is emitted the previous one. While pipelining in general increases throughput, in practice there is a cost of passing an element through the asynchronous (and therefore thread crossing) boundary which is significant. To amortize this cost Akka Streams uses -a *windowed*, *batching* backpressure strategy internally. It is windowed because as opposed to a `Stop-And-Wait`_ +a *windowed*, *batching* backpressure strategy internally. It is windowed because as opposed to a [Stop-And-Wait](https://en.wikipedia.org/wiki/Stop-and-wait_ARQ) protocol multiple elements might be "in-flight" concurrently with requests for elements. It is also batching because a new element is not immediately requested once an element has been drained from the window-buffer but multiple elements are requested after multiple elements have been drained. This batching strategy reduces the communication cost of @@ -53,11 +47,7 @@ the throughput of the connected chain. There are tools in Akka Streams however t of a processing chain to be "detached" or to define the maximum throughput of the stream through external timing sources. These situations are exactly those where the internal batching buffering strategy suddenly becomes non-transparent. -.. _Stop-And-Wait: https://en.wikipedia.org/wiki/Stop-and-wait_ARQ - - -Internal buffers and their effect ---------------------------------- +### Internal buffers and their effect As we have explained, for performance reasons Akka Streams introduces a buffer for every asynchronous processing stage. The purpose of these buffers is solely optimization, in fact the size of 1 would be the most natural choice if there @@ -65,37 +55,38 @@ would be no need for throughput improvements. Therefore it is recommended to kee and increase them only to a level suitable for the throughput requirements of the application. Default buffer sizes can be set through configuration: -:: +``` +akka.stream.materializer.max-input-buffer-size = 16 +``` - akka.stream.materializer.max-input-buffer-size = 16 +Alternatively they can be set by passing a `ActorMaterializerSettings` to the materializer: -Alternatively they can be set by passing a :class:`ActorMaterializerSettings` to the materializer: +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #materializer-buffer } -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#materializer-buffer +If the buffer size needs to be set for segments of a `Flow` only, it is possible by defining a separate +`Flow` with these attributes: -If the buffer size needs to be set for segments of a :class:`Flow` only, it is possible by defining a separate -:class:`Flow` with these attributes: - -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#section-buffer +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #section-buffer } Here is an example of a code that demonstrate some of the issues caused by internal buffers: -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#buffering-abstraction-leak +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #buffering-abstraction-leak } -Running the above example one would expect the number *3* to be printed in every 3 seconds (the ``conflateWithSeed`` -step here is configured so that it counts the number of elements received before the downstream ``ZipWith`` consumes +Running the above example one would expect the number *3* to be printed in every 3 seconds (the `conflateWithSeed` +step here is configured so that it counts the number of elements received before the downstream `ZipWith` consumes them). What is being printed is different though, we will see the number *1*. The reason for this is the internal -buffer which is by default 16 elements large, and prefetches elements before the ``ZipWith`` starts consuming them. -It is possible to fix this issue by changing the buffer size of ``ZipWith`` (or the whole graph) to 1. We will still see -a leading 1 though which is caused by an initial prefetch of the ``ZipWith`` element. +buffer which is by default 16 elements large, and prefetches elements before the `ZipWith` starts consuming them. +It is possible to fix this issue by changing the buffer size of `ZipWith` (or the whole graph) to 1. We will still see +a leading 1 though which is caused by an initial prefetch of the `ZipWith` element. -.. note:: - In general, when time or rate driven processing stages exhibit strange behavior, one of the first solutions to try - should be to decrease the input buffer of the affected elements to 1. +@@@ note +In general, when time or rate driven processing stages exhibit strange behavior, one of the first solutions to try +should be to decrease the input buffer of the affected elements to 1. -Buffers in Akka Streams -======================= +@@@ + +## Buffers in Akka Streams In this section we will discuss *explicit* user defined buffers that are part of the domain logic of the stream processing pipeline of an application. @@ -103,7 +94,7 @@ pipeline of an application. The example below will ensure that 1000 jobs (but not more) are dequeued from an external (imaginary) system and stored locally in memory - relieving the external system: -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#explicit-buffers-backpressure +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #explicit-buffers-backpressure } The next example will also queue up 1000 jobs locally, but if there are more jobs waiting in the imaginary external systems, it makes space for the new element by @@ -111,12 +102,12 @@ dropping one element from the *tail* of the buffer. Dropping from the tail is a it must be noted that this will drop the *youngest* waiting job. If some "fairness" is desired in the sense that we want to be nice to jobs that has been waiting for long, then this option can be useful. -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#explicit-buffers-droptail +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #explicit-buffers-droptail } Instead of dropping the youngest element from the tail of the buffer a new element can be dropped without enqueueing it to the buffer at all. -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#explicit-buffers-dropnew +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #explicit-buffers-dropnew } Here is another example with a queue of 1000 jobs, but it makes space for the new element by dropping one element from the *head* of the buffer. This is the *oldest* @@ -125,13 +116,13 @@ resent if not processed in a certain period. The oldest element will be retransmitted soon, (in fact a retransmitted duplicate might be already in the queue!) so it makes sense to drop it first. -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#explicit-buffers-drophead +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #explicit-buffers-drophead } Compared to the dropping strategies above, dropBuffer drops all the 1000 jobs it has enqueued once the buffer gets full. This aggressive strategy is useful when dropping jobs is preferred to delaying jobs. -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#explicit-buffers-dropbuffer +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #explicit-buffers-dropbuffer } If our imaginary external job provider is a client using our API, we might want to enforce that the client cannot have more than 1000 queued jobs @@ -139,48 +130,43 @@ otherwise we consider it flooding and terminate the connection. This is easily achievable by the error strategy which simply fails the stream once the buffer gets full. -.. includecode:: ../code/docs/stream/StreamBuffersRateSpec.scala#explicit-buffers-fail +@@snip [StreamBuffersRateSpec.scala](../code/docs/stream/StreamBuffersRateSpec.scala) { #explicit-buffers-fail } -Rate transformation -=================== +## Rate transformation -Understanding conflate ----------------------- +### Understanding conflate -When a fast producer can not be informed to slow down by backpressure or some other signal, ``conflate`` might be +When a fast producer can not be informed to slow down by backpressure or some other signal, `conflate` might be useful to combine elements from a producer until a demand signal comes from a consumer. Below is an example snippet that summarizes fast stream of elements to a standart deviation, mean and count of elements that have arrived while the stats have been calculated. -.. includecode:: ../code/docs/stream/RateTransformationDocSpec.scala#conflate-summarize +@@snip [RateTransformationDocSpec.scala](../code/docs/stream/RateTransformationDocSpec.scala) { #conflate-summarize } This example demonstrates that such flow's rate is decoupled. The element rate at the start of the flow can be much higher that the element rate at the end of the flow. -Another possible use of ``conflate`` is to not consider all elements for summary when producer starts getting too fast. -Example below demonstrates how ``conflate`` can be used to implement random drop of elements when consumer is not able +Another possible use of `conflate` is to not consider all elements for summary when producer starts getting too fast. +Example below demonstrates how `conflate` can be used to implement random drop of elements when consumer is not able to keep up with the producer. -.. includecode:: ../code/docs/stream/RateTransformationDocSpec.scala#conflate-sample +@@snip [RateTransformationDocSpec.scala](../code/docs/stream/RateTransformationDocSpec.scala) { #conflate-sample } -Understanding expand --------------------- +### Understanding expand Expand helps to deal with slow producers which are unable to keep up with the demand coming from consumers. Expand allows to extrapolate a value to be sent as an element to a consumer. -As a simple use of ``expand`` here is a flow that sends the same element to consumer when producer does not send +As a simple use of `expand` here is a flow that sends the same element to consumer when producer does not send any new elements. -.. includecode:: ../code/docs/stream/RateTransformationDocSpec.scala#expand-last +@@snip [RateTransformationDocSpec.scala](../code/docs/stream/RateTransformationDocSpec.scala) { #expand-last } Expand also allows to keep some state between demand requests from the downstream. Leveraging this, here is a flow that tracks and reports a drift between fast consumer and slow producer. -.. includecode:: ../code/docs/stream/RateTransformationDocSpec.scala#expand-drift - -Note that all of the elements coming from upstream will go through ``expand`` at least once. This means that the -output of this flow is going to report a drift of zero if producer is fast enough, or a larger drift otherwise. - +@@snip [RateTransformationDocSpec.scala](../code/docs/stream/RateTransformationDocSpec.scala) { #expand-drift } +Note that all of the elements coming from upstream will go through `expand` at least once. This means that the +output of this flow is going to report a drift of zero if producer is fast enough, or a larger drift otherwise. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/stream/stream-testkit.md b/akka-docs/src/main/paradox/scala/stream/stream-testkit.md index 293fecf3f0..72e9a23faa 100644 --- a/akka-docs/src/main/paradox/scala/stream/stream-testkit.md +++ b/akka-docs/src/main/paradox/scala/stream/stream-testkit.md @@ -1,118 +1,112 @@ -.. _stream-testkit-scala: - -############### -Testing streams -############### +# Testing streams Verifying behaviour of Akka Stream sources, flows and sinks can be done using various code patterns and libraries. Here we will discuss testing these elements using: -- simple sources, sinks and flows; -- sources and sinks in combination with :class:`TestProbe` from the :mod:`akka-testkit` module; -- sources and sinks specifically crafted for writing tests from the :mod:`akka-stream-testkit` module. + * simple sources, sinks and flows; + * sources and sinks in combination with `TestProbe` from the `akka-testkit` module; + * sources and sinks specifically crafted for writing tests from the `akka-stream-testkit` module. It is important to keep your data processing pipeline as separate sources, flows and sinks. This makes them easily testable by wiring them up to other -sources or sinks, or some test harnesses that :mod:`akka-testkit` or -:mod:`akka-stream-testkit` provide. +sources or sinks, or some test harnesses that `akka-testkit` or +`akka-stream-testkit` provide. -Built in sources, sinks and combinators -======================================= +## Built in sources, sinks and combinators Testing a custom sink can be as simple as attaching a source that emits elements from a predefined collection, running a constructed test flow and asserting on the results that sink produced. Here is an example of a test for a sink: -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#strict-collection +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #strict-collection } The same strategy can be applied for sources as well. In the next example we have a source that produces an infinite stream of elements. Such source can be tested by asserting that first arbitrary number of elements hold some -condition. Here the ``take`` combinator and ``Sink.seq`` are very useful. +condition. Here the `take` combinator and `Sink.seq` are very useful. -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#grouped-infinite +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #grouped-infinite } When testing a flow we need to attach a source and a sink. As both stream ends are under our control, we can choose sources that tests various edge cases of the flow and sinks that ease assertions. -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#folded-stream +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #folded-stream } -TestKit -======= +## TestKit Akka Stream offers integration with Actors out of the box. This support can be -used for writing stream tests that use familiar :class:`TestProbe` from the -:mod:`akka-testkit` API. +used for writing stream tests that use familiar `TestProbe` from the +`akka-testkit` API. One of the more straightforward tests would be to materialize stream to a -:class:`Future` and then use ``pipe`` pattern to pipe the result of that future +`Future` and then use `pipe` pattern to pipe the result of that future to the probe. -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#pipeto-testprobe +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #pipeto-testprobe } -Instead of materializing to a future, we can use a :class:`Sink.actorRef` that -sends all incoming elements to the given :class:`ActorRef`. Now we can use -assertion methods on :class:`TestProbe` and expect elements one by one as they +Instead of materializing to a future, we can use a `Sink.actorRef` that +sends all incoming elements to the given `ActorRef`. Now we can use +assertion methods on `TestProbe` and expect elements one by one as they arrive. We can also assert stream completion by expecting for -``onCompleteMessage`` which was given to :class:`Sink.actorRef`. +`onCompleteMessage` which was given to `Sink.actorRef`. -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#sink-actorref +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #sink-actorref } -Similarly to :class:`Sink.actorRef` that provides control over received -elements, we can use :class:`Source.actorRef` and have full control over +Similarly to `Sink.actorRef` that provides control over received +elements, we can use `Source.actorRef` and have full control over elements to be sent. -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#source-actorref +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #source-actorref } -Streams TestKit -=============== +## Streams TestKit You may have noticed various code patterns that emerge when testing stream -pipelines. Akka Stream has a separate :mod:`akka-stream-testkit` module that +pipelines. Akka Stream has a separate `akka-stream-testkit` module that provides tools specifically for writing stream tests. This module comes with -two main components that are :class:`TestSource` and :class:`TestSink` which +two main components that are `TestSource` and `TestSink` which provide sources and sinks that materialize to probes that allow fluent API. -.. note:: +@@@ note - Be sure to add the module :mod:`akka-stream-testkit` to your dependencies. +Be sure to add the module `akka-stream-testkit` to your dependencies. -A sink returned by ``TestSink.probe`` allows manual control over demand and +@@@ + +A sink returned by `TestSink.probe` allows manual control over demand and assertions over elements coming downstream. -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#test-sink-probe +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #test-sink-probe } -A source returned by ``TestSource.probe`` can be used for asserting demand or +A source returned by `TestSource.probe` can be used for asserting demand or controlling when stream is completed or ended with an error. -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#test-source-probe +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #test-source-probe } You can also inject exceptions and test sink behaviour on error conditions. -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#injecting-failure +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #injecting-failure } Test source and sink can be used together in combination when testing flows. -.. includecode:: ../code/docs/stream/StreamTestKitDocSpec.scala#test-source-and-sink +@@snip [StreamTestKitDocSpec.scala](../code/docs/stream/StreamTestKitDocSpec.scala) { #test-source-and-sink } - -Fuzzing Mode -============ +## Fuzzing Mode For testing, it is possible to enable a special stream execution mode that exercises concurrent execution paths more aggressively (at the cost of reduced performance) and therefore helps exposing race conditions in tests. To enable this setting add the following line to your configuration: -:: +``` +akka.stream.materializer.debug.fuzzing-mode = on +``` - akka.stream.materializer.debug.fuzzing-mode = on +@@@ warning +Never use this setting in production or benchmarks. This is a testing tool to provide more coverage of your code +during tests, but it reduces the throughput of streams. A warning message will be logged if you have this setting +enabled. -.. warning:: - - Never use this setting in production or benchmarks. This is a testing tool to provide more coverage of your code - during tests, but it reduces the throughput of streams. A warning message will be logged if you have this setting - enabled. \ No newline at end of file +@@@ \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/testing.md b/akka-docs/src/main/paradox/scala/testing.md index 3c9964ba42..364d3bbff6 100644 --- a/akka-docs/src/main/paradox/scala/testing.md +++ b/akka-docs/src/main/paradox/scala/testing.md @@ -1,29 +1,30 @@ -.. _akka-testkit: +# Testing Actor Systems -##################### -Testing Actor Systems -##################### +@@toc -.. toctree:: +@@@ index - testkit-example +* [testkit-example](testkit-example.md) + +@@@ As with any piece of software, automated tests are a very important part of the development cycle. The actor model presents a different view on how units of code are delimited and how they interact, which has an influence on how to perform tests. -Akka comes with a dedicated module :mod:`akka-testkit` for supporting tests at +Akka comes with a dedicated module `akka-testkit` for supporting tests at different levels, which fall into two clearly distinct categories: - - Testing isolated pieces of code without involving the actor model, meaning - without multiple threads; this implies completely deterministic behavior - concerning the ordering of events and no concurrency concerns and will be - called **Unit Testing** in the following. - - Testing (multiple) encapsulated actors including multi-threaded scheduling; - this implies non-deterministic order of events but shielding from - concurrency concerns by the actor model and will be called **Integration - Testing** in the following. +> + * Testing isolated pieces of code without involving the actor model, meaning +without multiple threads; this implies completely deterministic behavior +concerning the ordering of events and no concurrency concerns and will be +called **Unit Testing** in the following. + * Testing (multiple) encapsulated actors including multi-threaded scheduling; +this implies non-deterministic order of events but shielding from +concurrency concerns by the actor model and will be called **Integration +Testing** in the following. There are of course variations on the granularity of tests in both categories, where unit testing reaches down to white-box tests and integration testing can @@ -31,145 +32,147 @@ encompass functional tests of complete actor networks. The important distinction lies in whether concurrency concerns are part of the test or not. The tools offered are described in detail in the following sections. -.. note:: +@@@ note - Be sure to add the module :mod:`akka-testkit` to your dependencies. +Be sure to add the module `akka-testkit` to your dependencies. -Synchronous Unit Testing with :class:`TestActorRef` -=================================================== +@@@ -Testing the business logic inside :class:`Actor` classes can be divided into +## Synchronous Unit Testing with `TestActorRef` + +Testing the business logic inside `Actor` classes can be divided into two parts: first, each atomic operation must work in isolation, then sequences of incoming events must be processed correctly, even in the presence of some possible variability in the ordering of events. The former is the primary use case for single-threaded unit testing, while the latter can only be verified in integration tests. -Normally, the :class:`ActorRef` shields the underlying :class:`Actor` instance +Normally, the `ActorRef` shields the underlying `Actor` instance from the outside, the only communications channel is the actor's mailbox. This restriction is an impediment to unit testing, which led to the inception of the -:class:`TestActorRef`. This special type of reference is designed specifically +`TestActorRef`. This special type of reference is designed specifically for test purposes and allows access to the actor in two ways: either by obtaining a reference to the underlying actor instance, or by invoking or -querying the actor's behaviour (:meth:`receive`). Each one warrants its own +querying the actor's behaviour (`receive`). Each one warrants its own section below. -.. note:: - It is highly recommended to stick to traditional behavioural testing (using messaging - to ask the Actor to reply with the state you want to run assertions against), - instead of using ``TestActorRef`` whenever possible. +@@@ note -.. warning:: - Due to the synchronous nature of ``TestActorRef`` it will **not** work with some support - traits that Akka provides as they require asynchronous behaviours to function properly. - Examples of traits that do not mix well with test actor refs are :ref:`PersistentActor ` - and :ref:`AtLeastOnceDelivery ` provided by :ref:`Akka Persistence `. +It is highly recommended to stick to traditional behavioural testing (using messaging +to ask the Actor to reply with the state you want to run assertions against), +instead of using `TestActorRef` whenever possible. -Obtaining a Reference to an :class:`Actor` ------------------------------------------- +@@@ -Having access to the actual :class:`Actor` object allows application of all +@@@ warning + +Due to the synchronous nature of `TestActorRef` it will **not** work with some support +traits that Akka provides as they require asynchronous behaviours to function properly. +Examples of traits that do not mix well with test actor refs are @ref:[PersistentActor](persistence.md#event-sourcing-scala) +and @ref:[AtLeastOnceDelivery](persistence.md#at-least-once-delivery-scala) provided by @ref:[Akka Persistence](persistence.md). + +@@@ + +### Obtaining a Reference to an `Actor` + +Having access to the actual `Actor` object allows application of all traditional unit testing techniques on the contained methods. Obtaining a reference is done like this: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#test-actor-ref +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-actor-ref } -Since :class:`TestActorRef` is generic in the actor type it returns the +Since `TestActorRef` is generic in the actor type it returns the underlying actor with its proper static type. From this point on you may bring any unit testing tool to bear on your actor as usual. -.. _TestFSMRef: + +### Testing Finite State Machines -Testing Finite State Machines ------------------------------ - -If your actor under test is a :class:`FSM`, you may use the special -:class:`TestFSMRef` which offers all features of a normal :class:`TestActorRef` +If your actor under test is a `FSM`, you may use the special +`TestFSMRef` which offers all features of a normal `TestActorRef` and in addition allows access to the internal state: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#test-fsm-ref +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-fsm-ref } Due to a limitation in Scala’s type inference, there is only the factory method -shown above, so you will probably write code like ``TestFSMRef(new MyFSM)`` -instead of the hypothetical :class:`ActorRef`-inspired ``TestFSMRef[MyFSM]``. +shown above, so you will probably write code like `TestFSMRef(new MyFSM)` +instead of the hypothetical `ActorRef`-inspired `TestFSMRef[MyFSM]`. All methods shown above directly access the FSM state without any -synchronization; this is perfectly alright if the :class:`CallingThreadDispatcher` +synchronization; this is perfectly alright if the `CallingThreadDispatcher` is used and no other threads are involved, but it may lead to surprises if you were to actually exercise timer events, because those are executed on the -:obj:`Scheduler` thread. +`Scheduler` thread. -Testing the Actor's Behavior ----------------------------- +### Testing the Actor's Behavior When the dispatcher invokes the processing behavior of an actor on a message, -it actually calls :meth:`apply` on the current behavior registered for the -actor. This starts out with the return value of the declared :meth:`receive` -method, but it may also be changed using :meth:`become` and :meth:`unbecome` in +it actually calls `apply` on the current behavior registered for the +actor. This starts out with the return value of the declared `receive` +method, but it may also be changed using `become` and `unbecome` in response to external messages. All of this contributes to the overall actor -behavior and it does not lend itself to easy testing on the :class:`Actor` -itself. Therefore the :class:`TestActorRef` offers a different mode of -operation to complement the :class:`Actor` testing: it supports all operations -also valid on normal :class:`ActorRef`. Messages sent to the actor are +behavior and it does not lend itself to easy testing on the `Actor` +itself. Therefore the `TestActorRef` offers a different mode of +operation to complement the `Actor` testing: it supports all operations +also valid on normal `ActorRef`. Messages sent to the actor are processed synchronously on the current thread and answers may be sent back as -usual. This trick is made possible by the :class:`CallingThreadDispatcher` -described below (see `CallingThreadDispatcher`_); this dispatcher is set -implicitly for any actor instantiated into a :class:`TestActorRef`. +usual. This trick is made possible by the `CallingThreadDispatcher` +described below (see [CallingThreadDispatcher](#callingthreaddispatcher)); this dispatcher is set +implicitly for any actor instantiated into a `TestActorRef`. -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#test-behavior +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-behavior } -As the :class:`TestActorRef` is a subclass of :class:`LocalActorRef` with a few +As the `TestActorRef` is a subclass of `LocalActorRef` with a few special extras, also aspects like supervision and restarting work properly, but beware that execution is only strictly synchronous as long as all actors -involved use the :class:`CallingThreadDispatcher`. As soon as you add elements +involved use the `CallingThreadDispatcher`. As soon as you add elements which include more sophisticated scheduling you leave the realm of unit testing as you then need to think about asynchronicity again (in most cases the problem will be to wait until the desired effect had a chance to happen). One more special aspect which is overridden for single-threaded tests is the -:meth:`receiveTimeout`, as including that would entail asynchronous queuing of -:obj:`ReceiveTimeout` messages, violating the synchronous contract. +`receiveTimeout`, as including that would entail asynchronous queuing of +`ReceiveTimeout` messages, violating the synchronous contract. -.. note:: +@@@ note - To summarize: :class:`TestActorRef` overwrites two fields: it sets the - dispatcher to :obj:`CallingThreadDispatcher.global` and it sets the - :obj:`receiveTimeout` to None. +To summarize: `TestActorRef` overwrites two fields: it sets the +dispatcher to `CallingThreadDispatcher.global` and it sets the +`receiveTimeout` to None. -The Way In-Between: Expecting Exceptions ----------------------------------------- +@@@ + +### The Way In-Between: Expecting Exceptions If you want to test the actor behavior, including hotswapping, but without -involving a dispatcher and without having the :class:`TestActorRef` swallow +involving a dispatcher and without having the `TestActorRef` swallow any thrown exceptions, then there is another mode available for you: just use -the :meth:`receive` method on :class:`TestActorRef`, which will be forwarded to the +the `receive` method on `TestActorRef`, which will be forwarded to the underlying actor: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#test-expecting-exceptions +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-expecting-exceptions } -Use Cases ---------- +### Use Cases -You may of course mix and match both modi operandi of :class:`TestActorRef` as +You may of course mix and match both modi operandi of `TestActorRef` as suits your test needs: - - one common use case is setting up the actor into a specific internal state - before sending the test message - - another is to verify correct internal state transitions after having sent - the test message +> + * one common use case is setting up the actor into a specific internal state +before sending the test message + * another is to verify correct internal state transitions after having sent +the test message Feel free to experiment with the possibilities, and if you find useful patterns, don't hesitate to let the Akka forums know about them! Who knows, common operations might even be worked into nice DSLs. -.. _async-integration-testing-scala: - -Asynchronous Integration Testing with :class:`TestKit` -====================================================== + +## Asynchronous Integration Testing with `TestKit` When you are reasonably sure that your actor's business logic is correct, the next step is verifying that it works correctly within its intended environment (if the individual actors are simple enough, possibly because they use the -:mod:`FSM` module, this might also be the first step). The definition of the +`FSM` module, this might also be the first step). The definition of the environment depends of course very much on the problem at hand and the level at which you intend to test, ranging for functional/integration tests to full system tests. The minimal setup consists of the test procedure, which provides @@ -179,195 +182,178 @@ stimuli at varying injection points and arrange results to be sent from different emission points, but the basic principle stays the same in that a single procedure drives the test. -The :class:`TestKit` class contains a collection of tools which makes this +The `TestKit` class contains a collection of tools which makes this common task easy. -.. includecode:: code/docs/testkit/PlainWordSpec.scala#plain-spec +@@snip [PlainWordSpec.scala](code/docs/testkit/PlainWordSpec.scala) { #plain-spec } -The :class:`TestKit` contains an actor named :obj:`testActor` which is the -entry point for messages to be examined with the various ``expectMsg...`` -assertions detailed below. When mixing in the trait ``ImplicitSender`` this +The `TestKit` contains an actor named `testActor` which is the +entry point for messages to be examined with the various `expectMsg...` +assertions detailed below. When mixing in the trait `ImplicitSender` this test actor is implicitly used as sender reference when dispatching messages -from the test procedure. The :obj:`testActor` may also be passed to +from the test procedure. The `testActor` may also be passed to other actors as usual, usually subscribing it as notification listener. There is a whole set of examination methods, e.g. receiving all consecutive messages matching certain criteria, receiving a whole sequence of fixed messages or classes, receiving nothing for some time, etc. The ActorSystem passed in to the constructor of TestKit is accessible via the -:obj:`system` member. Remember to shut down the actor system after the test is +`system` member. Remember to shut down the actor system after the test is finished (also in case of failure) so that all actors—including the test actor—are stopped. -Built-In Assertions -------------------- +### Built-In Assertions -The above mentioned :meth:`expectMsg` is not the only method for formulating +The above mentioned `expectMsg` is not the only method for formulating assertions concerning received messages. Here is the full list: - * :meth:`expectMsg[T](d: Duration, msg: T): T` - - The given message object must be received within the specified time; the - object will be returned. - - * :meth:`expectMsgPF[T](d: Duration)(pf: PartialFunction[Any, T]): T` - - Within the given time period, a message must be received and the given - partial function must be defined for that message; the result from applying - the partial function to the received message is returned. The duration may - be left unspecified (empty parentheses are required in this case) to use - the deadline from the innermost enclosing :ref:`within ` - block instead. - - * :meth:`expectMsgClass[T](d: Duration, c: Class[T]): T` - - An object which is an instance of the given :class:`Class` must be received - within the allotted time frame; the object will be returned. Note that this - does a conformance check; if you need the class to be equal, have a look at - :meth:`expectMsgAllClassOf` with a single given class argument. - - * :meth:`expectMsgType[T: Manifest](d: Duration)` - - An object which is an instance of the given type (after erasure) must be - received within the allotted time frame; the object will be returned. This - method is approximately equivalent to - ``expectMsgClass(implicitly[ClassTag[T]].runtimeClass)``. - - * :meth:`expectMsgAnyOf[T](d: Duration, obj: T*): T` - - An object must be received within the given time, and it must be equal ( - compared with ``==``) to at least one of the passed reference objects; the - received object will be returned. - - * :meth:`expectMsgAnyClassOf[T](d: Duration, obj: Class[_ <: T]*): T` - - An object must be received within the given time, and it must be an - instance of at least one of the supplied :class:`Class` objects; the - received object will be returned. - - * :meth:`expectMsgAllOf[T](d: Duration, obj: T*): Seq[T]` - - A number of objects matching the size of the supplied object array must be - received within the given time, and for each of the given objects there - must exist at least one among the received ones which equals (compared with - ``==``) it. The full sequence of received objects is returned. - - * :meth:`expectMsgAllClassOf[T](d: Duration, c: Class[_ <: T]*): Seq[T]` - - A number of objects matching the size of the supplied :class:`Class` array - must be received within the given time, and for each of the given classes - there must exist at least one among the received objects whose class equals - (compared with ``==``) it (this is *not* a conformance check). The full - sequence of received objects is returned. - - * :meth:`expectMsgAllConformingOf[T](d: Duration, c: Class[_ <: T]*): Seq[T]` - - A number of objects matching the size of the supplied :class:`Class` array - must be received within the given time, and for each of the given classes - there must exist at least one among the received objects which is an - instance of this class. The full sequence of received objects is returned. - - * :meth:`expectNoMsg(d: Duration)` - - No message must be received within the given time. This also fails if a - message has been received before calling this method which has not been - removed from the queue using one of the other methods. - - * :meth:`receiveN(n: Int, d: Duration): Seq[AnyRef]` - - ``n`` messages must be received within the given time; the received - messages are returned. - - * :meth:`fishForMessage(max: Duration, hint: String)(pf: PartialFunction[Any, Boolean]): Any` - - Keep receiving messages as long as the time is not used up and the partial - function matches and returns ``false``. Returns the message received for - which it returned ``true`` or throws an exception, which will include the - provided hint for easier debugging. +> + * + `expectMsg[T](d: Duration, msg: T): T` + The given message object must be received within the specified time; the +object will be returned. + * + `expectMsgPF[T](d: Duration)(pf: PartialFunction[Any, T]): T` + Within the given time period, a message must be received and the given +partial function must be defined for that message; the result from applying +the partial function to the received message is returned. The duration may +be left unspecified (empty parentheses are required in this case) to use +the deadline from the innermost enclosing [within](#testkit-within) +block instead. + * + `expectMsgClass[T](d: Duration, c: Class[T]): T` + An object which is an instance of the given `Class` must be received +within the allotted time frame; the object will be returned. Note that this +does a conformance check; if you need the class to be equal, have a look at +`expectMsgAllClassOf` with a single given class argument. + * + `expectMsgType[T: Manifest](d: Duration)` + An object which is an instance of the given type (after erasure) must be +received within the allotted time frame; the object will be returned. This +method is approximately equivalent to +`expectMsgClass(implicitly[ClassTag[T]].runtimeClass)`. + * + `expectMsgAnyOf[T](d: Duration, obj: T*): T` + An object must be received within the given time, and it must be equal ( +compared with `==`) to at least one of the passed reference objects; the +received object will be returned. + * + `expectMsgAnyClassOf[T](d: Duration, obj: Class[_ <: T]*): T` + An object must be received within the given time, and it must be an +instance of at least one of the supplied `Class` objects; the +received object will be returned. + * + `expectMsgAllOf[T](d: Duration, obj: T*): Seq[T]` + A number of objects matching the size of the supplied object array must be +received within the given time, and for each of the given objects there +must exist at least one among the received ones which equals (compared with +`==`) it. The full sequence of received objects is returned. + * + `expectMsgAllClassOf[T](d: Duration, c: Class[_ <: T]*): Seq[T]` + A number of objects matching the size of the supplied `Class` array +must be received within the given time, and for each of the given classes +there must exist at least one among the received objects whose class equals +(compared with `==`) it (this is *not* a conformance check). The full +sequence of received objects is returned. + * + `expectMsgAllConformingOf[T](d: Duration, c: Class[_ <: T]*): Seq[T]` + A number of objects matching the size of the supplied `Class` array +must be received within the given time, and for each of the given classes +there must exist at least one among the received objects which is an +instance of this class. The full sequence of received objects is returned. + * + `expectNoMsg(d: Duration)` + No message must be received within the given time. This also fails if a +message has been received before calling this method which has not been +removed from the queue using one of the other methods. + * + `receiveN(n: Int, d: Duration): Seq[AnyRef]` + `n` messages must be received within the given time; the received +messages are returned. + * + `fishForMessage(max: Duration, hint: String)(pf: PartialFunction[Any, Boolean]): Any` + Keep receiving messages as long as the time is not used up and the partial +function matches and returns `false`. Returns the message received for +which it returned `true` or throws an exception, which will include the +provided hint for easier debugging. In addition to message reception assertions there are also methods which help with message flows: - * :meth:`receiveOne(d: Duration): AnyRef` - - Tries to receive one message for at most the given time interval and - returns ``null`` in case of failure. If the given Duration is zero, the - call is non-blocking (polling mode). - - * :meth:`receiveWhile[T](max: Duration, idle: Duration, messages: Int)(pf: PartialFunction[Any, T]): Seq[T]` - - Collect messages as long as - +> + * + `receiveOne(d: Duration): AnyRef` + Tries to receive one message for at most the given time interval and +returns `null` in case of failure. If the given Duration is zero, the +call is non-blocking (polling mode). + * + `receiveWhile[T](max: Duration, idle: Duration, messages: Int)(pf: PartialFunction[Any, T]): Seq[T]` + Collect messages as long as * they are matching the given partial function * the given time interval is not used up * the next message is received within the idle timeout * the number of messages has not yet reached the maximum + All collected messages are returned. The maximum duration defaults to the +time remaining in the innermost enclosing [within](#testkit-within) +block and the idle duration defaults to infinity (thereby disabling the +idle timeout feature). The number of expected messages defaults to +`Int.MaxValue`, which effectively disables this limit. + * + `awaitCond(p: => Boolean, max: Duration, interval: Duration)` + Poll the given condition every `interval` until it returns `true` or +the `max` duration is used up. The interval defaults to 100 ms and the +maximum defaults to the time remaining in the innermost enclosing +[within](#testkit-within) block. + * + `awaitAssert(a: => Any, max: Duration, interval: Duration)` + Poll the given assert function every `interval` until it does not throw +an exception or the `max` duration is used up. If the timeout expires the +last exception is thrown. The interval defaults to 100 ms and the maximum defaults +to the time remaining in the innermost enclosing [within](#testkit-within) +block.The interval defaults to 100 ms and the maximum defaults to the time +remaining in the innermost enclosing [within](#testkit-within) block. + * + `ignoreMsg(pf: PartialFunction[AnyRef, Boolean])` + `ignoreNoMsg` + The internal `testActor` contains a partial function for ignoring +messages: it will only enqueue messages which do not match the function or +for which the function returns `false`. This function can be set and +reset using the methods given above; each invocation replaces the previous +function, they are not composed. + This feature is useful e.g. when testing a logging system, where you want +to ignore regular messages and are only interested in your specific ones. - All collected messages are returned. The maximum duration defaults to the - time remaining in the innermost enclosing :ref:`within ` - block and the idle duration defaults to infinity (thereby disabling the - idle timeout feature). The number of expected messages defaults to - ``Int.MaxValue``, which effectively disables this limit. - - * :meth:`awaitCond(p: => Boolean, max: Duration, interval: Duration)` - - Poll the given condition every :obj:`interval` until it returns ``true`` or - the :obj:`max` duration is used up. The interval defaults to 100 ms and the - maximum defaults to the time remaining in the innermost enclosing - :ref:`within ` block. - - * :meth:`awaitAssert(a: => Any, max: Duration, interval: Duration)` - - Poll the given assert function every :obj:`interval` until it does not throw - an exception or the :obj:`max` duration is used up. If the timeout expires the - last exception is thrown. The interval defaults to 100 ms and the maximum defaults - to the time remaining in the innermost enclosing :ref:`within ` - block.The interval defaults to 100 ms and the maximum defaults to the time - remaining in the innermost enclosing :ref:`within ` block. - - * :meth:`ignoreMsg(pf: PartialFunction[AnyRef, Boolean])` - - :meth:`ignoreNoMsg` - - The internal :obj:`testActor` contains a partial function for ignoring - messages: it will only enqueue messages which do not match the function or - for which the function returns ``false``. This function can be set and - reset using the methods given above; each invocation replaces the previous - function, they are not composed. - - This feature is useful e.g. when testing a logging system, where you want - to ignore regular messages and are only interested in your specific ones. - -Expecting Log Messages ----------------------- +### Expecting Log Messages Since an integration test does not allow to the internal processing of the participating actors, verifying expected exceptions cannot be done directly. Instead, use the logging system for this purpose: replacing the normal event -handler with the :class:`TestEventListener` and using an :class:`EventFilter` +handler with the `TestEventListener` and using an `EventFilter` allows assertions on log messages, including those which are generated by exceptions: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#event-filter +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #event-filter } -If a number of occurrences is specific—as demonstrated above—then ``intercept`` +If a number of occurrences is specific—as demonstrated above—then `intercept` will block until that number of matching messages have been received or the -timeout configured in ``akka.test.filter-leeway`` is used up (time starts +timeout configured in `akka.test.filter-leeway` is used up (time starts counting after the passed-in block of code returns). In case of a timeout the test fails. -.. note:: +@@@ note - Be sure to exchange the default logger with the - :class:`TestEventListener` in your ``application.conf`` to enable this - function:: +Be sure to exchange the default logger with the +`TestEventListener` in your `application.conf` to enable this +function: - akka.loggers = [akka.testkit.TestEventListener] +``` +akka.loggers = [akka.testkit.TestEventListener] +``` -.. _TestKit.within: +@@@ -Timing Assertions ------------------ + +### Timing Assertions Another important part of functional testing concerns timing: certain events must not happen immediately (like a timer), others need to happen before a @@ -376,264 +362,243 @@ the positive or negative result must be obtained. Lower time limits need to be checked external to the examination, which is facilitated by a new construct for managing time constraints: -.. code-block:: scala +```scala +within([min, ]max) { + ... +} +``` - within([min, ]max) { - ... - } - -The block given to :meth:`within` must complete after a :ref:`Duration` which -is between :obj:`min` and :obj:`max`, where the former defaults to zero. The -deadline calculated by adding the :obj:`max` parameter to the block's start +The block given to `within` must complete after a @ref:[Duration](../common/duration.md) which +is between `min` and `max`, where the former defaults to zero. The +deadline calculated by adding the `max` parameter to the block's start time is implicitly available within the block to all examination methods, if you do not specify it, it is inherited from the innermost enclosing -:meth:`within` block. +`within` block. It should be noted that if the last message-receiving assertion of the block is -:meth:`expectNoMsg` or :meth:`receiveWhile`, the final check of the -:meth:`within` is skipped in order to avoid false positives due to wake-up +`expectNoMsg` or `receiveWhile`, the final check of the +`within` is skipped in order to avoid false positives due to wake-up latencies. This means that while individual contained assertions still use the maximum time bound, the overall block may take arbitrarily longer in this case. -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#test-within +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-within } -.. note:: +@@@ note - All times are measured using ``System.nanoTime``, meaning that they describe - wall time, not CPU time. +All times are measured using `System.nanoTime`, meaning that they describe +wall time, not CPU time. + +@@@ Ray Roestenburg has written a great article on using the TestKit: -``_. -His full example is also available :ref:`here `. +[http://roestenburg.agilesquad.com/2011/02/unit-testing-akka-actors-with-testkit_12.html](http://roestenburg.agilesquad.com/2011/02/unit-testing-akka-actors-with-testkit_12.html). +His full example is also available @ref:[here](testkit-example.md). -Accounting for Slow Test Systems -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Accounting for Slow Test Systems The tight timeouts you use during testing on your lightning-fast notebook will invariably lead to spurious test failures on the heavily loaded Jenkins server (or similar). To account for this situation, all maximum durations are -internally scaled by a factor taken from the :ref:`configuration`, -``akka.test.timefactor``, which defaults to 1. +internally scaled by a factor taken from the [Configuration](), +`akka.test.timefactor`, which defaults to 1. You can scale other durations with the same factor by using the implicit conversion -in ``akka.testkit`` package object to add dilated function to :class:`Duration`. +in `akka.testkit` package object to add dilated function to `Duration`. -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#duration-dilation +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #duration-dilation } -Resolving Conflicts with Implicit ActorRef ------------------------------------------- +### Resolving Conflicts with Implicit ActorRef -If you want the sender of messages inside your TestKit-based tests to be the ``testActor`` -simply mix in ``ImplicitSender`` into your test. +If you want the sender of messages inside your TestKit-based tests to be the `testActor` +simply mix in `ImplicitSender` into your test. -.. includecode:: code/docs/testkit/PlainWordSpec.scala#implicit-sender +@@snip [PlainWordSpec.scala](code/docs/testkit/PlainWordSpec.scala) { #implicit-sender } -Using Multiple Probe Actors ---------------------------- +### Using Multiple Probe Actors When the actors under test are supposed to send various messages to different destinations, it may be difficult distinguishing the message streams arriving -at the :obj:`testActor` when using the :class:`TestKit` as a mixin. Another +at the `testActor` when using the `TestKit` as a mixin. Another approach is to use it for creation of simple probe actors to be inserted in the message flows. To make this more powerful and convenient, there is a concrete -implementation called :class:`TestProbe`. The functionality is best explained +implementation called `TestProbe`. The functionality is best explained using a small example: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala - :include: imports-test-probe +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #imports-test-probe } -.. includecode:: code/docs/testkit/TestkitDocSpec.scala - :include: my-double-echo +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #my-double-echo } -.. includecode:: code/docs/testkit/TestkitDocSpec.scala - :include: test-probe +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-probe } -Here a the system under test is simulated by :class:`MyDoubleEcho`, which is +Here a the system under test is simulated by `MyDoubleEcho`, which is supposed to mirror its input to two outputs. Attaching two test probes enables verification of the (simplistic) behavior. Another example would be two actors A and B which collaborate by A sending messages to B. In order to verify this -message flow, a :class:`TestProbe` could be inserted as target of A, using the +message flow, a `TestProbe` could be inserted as target of A, using the forwarding capabilities or auto-pilot described below to include a real B in the test setup. If you have many test probes, you can name them to get meaningful actor names in test logs and assertions: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#test-probe-with-custom-name +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-probe-with-custom-name } Probes may also be equipped with custom assertions to make your test code even more concise and clear: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala - :include: test-special-probe +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-special-probe } -You have complete flexibility here in mixing and matching the :class:`TestKit` +You have complete flexibility here in mixing and matching the `TestKit` facilities with your own checks and choosing an intuitive name for it. In real life your code will probably be a bit more complicated than the example given above; just use the power! -.. warning:: +@@@ warning - Any message send from a ``TestProbe`` to another actor which runs on the - CallingThreadDispatcher runs the risk of dead-lock, if that other actor might - also send to this probe. The implementation of :meth:`TestProbe.watch` and - :meth:`TestProbe.unwatch` will also send a message to the watchee, which - means that it is dangerous to try watching e.g. :class:`TestActorRef` from a - :meth:`TestProbe`. +Any message send from a `TestProbe` to another actor which runs on the +CallingThreadDispatcher runs the risk of dead-lock, if that other actor might +also send to this probe. The implementation of `TestProbe.watch` and +`TestProbe.unwatch` will also send a message to the watchee, which +means that it is dangerous to try watching e.g. `TestActorRef` from a +`TestProbe`. -Watching Other Actors from Probes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@@ -A :class:`TestProbe` can register itself for DeathWatch of any other actor: +#### Watching Other Actors from Probes -.. includecode:: code/docs/testkit/TestkitDocSpec.scala - :include: test-probe-watch +A `TestProbe` can register itself for DeathWatch of any other actor: -Replying to Messages Received by Probes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-probe-watch } + +#### Replying to Messages Received by Probes The probes keep track of the communications channel for replies, if possible, so they can also reply: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#test-probe-reply +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-probe-reply } -Forwarding Messages Received by Probes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Forwarding Messages Received by Probes -Given a destination actor ``dest`` which in the nominal actor network would -receive a message from actor ``source``. If you arrange for the message to be -sent to a :class:`TestProbe` ``probe`` instead, you can make assertions +Given a destination actor `dest` which in the nominal actor network would +receive a message from actor `source`. If you arrange for the message to be +sent to a `TestProbe` `probe` instead, you can make assertions concerning volume and timing of the message flow while still keeping the network functioning: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala - :include: test-probe-forward-actors +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-probe-forward-actors } -.. includecode:: code/docs/testkit/TestkitDocSpec.scala - :include: test-probe-forward +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-probe-forward } -The ``dest`` actor will receive the same message invocation as if no test probe +The `dest` actor will receive the same message invocation as if no test probe had intervened. -Auto-Pilot -^^^^^^^^^^ +#### Auto-Pilot Receiving messages in a queue for later inspection is nice, but in order to keep a test running and verify traces later you can also install an -:class:`AutoPilot` in the participating test probes (actually in any -:class:`TestKit`) which is invoked before enqueueing to the inspection queue. -This code can be used to forward messages, e.g. in a chain ``A --> Probe --> -B``, as long as a certain protocol is obeyed. +`AutoPilot` in the participating test probes (actually in any +`TestKit`) which is invoked before enqueueing to the inspection queue. +This code can be used to forward messages, e.g. in a chain `A --> Probe --> +B`, as long as a certain protocol is obeyed. -.. includecode:: ../../../akka-testkit/src/test/scala/akka/testkit/TestProbeSpec.scala#autopilot +@@snip [TestProbeSpec.scala]../../../../../akka-testkit/src/test/scala/akka/testkit/TestProbeSpec.scala) { #autopilot } -The :meth:`run` method must return the auto-pilot for the next message, which -may be :class:`KeepRunning` to retain the current one or :class:`NoAutoPilot` +The `run` method must return the auto-pilot for the next message, which +may be `KeepRunning` to retain the current one or `NoAutoPilot` to switch it off. -Caution about Timing Assertions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Caution about Timing Assertions -The behavior of :meth:`within` blocks when using test probes might be perceived +The behavior of `within` blocks when using test probes might be perceived as counter-intuitive: you need to remember that the nicely scoped deadline as -described :ref:`above ` is local to each probe. Hence, probes +described [above](#testkit-within) is local to each probe. Hence, probes do not react to each other's deadlines or to the deadline set in an enclosing -:class:`TestKit` instance: +`TestKit` instance: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#test-within-probe +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-within-probe } -Here, the ``expectMsg`` call will use the default timeout. +Here, the `expectMsg` call will use the default timeout. -Testing parent-child relationships ----------------------------------- +### Testing parent-child relationships The parent of an actor is always the actor that created it. At times this leads to a coupling between the two that may not be straightforward to test. There are several approaches to improve testability of a child actor that needs to refer to its parent: -1. when creating a child, pass an explicit reference to its parent -2. create the child with a ``TestProbe`` as parent -3. create a fabricated parent when testing + 1. when creating a child, pass an explicit reference to its parent + 2. create the child with a `TestProbe` as parent + 3. create a fabricated parent when testing Conversely, a parent's binding to its child can be lessened as follows: -4. when creating a parent, tell the parent how to create its child + 4. when creating a parent, tell the parent how to create its child For example, the structure of the code you want to test may follow this pattern: -.. includecode:: code/docs/testkit/ParentChildSpec.scala#test-example +@@snip [ParentChildSpec.scala](code/docs/testkit/ParentChildSpec.scala) { #test-example } -Introduce child to its parent -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Introduce child to its parent -The first option is to avoid use of the :meth:`context.parent` function and create +The first option is to avoid use of the `context.parent` function and create a child with a custom parent by passing an explicit reference to its parent instead. -.. includecode:: code/docs/testkit/ParentChildSpec.scala#test-dependentchild +@@snip [ParentChildSpec.scala](code/docs/testkit/ParentChildSpec.scala) { #test-dependentchild } -Create the child using TestProbe -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Create the child using TestProbe -The ``TestProbe`` class can in fact create actors that will run with the test probe as parent. -This will cause any messages the child actor sends to `context.parent` to +The `TestProbe` class can in fact create actors that will run with the test probe as parent. +This will cause any messages the child actor sends to *context.parent* to end up in the test probe. -.. includecode:: code/docs/testkit/ParentChildSpec.scala#test-TestProbe-parent +@@snip [ParentChildSpec.scala](code/docs/testkit/ParentChildSpec.scala) { #test-TestProbe-parent } -Using a fabricated parent -^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Using a fabricated parent If you prefer to avoid modifying the parent or child constructor you can create a fabricated parent in your test. This, however, does not enable you to test the parent actor in isolation. -.. includecode:: code/docs/testkit/ParentChildSpec.scala#test-fabricated-parent +@@snip [ParentChildSpec.scala](code/docs/testkit/ParentChildSpec.scala) { #test-fabricated-parent } -Externalize child making from the parent -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +#### Externalize child making from the parent Alternatively, you can tell the parent how to create its child. There are two ways -to do this: by giving it a :class:`Props` object or by giving it a function which takes care of creating the child actor: +to do this: by giving it a `Props` object or by giving it a function which takes care of creating the child actor: -.. includecode:: code/docs/testkit/ParentChildSpec.scala#test-dependentparent +@@snip [ParentChildSpec.scala](code/docs/testkit/ParentChildSpec.scala) { #test-dependentparent } -Creating the :class:`Props` is straightforward and the function may look like this in your test code: +Creating the `Props` is straightforward and the function may look like this in your test code: -.. includecode:: code/docs/testkit/ParentChildSpec.scala#child-maker-test +@@snip [ParentChildSpec.scala](code/docs/testkit/ParentChildSpec.scala) { #child-maker-test } And like this in your application code: -.. includecode:: code/docs/testkit/ParentChildSpec.scala#child-maker-prod - +@@snip [ParentChildSpec.scala](code/docs/testkit/ParentChildSpec.scala) { #child-maker-prod } Which of these methods is the best depends on what is most important to test. The most generic option is to create the parent actor by passing it a function that is responsible for the Actor creation, but the fabricated parent is often sufficient. -.. _Scala-CallingThreadDispatcher: + +## CallingThreadDispatcher -CallingThreadDispatcher -======================= - -The :class:`CallingThreadDispatcher` serves good purposes in unit testing, as +The `CallingThreadDispatcher` serves good purposes in unit testing, as described above, but originally it was conceived in order to allow contiguous stack traces to be generated in case of an error. As this special dispatcher runs everything which would normally be queued directly on the current thread, the full history of a message's processing chain is recorded on the call stack, so long as all intervening actors run on this dispatcher. -How to use it -------------- +### How to use it Just set the dispatcher as you normally would: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#calling-thread-dispatcher +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #calling-thread-dispatcher } -How it works ------------- +### How it works -When receiving an invocation, the :class:`CallingThreadDispatcher` checks +When receiving an invocation, the `CallingThreadDispatcher` checks whether the receiving actor is already active on the current thread. The simplest example for this situation is an actor which sends a message to itself. In this case, processing cannot continue immediately as that would @@ -644,7 +609,7 @@ previous work. In the other case, the invocation is simply processed immediately on the current thread. Futures scheduled via this dispatcher are also executed immediately. -This scheme makes the :class:`CallingThreadDispatcher` work like a general +This scheme makes the `CallingThreadDispatcher` work like a general purpose dispatcher for any actors which never block on external events. In the presence of multiple threads it may happen that two invocations of an @@ -658,93 +623,95 @@ concurrency. The other remaining difficulty is correct handling of suspend and resume: when an actor is suspended, subsequent invocations will be queued in thread-local queues (the same ones used for queuing in the normal case). The call to -:meth:`resume`, however, is done by one specific thread, and all other threads +`resume`, however, is done by one specific thread, and all other threads in the system will probably not be executing this specific actor, which leads to the problem that the thread-local queues cannot be emptied by their native -threads. Hence, the thread calling :meth:`resume` will collect all currently +threads. Hence, the thread calling `resume` will collect all currently queued invocations from all threads into its own queue and process them. -Limitations ------------ +### Limitations -.. warning:: +@@@ warning - In case the CallingThreadDispatcher is used for top-level actors, but - without going through TestActorRef, then there is a time window during which - the actor is awaiting construction by the user guardian actor. Sending - messages to the actor during this time period will result in them being - enqueued and then executed on the guardian’s thread instead of the caller’s - thread. To avoid this, use TestActorRef. +In case the CallingThreadDispatcher is used for top-level actors, but +without going through TestActorRef, then there is a time window during which +the actor is awaiting construction by the user guardian actor. Sending +messages to the actor during this time period will result in them being +enqueued and then executed on the guardian’s thread instead of the caller’s +thread. To avoid this, use TestActorRef. + +@@@ If an actor's behavior blocks on a something which would normally be affected by the calling actor after having sent the message, this will obviously dead-lock when using this dispatcher. This is a common scenario in actor tests -based on :class:`CountDownLatch` for synchronization: +based on `CountDownLatch` for synchronization: -.. code-block:: scala - - val latch = new CountDownLatch(1) - actor ! startWorkAfter(latch) // actor will call latch.await() before proceeding - doSomeSetupStuff() - latch.countDown() +```scala +val latch = new CountDownLatch(1) +actor ! startWorkAfter(latch) // actor will call latch.await() before proceeding +doSomeSetupStuff() +latch.countDown() +``` The example would hang indefinitely within the message processing initiated on the second line and never reach the fourth line, which would unblock it on a normal dispatcher. -Thus, keep in mind that the :class:`CallingThreadDispatcher` is not a +Thus, keep in mind that the `CallingThreadDispatcher` is not a general-purpose replacement for the normal dispatchers. On the other hand it may be quite useful to run your actor network on it for testing, because if it runs without dead-locking chances are very high that it will not dead-lock in production. -.. warning:: +@@@ warning - The above sentence is unfortunately not a strong guarantee, because your - code might directly or indirectly change its behavior when running on a - different dispatcher. If you are looking for a tool to help you debug - dead-locks, the :class:`CallingThreadDispatcher` may help with certain error - scenarios, but keep in mind that it has may give false negatives as well as - false positives. +The above sentence is unfortunately not a strong guarantee, because your +code might directly or indirectly change its behavior when running on a +different dispatcher. If you are looking for a tool to help you debug +dead-locks, the `CallingThreadDispatcher` may help with certain error +scenarios, but keep in mind that it has may give false negatives as well as +false positives. -Thread Interruptions --------------------- +@@@ + +### Thread Interruptions If the CallingThreadDispatcher sees that the current thread has its -``isInterrupted()`` flag set when message processing returns, it will throw an -:class:`InterruptedException` after finishing all its processing (i.e. all +`isInterrupted()` flag set when message processing returns, it will throw an +`InterruptedException` after finishing all its processing (i.e. all messages which need processing as described above are processed before this -happens). As :meth:`tell` cannot throw exceptions due to its contract, this +happens). As `tell` cannot throw exceptions due to its contract, this exception will then be caught and logged, and the thread’s interrupted status will be set again. -If during message processing an :class:`InterruptedException` is thrown then it +If during message processing an `InterruptedException` is thrown then it will be caught inside the CallingThreadDispatcher’s message handling loop, the thread’s interrupted flag will be set and processing continues normally. -.. note:: +@@@ note - The summary of these two paragraphs is that if the current thread is - interrupted while doing work under the CallingThreadDispatcher, then that - will result in the ``isInterrupted`` flag to be ``true`` when the message - send returns and no :class:`InterruptedException` will be thrown. +The summary of these two paragraphs is that if the current thread is +interrupted while doing work under the CallingThreadDispatcher, then that +will result in the `isInterrupted` flag to be `true` when the message +send returns and no `InterruptedException` will be thrown. -Benefits --------- +@@@ -To summarize, these are the features with the :class:`CallingThreadDispatcher` +### Benefits + +To summarize, these are the features with the `CallingThreadDispatcher` has to offer: - - Deterministic execution of single-threaded tests while retaining nearly full - actor semantics - - Full message processing history leading up to the point of failure in - exception stack traces - - Exclusion of certain classes of dead-lock scenarios +> + * Deterministic execution of single-threaded tests while retaining nearly full +actor semantics + * Full message processing history leading up to the point of failure in +exception stack traces + * Exclusion of certain classes of dead-lock scenarios -.. _actor.logging-scala: - -Tracing Actor Invocations -========================= + +## Tracing Actor Invocations The testing facilities described up to this point were aiming at formulating assertions about a system’s behavior. If a test fails, it is usually your job @@ -752,58 +719,57 @@ to find the cause, fix it and verify the test again. This process is supported by debuggers as well as logging, where the Akka toolkit offers the following options: -* *Logging of exceptions thrown within Actor instances* + * + *Logging of exceptions thrown within Actor instances* + This is always on; in contrast to the other logging mechanisms, this logs at +`ERROR` level. + * + *Logging of message invocations on certain actors* + This is enabled by a setting in the [Configuration]() — namely +`akka.actor.debug.receive` — which enables the `loggable` +statement to be applied to an actor’s `receive` function: - This is always on; in contrast to the other logging mechanisms, this logs at - ``ERROR`` level. +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #logging-receive } -* *Logging of message invocations on certain actors* - - This is enabled by a setting in the :ref:`configuration` — namely - ``akka.actor.debug.receive`` — which enables the :meth:`loggable` - statement to be applied to an actor’s :meth:`receive` function: - -.. includecode:: code/docs/testkit/TestkitDocSpec.scala#logging-receive - -If the aforementioned setting is not given in the :ref:`configuration`, this method will -pass through the given :class:`Receive` function unmodified, meaning that +If the aforementioned setting is not given in the [Configuration](), this method will +pass through the given `Receive` function unmodified, meaning that there is no runtime cost unless actually enabled. The logging feature is coupled to this specific local mark-up because enabling it uniformly on all actors is not usually what you need, and it would lead to endless loops if it were applied to event bus logger listeners. -* *Logging of special messages* + * + *Logging of special messages* + Actors handle certain special messages automatically, e.g. `Kill`, +`PoisonPill`, etc. Tracing of these message invocations is enabled by +the setting `akka.actor.debug.autoreceive`, which enables this on all +actors. + * + *Logging of the actor lifecycle* + Actor creation, start, restart, monitor start, monitor stop and stop may be traced by +enabling the setting `akka.actor.debug.lifecycle`; this, too, is enabled +uniformly on all actors. - Actors handle certain special messages automatically, e.g. :obj:`Kill`, - :obj:`PoisonPill`, etc. Tracing of these message invocations is enabled by - the setting ``akka.actor.debug.autoreceive``, which enables this on all - actors. +All these messages are logged at `DEBUG` level. To summarize, you can enable +full logging of actor activities using this configuration fragment: -* *Logging of the actor lifecycle* - - Actor creation, start, restart, monitor start, monitor stop and stop may be traced by - enabling the setting ``akka.actor.debug.lifecycle``; this, too, is enabled - uniformly on all actors. - -All these messages are logged at ``DEBUG`` level. To summarize, you can enable -full logging of actor activities using this configuration fragment:: - - akka { - loglevel = "DEBUG" - actor { - debug { - receive = on - autoreceive = on - lifecycle = on - } +``` +akka { + loglevel = "DEBUG" + actor { + debug { + receive = on + autoreceive = on + lifecycle = on } } +} +``` -Different Testing Frameworks -============================ +## Different Testing Frameworks -Akka’s own test suite is written using `ScalaTest `_, +Akka’s own test suite is written using [ScalaTest](http://scalatest.org), which also shines through in documentation examples. However, the TestKit and its facilities do not depend on that framework, you can essentially use whichever suits your development style best. @@ -812,53 +778,46 @@ This section contains a collection of known gotchas with some other frameworks, which is by no means exhaustive and does not imply endorsement or special support. -When you need it to be a trait ------------------------------- +### When you need it to be a trait -If for some reason it is a problem to inherit from :class:`TestKit` due to it -being a concrete class instead of a trait, there’s :class:`TestKitBase`: +If for some reason it is a problem to inherit from `TestKit` due to it +being a concrete class instead of a trait, there’s `TestKitBase`: -.. includecode:: code/docs/testkit/TestkitDocSpec.scala - :include: test-kit-base - :exclude: put-your-test-code-here +@@snip [TestkitDocSpec.scala](code/docs/testkit/TestkitDocSpec.scala) { #test-kit-base } -The ``implicit lazy val system`` must be declared exactly like that (you can of +The `implicit lazy val system` must be declared exactly like that (you can of course pass arguments to the actor system factory as needed) because trait -:class:`TestKitBase` needs the system during its construction. +`TestKitBase` needs the system during its construction. -.. warning:: +@@@ warning - Use of the trait is discouraged because of potential issues with binary - backwards compatibility in the future, use at own risk. +Use of the trait is discouraged because of potential issues with binary +backwards compatibility in the future, use at own risk. -Specs2 ------- +@@@ -Some `Specs2 `_ users have contributed examples of how to work around some clashes which may arise: +### Specs2 -* Mixing TestKit into :class:`org.specs2.mutable.Specification` results in a - name clash involving the ``end`` method (which is a private variable in - TestKit and an abstract method in Specification); if mixing in TestKit first, - the code may compile but might then fail at runtime. The work-around—which is - actually beneficial also for the third point—is to apply the TestKit together - with :class:`org.specs2.specification.Scope`. -* The Specification traits provide a :class:`Duration` DSL which uses partly - the same method names as :class:`scala.concurrent.duration.Duration`, resulting in ambiguous - implicits if ``scala.concurrent.duration._`` is imported. There are two workarounds: +Some [Specs2](http://specs2.org) users have contributed examples of how to work around some clashes which may arise: - * either use the Specification variant of Duration and supply an implicit - conversion to the Akka Duration. This conversion is not supplied with the - Akka distribution because that would mean that our JAR files would depend on - Specs2, which is not justified by this little feature. + * Mixing TestKit into `org.specs2.mutable.Specification` results in a +name clash involving the `end` method (which is a private variable in +TestKit and an abstract method in Specification); if mixing in TestKit first, +the code may compile but might then fail at runtime. The work-around—which is +actually beneficial also for the third point—is to apply the TestKit together +with `org.specs2.specification.Scope`. + * The Specification traits provide a `Duration` DSL which uses partly +the same method names as `scala.concurrent.duration.Duration`, resulting in ambiguous +implicits if `scala.concurrent.duration._` is imported. There are two workarounds: + * either use the Specification variant of Duration and supply an implicit +conversion to the Akka Duration. This conversion is not supplied with the +Akka distribution because that would mean that our JAR files would depend on +Specs2, which is not justified by this little feature. + * or mix `org.specs2.time.NoTimeConversions` into the Specification. + * Specifications are by default executed concurrently, which requires some care +when writing the tests or alternatively the `sequential` keyword. - * or mix :class:`org.specs2.time.NoTimeConversions` into the Specification. - -* Specifications are by default executed concurrently, which requires some care - when writing the tests or alternatively the ``sequential`` keyword. - -Configuration -============= +## Configuration There are several configuration properties for the TestKit module, please refer -to the :ref:`reference configuration `. - +to the @ref:[reference configuration](../general/configuration.md#config-akka-testkit). \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/testkit-example.md b/akka-docs/src/main/paradox/scala/testkit-example.md index f74bcbe417..f59baf753c 100644 --- a/akka-docs/src/main/paradox/scala/testkit-example.md +++ b/akka-docs/src/main/paradox/scala/testkit-example.md @@ -1,10 +1,5 @@ -.. _testkit-example: +# TestKit Example -######################## -TestKit Example -######################## - -Ray Roestenburg's example code from `his blog `_ adapted to work with Akka 2.x. - -.. includecode:: code/docs/testkit/TestKitUsageSpec.scala#testkit-usage +Ray Roestenburg's example code from [his blog](http://roestenburg.agilesquad.com/2011/02/unit-testing-akka-actors-with-testkit_12.html) adapted to work with Akka 2.x. +@@snip [TestKitUsageSpec.scala](code/docs/testkit/TestKitUsageSpec.scala) { #testkit-usage } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/typed-actors.md b/akka-docs/src/main/paradox/scala/typed-actors.md index cf01a5a225..037541dce9 100644 --- a/akka-docs/src/main/paradox/scala/typed-actors.md +++ b/akka-docs/src/main/paradox/scala/typed-actors.md @@ -1,14 +1,13 @@ -.. _typed-actors-scala: +# Typed Actors -Typed Actors -==================== +@@@ note -.. note:: +This module will be deprecated as it will be superseded by the @ref:[Akka Typed](typed.md) +project which is currently being developed in open preview mode. - This module will be deprecated as it will be superseded by the :ref:`typed-scala` - project which is currently being developed in open preview mode. +@@@ -Akka Typed Actors is an implementation of the `Active Objects `_ pattern. +Akka Typed Actors is an implementation of the [Active Objects](http://en.wikipedia.org/wiki/Active_object) pattern. Essentially turning method invocations into asynchronous dispatch instead of synchronous that has been the default way since Smalltalk came out. Typed Actors consist of 2 "parts", a public interface and an implementation, and if you've done any work in "enterprise" Java, this will be very familiar to you. As with normal Actors you have an external API (the public interface instance) that will delegate method calls asynchronously to @@ -17,241 +16,214 @@ a private instance of the implementation. The advantage of Typed Actors vs. Actors is that with TypedActors you have a static contract, and don't need to define your own messages, the downside is that it places some limitations on what you can do and what you can't, i.e. you -cannot use :meth:`become`/:meth:`unbecome`. +cannot use `become`/`unbecome`. -Typed Actors are implemented using `JDK Proxies `_ which provide a pretty easy-worked API to intercept method calls. +Typed Actors are implemented using [JDK Proxies](http://docs.oracle.com/javase/6/docs/api/java/lang/reflect/Proxy.html) which provide a pretty easy-worked API to intercept method calls. -.. note:: +@@@ note - Just as with regular Akka Actors, Typed Actors process one call at a time. +Just as with regular Akka Actors, Typed Actors process one call at a time. -When to use Typed Actors ------------------------- +@@@ + +## When to use Typed Actors Typed actors are nice for bridging between actor systems (the “inside”) and non-actor code (the “outside”), because they allow you to write normal OO-looking code on the outside. Think of them like doors: their practicality lies in interfacing between private sphere and the public, but you don’t want -that many doors inside your house, do you? For a longer discussion see `this -blog post `_. +that many doors inside your house, do you? For a longer discussion see [this +blog post](http://letitcrash.com/post/19074284309/when-to-use-typedactors). A bit more background: TypedActors can easily be abused as RPC, and that -is an abstraction which is `well-known -`_ +is an abstraction which is [well-known](http://doc.akka.io/docs/misc/smli_tr-94-29.pdf) to be leaky. Hence TypedActors are not what we think of first when we talk about making highly scalable concurrent software easier to write correctly. They have their niche, use them sparingly. -The tools of the trade ----------------------- +## The tools of the trade Before we create our first Typed Actor we should first go through the tools that we have at our disposal, -it's located in ``akka.actor.TypedActor``. +it's located in `akka.actor.TypedActor`. -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-extension-tools +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-extension-tools } -.. warning:: +@@@ warning - Same as not exposing ``this`` of an Akka Actor, it's important not to expose ``this`` of a Typed Actor, - instead you should pass the external proxy reference, which is obtained from within your Typed Actor as - ``TypedActor.self``, this is your external identity, as the ``ActorRef`` is the external identity of - an Akka Actor. +Same as not exposing `this` of an Akka Actor, it's important not to expose `this` of a Typed Actor, +instead you should pass the external proxy reference, which is obtained from within your Typed Actor as +`TypedActor.self`, this is your external identity, as the `ActorRef` is the external identity of +an Akka Actor. -Creating Typed Actors ---------------------- +@@@ + +## Creating Typed Actors To create a Typed Actor you need to have one or more interfaces, and one implementation. The following imports are assumed: -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: imports +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #imports } Our example interface: -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-iface - :exclude: typed-actor-iface-methods +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-iface } Our example implementation of that interface: -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-impl - :exclude: typed-actor-impl-methods +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-impl } The most trivial way of creating a Typed Actor instance -of our ``Squarer``: +of our `Squarer`: -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-create1 +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-create1 } First type is the type of the proxy, the second type is the type of the implementation. If you need to call a specific constructor you do it like this: -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-create2 +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-create2 } -Since you supply a ``Props``, you can specify which dispatcher to use, what the default timeout should be used and more. -Now, our ``Squarer`` doesn't have any methods, so we'd better add those. +Since you supply a `Props`, you can specify which dispatcher to use, what the default timeout should be used and more. +Now, our `Squarer` doesn't have any methods, so we'd better add those. -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-iface +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-iface } Alright, now we've got some methods we can call, but we need to implement those in SquarerImpl. -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-impl +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-impl } Excellent, now we have an interface and an implementation of that interface, and we know how to create a Typed Actor from that, so let's look at calling these methods. -Method dispatch semantics -------------------------- +## Method dispatch semantics Methods returning: - * ``Unit`` will be dispatched with ``fire-and-forget`` semantics, exactly like ``ActorRef.tell`` - * ``scala.concurrent.Future[_]`` will use ``send-request-reply`` semantics, exactly like ``ActorRef.ask`` - * ``scala.Option[_]`` will use ``send-request-reply`` semantics, but *will* block to wait for an answer, - and return ``scala.None`` if no answer was produced within the timeout, or ``scala.Some[_]`` containing the result otherwise. - Any exception that was thrown during this call will be rethrown. - * Any other type of value will use ``send-request-reply`` semantics, but *will* block to wait for an answer, - throwing ``java.util.concurrent.TimeoutException`` if there was a timeout or rethrow any exception that was thrown during this call. +> + * `Unit` will be dispatched with `fire-and-forget` semantics, exactly like `ActorRef.tell` + * `scala.concurrent.Future[_]` will use `send-request-reply` semantics, exactly like `ActorRef.ask` + * `scala.Option[_]` will use `send-request-reply` semantics, but *will* block to wait for an answer, +and return `scala.None` if no answer was produced within the timeout, or `scala.Some[_]` containing the result otherwise. +Any exception that was thrown during this call will be rethrown. + * Any other type of value will use `send-request-reply` semantics, but *will* block to wait for an answer, +throwing `java.util.concurrent.TimeoutException` if there was a timeout or rethrow any exception that was thrown during this call. -Messages and immutability -------------------------- +## Messages and immutability While Akka cannot enforce that the parameters to the methods of your Typed Actors are immutable, we *strongly* recommend that parameters passed are immutable. -One-way message send -^^^^^^^^^^^^^^^^^^^^ +### One-way message send -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-call-oneway +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-call-oneway } As simple as that! The method will be executed on another thread; asynchronously. -Request-reply message send -^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Request-reply message send -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-call-option +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-call-option } This will block for as long as the timeout that was set in the Props of the Typed Actor, -if needed. It will return ``None`` if a timeout occurs. +if needed. It will return `None` if a timeout occurs. -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-call-strict +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-call-strict } This will block for as long as the timeout that was set in the Props of the Typed Actor, -if needed. It will throw a ``java.util.concurrent.TimeoutException`` if a timeout occurs. +if needed. It will throw a `java.util.concurrent.TimeoutException` if a timeout occurs. -Request-reply-with-future message send -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +### Request-reply-with-future message send -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-call-future +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-call-future } This call is asynchronous, and the Future returned can be used for asynchronous composition. -Stopping Typed Actors ---------------------- +## Stopping Typed Actors Since Akka's Typed Actors are backed by Akka Actors they must be stopped when they aren't needed anymore. -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-stop +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-stop } This asynchronously stops the Typed Actor associated with the specified proxy ASAP. -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-poisonpill +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-poisonpill } This asynchronously stops the Typed Actor associated with the specified proxy after it's done with all calls that were made prior to this call. -Typed Actor Hierarchies ------------------------ +## Typed Actor Hierarchies -Since you can obtain a contextual Typed Actor Extension by passing in an ``ActorContext`` -you can create child Typed Actors by invoking ``typedActorOf(..)`` on that: +Since you can obtain a contextual Typed Actor Extension by passing in an `ActorContext` +you can create child Typed Actors by invoking `typedActorOf(..)` on that: -.. includecode:: code/docs/actor/TypedActorDocSpec.scala - :include: typed-actor-hierarchy +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-hierarchy } -You can also create a child Typed Actor in regular Akka Actors by giving the ``ActorContext`` +You can also create a child Typed Actor in regular Akka Actors by giving the `ActorContext` as an input parameter to TypedActor.get(…). -Supervisor Strategy -------------------- +## Supervisor Strategy -By having your Typed Actor implementation class implement ``TypedActor.Supervisor`` +By having your Typed Actor implementation class implement `TypedActor.Supervisor` you can define the strategy to use for supervising child actors, as described in -:ref:`supervision` and :ref:`fault-tolerance-scala`. + supervision and @ref:[Fault Tolerance](fault-tolerance.md). -Lifecycle callbacks -------------------- +## Lifecycle callbacks By having your Typed Actor implementation class implement any and all of the following: - * ``TypedActor.PreStart`` - * ``TypedActor.PostStop`` - * ``TypedActor.PreRestart`` - * ``TypedActor.PostRestart`` +> +> + * `TypedActor.PreStart` + * `TypedActor.PostStop` + * `TypedActor.PreRestart` + * `TypedActor.PostRestart` +> +You can hook into the lifecycle of your Typed Actor. - You can hook into the lifecycle of your Typed Actor. +## Receive arbitrary messages -Receive arbitrary messages --------------------------- +If your implementation class of your TypedActor extends `akka.actor.TypedActor.Receiver`, +all messages that are not `MethodCall` instances will be passed into the `onReceive`-method. -If your implementation class of your TypedActor extends ``akka.actor.TypedActor.Receiver``, -all messages that are not ``MethodCall`` instances will be passed into the ``onReceive``-method. - -This allows you to react to DeathWatch ``Terminated``-messages and other types of messages, +This allows you to react to DeathWatch `Terminated`-messages and other types of messages, e.g. when interfacing with untyped actors. -Proxying --------- +## Proxying -You can use the ``typedActorOf`` that takes a TypedProps and an ActorRef to proxy the given ActorRef as a TypedActor. -This is usable if you want to communicate remotely with TypedActors on other machines, just pass the ``ActorRef`` to ``typedActorOf``. +You can use the `typedActorOf` that takes a TypedProps and an ActorRef to proxy the given ActorRef as a TypedActor. +This is usable if you want to communicate remotely with TypedActors on other machines, just pass the `ActorRef` to `typedActorOf`. -.. note:: +@@@ note - The ActorRef needs to accept ``MethodCall`` messages. +The ActorRef needs to accept `MethodCall` messages. -Lookup & Remoting ------------------ +@@@ -Since ``TypedActors`` are backed by ``Akka Actors``, you can use ``typedActorOf`` to proxy ``ActorRefs`` potentially residing on remote nodes. +## Lookup & Remoting -.. includecode:: code/docs/actor/TypedActorDocSpec.scala#typed-actor-remote +Since `TypedActors` are backed by `Akka Actors`, you can use `typedActorOf` to proxy `ActorRefs` potentially residing on remote nodes. -Supercharging -------------- +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-remote } + +## Supercharging Here's an example on how you can use traits to mix in behavior in your Typed Actors. -.. includecode:: code/docs/actor/TypedActorDocSpec.scala#typed-actor-supercharge +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-supercharge } -.. includecode:: code/docs/actor/TypedActorDocSpec.scala#typed-actor-supercharge-usage +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-actor-supercharge-usage } -Typed Router pattern --------------------- +## Typed Router pattern -Sometimes you want to spread messages between multiple actors. The easiest way to achieve this in Akka is to use a :ref:`Router `, -which can implement a specific routing logic, such as ``smallest-mailbox`` or ``consistent-hashing`` etc. +Sometimes you want to spread messages between multiple actors. The easiest way to achieve this in Akka is to use a @ref:[Router](routing.md), +which can implement a specific routing logic, such as `smallest-mailbox` or `consistent-hashing` etc. Routers are not provided directly for typed actors, but it is really easy to leverage an untyped router and use a typed proxy in front of it. -To showcase this let's create typed actors that assign themselves some random ``id``, so we know that in fact, the router has sent the message to different actors: +To showcase this let's create typed actors that assign themselves some random `id`, so we know that in fact, the router has sent the message to different actors: -.. includecode:: code/docs/actor/TypedActorDocSpec.scala#typed-router-types +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-router-types } In order to round robin among a few instances of such actors, you can simply create a plain untyped router, -and then facade it with a ``TypedActor`` like shown in the example below. This works because typed actors of course -communicate using the same mechanisms as normal actors, and methods calls on them get transformed into message sends of ``MethodCall`` messages. - -.. includecode:: code/docs/actor/TypedActorDocSpec.scala#typed-router +and then facade it with a `TypedActor` like shown in the example below. This works because typed actors of course +communicate using the same mechanisms as normal actors, and methods calls on them get transformed into message sends of `MethodCall` messages. +@@snip [TypedActorDocSpec.scala](code/docs/actor/TypedActorDocSpec.scala) { #typed-router } \ No newline at end of file diff --git a/akka-docs/src/main/paradox/scala/typed.md b/akka-docs/src/main/paradox/scala/typed.md index 75feb168fb..f984b81bcf 100644 --- a/akka-docs/src/main/paradox/scala/typed.md +++ b/akka-docs/src/main/paradox/scala/typed.md @@ -1,48 +1,46 @@ -.. _typed-scala: +# Akka Typed -########## -Akka Typed -########## +@@@ warning -.. warning:: +This module is currently marked as @ref:[may change](../common/may-change.md) in the sense +of being the subject of active research. This means that API or semantics can +change without warning or deprecation period and it is not recommended to use +this module in production just yet—you have been warned. - This module is currently marked as :ref:`may change ` in the sense - of being the subject of active research. This means that API or semantics can - change without warning or deprecation period and it is not recommended to use - this module in production just yet—you have been warned. +@@@ -As discussed in :ref:`actor-systems` (and following chapters) Actors are about +As discussed in @ref:[Actor Systems](../general/actor-systems.md) (and following chapters) Actors are about sending messages between independent units of computation, but how does that look like? In all of the following these imports are assumed: -.. includecode:: code/docs/akka/typed/IntroSpec.scala#imports +@@snip [IntroSpec.scala](code/docs/akka/typed/IntroSpec.scala) { #imports } With these in place we can define our first Actor, and of course it will say hello! -.. includecode:: code/docs/akka/typed/IntroSpec.scala#hello-world-actor +@@snip [IntroSpec.scala](code/docs/akka/typed/IntroSpec.scala) { #hello-world-actor } This small piece of code defines two message types, one for commanding the Actor to greet someone and one that the Actor will use to confirm that it has -done so. The :class:`Greet` type contains not only the information of whom to -greet, it also holds an :class:`ActorRef` that the sender of the message -supplies so that the :class:`HelloWorld` Actor can send back the confirmation +done so. The `Greet` type contains not only the information of whom to +greet, it also holds an `ActorRef` that the sender of the message +supplies so that the `HelloWorld` Actor can send back the confirmation message. -The behavior of the Actor is defined as the :meth:`greeter` value with the help -of the :class:`Stateless` behavior constructor—there are many different ways of +The behavior of the Actor is defined as the `greeter` value with the help +of the `Stateless` behavior constructor—there are many different ways of formulating behaviors as we shall see in the following. The “stateless” behavior is not capable of changing in response to a message, it will stay the same until the Actor is stopped by its parent. The type of the messages handled by this behavior is declared to be of class -:class:`Greet`, which implies that the supplied function’s ``msg`` argument is -also typed as such. This is why we can access the ``whom`` and ``replyTo`` +`Greet`, which implies that the supplied function’s `msg` argument is +also typed as such. This is why we can access the `whom` and `replyTo` members without needing to use a pattern match. -On the last line we see the :class:`HelloWorld` Actor send a message to another -Actor, which is done using the ``!`` operator (pronounced “tell”). Since the -``replyTo`` address is declared to be of type ``ActorRef[Greeted]`` the +On the last line we see the `HelloWorld` Actor send a message to another +Actor, which is done using the `!` operator (pronounced “tell”). Since the +`replyTo` address is declared to be of type `ActorRef[Greeted]` the compiler will only permit us to send messages of this type, other usage will not be accepted. @@ -50,61 +48,52 @@ The accepted message types of an Actor together with all reply types defines the protocol spoken by this Actor; in this case it is a simple request–reply protocol but Actors can model arbitrarily complex protocols when needed. The protocol is bundled together with the behavior that implements it in a nicely -wrapped scope—the ``HelloWorld`` object. +wrapped scope—the `HelloWorld` object. Now we want to try out this Actor, so we must start an ActorSystem to host it: -.. includecode:: code/docs/akka/typed/IntroSpec.scala#hello-world +@@snip [IntroSpec.scala](code/docs/akka/typed/IntroSpec.scala) { #hello-world } After importing the Actor’s protocol definition we start an Actor system from the defined behavior. As Carl Hewitt said, one Actor is no Actor—it would be quite lonely with nobody to talk to. In this sense the example is a little cruel because we only -give the ``HelloWorld`` Actor a fake person to talk to—the “ask” pattern -(represented by the ``?`` operator) can be used to send a message such that the +give the `HelloWorld` Actor a fake person to talk to—the “ask” pattern +(represented by the `?` operator) can be used to send a message such that the reply fulfills a Promise to which we get back the corresponding Future. -Note that the :class:`Future` that is returned by the “ask” operation is +Note that the `Future` that is returned by the “ask” operation is properly typed already, no type checks or casts needed. This is possible due to -the type information that is part of the message protocol: the ``?`` operator -takes as argument a function that accepts an :class:`ActorRef[U]` (which -explains the ``_`` hole in the expression on line 7 above) and the ``replyTo`` -parameter which we fill in is of type ``ActorRef[Greeted]``, which -means that the value that fulfills the :class:`Promise` can only be of type -:class:`Greeted`. +the type information that is part of the message protocol: the `?` operator +takes as argument a function that accepts an `ActorRef[U]` (which +explains the `_` hole in the expression on line 7 above) and the `replyTo` +parameter which we fill in is of type `ActorRef[Greeted]`, which +means that the value that fulfills the `Promise` can only be of type +`Greeted`. -We use this here to send the :class:`Greet` command to the Actor and when the +We use this here to send the `Greet` command to the Actor and when the reply comes back we will print it out and tell the actor system to shut down. -Once that is done as well we print the ``"system terminated"`` messages and the -program ends. The ``recovery`` combinator on the original :class:`Future` is +Once that is done as well we print the `"system terminated"` messages and the +program ends. The `recovery` combinator on the original `Future` is needed in order to ensure proper system shutdown even in case something went -wrong; the ``flatMap`` and ``map`` combinators that the ``for`` expression gets -turned into care only about the “happy path” and if the ``future`` failed with -a timeout then no ``greeting`` would be extracted and nothing would happen. +wrong; the `flatMap` and `map` combinators that the `for` expression gets +turned into care only about the “happy path” and if the `future` failed with +a timeout then no `greeting` would be extracted and nothing would happen. This shows that there are aspects of Actor messaging that can be type-checked by the compiler, but this ability is not unlimited, there are bounds to what we can statically express. Before we go on with a more complex (and realistic) example we make a small detour to highlight some of the theory behind this. -A Little Bit of Theory -====================== +## A Little Bit of Theory -The `Actor Model`_ as defined by Hewitt, Bishop and Steiger in 1973 is a +The [Actor Model](http://en.wikipedia.org/wiki/Actor_model1. send a finite number of messages to Actors it knows2. create a finite number of new Actors3. designate the behavior to be applied to the next message) as defined by Hewitt, Bishop and Steiger in 1973 is a computational model that expresses exactly what it means for computation to be distributed. The processing units—Actors—can only communicate by exchanging messages and upon reception of a message an Actor can do the following three fundamental actions: -.. _`Actor Model`: http://en.wikipedia.org/wiki/Actor_model - - 1. send a finite number of messages to Actors it knows - - 2. create a finite number of new Actors - - 3. designate the behavior to be applied to the next message - The Akka Typed project expresses these actions using behaviors and addresses. Messages can be sent to an address and behind this façade there is a behavior that receives the message and acts upon it. The binding between address and @@ -148,24 +137,23 @@ just given by the last message type that was received or sent. In the next section we demonstrate this on a more realistic example. -A More Complex Example -====================== +## A More Complex Example Consider an Actor that runs a chat room: client Actors may connect by sending a message that contains their screen name and then they can post messages. The chat room Actor will disseminate all posted messages to all currently connected client Actors. The protocol definition could look like the following: -.. includecode:: code/docs/akka/typed/IntroSpec.scala#chatroom-protocol +@@snip [IntroSpec.scala](code/docs/akka/typed/IntroSpec.scala) { #chatroom-protocol } -Initially the client Actors only get access to an ``ActorRef[GetSession]`` +Initially the client Actors only get access to an `ActorRef[GetSession]` which allows them to make the first step. Once a client’s session has been -established it gets a :class:`SessionGranted` message that contains a ``handle`` to -unlock the next protocol step, posting messages. The :class:`PostMessage` +established it gets a `SessionGranted` message that contains a `handle` to +unlock the next protocol step, posting messages. The `PostMessage` command will need to be sent to this particular address that represents the session that has been added to the chat room. The other aspect of a session is -that the client has revealed its own address, via the ``replyTo`` argument, so that subsequent -:class:`MessagePosted` events can be sent to it. +that the client has revealed its own address, via the `replyTo` argument, so that subsequent +`MessagePosted` events can be sent to it. This illustrates how Actors can express more than just the equivalent of method calls on Java objects. The declared message types and their contents describe a @@ -173,65 +161,64 @@ full protocol that can involve multiple Actors and that can evolve over multiple steps. The implementation of the chat room protocol would be as simple as the following: -.. includecode:: code/docs/akka/typed/IntroSpec.scala#chatroom-behavior +@@snip [IntroSpec.scala](code/docs/akka/typed/IntroSpec.scala) { #chatroom-behavior } The core of this behavior is stateful, the chat room itself does not change into something else when sessions are established, but we introduce a variable -that tracks the opened sessions. Note that by using a method parameter a ``var`` -is not needed. When a new :class:`GetSession` command comes in we add that client to the +that tracks the opened sessions. Note that by using a method parameter a `var` +is not needed. When a new `GetSession` command comes in we add that client to the list that is in the returned behavior. Then we also need to create the session’s -:class:`ActorRef` that will be used to post messages. In this case we want to -create a very simple Actor that just repackages the :class:`PostMessage` -command into a :class:`PostSessionMessage` command which also includes the +`ActorRef` that will be used to post messages. In this case we want to +create a very simple Actor that just repackages the `PostMessage` +command into a `PostSessionMessage` command which also includes the screen name. Such a wrapper Actor can be created by using the -:meth:`spawnAdapter` method on the :class:`ActorContext`, so that we can then -go on to reply to the client with the :class:`SessionGranted` result. +`spawnAdapter` method on the `ActorContext`, so that we can then +go on to reply to the client with the `SessionGranted` result. -The behavior that we declare here can handle both subtypes of :class:`Command`. -:class:`GetSession` has been explained already and the -:class:`PostSessionMessage` commands coming from the wrapper Actors will +The behavior that we declare here can handle both subtypes of `Command`. +`GetSession` has been explained already and the +`PostSessionMessage` commands coming from the wrapper Actors will trigger the dissemination of the contained chat room message to all connected clients. But we do not want to give the ability to send -:class:`PostSessionMessage` commands to arbitrary clients, we reserve that +`PostSessionMessage` commands to arbitrary clients, we reserve that right to the wrappers we create—otherwise clients could pose as completely -different screen names (imagine the :class:`GetSession` protocol to include -authentication information to further secure this). Therefore :class:`PostSessionMessage` -has ``private`` visibility and can't be created outside the actor. +different screen names (imagine the `GetSession` protocol to include +authentication information to further secure this). Therefore `PostSessionMessage` +has `private` visibility and can't be created outside the actor. If we did not care about securing the correspondence between a session and a -screen name then we could change the protocol such that :class:`PostMessage` is -removed and all clients just get an :class:`ActorRef[PostSessionMessage]` to +screen name then we could change the protocol such that `PostMessage` is +removed and all clients just get an `ActorRef[PostSessionMessage]` to send to. In this case no wrapper would be needed and we could just use -``ctx.self``. The type-checks work out in that case because -:class:`ActorRef[-T]` is contravariant in its type parameter, meaning that we -can use a :class:`ActorRef[Command]` wherever an -:class:`ActorRef[PostSessionMessage]` is needed—this makes sense because the +`ctx.self`. The type-checks work out in that case because +`ActorRef[-T]` is contravariant in its type parameter, meaning that we +can use a `ActorRef[Command]` wherever an +`ActorRef[PostSessionMessage]` is needed—this makes sense because the former simply speaks more languages than the latter. The opposite would be -problematic, so passing an :class:`ActorRef[PostSessionMessage]` where -:class:`ActorRef[Command]` is required will lead to a type error. +problematic, so passing an `ActorRef[PostSessionMessage]` where +`ActorRef[Command]` is required will lead to a type error. -Trying it out -------------- +### Trying it out In order to see this chat room in action we need to write a client Actor that can use it: -.. includecode:: code/docs/akka/typed/IntroSpec.scala#chatroom-gabbler +@@snip [IntroSpec.scala](code/docs/akka/typed/IntroSpec.scala) { #chatroom-gabbler } From this behavior we can create an Actor that will accept a chat room session, post a message, wait to see it published, and then terminate. The last step requires the ability to change behavior, we need to transition from the normal running behavior into the terminated state. This is why this Actor uses a -different behavior constructor named :class:`Total`. This constructor takes as +different behavior constructor named `Total`. This constructor takes as argument a function from the handled message type, in this case -:class:`SessionEvent`, to the next behavior. That next behavior must again be +`SessionEvent`, to the next behavior. That next behavior must again be of the same type as we discussed in the theory section above. Here we either stay in the very same behavior or we terminate, and both of these cases are so -common that there are special values ``Same`` and ``Stopped`` that can be used. +common that there are special values `Same` and `Stopped` that can be used. The behavior is named “total” (as opposed to “partial”) because the declared -function must handle all values of its input type. Since :class:`SessionEvent` +function must handle all values of its input type. Since `SessionEvent` is a sealed trait the Scala compiler will warn us if we forget to handle one of the subtypes; in this case it reminded us that alternatively to -:class:`SessionGranted` we may also receive a :class:`SessionDenied` event. +`SessionGranted` we may also receive a `SessionDenied` event. Now to try things out we must start both a chat room and a gabbler and of course we do this inside an Actor system. Since there can be only one guardian @@ -240,63 +227,61 @@ want—it complicates its logic) or the gabbler from the chat room (which is nonsensical) or we start both of them from a third Actor—our only sensible choice: -.. includecode:: code/docs/akka/typed/IntroSpec.scala#chatroom-main +@@snip [IntroSpec.scala](code/docs/akka/typed/IntroSpec.scala) { #chatroom-main } -In good tradition we call the ``main`` Actor what it is, it directly -corresponds to the ``main`` method in a traditional Java application. This +In good tradition we call the `main` Actor what it is, it directly +corresponds to the `main` method in a traditional Java application. This Actor will perform its job on its own accord, we do not need to send messages -from the outside, so we declare it to be of type ``NotUsed``. Actors receive not +from the outside, so we declare it to be of type `NotUsed`. Actors receive not only external messages, they also are notified of certain system events, so-called Signals. In order to get access to those we choose to implement this -particular one using the :class:`Stateful` behavior decorator. The -provided ``signal`` function will be invoked for signals (subclasses of :class:`Signal`) -or the ``mesg`` function for user messages. +particular one using the `Stateful` behavior decorator. The +provided `signal` function will be invoked for signals (subclasses of `Signal`) +or the `mesg` function for user messages. This particular main Actor reacts to two signals: when it is started it will -first receive the :class:`PreStart` signal, upon which the chat room and the +first receive the `PreStart` signal, upon which the chat room and the gabbler are created and the session between them is initiated, and when the -gabbler is finished we will receive the :class:`Terminated` event due to having -called ``ctx.watch`` for it. This allows us to shut down the Actor system: when +gabbler is finished we will receive the `Terminated` event due to having +called `ctx.watch` for it. This allows us to shut down the Actor system: when the main Actor terminates there is nothing more to do. -Therefore after creating the Actor system with the ``main`` Actor’s -:class:`Props` we just await its termination. +Therefore after creating the Actor system with the `main` Actor’s +`Props` we just await its termination. -Status of this Project and Relation to Akka Actors -================================================== +## Status of this Project and Relation to Akka Actors Akka Typed is the result of many years of research and previous attempts (including Typed Channels in the 2.2.x series) and it is on its way to stabilization, but maturing such a profound change to the core concept of Akka will take a long time. We expect that this module will stay marked -:ref:`may change ` for multiple major releases of Akka and the -plain ``akka.actor.Actor`` will not be deprecated or go away anytime soon. +@ref:[may change](../common/may-change.md) for multiple major releases of Akka and the +plain `akka.actor.Actor` will not be deprecated or go away anytime soon. Being a research project also entails that the reference documentation is not as detailed as it will be for a final version, please refer to the API documentation for greater depth and finer detail. -Main Differences ----------------- +### Main Differences -The most prominent difference is the removal of the ``sender()`` functionality. +The most prominent difference is the removal of the `sender()` functionality. This turned out to be the Achilles heel of the Typed Channels project, it is the feature that makes its type signatures and macros too complex to be viable. The solution chosen in Akka Typed is to explicitly include the properly typed reply-to address in the message, which both burdens the user with this task but also places this aspect of protocol design where it belongs. -The other prominent difference is the removal of the :class:`Actor` trait. In +The other prominent difference is the removal of the `Actor` trait. In order to avoid closing over unstable references from different execution contexts (e.g. Future transformations) we turned all remaining methods that were on this trait into messages: the behavior receives the -:class:`ActorContext` as an argument during processing and the lifecycle hooks +`ActorContext` as an argument during processing and the lifecycle hooks have been converted into Signals. A side-effect of this is that behaviors can now be tested in isolation without having to be packaged into an Actor, tests can run fully synchronously without having to worry about timeouts and spurious failures. Another side-effect is -that behaviors can nicely be composed and decorated, see the :class:`And`, -:class:`Or`, :class:`Widened`, :class:`ContextAware` combinators; nothing about +that behaviors can nicely be composed and decorated, see the `And`, +`Or`, `Widened`, `ContextAware` combinators; nothing about these is special or internal, new combinators can be written as external -libraries or tailor-made for each project. +libraries or tailor-made for each project. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/security/2017-02-10-java-serialization.md b/akka-docs/src/main/paradox/security/2017-02-10-java-serialization.md index 7960765721..7a9a86d723 100644 --- a/akka-docs/src/main/paradox/security/2017-02-10-java-serialization.md +++ b/akka-docs/src/main/paradox/security/2017-02-10-java-serialization.md @@ -1,62 +1,55 @@ -Java Serialization, Fixed in Akka 2.4.17 -======================================== +# Java Serialization, Fixed in Akka 2.4.17 -Date ----- +## Date 10 Feburary 2017 -Description of Vulnerability -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Description of Vulnerability -An attacker that can connect to an ``ActorSystem`` exposed via Akka Remote over TCP can gain remote code execution +An attacker that can connect to an `ActorSystem` exposed via Akka Remote over TCP can gain remote code execution capabilities in the context of the JVM process that runs the ActorSystem if: -* ``JavaSerializer`` is enabled (default in Akka 2.4.x) -* and TLS is disabled *or* TLS is enabled with ``akka.remote.netty.ssl.security.require-mutual-authentication = false`` - (which is still the default in Akka 2.4.x) -* or if TLS is enabled with mutual authentication and the authentication keys of a host that is allowed to connect have been compromised, an attacker gained access to a valid certificate (e.g. by compromising a node with certificates issued by the same internal PKI tree to get access of the certificate) -* regardless of whether ``untrusted`` mode is enabled or not + * `JavaSerializer` is enabled (default in Akka 2.4.x) + * and TLS is disabled *or* TLS is enabled with `akka.remote.netty.ssl.security.require-mutual-authentication = false` +(which is still the default in Akka 2.4.x) + * or if TLS is enabled with mutual authentication and the authentication keys of a host that is allowed to connect have been compromised, an attacker gained access to a valid certificate (e.g. by compromising a node with certificates issued by the same internal PKI tree to get access of the certificate) + * regardless of whether `untrusted` mode is enabled or not -Java deserialization is `known to be vulnerable `_ to attacks when attacker can provide arbitrary types. +Java deserialization is [known to be vulnerable](https://community.hpe.com/t5/Security-Research/The-perils-of-Java-deserialization/ba-p/6838995) to attacks when attacker can provide arbitrary types. Akka Remoting uses Java serialiser as default configuration which makes it vulnerable in its default form. The documentation of how to disable Java serializer was not complete. The documentation of how to enable mutual authentication was missing (only described in reference.conf). -To protect against such attacks the system should be updated to Akka `2.4.17` or later and be configured with -:ref:`disabled Java serializer `. Additional protection can be achieved when running in an -untrusted network by enabling :ref:`TLS with mutual authentication `. +To protect against such attacks the system should be updated to Akka *2.4.17* or later and be configured with +@ref:[disabled Java serializer](../scala/remoting.md#disable-java-serializer-scala). Additional protection can be achieved when running in an +untrusted network by enabling @ref:[TLS with mutual authentication](../scala/remoting.md#remote-tls-scala). -Please subscribe to the `akka-security `_ mailing list to be notified promptly about future security issues. +Please subscribe to the [akka-security](https://groups.google.com/forum/#!forum/akka-security) mailing list to be notified promptly about future security issues. -Severity -~~~~~~~~ +### Severity -The `CVSS `_ score of this vulnerability is 6.8 (Medium), based on vector `AV:A/AC:M/Au:N/C:C/I:C/A:C/E:F/RL:TF/RC:C `_. +The [CVSS](https://en.wikipedia.org/wiki/CVSS) score of this vulnerability is 6.8 (Medium), based on vector [AV:A/AC:M/Au:N/C:C/I:C/A:C/E:F/RL:TF/RC:C](https://nvd.nist.gov/cvss.cfm?calculator&version=2&vector=\(AV:A/AC:M/Au:N/C:C/I:C/A:C/E:F/RL:TF/RC:C\)). Rationale for the score: -* AV:A - Best practice is that Akka remoting nodes should only be accessible from the adjacent network, so in good setups, this will be adjacent. -* AC:M - Any one in the adjacent network can launch the attack with non-special access privileges. -* C:C, I:C, A:C - Remote Code Execution vulnerabilities are by definition CIA:C. + * AV:A - Best practice is that Akka remoting nodes should only be accessible from the adjacent network, so in good setups, this will be adjacent. + * AC:M - Any one in the adjacent network can launch the attack with non-special access privileges. + * C:C, I:C, A:C - Remote Code Execution vulnerabilities are by definition CIA:C. -Affected Versions -~~~~~~~~~~~~~~~~~ +### Affected Versions -- Akka `2.4.16` and prior -- Akka `2.5-M1` (milestone not intended for production) + * Akka *2.4.16* and prior + * Akka *2.5-M1* (milestone not intended for production) -Fixed Versions -~~~~~~~~~~~~~~ +### Fixed Versions We have prepared patches for the affected versions, and have released the following versions which resolve the issue: -- Akka `2.4.17` (Scala 2.11, 2.12) + * Akka *2.4.17* (Scala 2.11, 2.12) Binary and source compatibility has been maintained for the patched releases so the upgrade procedure is as simple as changing the library dependency. It will also be fixed in 2.5-M2 or 2.5.0-RC1. -Acknowledgements -~~~~~~~~~~~~~~~~ +### Acknowledgements We would like to thank Alvaro Munoz at Hewlett Packard Enterprise Security & Adrian Bravo at Workday for their thorough investigation and bringing this issue to our attention. \ No newline at end of file diff --git a/akka-docs/src/main/paradox/security/index.md b/akka-docs/src/main/paradox/security/index.md index d7a48f318b..5dc5a917b3 100644 --- a/akka-docs/src/main/paradox/security/index.md +++ b/akka-docs/src/main/paradox/security/index.md @@ -1,36 +1,33 @@ -Security Announcements -====================== +# Security Announcements -Receiving Security Advisories ------------------------------ +## Receiving Security Advisories -The best way to receive any and all security announcements is to subscribe to the `Akka security list `_. +The best way to receive any and all security announcements is to subscribe to the [Akka security list](https://groups.google.com/forum/#!forum/akka-security). The mailing list is very low traffic, and receives notifications only after security reports have been managed by the core team and fixes are publicly available. -Reporting Vulnerabilities -------------------------- +## Reporting Vulnerabilities We strongly encourage people to report such problems to our private security mailing list first, before disclosing them in a public forum. Following best practice, we strongly encourage anyone to report potential security -vulnerabilities to security@akka.io before disclosing them in a public forum like the mailing list or as a Github issue. +vulnerabilities to [security@akka.io](mailto:security@akka.io) before disclosing them in a public forum like the mailing list or as a Github issue. Reports to this email address will be handled by our security team, who will work together with you to ensure that a fix can be provided without delay. -Security Related Documentation ------------------------------- +## Security Related Documentation -* :ref:`disable-java-serializer-scala` -* :ref:`remote-deployment-whitelist-scala` -* :ref:`remote-security-scala` + * @ref:[Disabling the Java Serializer](../scala/remoting.md#disable-java-serializer-scala) + * @ref:[Remote deployment whitelist](../scala/remoting.md#remote-deployment-whitelist-scala) + * @ref:[Remote Security](../scala/remoting.md#remote-security-scala) +## Fixed Security Vulnerabilities -Fixed Security Vulnerabilities ------------------------------- +@@toc { depth=1 } -.. toctree:: - :maxdepth: 1 +@@@ index - 2017-02-10-java-serialization \ No newline at end of file +* [2017-02-10-java-serialization](2017-02-10-java-serialization.md) + +@@@ \ No newline at end of file