!str,htc remove FlexiMerge/Route
- replace all occurrences with equivalent GraphStage implementations This commit introduces a mini-DSL for GraphStage that allows emitting or reading multiple elements to/from a port with one statement, installing stateful handlers on the port to make it work. The emitting side allows stacked continuations, meaning that while an emit() is ongoing (waiting for demand) another one can be added to the queue; this allows convenient formualation of merge-type stages.
This commit is contained in:
Roland Kuhn
parent
dc07fd250c
commit
02810cfa64
45 changed files with 1001 additions and 3227 deletions
|
|
@ -258,178 +258,9 @@ The following code example demonstrates the buffer class corresponding to the me
|
|||
Custom graph processing junctions
|
||||
=================================
|
||||
|
||||
To extend available fan-in and fan-out structures (graph stages) Akka Streams include :class:`FlexiMerge` and
|
||||
:class:`FlexiRoute` which provide an intuitive DSL which allows to describe which upstream or downstream stream
|
||||
elements should be pulled from or emitted to.
|
||||
|
||||
Using FlexiMerge
|
||||
----------------
|
||||
:class:`FlexiMerge` can be used to describe a fan-in element which contains some logic about which upstream stage the
|
||||
merge should consume elements. It is recommended to create your custom fan-in stage as a separate class, name it
|
||||
appropriately to the behavior it is exposing and reuse it this way – similarly as you would use built-in fan-in stages.
|
||||
|
||||
The first flexi merge example we are going to implement is a so-called "preferring merge", in which one
|
||||
of the input ports is *preferred*, e.g. if the merge could pull from the preferred or another secondary input port,
|
||||
it will pull from the preferred port, only pulling from the secondary ports once the preferred one does not have elements
|
||||
available.
|
||||
|
||||
Implementing a custom merge stage is done by extending the :class:`FlexiMerge` trait, exposing its input ports and finally
|
||||
defining the logic which will decide how this merge should behave. First we need to create the ports which are used
|
||||
to wire up the fan-in element in a :class:`FlowGraph`. These input ports *must* be properly typed and their names should
|
||||
indicate what kind of port it is.
|
||||
|
||||
.. includecode:: code/docs/stream/FlexiDocSpec.scala#flexi-preferring-merge-ports
|
||||
|
||||
Next we implement the ``createMergeLogic`` method, which will be used as factory of merges :class:`MergeLogic`.
|
||||
A new :class:`MergeLogic` object will be created for each materialized stream, so it is allowed to be stateful.
|
||||
|
||||
.. includecode:: code/docs/stream/FlexiDocSpec.scala#flexi-preferring-merge
|
||||
|
||||
The :class:`MergeLogic` defines the behaviour of our merge stage, and may be *stateful* (for example to buffer some elements
|
||||
internally).
|
||||
|
||||
.. warning::
|
||||
While a :class:`MergeLogic` instance *may* be stateful, the :class:`FlexiMerge` instance
|
||||
*must not* hold any mutable state, since it may be shared across several materialized ``FlowGraph`` instances.
|
||||
|
||||
Next we implement the ``initialState`` method, which returns the behaviour of the merge stage. A ``MergeLogic#State``
|
||||
defines the behaviour of the merge by signaling which input ports it is interested in consuming, and how to handle
|
||||
the element once it has been pulled from its upstream. Signalling which input port we are interested in pulling data
|
||||
from is done by using an appropriate *read condition*. Available *read conditions* include:
|
||||
|
||||
- ``Read(input)`` - reads from only the given input,
|
||||
- ``ReadAny(inputs)`` – reads from any of the given inputs,
|
||||
- ``ReadPreferred(preferred)(secondaries)`` – reads from the preferred input if elements available, otherwise from one of the secondaries,
|
||||
- ``ReadAll(inputs)`` – reads from *all* given inputs (like ``Zip``), and offers an :class:`ReadAllInputs` as the ``element`` passed into the state function, which allows to obtain the pulled element values in a type-safe way.
|
||||
|
||||
In our case we use the :class:`ReadPreferred` read condition which has the exact semantics which we need to implement
|
||||
our preferring merge – it pulls elements from the preferred input port if there are any available, otherwise reverting
|
||||
to pulling from the secondary inputs. The context object passed into the state function allows us to interact with the
|
||||
connected streams, for example by emitting an ``element``, which was just pulled from the given ``input``, or signalling
|
||||
completion or failure to the merges downstream stage.
|
||||
|
||||
The state function must always return the next behaviour to be used when an element should be pulled from its upstreams,
|
||||
we use the special :class:`SameState` object which signals :class:`FlexiMerge` that no state transition is needed.
|
||||
|
||||
.. note::
|
||||
As response to an input element it is allowed to emit at most one output element.
|
||||
|
||||
Implementing Zip-like merges
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
More complex fan-in junctions may require not only multiple States but also sharing state between those states.
|
||||
As :class:`MergeLogic` is allowed to be stateful, it can be easily used to hold the state of the merge junction.
|
||||
|
||||
We now implement the equivalent of the built-in ``Zip`` junction by using the property that a the MergeLogic can be stateful
|
||||
and that each read is followed by a state transition (much like in Akka FSM or ``Actor#become``).
|
||||
|
||||
.. includecode:: code/docs/stream/FlexiDocSpec.scala#fleximerge-zip-states
|
||||
|
||||
The above style of implementing complex flexi merges is useful when we need fine grained control over consuming from certain
|
||||
input ports. Sometimes however it is simpler to strictly consume all of a given set of inputs. In the ``Zip`` rewrite below
|
||||
we use the :class:`ReadAll` read condition, which behaves slightly differently than the other read conditions, as the element
|
||||
it is emitting is of the type :class:`ReadAllInputs` instead of directly handing over the pulled elements:
|
||||
|
||||
.. includecode:: code/docs/stream/FlexiDocSpec.scala#fleximerge-zip-readall
|
||||
|
||||
Thanks to being handed a :class:`ReadAllInputs` instance instead of the elements directly it is possible to pick elements
|
||||
in a type-safe way based on their input port.
|
||||
|
||||
Connecting your custom junction is as simple as creating an instance and connecting Sources and Sinks to its ports
|
||||
(notice that the merged output port is named ``out``):
|
||||
|
||||
.. includecode:: code/docs/stream/FlexiDocSpec.scala#fleximerge-zip-connecting
|
||||
|
||||
.. _flexi-merge-completion-handling-scala:
|
||||
|
||||
Completion handling
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
Completion handling in :class:`FlexiMerge` is defined by an :class:`CompletionHandling` object which can react on
|
||||
completion and failure signals from its upstream input ports. The default strategy is to remain running while at-least-one
|
||||
upstream input port which are declared to be consumed in the current state is still running (i.e. has not signalled
|
||||
completion or failure).
|
||||
|
||||
Customising completion can be done via overriding the ``MergeLogic#initialCompletionHandling`` method, or from within
|
||||
a :class:`State` by calling ``ctx.changeCompletionHandling(handling)``. Other than the default completion handling (as
|
||||
late as possible) :class:`FlexiMerge` also provides an ``eagerClose`` completion handling which completes (or fails) its
|
||||
downstream as soon as at least one of its upstream inputs completes (or fails).
|
||||
|
||||
In the example below the we implement an ``ImportantWithBackups`` fan-in stage which can only keep operating while
|
||||
the ``important`` and at-least-one of the ``replica`` inputs are active. Therefore in our custom completion strategy we
|
||||
have to investigate which input has completed or failed and act accordingly. If the important input completed or failed
|
||||
we propagate this downstream completing the stream, on the other hand if the first replicated input fails, we log the
|
||||
exception and instead of failing the downstream swallow this exception (as one failed replica is still acceptable).
|
||||
Then we change the completion strategy to ``eagerClose`` which will propagate any future completion or failure event right
|
||||
to this stages downstream effectively shutting down the stream.
|
||||
|
||||
.. includecode:: code/docs/stream/FlexiDocSpec.scala#fleximerge-completion
|
||||
|
||||
In case you want to change back to the default completion handling, it is available as ``MergeLogic#defaultCompletionHandling``.
|
||||
|
||||
It is not possible to emit elements from the completion handling, since completion
|
||||
handlers may be invoked at any time (without regard to downstream demand being available).
|
||||
|
||||
Using FlexiRoute
|
||||
----------------
|
||||
Similarily to using :class:`FlexiMerge`, implementing custom fan-out stages requires extending the :class:`FlexiRoute` class
|
||||
and with a :class:`RouteLogic` object which determines how the route should behave.
|
||||
|
||||
The first flexi route stage that we are going to implement is ``Unzip``, which consumes a stream of pairs and splits
|
||||
it into two streams of the first and second elements of each tuple.
|
||||
|
||||
A :class:`FlexiRoute` has exactly-one input port (in our example, type parameterized as ``(A,B)``), and may have multiple
|
||||
output ports, all of which must be created beforehand (they can not be added dynamically). First we need to create the
|
||||
ports which are used to wire up the fan-in element in a :class:`FlowGraph`.
|
||||
|
||||
.. includecode:: code/docs/stream/FlexiDocSpec.scala#flexiroute-unzip
|
||||
|
||||
Next we implement ``RouteLogic#initialState`` by providing a State that uses the :class:`DemandFromAll` *demand condition*
|
||||
to signal to flexi route that elements can only be emitted from this stage when demand is available from all given downstream
|
||||
output ports. Other available demand conditions are:
|
||||
|
||||
- ``DemandFrom(output)`` - triggers when the given output port has pending demand,
|
||||
- ``DemandFromAny(outputs)`` - triggers when any of the given output ports has pending demand,
|
||||
- ``DemandFromAll(outputs)`` - triggers when *all* of the given output ports has pending demand.
|
||||
|
||||
Since the ``Unzip`` junction we're implementing signals both downstreams stages at the same time, we use ``DemandFromAll``,
|
||||
unpack the incoming tuple in the state function and signal its first element to the ``left`` stream, and the second element
|
||||
of the tuple to the ``right`` stream. Notice that since we are emitting values of different types (``A`` and ``B``),
|
||||
the type parameter of this ``State[_]`` must be set to ``Any``. This type can be utilised more efficiently when a junction
|
||||
is emitting the same type of element to its downstreams e.g. in all *strictly routing* stages.
|
||||
|
||||
The state function must always return the next behaviour to be used when an element should be emitted,
|
||||
we use the special :class:`SameState` object which signals :class:`FlexiRoute` that no state transition is needed.
|
||||
|
||||
.. warning::
|
||||
While a :class:`RouteLogic` instance *may* be stateful, the :class:`FlexiRoute` instance
|
||||
*must not* hold any mutable state, since it may be shared across several materialized ``FlowGraph`` instances.
|
||||
|
||||
.. note::
|
||||
It is only allowed to `emit` at most one element to each output in response to `onInput`, `IllegalStateException` is thrown.
|
||||
|
||||
Completion handling
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
Completion handling in :class:`FlexiRoute` is handled similarly to :class:`FlexiMerge` (which is explained in depth in
|
||||
:ref:`flexi-merge-completion-handling-scala`), however in addition to reacting to its upstreams *completion* or *failure*
|
||||
it can also react to its downstream stages *cancelling* their subscriptions. The default completion handling for
|
||||
:class:`FlexiRoute` (defined in ``RouteLogic#defaultCompletionHandling``) is to continue running until all of its
|
||||
downstreams have cancelled their subscriptions, or the upstream has completed / failed.
|
||||
|
||||
In order to customise completion handling we can override overriding the ``RouteLogic#initialCompletionHandling`` method,
|
||||
or call ``ctx.changeCompletionHandling(handling)`` from within a :class:`State`. Other than the default completion handling
|
||||
(as late as possible) :class:`FlexiRoute` also provides an ``eagerClose`` completion handling which completes all its
|
||||
downstream streams as well as cancels its upstream as soon as *any* of its downstream stages cancels its subscription.
|
||||
|
||||
In the example below we implement a custom completion handler which completes the entire stream eagerly if the ``important``
|
||||
downstream cancels, otherwise (if any other downstream cancels their subscription) the :class:`ImportantRoute` keeps running.
|
||||
|
||||
.. includecode:: code/docs/stream/FlexiDocSpec.scala#flexiroute-completion
|
||||
|
||||
Notice that State changes are only allowed in reaction to downstream cancellations, and not in the upstream completion/failure
|
||||
cases. This is because since there is only one upstream, there is nothing else to do than possibly flush buffered elements
|
||||
and continue with shutting down the entire stream.
|
||||
|
||||
It is not possible to emit elements from the completion handling, since completion
|
||||
handlers may be invoked at any time (without regard to downstream demand being available).
|
||||
To extend available fan-in and fan-out structures (graph stages) Akka Streams include :class:`GraphStage`. This is an
|
||||
advanced usage DSL that should only be needed in rare and special cases, documentation will be forthcoming in one of the
|
||||
next releases.
|
||||
|
||||
Thread safety of custom processing stages
|
||||
=========================================
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ strategies, but the semantics have been adapted to the domain of stream processi
|
|||
|
||||
.. warning::
|
||||
|
||||
*ZipWith*, *FlexiMerge*, *FlexiRoute* junction, *ActorPublisher* source and *ActorSubscriber* sink
|
||||
*ZipWith*, *GraphStage* junction, *ActorPublisher* source and *ActorSubscriber* sink
|
||||
components do not honour the supervision strategy attribute yet.
|
||||
|
||||
Supervision Strategies
|
||||
|
|
|
|||
|
|
@ -251,5 +251,5 @@ such as ``Zip`` however *do guarantee* their outputs order, as each output eleme
|
|||
been signalled already – thus the ordering in the case of zipping is defined by this property.
|
||||
|
||||
If you find yourself in need of fine grained control over order of emitted elements in fan-in
|
||||
scenarios consider using :class:`MergePreferred` or :class:`FlexiMerge` – which gives you full control over how the
|
||||
scenarios consider using :class:`MergePreferred` or :class:`GraphStage` – which gives you full control over how the
|
||||
merge is performed.
|
||||
|
|
|
|||
|
|
@ -32,7 +32,6 @@ Akka Streams currently provide these junctions (for a detailed list see :ref:`st
|
|||
- ``Balance[T]`` – *(1 input, N outputs)* given an input element emits to one of its output ports
|
||||
- ``UnzipWith[In,A,B,...]`` – *(1 input, N outputs)* takes a function of 1 input that given a value for each input emits N output elements (where N <= 20)
|
||||
- ``UnZip[A,B]`` – *(1 input, 2 outputs)* splits a stream of ``(A,B)`` tuples into two streams, one of type ``A`` and one of type ``B``
|
||||
- ``FlexiRoute[In]`` – *(1 input, N outputs)* enables writing custom fan out elements using a simple DSL
|
||||
|
||||
* **Fan-in**
|
||||
|
||||
|
|
@ -41,7 +40,6 @@ Akka Streams currently provide these junctions (for a detailed list see :ref:`st
|
|||
- ``ZipWith[A,B,...,Out]`` – *(N inputs, 1 output)* which takes a function of N inputs that given a value for each input emits 1 output element
|
||||
- ``Zip[A,B]`` – *(2 inputs, 1 output)* is a :class:`ZipWith` specialised to zipping input streams of ``A`` and ``B`` into an ``(A,B)`` tuple stream
|
||||
- ``Concat[A]`` – *(2 inputs, 1 output)* concatenates two streams (first consume one, then the second one)
|
||||
- ``FlexiMerge[Out]`` – *(N inputs, 1 output)* enables writing custom fan-in elements using a simple DSL
|
||||
|
||||
One of the goals of the FlowGraph DSL is to look similar to how one would draw a graph on a whiteboard, so that it is
|
||||
simple to translate a design from whiteboard to code and be able to relate those two. Let's illustrate this by translating
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue