Added MergeSequence graph stage (#29247)

Fixes #28769

Use case for this is if you have a sequence of elements that has been
partitioned across multiple streams, and you want to merge them back
together in order. It will typically be used in combination with
`zipWithIndex` to define the index for the sequence, followed by a
`Partition`, followed by the processing of different substreams with
different flows (each flow emitting exactly one output for each input),
and then merging with this stage, using the index from `zipWithIndex`.

A more concrete use case is if you're consuming messages from a message
broker, and you have a flow that you wish to apply to some messages, but
not others, you can partition the message stream according to which
should be processed by the flow and which should bypass it, and then
bring the elements back together acknowledgement. If an ordinary merge
was used rather than this, the messages that bypass the processing flow
would likely overtake the messages going through the processing flow,
and the result would be out of order offset acknowledgement which would
lead to dropping messages on failure.

I've included a minimal version of the above example in the documentation.
This commit is contained in:
James Roper 2020-07-10 01:52:46 +10:00 committed by GitHub
parent 1898216b0d
commit 558160702b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 471 additions and 0 deletions

View file

@ -171,6 +171,7 @@ object StreamOperatorsIndexGenerator extends AutoPlugin {
.map(method => (element, method))
} ++ List(
(noElement, "Partition"),
(noElement, "MergeSequence"),
(noElement, "Broadcast"),
(noElement, "Balance"),
(noElement, "Unzip"),