Added MergeSequence graph stage (#29247)
Fixes #28769 Use case for this is if you have a sequence of elements that has been partitioned across multiple streams, and you want to merge them back together in order. It will typically be used in combination with `zipWithIndex` to define the index for the sequence, followed by a `Partition`, followed by the processing of different substreams with different flows (each flow emitting exactly one output for each input), and then merging with this stage, using the index from `zipWithIndex`. A more concrete use case is if you're consuming messages from a message broker, and you have a flow that you wish to apply to some messages, but not others, you can partition the message stream according to which should be processed by the flow and which should bypass it, and then bring the elements back together acknowledgement. If an ordinary merge was used rather than this, the messages that bypass the processing flow would likely overtake the messages going through the processing flow, and the result would be out of order offset acknowledgement which would lead to dropping messages on failure. I've included a minimal version of the above example in the documentation.
This commit is contained in:
parent
1898216b0d
commit
558160702b
9 changed files with 471 additions and 0 deletions
|
|
@ -171,6 +171,7 @@ object StreamOperatorsIndexGenerator extends AutoPlugin {
|
|||
.map(method => (element, method))
|
||||
} ++ List(
|
||||
(noElement, "Partition"),
|
||||
(noElement, "MergeSequence"),
|
||||
(noElement, "Broadcast"),
|
||||
(noElement, "Balance"),
|
||||
(noElement, "Unzip"),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue