Added MergeSequence graph stage (#29247)

Fixes #28769

Use case for this is if you have a sequence of elements that has been
partitioned across multiple streams, and you want to merge them back
together in order. It will typically be used in combination with
`zipWithIndex` to define the index for the sequence, followed by a
`Partition`, followed by the processing of different substreams with
different flows (each flow emitting exactly one output for each input),
and then merging with this stage, using the index from `zipWithIndex`.

A more concrete use case is if you're consuming messages from a message
broker, and you have a flow that you wish to apply to some messages, but
not others, you can partition the message stream according to which
should be processed by the flow and which should bypass it, and then
bring the elements back together acknowledgement. If an ordinary merge
was used rather than this, the messages that bypass the processing flow
would likely overtake the messages going through the processing flow,
and the result would be out of order offset acknowledgement which would
lead to dropping messages on failure.

I've included a minimal version of the above example in the documentation.
This commit is contained in:
James Roper 2020-07-10 01:52:46 +10:00 committed by GitHub
parent 1898216b0d
commit 558160702b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 471 additions and 0 deletions

View file

@ -0,0 +1,52 @@
/*
* Copyright (C) 2020 Lightbend Inc. <https://www.lightbend.com>
*/
package docs.stream.operators
import akka.actor.ActorSystem
object MergeSequenceDocExample {
implicit val system: ActorSystem = ???
// #merge-sequence
import akka.NotUsed
import akka.stream.ClosedShape
import akka.stream.scaladsl.{ Flow, GraphDSL, MergeSequence, Partition, RunnableGraph, Sink, Source }
val subscription: Source[Message, NotUsed] = createSubscription()
val messageProcessor: Flow[(Message, Long), (Message, Long), NotUsed] =
createMessageProcessor()
val messageAcknowledger: Sink[Message, NotUsed] = createMessageAcknowledger()
RunnableGraph
.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
// Partitions stream into messages that should or should not be processed
val partition = builder.add(Partition[(Message, Long)](2, {
case (message, _) if shouldProcess(message) => 0
case _ => 1
}))
// Merges stream by the index produced by zipWithIndex
val merge = builder.add(MergeSequence[(Message, Long)](2)(_._2))
subscription.zipWithIndex ~> partition.in
// First goes through message processor
partition.out(0) ~> messageProcessor ~> merge
// Second partition bypasses message processor
partition.out(1) ~> merge
merge.out.map(_._1) ~> messageAcknowledger
ClosedShape
})
.run()
// #merge-sequence
def shouldProcess(message: Message): Boolean = true
trait Message
def createSubscription(): Source[Message, NotUsed] = ???
def createMessageProcessor(): Flow[(Message, Long), (Message, Long), NotUsed] = ???
def createMessageAcknowledger(): Sink[Message, NotUsed] = ???
}