Chunked messages in reliable delivery, #24276 (#28915)

* To avoid head of line blocking from serialization and transfer of large messages
  this can be enabled.
* ProducerController setting to chunk messages
* split up large messages in chunks in ProducerController
  and assemble again in ConsumerController
* serialization moved to these actors instead of in the Artery stream
* other messages (for other actors) can interleave with the chunks
* serializer for ChunkedMessage in SequencedMessage and MessageSent
* cleanup partially stored chunked messages
* reference docs
* mima filters
* additional test for sending the Request after half window size
* enforce that chunk-large-messages=off for sharding and work-pulling

akka-docs/src/main/paradox/remoting-artery.md

@@ -664,7 +664,7 @@ See `inbound-lanes` and `outbound-lanes` in the @ref:[reference configuration](g
 
 All the communication between user defined remote actors are isolated from the channel of Akka internal messages so
 a large user message cannot block an urgent system message. While this provides good isolation for Akka services, all
-user communications by default happen through a shared network connection (an Aeron stream). When some actors
+user communications by default happen through a shared network connection. When some actors
 send large messages this can cause other messages to suffer higher latency as they need to wait until the full
 message has been transported on the shared channel (and hence, shared bottleneck). In these cases it is usually
 helpful to separate actors that have different QoS requirements: large messages vs. low latency.
@@ -695,6 +695,11 @@ This means that all messages sent to the following actors will pass through the
 
 Messages destined for actors not matching any of these patterns are sent using the default channel as before.
 
+The large messages channel can still not be used for extremely large messages, a few MB per message at most.
+An alternative is to use the @ref:[Reliable delivery](typed/reliable-delivery.md) that has support for 
+automatically @ref[splitting up large messages](typed/reliable-delivery.md#chunk-large-messages) and assemble
+them again on the receiving side.
+
 ### External, shared Aeron media driver
 
 The Aeron transport is running in a so called [media driver](https://github.com/real-logic/Aeron/wiki/Media-Driver-Operation).

akka-docs/src/main/paradox/typed/reliable-delivery.md

@@ -50,6 +50,10 @@ There are 3 supported patterns, which are described in the following sections:
 * @ref:[Work pulling](#work-pulling)
 * @ref:[Sharding](#sharding)
 
+The Point-to-Point pattern has support for automatically @ref:[splitting up large messages](#chunk-large-messages)
+and assemble them again on the consumer side. This feature is useful for avoiding head of line blocking from
+serialization and transfer of large messages.
+
 ## Point-to-point
 
 This pattern implements point-to-point reliable delivery between a single producer actor sending messages and a single consumer actor
@@ -410,6 +414,25 @@ This can be more efficient since messages don't have to be kept in memory in the
 they have been confirmed, but the drawback is that lost messages will not be delivered. See configuration
 `only-flow-control` of the `ConsumerController`.
 
+## Chunk large messages
+
+To avoid head of line blocking from serialization and transfer of large messages the @ref:[Point-to-Point](#point-to-point) 
+pattern has support for automatically @ref:[splitting up large messages](#chunk-large-messages) and assemble them
+again on the consumer side.
+
+Serialization and deserialization is performed by the `ProducerController` and `ConsumerController` respectively
+instead of in the remote transport layer.
+
+This is enabled by configuration `akka.reliable-delivery.producer-controller.chunk-large-messages` and defines
+the maximum size in bytes of the chunked pieces. Messages smaller than the configured size are not chunked, but
+serialization still takes place in the `ProducerController` and `ConsumerController`. 
+
+Aside from the configuration the API is the same as the @ref:[Point-to-point](#point-to-point) pattern. If
+@ref:[Durable producer](#durable-producer) is enabled the chunked pieces are stored rather than the full large
+message.
+
+This feature is not implemented for @ref:[Work pulling](#work-pulling) and @ref:[Sharding](#sharding) yet.
+
 ## Configuration
 
 There are several configuration properties, please refer to `akka.reliable-delivery` config section in the

akka-docs/src/test/scala/docs/persistence/PersistenceSchemaEvolutionDocSpec.scala

@@ -5,8 +5,6 @@
 package docs.persistence
 
 import java.io.NotSerializableException
-
-import scala.language.reflectiveCalls
 import java.nio.charset.Charset
 
 import akka.actor.ActorSystem
@@ -16,8 +14,8 @@ import akka.testkit.TestKit
 import com.typesafe.config._
 import org.scalatest.wordspec.AnyWordSpec
 import spray.json.JsObject
-
 import scala.concurrent.duration._
+
 import docs.persistence.proto.FlightAppModels
 
 class PersistenceSchemaEvolutionDocSpec extends AnyWordSpec {
@@ -284,7 +282,7 @@ class RemovedEventsAwareSerializer extends SerializerWithStringManifest {
       case m if SkipEventManifestsEvents.contains(m) =>
         EventDeserializationSkipped
 
-      case other => new String(bytes, utf8)
+      case _ => new String(bytes, utf8)
     }
 }
 //#string-serializer-skip-deleved-event-by-manifest