+htp #18837 completely rewrite EntityStreamingSupport
added CSV examples updated docs EntityStreamingSupport is now an entry point, to all streaming things both read and write side use it it's easy to extend as well
This commit is contained in:
Konrad Malawski
parent
6562ddd2df
commit
9cc32c3aba
28 changed files with 862 additions and 847 deletions
|
|
@ -4,7 +4,7 @@ Source Streaming
|
|||
================
|
||||
|
||||
Akka HTTP supports completing a request with an Akka ``Source[T, _]``, which makes it possible to easily build
|
||||
streaming end-to-end APIs which apply back-pressure throughout the entire stack.
|
||||
and consume streaming end-to-end APIs which apply back-pressure throughout the entire stack.
|
||||
|
||||
It is possible to complete requests with raw ``Source[ByteString, _]``, however often it is more convenient to
|
||||
stream on an element-by-element basis, and allow Akka HTTP to handle the rendering internally - for example as a JSON array,
|
||||
|
|
@ -13,9 +13,6 @@ or CSV stream (where each element is separated by a new-line).
|
|||
In the following sections we investigate how to make use of the JSON Streaming infrastructure,
|
||||
however the general hints apply to any kind of element-by-element streaming you could imagine.
|
||||
|
||||
It is possible to implement your own framing for any content type you might need, including bianary formats
|
||||
by implementing :class:`FramingWithContentType`.
|
||||
|
||||
JSON Streaming
|
||||
==============
|
||||
|
||||
|
|
@ -24,7 +21,7 @@ objects as a continuous HTTP request or response. The elements are most often se
|
|||
however do not have to be. Concatenating elements side-by-side or emitting "very long" JSON array is also another
|
||||
use case.
|
||||
|
||||
In the below examples, we'll be refering to the ``User`` and ``Measurement`` case classes as our model, which are defined as:
|
||||
In the below examples, we'll be refering to the ``Tweet`` and ``Measurement`` case classes as our model, which are defined as:
|
||||
|
||||
.. includecode2:: ../../code/docs/http/scaladsl/server/directives/JsonStreamingExamplesSpec.scala
|
||||
:snippet: models
|
||||
|
|
@ -47,32 +44,53 @@ Akka Streams ``Source[T,_]``. One such trait, containing the needed marshallers
|
|||
spray-json (a high performance json parser library), and is shipped as part of Akka HTTP in the
|
||||
``akka-http-spray-json-experimental`` module.
|
||||
|
||||
Next we import our model's marshallers, generated by spray-json.
|
||||
Once the general infrastructure is prepared we import our model's marshallers, generated by spray-json (Step 1),
|
||||
and enable JSON Streaming by making an implicit ``EntityStreamingSupport`` instance available (Step 2).
|
||||
Akka HTTP pre-packages JSON and CSV entity streaming support, however it is simple to add your own, in case you'd
|
||||
like to stream a different content type (for example plists or protobuf).
|
||||
|
||||
The last bit of setup, before we can render a streaming json response
|
||||
The final step is simply completing a request using a Source of tweets, as simple as that:
|
||||
|
||||
.. includecode2:: ../../code/docs/http/scaladsl/server/directives/JsonStreamingExamplesSpec.scala
|
||||
:snippet: spray-json-response-streaming
|
||||
|
||||
The reason the ``EntityStreamingSupport`` has to be enabled explicitly is that one might want to configure how the
|
||||
stream should be rendered. We'll dicuss this in depth in the next section though.
|
||||
|
||||
.. _Streaming API: https://dev.twitter.com/streaming/overview
|
||||
|
||||
Customising response rendering mode
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
The mode in which a response is marshalled and then rendered to the HttpResponse from the provided ``Source[T,_]``
|
||||
is customisable (thanks to conversions originating from ``Directives`` via ``EntityStreamingDirectives``).
|
||||
Since it is not always possible to directly and confidently answer the question of how a stream of ``T`` should look on
|
||||
the wire, the ``EntityStreamingSupport`` traits come into play and allow fine-tuning the streams rendered representation.
|
||||
|
||||
Since Marshalling is a potentially asynchronous operation in Akka HTTP (because transforming ``T`` to ``JsValue`` may
|
||||
potentially take a long time (depending on your definition of "long time"), we allow to run marshalling concurrently
|
||||
(up to ``parallelism`` concurrent marshallings) by using the ``renderAsync(parallelism)`` mode:
|
||||
For example, in case of JSON Streaming, there isn't really one standard about rendering the response. Some APIs prefer
|
||||
to render multiple JSON objects in a line-by-line fashion (Twitter's streaming APIs for example), while others simply return
|
||||
very large arrays, which could be streamed as well.
|
||||
|
||||
Akka defaults to the second one (streaming a JSON Array), as it is correct JSON and clients not expecting
|
||||
a streaming API would still be able to consume it in a naive way if they'd want to.
|
||||
|
||||
The line-by-line aproach however is also pretty popular even though it is not valid JSON. It's relatively simplicity for
|
||||
client-side parsing is a strong point in case to pick this format for your Streaming APIs.
|
||||
Below we demonstrate how to reconfigure the support trait to render the JSON as
|
||||
|
||||
.. includecode2:: ../../code/docs/http/scaladsl/server/directives/JsonStreamingExamplesSpec.scala
|
||||
:snippet: line-by-line-json-response-streaming
|
||||
|
||||
Another interesting feature is parallel marshalling. Since marshalling can potentially take much time,
|
||||
it is possible to marshal multiple elements of the stream in parallel. This is simply a configuration
|
||||
option on ``EntityStreamingSupport`` and is configurable like this:
|
||||
|
||||
.. includecode2:: ../../code/docs/http/scaladsl/server/directives/JsonStreamingExamplesSpec.scala
|
||||
:snippet: async-rendering
|
||||
|
||||
The ``renderAsync`` mode perserves ordering of the Source's elements, which may sometimes be a required property,
|
||||
The above shown mode perserves ordering of the Source's elements, which may sometimes be a required property,
|
||||
for example when streaming a strictly ordered dataset. Sometimes the contept of strict-order does not apply to the
|
||||
data being streamed though, which allows us to explit this property and use ``renderAsyncUnordered(parallelism)``,
|
||||
which will concurrently marshall up to ``parallelism`` elements and emit the first which is marshalled onto
|
||||
the HttpResponse:
|
||||
data being streamed though, which allows us to exploit this property and use an ``unordered`` rendering.
|
||||
|
||||
This also is a configuration option and is used as shown below. Effectively this will allow Akka's marshalling infrastructure
|
||||
to concurrently marshallup to ``parallelism`` elements and emit the first which is marshalled onto the ``HttpResponse``:
|
||||
|
||||
.. includecode2:: ../../code/docs/http/scaladsl/server/directives/JsonStreamingExamplesSpec.scala
|
||||
:snippet: async-unordered-rendering
|
||||
|
|
@ -94,18 +112,25 @@ will be applied automatically thanks to using Akka HTTP/Streams).
|
|||
.. includecode2:: ../../code/docs/http/scaladsl/server/directives/JsonStreamingExamplesSpec.scala
|
||||
:snippet: spray-json-request-streaming
|
||||
|
||||
Implementing custom (Un)Marshaller support for JSON streaming
|
||||
-------------------------------------------------------------
|
||||
Simple CSV streaming example
|
||||
----------------------------
|
||||
|
||||
While not provided by Akka HTTP directly, the infrastructure is extensible and by investigating how ``SprayJsonSupport``
|
||||
is implemented it is certainly possible to provide the same infrastructure for other marshaller implementations (such as
|
||||
Play JSON, or Jackson directly for example). Such support traits will want to extend the ``EntityStreamingSupport`` trait.
|
||||
Akka HTTP provides another ``EntityStreamingSupport`` out of the box, namely ``csv`` (comma-separated values).
|
||||
For completeness, we demonstrate its usage in the below snippet. As you'll notice, switching betweeen streaming
|
||||
modes is fairly simple, one only has to make sure that an implicit ``Marshaller`` of the requested type is available,
|
||||
and that the streaming support operates on the same ``Content-Type`` as the rendered values. Otherwise you'll see
|
||||
an error during runtime that the marshaller did not expose the expected content type and thus we can not render
|
||||
the streaming response).
|
||||
|
||||
The following types that may need to be implemented by a custom framed-streaming support library are:
|
||||
.. includecode2:: ../../code/docs/http/scaladsl/server/directives/JsonStreamingExamplesSpec.scala
|
||||
:snippet: csv-example
|
||||
|
||||
- ``SourceRenderingMode`` which can customise how to render the begining / between-elements and ending of such stream (while writing a response, i.e. by calling ``complete(source)``).
|
||||
Implementations for JSON are available in ``akka.http.scaladsl.server.JsonSourceRenderingMode``.
|
||||
- ``FramingWithContentType`` which is needed to be able to split incoming ``ByteString`` chunks into frames
|
||||
of the higher-level data type format that is understood by the provided unmarshallers.
|
||||
In the case of JSON it means chunking up ByteStrings such that each emitted element corresponds to exactly one JSON object,
|
||||
this framing is implemented in ``EntityStreamingSupport``.
|
||||
Implementing custom EntityStreamingSupport traits
|
||||
-------------------------------------------------
|
||||
|
||||
The ``EntityStreamingSupport`` infrastructure is open for extension and not bound to any single format, content type
|
||||
or marshalling library. The provided JSON support does not rely on Spray JSON directly, but uses ``Marshaller[T, ByteString]``
|
||||
instances, which can be provided using any JSON marshalling library (such as Circe, Jawn or Play JSON).
|
||||
|
||||
When implementing a custom support trait, one should simply extend the ``EntityStreamingSupport`` abstract class,
|
||||
and implement all of it's methods. It's best to use the existing implementations as a guideline.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue