Throughput tuning with Parallelism in Akka Streams

When we using Akka Streams to do stream processing with infinity elements, it’s been processed 1 after 1 by default. With back-pressure provided by Akka Stream, we can notify the producer to slow down to solve the fast producer vs. slow consumer problem. But sometimes we can’t control the production speed, like Kafka messages, we need to find some way to fasten the consuming speed. Let’s say a Kafka topic produce 100 messages per second per partition while consumer will spend 1 second to consume each message.

Rate limit (throttle) in Akka Streams

Recently, I got a chance to contribute to Akka by adding unprovided throttle operator into FlowWithContext, which can carry context in a Stream without care about it, like Kafka offset. It’s a very good chance to learn some implementation details about Akka Streams, and here is what I’ve learned.