Architecting Queues with RabbitMQ and Kinesis

Approximately remain in this minutes read.

Architecting Queues with RabbitMQ and Kinesis

Written by

If you read our article comparing RabbitMQ and Amazon’s SQS, you already know the basics of what queues are about. In short, they are a perfect tool to use when you want to make sure various services in your application can communicate and exchange messages and tasks. But the world of queues is a very complicated one, as we explore more below…

How Do Queues Differ?

AWS, for example, offers three services that can be thought of as queues. One of them, Simple Queue Service (SQS), which we previously discussed, deals with the most popular use-case of applications publishing to the queues and others reading from the queues. Another, Simple Notification Service, provides the benefits of other topologies like fanout or Pub/Sub. It is mainly a notification framework as the name suggests, but one that is often used together with SQS, so it’s included here. The third one, which this article deals with, is Kinesis, an event-streaming system.

Do You Process Messages or Data?

When deciding which queuing solution you need, the most important question to ask yourself is: Am I processing messages or data? Messages, in our case, mean small pieces of information, usually imperatives for the workers to follow. Data, on the other hand, means larger chunks that need to get processed.

RabbitMQ is a universal general purpose message broker. It can be used in any way the developer sees fit. But if there are use cases where specialized systems are required, then the amount of overhead needed to mold RabbitMQ into desired shape may be too much.

On the other hand, Kinesis is designed with a clear objective in mind and optimized to do it in the best way possible. It is nowhere near as flexible as RabbitMQ, but it doesn’t have to be, as its aim is to allow data to be streamed, stored, and possibly replayed.

Keep in mind that Rabbit’s universality is advantageous in more than one way. When your system calls for more than one particular use case, Rabbit can handle them all without the need to manage several different services. The only limiting factor may be the size of the cluster and processing capabilities.

But when should you make use of the different solutions? In the following section, we provide two examples to help you decide between the use cases.

Read here about how to build a private cloud.

Queuing Messages

Imagine that your application contains several services. Some of them react to real-time events, while others run background jobs. Those background jobs can be anything like generating reports, creating PDFs, or drawing charts. They take some time to complete, usually a longer time than an HTTP request allows, and their results are not time-critical. For example, a user who wants to generate a report is fine with waiting two minutes until the report is generated. He or she will be notified asynchronously when it is ready so they can view or download it.

For these background tasks, either most of the data is already contained on some storage (like block devices, network filesystems, or databases), or they do not require specific input at all. The main point is that they do not necessarily work on data the user provided at the moment of scheduling a task.

Such asynchronous and possibly long-running tasks with variable response times are a perfect fit for traditional job queues. The front-end part of the application acts as a producer of messages reacting to a user’s behavior. Once a user clicks “generate report,” a message, “generate a report,” is put into the queue. This message can carry additional data, like user name, current date and time, or which topic does the report cover. On top of that, the scheduling application can select a queue where the message will be put. This way you can have priority queues, location-based queues, or some other arbitrary distinction to help with processing. If you feel that this reflects your actual use case, than RabbitMQ is likely the right fit for you.

Streaming Data

Now imagine a different situation: You run an advertising server handling thousands of requests each second. For each request, you want to ensure that the proper data can be gathered and processed. You want to know whether the user displayed your ad for the first time or whether it was a recurring visit. You want to know which part the user clicked on, at what time of the day, and into which page this ad was embedded. Though the amount of the data to be processed is not necessarily big, the sheer amount and the potential spikes in traffic can give your machines a hard time.

You don’t want to lose any data, but also you don’t want to drop any incoming requests, so you have to process each request as soon as possible. You are okay with the more costly processing occurring asynchronously, and you may event want to run the same incoming data through different sets of data processing engines. All or at least most of the data you need to deal with is actually contained in the incoming event stream.

If this description is accurate, then you may be interested in event streaming. The scenario could look like this: Each user request is looked up in the database (probably an in-memory cache) and responded to instantly. Meanwhile, all of the associated data is wrapped as an event and put into an event stream. Data processing engines are registered for such events and handle them as they come. The architecture is therefore event-driven, as all the processing happens in response to the generated events. Event streams also store logs of the messages as they arrive, so you can later playback the traffic routing it through different algorithms.

AWS Kinesis is an example of one such event streaming service. It can be utilized even if you don’t want to store your whole service in AWS cloud, though of course having most of the building blocks in the same place can reduce latency and bandwidth costs.

See what the hype is about! Watch our demo and take a test drive.

Managed Versus On-Premise: How Rabbit Compares to AWS Kinesis

Taking into consideration AWS’s approach as a building blocks provider, one question every architect needs to answer at the beginning is: Which software will I use as a service and which will I host on my own? Kinesis, as a proprietary AWS solution, is only available for the former.

If you would like to use RabbitMQ options, then your next choice is whether to manage your own cluster or use services from SaaS providers, like CloudAMQP. Here, the choice may depend on the latency you want to maintain, the scale you’d like to achieve, and some other factors as well, such as the cost of each.

Possible Alternatives

Even though RabbitMQ is the most popular open source message broker, its universality also means it is not best suited for every possible workload available. For the specific job of event streaming, a better alternative might be Kafka from the Apache Software Foundation.

Engineered at LinkedIn, Kafka was built from the ground up with horizontal scaling in mind and features like persistent log and replay capability. If you require either of those features and don’t feel convinced with AWS’ offer, you can consider trying Kafka over RabbitMQ.


RabbitMQ is a great starting point in a world of queues. It’s open source, universal, and offers great documentation. It can be used in a variety of scenarios, though it may not necessarily be the best choice for each one.

AWS Kinesis is a Software as a Service (SaaS) event-streaming service. While it shares some of the same features with RabbitMQ, it is by no means its competition. Kinesis’ use case is quite specialized and as such it does a much better job in its specific areas.

Your choice to use one of the two over the other should not be based on price or the features available. Rather, it should come from the high-level architecture of your service: Do you need to pass messages around or do you want to process high-ingress streams of data? RabbitMQ is the answer to the former, while Kinesis is the answer to the latter.

October 23, 2017

Simple Share Buttons
Simple Share Buttons