Overview

Finally! You get to build a new system from scratch! Now is your chance to amaze your peers. Show them how much you know about: Distributed, Resilient, Functional, Scalable architecture; did I forget a buzzword? With the proper use-case, timeline and budget any system can be developed with n+1 adjectives.

Since we exist in the real world, we’re going to take a “real-world” look at a major component commonly found in these fabled architectures, the Message Queue (MQ). When implemented properly, MQs are an amazing tool that can, in fact, add several adjectives to your architecture. However, adding buzzwords to your tech-stack by leveraging an MQ does not mean it was the right choice. Let’s examine several ways adding an MQ into your architecture can make your new system a pain to implement and maintain.

Hosting Expenses

Before discussing the looming technical complications, first, consider the runtime and development cost of implementing an MQ or Topic – think RabbitMQ or Kafka. For various reasons, it’s unlikely that you will be using the same server to host your application as well as your MQ. That means you have to either host it on a local server or pay for remote hosting.

A few of the more popular MQ/Topic as a Service providers are Amazon MQ and Confluent for Kafka. Using one of these providers significantly reduces the required configuration, but, at what cost?

  • Amazon MQ
    Running a single Broker with only 1GB of storage can cost between $21-$290/month using the provided example of the Amazon MQ pricing calculator
  • Confluent Apache Kafka
    For a similar, single Broker with 1GB of storage Confluent Cloud may cost between $37-$751/month for service comparable to Amazon MQ, see the pricing
  • Fine! I’ll host it myself!
    No two local servers have the exact same cost, so, I won’t try to estimate. Keep in mind that there are costs associated with a self-hosting that can range from the operating system, memory, and electricity consumed by the CPU. There’s also a cost associated with the time that is spent on the configuration required to setup and maintain a local MQ/Topic. 

    Thinking that setting up your own Topic is easy as cake? Take a gander at the broker configuration for Kafka.

Dev Costs

When compared to a web services (HTTP) implementation – which is generally understood by developers and is very flexible – leveraging an MQ/Topic properly requires a specific set of conditions in order for it to make sense. Observe the pre-implementation conditions below.

In order to operate at peak efficiency, you should only use an MQ/Topic under specific conditions, otherwise, there is no ROI for the cost and configuration.

Do NOT use a Message Queue / Topic:

  • If, Bi-directional communication is required
    Unlike sync and async HTTP, MQ/Topic communication is optimized for throughput and not bidirectional communication. If you wanted to imitate Request/Response functionality your application would have to manually match events received in one consumer as a response to an event sent by a producer elsewhere in the application and then return a Future, Promise or Observable. This becomes quite cumbersome in a distributed application where a client is waiting for a response.
  • If, Data must be “fresh”
    Because Messages are blasted to the MQ in async fashion and require a valid consumer to receive the message, most MQ/Topic cannot guarantee that the data which is being processed by the consumer is still valid. For example, a queue that contains stock ticker data could be receiving new messages so rapidly that by the time a user sees the data being read from the MQ, there are already new data for the same stock waiting to be processed in the queue.
  • if, All of the Consumers are known
    In the case that the messages or events are always consumed by a known service that is adequately abstracted behind a proxy / load-balancer or even within the same network, there is not much benefit in exposing the generic consumer functionality of an MQ when compared to sending the message/data directly with HTTP.
depiction of which scenario you should or should not use a message queue

You SHOULD use a Message Queue:

  • If, the Messages or Events are strictly formatted
    Many MQ/Topic technologies such as RabbitMQ and Kafka allow messages to be routed to different queues or topics based on the contents of the data. That way all messages for “billing” or “purchasing” end up in their respective queue. Having a strict data format increases the reliability of message/event routing.
  • If, some consumers are anonymous
    In a case where there could be N consumers of the messages/events, it’s more efficient to have simultaneous consumers “listening” on a topic or a fanout strategy of a queue VS sending an explicit HTTP request for each message to all of the potential consumers. As a side note, if the data is being sent to or gathered from a sensor such as a solar weather station, the energy cost of sending data to an MQ is much more efficient as the network connection is kept open vs being toggled.
  • If, consumers are intermittently available or go down frequently
    Believe it or not, servers have down-time. This occurs whether the server is hosting web services or MQ consumers; yet, if this is a common occurrence for your consumers, an MQ could prove to be more suitable than direct HTTP as the data will be pushed and stored in the Queue/topic for a period of time or until it is consumed. This is particularly helpful when working with producers or consumers with spotty network availability.
  • If, Messages are “fire and forget”
    MQ/Topics are optimized for throughput or “number of messages per second”. Part of this optimization is that publishing data is generally uni-directional. Since no response is required – other than the optional ACK – writing to a queue or topic has insane performance. For the same reason, consuming from an MQ is highly efficient when the reading strategy is tuned appropriately.