11.1.1.1 direct messaging from producers to consumers
- UDP multicast -- low latency, although UDP itself is unreliable, app-level can recover lost packets, widely used in financial industry
- brokerless messaging lib -- ZeroMQ, nanomsg ==> TCP or IP multicast
- StatsD, Brubeck -- UDP, collect metrics for monitoring
- if the consumer exposes a service, producers can make a direct HTTP or RPC request to push messages to the consumer
webhook pattern -- a callback URL of one service is registered with another service, and it makes a request to that URL whenever an event occurs
direct messaging problems
weak fault-tolerance -- possibility of message loss
- if a consumer is offline, producer can resend failed msgs
- but if the producer crashes, the buffer of unsent msgs would be lost
11.1.1.2 message brokers
msg broker (msg queue) -- a kind of DB optimized for handling msg streams
- tolerate clients' connect, disconnect and crash
- take in charge of the problem of durability by keeping msgs in memory / disk
- generally allow unbounded queueing for slow consumers
- with queueing, the consumers are generally async
11.1.1.3 msg brokers vs. DB
Traditional view of MQ
- standards : JMS, AMQP
- software: RabbitMQ, ActiveMQ, HornetMQ, Qpid, TIBCO EMS, IBM MQ, Azure Service Bus, Google Cloud Pub/Sub
| msg broker |
DB |
automatically delete a msg that has been delivered successfully to its consumer
==> not suitable for long-term data storage |
keep data until explicit delete |
- assume the queues are short
- if the Q is long, the overall throughput may degrade |
|
| support some way of subscribing to a subset of topics matching some pattern |
support secondary indexes & various ways of searching for data |
| notify clients when data changes |
the query result is based on a point-in-time snapshot of the data |
11.1.1.4 multiple consumers
load balancing
- 1 msg is delivered to 1 of the consumers (assigned arbitrarily)
- the consumers can share the work of processing the messages in the topic
- useful if the msgs are expensive to process
fan-out
- 1 msg is delivered to all consumers
- allows several independent consumers to process the same msgs
Figure 11-1. (a) Load balancing: sharing the work of consuming a topic among consumers; (b) fan-out: delivering each message to multiple consumers.
11.1.1.5 acknowledgements and redelivery
- to avoid message loss, MQs use acknowledgements
- if the connection is closed or times out, MQ assume that the msg was not processed
==> it delivers the msg again to another consumer
- when combined with load balancing, the redelivery inevitably leads to msg being reordered
Figure 11-2. Consumer 2 crashes while processing m3, so it is redelivered to consumer 1 at a later time.
Whether msg reordering is a problem depends on your context.
To avoid this issue, you can use a separate queue per consumer.
Reference
Designing Data-Intensive Applications by Martin Kleppman
