Message queues are infrastructure components that enable one software service to send a message to another service without requiring the sender and receiver to be available and communicating simultaneously. The sender publishes a message to the queue, where it is stored durably until the receiving service retrieves and processes it. This decoupling of sender and receiver is the foundational mechanism that enables asynchronous communication in payment platforms built on event-driven or microservices architectures.
In a payment system context, message queues serve as the reliable transport layer for payment events, instructions, compliance alerts, settlement notifications, and ledger posting triggers. They ensure that these messages are not lost if a receiving service is temporarily unavailable, experiencing high load, or being updated, and that every message is eventually delivered and processed exactly as intended, regardless of the timing of service availability.
As illustrated in a typical message queue interaction within a payment platform, the payment initiation service publishes a PaymentInitiated message to a designated queue or topic. The message is stored in the queue infrastructure immediately upon publication, with the durability guaranteed by the message broker’s storage mechanism.
The compliance screening service retrieves the message from the queue when it is ready to process it, performs the AML check, and publishes its result as a new message to a separate queue. If the compliance service is temporarily unavailable, the PaymentInitiated message remains in the queue until the service recovers, at which point it processes the message from where it left off. No message is lost, and no manual intervention is required to resume processing after the recovery.
Key Takeaways: #
- Message queues are the infrastructure layer that enables reliable, asynchronous communication between the services of a payment platform, storing messages durably until the receiving service is ready to process them and ensuring that no message is lost even if a service is temporarily unavailable;
- Apache Kafka and RabbitMQ are the two most widely adopted message queue and message streaming platforms in payment systems, each suited to different use cases. Kafka is optimised for high-throughput event streaming and long-term event log retention, making it the preferred choice for payment platforms with large transaction volumes and audit trail requirements. RabbitMQ is optimised for flexible message routing and task queue patterns, making it well suited to workflow orchestration and notification delivery use cases;
- Under DORA, the message queue infrastructure is a critical ICT component for payment institutions and e-money institutions operating event-driven payment platforms. Its availability, durability, and recovery characteristics must be assessed and managed within the institution’s ICT risk management framework, including documented recovery time and recovery point objectives.
Apache Kafka and RabbitMQ: The Two Primary Platforms #
The two most widely adopted message queue and message streaming platforms in payment systems are Apache Kafka and RabbitMQ. While both serve the fundamental purpose of enabling asynchronous message delivery between services, they are architecturally distinct and optimised for different use cases within payment platforms.
Apache Kafka: Apache Kafka is a distributed event streaming platform originally developed at LinkedIn and now maintained as an open-source Apache Software Foundation project. Kafka’s architecture is based on a persistent, append-only log structure in which messages, referred to as records, are written to topics and retained for a configurable period regardless of whether they have been consumed. Consumers read from a specific position in the topic log, known as an offset, and can replay messages from any point in the log, including from the beginning.
This log-based architecture makes Kafka particularly well suited to payment platforms for several reasons. The persistent event log provides a complete, ordered history of all payment events that can be replayed for audit, regulatory reporting, or system recovery purposes. The high-throughput architecture, capable of processing millions of messages per second across a distributed cluster, supports the transaction volumes of large-scale payment platforms.
Consumer groups allow multiple independent services to consume the same event stream simultaneously, with each consumer group maintaining its own offset position independently. Kafka’s durability guarantees, achieved through replication of each topic partition across multiple broker nodes, ensure that messages are not lost even in the event of individual node failures.
For payment institutions and e-money institutions, Kafka’s event log retention capability is particularly relevant to regulatory compliance. The complete, ordered event log provides the tamper-resistant audit trail required under AML frameworks and the ICT incident reconstruction capability required under DORA. The log can also be used to rebuild the state of the payment platform from scratch in a disaster recovery scenario, replaying all historical events to restore the system to its pre-incident state.
RabbitMQ: RabbitMQ is an open-source message broker based on the Advanced Message Queuing Protocol (AMQP), originally developed by Rabbit Technologies and now maintained under VMware. RabbitMQ’s architecture is based on exchanges and queues, in which producers publish messages to exchanges that route them to one or more queues based on configurable routing rules. Consumers retrieve messages from queues and acknowledge receipt upon successful processing, with the broker retaining the message and redelivering it if an acknowledgement is not received within a defined timeout.
RabbitMQ is optimised for flexible message routing, task queue patterns, and workflow orchestration. Its exchange and routing key model allows messages to be directed to different queues based on their content, enabling sophisticated routing logic that is well suited to payment workflow orchestration, where different payment types, currencies, or risk levels may need to be routed to different processing queues. RabbitMQ’s acknowledgement model ensures that each message is processed exactly once by the consuming service, making it well suited to use cases where duplicate processing would cause data integrity issues.
In payment platforms, RabbitMQ is commonly used for task queues that distribute processing work across multiple worker instances, notification delivery systems that route alerts to different channels based on message type or recipient preferences, and workflow orchestration systems that coordinate the sequential steps of a multi-stage payment processing workflow.
Key Functions of Message Queues in Payment Systems #
Reliable message delivery and durability: Message queues ensure that messages published by a producer are stored durably until they are successfully processed by a consumer, even if the consumer is temporarily unavailable. This durability guarantee is fundamental to payment system reliability, as it ensures that payment events, settlement notifications, and compliance alerts are not lost due to service failures or network interruptions. In a payment platform processing real-time transactions on rails such as SEPA Instant or Faster Payments, message durability is a prerequisite for meeting the processing reliability expectations of those payment schemes.
Backpressure and load management: Message queues absorb demand spikes by buffering messages when consumer services are processing at capacity. Rather than allowing a surge in payment volume to overwhelm downstream services or cause request rejections, the message queue accumulates the excess messages and delivers them to consumers as capacity becomes available. This backpressure mechanism enables payment platforms to handle transaction volume spikes gracefully, maintaining processing integrity during peak demand periods without requiring the system to be provisioned for maximum conceivable load at all times.
Decoupling and independent service operation: By routing messages through a queue rather than making direct service-to-service calls, message queues decouple the services of the payment platform from one another. A producer service publishes a message and continues processing without waiting for the consumer to acknowledge receipt or complete its processing. This decoupling allows each service to operate at its own pace, scale independently, and be updated or redeployed without coordinating with the services it communicates with. For payment institutions managing regulatory updates that require changes to specific services, this decoupling significantly reduces the operational risk of each deployment.
Exactly-once and at-least-once delivery guarantees: Message queue platforms provide configurable delivery guarantee semantics that determine how the broker and consumer handle message delivery and acknowledgement. At-least-once delivery ensures that every message is delivered to the consumer at least once, with redelivery occurring if the consumer does not acknowledge receipt within the timeout period. This guarantee is appropriate for idempotent operations where processing the same message twice produces the same result. Exactly-once delivery, supported by Kafka through its transactional API and by RabbitMQ through its acknowledgement and deduplication mechanisms, ensures that each message is processed precisely once, which is the required guarantee for ledger posting operations where duplicate processing would create erroneous financial entries.
Event ordering and sequencing: Payment systems often require that events are processed in the order in which they occurred, for example ensuring that a PaymentSettled event is processed after the corresponding PaymentSubmitted event, and that a RefundInitiated event is processed in the correct sequence relative to the original payment. Kafka maintains strict ordering within each topic partition, ensuring that events published to the same partition are consumed in the order they were published. This ordering guarantee is important for payment ledger posting, where out-of-order processing of events could produce incorrect account balances or settlement positions.
Message Queues and DORA Compliance #
For payment institutions and e-money institutions subject to DORA, the message queue infrastructure is a critical ICT component whose availability, durability, and recovery characteristics must be assessed and managed within the institution’s ICT risk management framework. DORA requires institutions to define recovery time and recovery point objectives for their critical ICT systems, and the message queue infrastructure directly affects both metrics. If the message queue becomes unavailable, payment processing across the entire event-driven platform is impaired. If messages stored in the queue are lost due to an infrastructure failure, the institution may be unable to reconstruct the state of payments in flight at the time of the incident.
Institutions should configure their message queue infrastructure with replication across multiple availability zones to ensure resilience against individual node or zone failures, define and test recovery procedures for message queue infrastructure failures as part of their DORA operational resilience testing, and document the message retention periods and replay capabilities of their chosen platform as part of their ICT risk management and business continuity documentation.
FAQ: #
What is the difference between Apache Kafka and RabbitMQ, and how should a payment institution choose between them?
- Kafka and RabbitMQ are optimised for different use cases. Kafka is a distributed event streaming platform built around a persistent, replayable log structure, offering very high throughput, long-term event retention, and the ability for multiple consumer services to read the same event stream independently. It is best suited to high-volume payment event streaming, audit log retention, and scenarios where the ability to replay historical events is important.
- RabbitMQ is a message broker built around flexible routing and task queue patterns, offering sophisticated message routing logic, per-message acknowledgement, and well-established workflow orchestration capabilities. It is best suited to payment workflow orchestration, notification routing, and task distribution use cases. Many payment platforms use both technologies together, with Kafka handling the high-volume event streaming layer and RabbitMQ handling specific workflow and routing use cases where its exchange and routing key model provides advantages over Kafka’s topic-based architecture.
What happens to messages in a queue if the message broker becomes unavailable?
- The impact of message broker unavailability depends on the durability configuration and replication architecture of the message queue infrastructure. In a production payment platform, the message broker should be deployed with replication across multiple nodes and availability zones, meaning that the failure of any individual node does not result in message loss or broker unavailability, as other nodes in the cluster continue to serve requests. If the entire broker cluster becomes unavailable, messages published after the outage begins will not be delivered until the broker recovers.
- Messages published before the outage that were already stored in the broker’s durable storage are retained and delivered to consumers when the broker recovers, provided the retention period has not expired. For payment institutions subject to DORA’s recovery time objective requirements, the recovery time of the message broker infrastructure must be assessed and documented as part of the institution’s ICT risk management framework.
