CTO and Principal Analyst, Intellyx
Jan 5, 2024 | 3 mins read
Transactional consistency keeps the state of the business in line with the real world by ensuring data updates either happen completely or not at all.
For example, transactions ensure physical inventory in the warehouse is consistent with the inventory count in the database so that you are sure you have the item you want to sell.
Handling events such as these typically means handling a series of linked state changes to the business, each of which have to be reliably captured and processed through a carefully orchestrated flow.
Payment Example
Let’s revisit the international payment processing example from the previous blog post and add transactions to each step to see how it ensures consistency.
First of all, transactions guarantee that messages either are sent successfully from one microservice to the next, or restore event queues to their previous state (i.e. the event is still in the queue) so the operation can be restarted by resending the event.
If the microservice, on the other hand, reads the event without starting a transaction, the event might be deleted from the queue. The customer would then have to submit the request again.
Ovals illustrate transactions that join data operations on the queues to data operations in the microservices. Transactions ensure events successfully move from one queue to another but also ensure success in processing the events.
Before reading from or writing to a queue, the microservice starts a transaction. Database operations the microservice performs are included within the scope of the same transaction. If a database operation fails for any reason, the entire transaction is rolled back and is restarted from
Tracing the Payment Flow
Let’s revisit the payment processing flow and see what happens when we add transactions to it.
When an organization submits a payment request, that request is stored transactionally in a queue. If an error occurs, the transaction rolls back and the microservice notifies the payment submitter that an error occurred, and they have to resubmit the request.
After the payment request is captured reliably, however, this initial event triggers the payment processing flow, which can now be completed with transactional consistency.
Transactions at each stage in the processing flow ensure that each step is executed correctly or retried in the case of any failure.
When the validation microservice reads in the payment request data, for example, it starts a transaction to reliably store the results. If any error occurs, the step is rolled back and restarted from the original event.
Similarly, the validation service writes its output to a database and a queue within a transaction. If any error occurs, the transaction rolls back the data to the original state. Otherwise, the microservice stores the updates persistent and processing moves to the next step.
Transactions guarantee consistency for globally distributed applications by either persisting the results or rolling data back to the original state, regardless of where in the network the application is running.
SAGAs or Compensation
What about when there’s an error invalidating the entire payment processing flow, and the entire flow has to be undone, not just a single step?
Define a compensation action for each microservice in the flow so that any previously committed update can be reversed.
For example, if an error occurs during the confirmation step that cannot be resolved (or resolved within a reasonable time period), the entire flow may need to be canceled and restarted.
This is where the SAGA pattern comes in. A SAGA is a long running process that undoes previously committed transactions in a flow, step by step. You can think of it as reversing the flow of transactions.
When an error triggers a SAGA, a reverse flow executes a compensating transaction associated with each step in the process.
Compensating transactions are familiar to us all through the process of reversing incorrect credit card charges. We’ve all seen or experienced this - if a store makes a mistake on a charge, the only way to correct it is to undo the previous transaction and submit a new one.
A SAGA traverses the set of event queues in reverse order to trigger compensation actions associated with each step in the payment processing flow.
In this example, the results of the fraud check, credit check, and validation step are reset to their prior values (basically zero or null, to indicate the payment had not been processed or needs to be reprocessed).
Most banks also record or log the results of the prior execution for auditing and compliance purposes.
Relationship to Systems of Record
For traditional, monolithic systems a shared database often becomes the system of record for one or more applications.
For distributed processing, in which each microservice handles its own data, there may be multiple systems of record, or a requirement to forward data from individual microservice databases to a common system of record for reporting and analytics processing.
An alternative uses the data stored in the queues of the event processing system as the system of record.
Such a system configures the queues to log or record all events persistently. The complete record of events contains the state of the business at any given point in time.
Solutions
Most event-driven messaging systems, such as Fiorano, offer transactions as an option for handling events and database updates consistently.
They also offer frameworks for constructing and executing compensation actions that follow the SAGA pattern. (SAGAs cannot be automated because compensation actions are application-dependent).
The Intellyx Take
Globally distributed systems are by their very definition susceptible to network errors and synchronization issues that monolithic systems are not.
Event-driven microservices are a good way to address such issues.
Adding transactions spanning the event queues and the microservices provide the consistency guarantees many applications need, such as international payment processing or retail order processing.
Fiorano’s integration of transactions with queued events provides a simple, straightforward way to get the transactional guarantees you need.
Copyright © Intellyx LLC. Intellyx is solely responsible for the content of this eBook. As of the time of writing, Fiorano is an Intellyx customer. No AI chatbots were used to write this content. Image by Kalan, via Wikimedia Commons.
