Microservices Architecture, The Hard Parts : Data Query using CQRS Pattern
Data Query involving multiple microservices to serve a single business use case is tough to design and implement. There are ways to handle this efficiently — one of the easy to implement approach being API Composition Pattern. You can read more about this pattern in my previous blog here.
But as you work with API Composition patterns, you would realize that there are some limitations with this pattern. It can handle only some of the types of use cases to “efficiently” query data across multiple microservices and you need another pattern to handle such use cases. This alternate pattern is called as the Command Query Responsibility Segregation (CQRS) Pattern.
Let’s look at a use case where we need such a pattern —
Assume as a business requirement, a user can view the list of published content on a social site and while rendering the content, the system also needs to display the Content Author’s first name and last name and count of reactions and comments made on the content page. Additionally, users can fetch content using author name or published during a specific date range.
The team has realized the domain problem using Microservices Architecture as below —
- Content Service — To manage content
- Comments Service — To manage comments on a content
- Reactions Service — To manage reactions on a content
- User Service — To manage users
Aren’t the given services too fine grained and some of them can be clubbed together — Well, as the saying goes, “It Depends” on the overall domain problem we are trying to address. And the purpose of this blog is to show, given a situation like this, how we can handle querying of data across multiple microservices.
Let’s see why API Composition cannot handle this use case “efficiently” —
As per Microservice data abstraction principle, Content microservice handles the Content domain and hence stores user’s unique identifier as the Author ID of the content in its own Database and User service manages the users data including the first name and the last name.
Scenario 1 — If we need to fetch all the Content without any filter condition applied, then API Composition component needs to first make a call to the Content service to get the list of contents -> and for each content — call User service to get the first name and last name to populate the Author’s name, call Comments service and Reactions service to fetch the count -> and finally aggregate the data and respond to the client.
Scenario 2 — If a user happens to apply a filter for getting contents published by a given user, then the API Composition component has to 1st make a call to the User service to get the User ID of the user based on the user name -> then call Content service to get all the contents published by the given user ID -> and for each content — make API calls to Reactions and Comments services to get the counts -> and finally aggregate the data and respond to the client.
In both the cases, for a single API call from the consumer client, API Composition component is doing the heavy work of making expensive multiple network calls to multiple services, performing in-memory data aggregation which leads to lots of compute and memory resource consumption. Overall, low performance and high latency in responding to the consumer client. So, technically, we could achieve what we wanted by using API Composition pattern but not in an “efficient” manner.
We can off-course optimize this implementation by making batch calls may be — get first name and last name of the many users in a single API by passing multiple user IDs. And the same can be done with reactions and comments service. But this is with the assumption that these domain services provide data retrieval using Bulk APIs, else the services need to be enhanced.
Let’s see how CQRS pattern can help to handle the same use case in an efficient way —
As the name pattern name says, it is about segregation of concerns — a component responsible for the “Command” aspect of a data model which includes create, update and delete operations and another component responsible for the “Query” aspect of a data model which includes query operations.
Let’s take a simple example to understand this better —
Let’s assume, a Content domain service leverages RDBMS database to store the content data. But with new sets of requirements pouring in related to advanced search use cases, it may not be possible to implement the queries “efficiently” using RDBMS and hence Content Team thinks of leveraging specialized datastore like Elastic Search to perform these advanced search queries.
With this thought process, the content creation/update/delete operations will be managed by a Command component using the RDBMS data store and search (query) operations will be managed by the Query component using Elastic Search engine. This leads to separation of concerns. And there will be enough mechanisms in place to ensure that data from the RDBMS data store flows to Elastic Search engine and is kept in sync, and to not have any stale data. One of the ways by which this can be implemented is using Business Domain Events — where the Command component publishes a business domain event when data is added/modified/delete in the RDBMS database and the Query component listens to these events and updates its datastore to keep data in sync.
BTW it is not necessary to have different kinds of datastore to have a CQRS pattern in place. For instance — A read Only database replica with lots of additional indexes applied compared to the writer database instance is also a kind of CQRS. But in general, you would see different data stores as part of CQRS implementation.
Now, lets see how CQRS can help to implement the use cases we were discussing above in an “efficient” manner —
For our use case, we can leverage the CQRS pattern where domain services publish events and a Query service listens to these events and stores the data in its own datastore as per the business need.
With the given approach, events from the required domain services flow into Content Query service. And this service can keep only the required attributes from the events and store them in its own datastore and discard rest of the data in the event. For instance — from User event, it can keep only the userID, first name and last name and discard rest of the attributes. Similarly, from comments and Reactions service, it can continue to update the comments and reactions aggregate count against a given content and discard rest of the data. But if a business needs additional data to be kept to serve other use cases, then the Query component can store the attributes as needed.
I am sure by now, you would have realized the advantage of the CQRS pattern. Because all the required data is directly available within the boundary of the View only datastore owned by the Query component, it can perform queries “efficiently” using the Datastore engine to get the right data to respond to the consumer client, with performance and low latency. No expensive network and in-memory calculations involved. Plus, the query service can be scaled independently as needed and is not coupled with the rest of the domain services.
And not to ignore the cons of this pattern — the complexity involved in designing and implementing CQRS pattern and ensuring the data is always kept in sync between the Command and the Query data stores.
Now that we have discussed both the patterns for querying data scattered across services, based on the Pros and Cons of each approach, you can make a decision on which approach to lean to :)
Hope you enjoyed reading this blog. Do share this blog on social media if this has helped you in any way.
Happy Blogging…Cheers!!!
#MicroservicesQuery #MicroservicesCQRS #MicroservicesAPICompsition #APICompositionPattern
