Solution Pattern: Event-Driven Sentiment Analysis with Kafka and AI/ML

Architecture

Introduction for the architecture of this solution pattern.

1. Common Challenges

Globex wants to be able to both moderate and analyse product reviews submitted via their mobile and web applications. Building a system to which can respond to customer reviews quickly is very critical to Globex because this would keep them ahead of their competitors.

They are aware that customer reviews is a true reflection of how they are perceived amongst competition, and so getting a clear picture of the sentiment behind the reviews is critical. They would also like to know how this varies over a period of time - even within the same product category. This could help them understand how new products are being received.

It is also important to moderate the reviews for abusive language, unwanted text and also ensure privacy and security of personal data. There is a need to moderate each of the reviews for language and other areas before they are published to the eCommerce application.

Adoption of a streaming technology such as Kafka becomes important to manage multiple systems which would consume the data from the data stream as events. But not all teams are well versed with using Kafka as a data stream. And this could potentially slowdown the implementation of features. The events generated and consumed will need to be monitored and measured, with a possibility of even securing access data streaming platform

2. Technology Stack

3. An in-depth look at the solution’s architecture

Intelligent Applications (AI) based on an Event Driven Architecture has been used build this solution. The following diagram represents the building blocks of this solution.

architecture blocks

4. About the Technology Stack

4.1. Intelligent Applications with Red Hat OpenShift Data Science

For building the Sentiment Analysis and Reviews Moderation Intelligent Applications based on AIML, the solution leverages Red Hat OpenShift Data Science platform. With OpenShift Data Science, the organisations can

  • Accelerate the delivery of intelligent applications and reduce time to market

  • Data scientists can rapidly develop, train, test, and deploy the models with an on-demand infrastructure

  • Deploy models to hybrid cloud environments, gaining the flexibility of running workloads wherever you need them

4.2. Event Driven Architecture with OpenShift Serverless

OpenShift Serverless, based on the upstream Knative project, helps with event-driven computing amongst other usecases. The Eventing functionality provides the capability to build Event-Driven applications that can connect to and from a number of systems. In this solution, all the services emit Cloud Events with specific ce-type and ce-source. These events are then fed into the Knative Broker. The associated triggers will invoke the right knative service to respond to the right event. Building all this is super easy using CRDs.

4.2.1. Cloud Events

With Knative Eventing, all the systems talk to each other using the Cloud Events which is a specification for describing event data in a common way. Cloud Events makes it easy for various actors (services, systems etc) to interact with each other in a standard way. Furthermore, this can be used for adopting Observability using Open Tracing.

broker trigger

4.3. Realtime Dashboard for Sentiment Analysis: InfluxDB and Grafana

Once the review is processed by the Sentiment Analysis service, the data is streamed into Kafka real-time. Another knative Service then consumes this data and pushed into InfluxDB. Grafana Dashboards can be configured to read from this TimeSeries DB and plot the sentiment analysis score on a dashboard.

grafana dashboard

4.4. Reviews moderation

The Reviews also pass through the Reviews Moderation Intelligent Application which filters the reviews for abusive or foul language. Only the reviews which are moderated are then persisted in the database and then be displayed on the product page.

moderated reviews