MENU
Services
Expertise
Solutions
Insights
Who we are
Careers
Contact
Contact

Building risk detection engine with Kafka and stream processing for top global retailer

7 minutes read

case-study
ClientNDA
IndustryRetail

Our client, a global retailer, aimed to enhance the security of their system since hackers carried out fraudulent activities on their E-commerce site. VirtusLab developed a risk engine platform for them using the Apache Kafka framework. This solution monitors, categorizes, and prevents malicious attacks. On the very first day after implementation, it detected and prevented thousands of attempted frauds.

Download this success story as PDF

Print it out, take it with you to read later, or share it with your peers.
Free download

The challenge 

With the prevalence of cloud computing and the ease of access to computing power, attackers were building massive botnets to imitate real customers and carry out fraudulent activities. The ideal solution would be able to counter that threat. Our client needed a way to recognize and prevent account takeovers in real-time, at scale. 

The solution 

Having previously worked with Virtuslab, the client chose to outsource this task to them. VirtusLab built a reliable and scalable risk engine platform using Apache Kafka. The main responsibilities of the platform included:

  • Categorizing authentication attempts.
  • Calculating statistics to observe malicious traffic patterns. 
  • Reacting proactively to prevent various types of attacks.

The platform offers several different types of analysis performed in parallel, using specific data pipelines. As the platform is event-driven, every new login event triggers each pipeline’s execution. An event is just a statement of the fact – something that has happened in the real world.

The platform utilized Kafka Streams, a library that can be used with any JVM application, and ksqlDB for specific stream processors. Kafka Streams-based applications do not have any specific requirements about the deployment platform, thus its infrastructure is built on top of Kubernetes. This allows for scaling up and down according to the traffic volume. 

Additionally, specific stream processors utilize ksqlDB while integrating with multiple third-party systems through Kafka Connect.

The results 

After implementing the risk detection platform, the system blocked around 30,000 IP addresses, of which about 1000 were unique, and locked approximately 500 fake user accounts within a single day.

These statistics demonstrate the effectiveness of the solution in preventing fraudulent activities and protecting user accounts. The platform also recognizes:

  • Login attempts from unknown and untrusted devices for a given user.
  • Login attempts from new locations for a given user.
  • Login attempts from botnet agents.
  • Brute-force attacks.

Tech stack

Framework: Apache Kafka, Kafka Streams, ksqlDB:

Infrastructure: Kubernetes

Integration and Communication: Kafka Connect