Building risk detection engine with Kafka and stream processing for top global retailer

7 minutes read
case-study
ClientNDA
IndustryRetail

Our client, a global retailer, aimed to enhance the security of their system since hackers carried out fraudulent activities on their E-commerce site. VirtusLab developed a risk engine platform for them using the Apache Kafka framework. This solution monitors, categorizes, and prevents malicious attacks. On the very first day after implementation, it detected and prevented thousands of attempted frauds.

Download this success story as PDF

Print it out, take it with you to read later, or share it with your peers.Free download
case-study

The challenge 

With the prevalence of cloud computing and the ease of access to computing power, attackers were building massive botnets to imitate real customers and carry out fraudulent activities. The ideal solution would be able to counter that threat. Our client needed a way to recognize and prevent account takeovers in real-time, at scale. 

The solution 

Having previously worked with Virtuslab, the client chose to outsource this task to them. VirtusLab built a reliable and scalable risk engine platform using Apache Kafka. The main responsibilities of the platform included:

  • Categorizing authentication attempts.
  • Calculating statistics to observe malicious traffic patterns. 
  • Reacting proactively to prevent various types of attacks.

The platform offers several different types of analysis performed in parallel, using specific data pipelines. As the platform is event-driven, every new login event triggers each pipeline’s execution. An event is just a statement of the fact – something that has happened in the real world.

The platform utilized Kafka Streams, a library that can be used with any JVM application, and ksqlDB for specific stream processors. Kafka Streams-based applications do not have any specific requirements about the deployment platform, thus its infrastructure is built on top of Kubernetes. This allows for scaling up and down according to the traffic volume. 

Additionally, specific stream processors utilize ksqlDB while integrating with multiple third-party systems through Kafka Connect.

The results 

After implementing the risk detection platform, the system blocked around 30,000 IP addresses, of which about 1000 were unique, and locked approximately 500 fake user accounts within a single day.

These statistics demonstrate the effectiveness of the solution in preventing fraudulent activities and protecting user accounts. The platform also recognizes:

  • Login attempts from unknown and untrusted devices for a given user.
  • Login attempts from new locations for a given user.
  • Login attempts from botnet agents.
  • Brute-force attacks.

Tech stack

Framework: Apache Kafka, Kafka Streams, ksqlDB:

Infrastructure: Kubernetes

Integration and Communication: Kafka Connect

Take the first step to a sustained competitive edge for your business

Let's connect

VirtusLab's work has met the mark several times over, and their latest project is no exception. The team is efficient, hard-working, and trustworthy. Customers can expect a proactive team that drives results.

Stephen Rooke
Stephen RookeDirector of Software Development @ Extreme Reach

VirtusLab's engineers are truly Strapi extensions experts. Their knowledge and expertise in the area of Strapi plugins gave us the opportunity to lift our multi-brand CMS implementation to a different level.

facile logo
Leonardo PoddaEngineering Manager @ Facile.it

VirtusLab has been an incredible partner since the early development of Scala 3, essential to a mature and stable Scala 3 ecosystem.

Martin_Odersky
Martin OderskyHead of Programming Research Group @ EPFL

VirtusLab's strength is its knowledge of the latest trends and technologies for creating UIs and its ability to design complex applications. The VirtusLab team's in-depth knowledge, understanding, and experience of MIS systems have been invaluable to us in developing our product. The team is professional and delivers on time – we greatly appreciated this efficiency when working with them.

Michael_Grant
Michael GrantDirector of Development @ Cyber Sec Company