The solution
Having previously worked with Virtuslab, the client chose to outsource this task to them. VirtusLab built a reliable and scalable risk engine platform using Apache Kafka. The main responsibilities of the platform included:
- Categorizing authentication attempts.
- Calculating statistics to observe malicious traffic patterns.
- Reacting proactively to prevent various types of attacks.
The platform offers several different types of analysis performed in parallel, using specific data pipelines. As the platform is event-driven, every new login event triggers each pipeline’s execution. An event is just a statement of the fact – something that has happened in the real world.
The platform utilized Kafka Streams, a library that can be used with any JVM application, and ksqlDB for specific stream processors. Kafka Streams-based applications do not have any specific requirements about the deployment platform, thus its infrastructure is built on top of Kubernetes. This allows for scaling up and down according to the traffic volume.
Additionally, specific stream processors utilize ksqlDB while integrating with multiple third-party systems through Kafka Connect.