Automatization of the end-to-end Machine Learning pipelines

The challenge

Our client aimed to automate their Machine Learning pipeline. It would allow them to take full advantage of existing customer behavior data and deliver a more personalized experience on their website. The client's Data Science team was already developing ML models but lacked the engineering expertise and manpower necessary to scale up their production. 

In order to maintain the project timeline of delivering new solutions to the end-user the existing pipeline had to be fully automated. This called for a holistic approach in multiple areas:

• Data ingestion
• Feature generation
• Model training 
• Model deployment and serving

Having previously collaborated with the client, VirtusLab assembled a dedicated team proficient in Big Data, software development, and data transformation. 

The solution

VirtusLab built a fully automated end-to-end Machine Learning process that delivers new models on demand. They created small, manageable code pipelines, using decoupled component methodology. VirtusLab designed the pipelines, making it easier to understand individual data transformations, features, or models. They leveraged PySpark and TensorFlow frameworks only where necessary.

VirtusLab introduced a composable configuration for ease of production and scalability. They also developed common building blocks to extract complex logic and ensure consistency across modules. Overall, they built a fully automated end-to-end Machine Learning process that delivers new models on demand.

Working in collaboration with the client's Data Science team, rigorous engineering practices were integrated, including:

• Unit and acceptance testing 
• Static type checking
• Linting 
• Code reviews 
• Continuous integration and deployment (CICD)

The results

VirtusLab’s solution helped the client's Data Science team successfully productionise 25 Machine Learning models and over 200 pipelines. 

Automated ML pipelines enabled our client to better personalize the online shopping experience. Our client introduced tailored product recommendations, customized promotions, and dynamic pricing that increased customer satisfaction and loyalty.

VirtusLab's small code pipelines enhanced the client's ability to manage complex data pipelines and Machine Learning processes. Using decoupled component methodology ensured that the system is:

• Robust 
• Adaptable 
• Capable of handling the demands of large-scale data processing and model deployment

VirtusLab also streamlined the process of monitoring the performance of ML models, making it easier to address issues as they arose. For example, the client is now effectively counteracting problems like data drift — an unexpected change in input data that can affect the quality of the model's result.

Tech stack

Programming language: Python

Libraries: Pandas 

Big Data Processing Framework: PySpark, Hadoop, Hive

Machine Learning Framework: TensorFlow

Cloud storage: S3

Other tools: Jenkins, Conda, mypy, Jinja, Oozie