The challenge
Our client aimed to automate their Machine Learning pipeline. It would allow them to take full advantage of existing customer behavior data and deliver a more personalized experience on their website. The client's Data Science team was already developing ML models but lacked the engineering expertise and manpower necessary to scale up their production.
In order to maintain the project timeline of delivering new solutions to the end-user the existing pipeline had to be fully automated. This called for a holistic approach in multiple areas:
- Data ingestion
- Feature generation
- Model training
- Model deployment and serving
Having previously collaborated with the client, VirtusLab assembled a dedicated team proficient in Big Data, software development, and data transformation.
The solution
VirtusLab built a fully automated end-to-end Machine Learning process that delivers new models on demand. They created small, manageable code pipelines, using decoupled component methodology. VirtusLab designed the pipelines, making it easier to understand individual data transformations, features, or models. They leveraged PySpark and TensorFlow frameworks only where necessary.
VirtusLab introduced a composable configuration for ease of production and scalability. They also developed common building blocks to extract complex logic and ensure consistency across modules. Overall, they built a fully automated end-to-end Machine Learning process that delivers new models on demand.
Working in collaboration with the client's Data Science team, rigorous engineering practices were integrated, including:
- Unit and acceptance testing
- Static type checking
- Linting
- Code reviews
- Continuous integration and deployment (CICD)
The results
VirtusLab’s solution helped the client's Data Science team successfully productionise 6 Machine Learning models and over 30 pipelines.
Automated ML pipelines enabled our client to better personalize the online shopping experience. Our client introduced tailored product recommendations, customized promotions, and dynamic pricing that increased customer satisfaction and loyalty.
VirtusLab's small code pipelines enhanced the client's ability to manage complex data pipelines and Machine Learning processes. Using decoupled component methodology ensured that the system is:
- Robust
- Adaptable
- Capable of handling the demands of large-scale data processing and model deployment
VirtusLab also streamlined the process of monitoring the performance of ML models, making it easier to address issues as they arose. For example, the client is now effectively counteracting problems like data drift — an unexpected change in input data that can affect the quality of the model's result.
Tech stack
Programming language: Python
Libraries: Pandas
Big Data Processing Framework: PySpark, Hadoop, Hive
Machine Learning Framework: TensorFlow
Cloud storage: S3
Other tools: Jenkins, Conda, mypy, Jinja, Oozie
Partner flexibly with VirtusLab
Use one or a combination of engagement models to suit your needs.