9 minutes read
Our client aimed to optimize their data analytics strategy. Doing so would reduce the manhours and other operational costs needed to complete their daily tasks.
VirtusLab prepared and executed a migration of the existing jobs to Spark 3 in a way that allowed our client’s platform to operate without disruption. It also enabled our client to update the platform to a more efficient version with a better Developer Experience and maintain its compatibility with legacy code.
In its current state, our client’s Big Data analytics platform relied on legacy code which made it inefficient and difficult to scale and maintain.
The platform was based on Spark 2 and Scala 2.11. Updating the platform required a newer version of Spark and Scala, which prompted our client to update them both. This meant conducting a full-scale migration to Spark 3.
The migration process carried a risk of service disruptions. Any loss of the platform’s availability carried a risk of significant revenue drops and reduced customer satisfaction. As a global retailer, our client was determined to avoid that to maintain their strong reputation.
Our client needed additional manpower and expertise to navigate this high-stakes migration as their engineering teams were occupied with their ongoing projects. They also lacked experience with systems transition.
Encouraged by our existing partnership, our client turned to VirtusLab to avoid the common pitfalls of large migration projects.
VirtusLab helped our client in executing their high-level migration plan. During the entire process, our engineers advised the retailer on some aspects of the overall migration and were involved in planning, tracking, and implementing 3 large projects within the platform:
Additionally, our engineers helped introduce a cross-compilation process to separate the upgrades of hundreds of ETL tasks from the platform's component upgrades.
Our client created jobs within their platform using multiple technologies and workflows. Their unification and refactoring into a common framework became an important part of the platform’s update plan.
The migration plan had a few assumptions on its own. For every piece of infrastructure, it allowed for the automatization of the migration, combined with the adoption of repeatable processes across different versions of the platform. It also added the ability to recover data for various error scenarios.
During the migration, our engineers also introduced additional improvements to the platform, such as optimizing the storage of raw data.
The client’s platform was partially based on technologies that were no longer supported. It forced our engineers to rewrite some of the legacy jobs, as they were unsuitable for migration.
Before switching jobs to Spark 3.0.2, tour engineers made the necessary preparations to make the deployment as efficient as possible.
Once all the prerequisites were met, our engineers could deploy the new version of the platform. They executed the migration in a way that simplifies future migrations and potential upgrades for the client.
Our engineers divided the migration of the existing jobs into smaller modules, which allowed for a shorter feedback loop. It also gave our engineers greater control over the process and made it more transparent for both users and developers working daily on the platform.
Additionally, the team provided proof that the new version was producing accurate results, ensuring no regression would take place during the migration. They achieved it through a combination of local testing and parallel production runs, with comparative analysis of results as needed.
Deployment was carried out in modules, one by one. Our engineers started with simpler jobs to see if the new version of the application operated properly without the risk of the most vital jobs crashing.
The migration of the platform itself was a matter of "flipping the switch" on the CI/CD pipeline and deploying. In the case of more critical jobs, our engineers conducted additional tests post-deployment or ran old and new versions simultaneously to ensure that both of them produced the same results.
VirtusLab migrated all the jobs to Spark 3.0.2 successfully. The platform stayed operational throughout the process and maintained its high availability. The migration and update of the platform yielded a range of benefits, both on a local and global scale.
Locally:
Globally:
Our engineers also provided proof that the new version is producing the correct results (so no regression upon migration happened) by a combination of local and staging tests as well as parallel production runs with comparing results whenever necessary.
Additionally, the adoption of cross-compilation added backward compatibility, meaning that the system works with its legacy version. It also made the platform’s code easier to work with and more modularized.