Scala Big Data Engineer (Expert)

SCROLL DOWN

Kraków, Rzeszów, Kielce, Wrocław, Katowice, Gdańsk + REMOTE

Scala Big Data Engineer (Expert)

B2B: 26000 - 31000

PLN

UOP: 20000 - 24000

PLN

Would you like to wrangle terabytes of data in minutes using cutting-edge technologies? Would you like to join a team of talented and passionate Software Engineers? Developing your programming skills, building and maintaining data lakes, data platforms, and applications for various modern and highly-funded companies? Using FP and type safety to make sure your applications are correct and do that in a company developing Scala 3? Whether you’re an expert, or you are a Scala developer who would like to get into Big Data or a Data Engineer who wants to learn more about Scala (or both!) then we are looking for someone just like you.

Location

Kraków, Rzeszów, Kielce, Wrocław, Katowice, Gdańsk

Skills required

Scala

Expert

Spark

Expert

Hadoop or Cloud solutions

Advanced

English

Advanced

Available projects

Project Conway

Project scope

Our main goal is to develop a generic Big Data ingestion and transformation framework for a new, highly funded, UK-based, insurance company. We work across two teams: one responsible for developing the framework and extending the capabilities of the platform and another one focusing on using the framework to ingest hundreds of datasets from various data sources.
There are lots of greenfield areas and possibilities to impact the whole company.

Tech stack

Scala, Spark (mostly Core, some SQL), AWS (EMR, EC2, S3, Codebuild)

Challenges

- Creating a generic Big Data framework on top of Spark being able to ingest and transform hundreds of datasets from various sources.
- Making Apache Spark reliable and conforming to Functional Programming style.
- Automating any possible part of the development process - the less manual work the better.

Team

There are around 20 engineers (both Scala/Spark and Snowflake) in the team, of which about a half are from well known consultancies.
Most of the Engineers have Scala background and put a lot of focus on Functional Programming - some of the consultants have been in the Scala community for more than 10 years.

Project Sherlock

Project scope

We are developing an analytical platform (e-commerce industry), operating on a cluster of hundreds of machines, with hundreads of RAM’s terabytes, as well as thousands of cores. We integrate, process and analyze data. We need all this to run and test the applications we have developed that use state-of-the-art technologies from the Big Data world.

We are also optimizing complex machine learning applications (ML pipelines), improving the generation of huge analytical views (joining tables with millions of records in a few minutes is our speciality!). We have an enormous cluster available to test all the developed apps.

Tech Stack

Apache Spark (core, SQL, pySpark, Streaming, mllib), Scala, Kafka, Hive, HBase, Hadoop, Teradata, Azure, Jenkins, Ansible, SBT, GIT.

Team

Over thirty developers with experience in building Big Data solutions divided into teams of 4-6 people. We have a real influence on the selection of tools or the possibility of architectural changes.

What we expect in general

Able and eager to lead by example
Hands-on experience in designing and developing scalable, distributed, highly available solutions
Interest in solving challenging data engineering problems using state-of-the-art techniques
Grounded knowledge and understanding of data structures, algorithms, and distributed computing
Fluent with Scala and/or Spark
A functional Programming mindset is nice to have
Able to develop and maintain high-quality code
Can communicate with the business using good English (both written and verbal)
Experienced in using big data and cloud solutions (eg Hadoop, AWS, Azure)
Has a deep understanding of data-intensive distributed systems

What's on offer?

community

Building tech community

We know how important it is to share knowledge and exchange experience, especially between the nearest tech community. We run 9 meetup groups in Kraków, Kielce, and Rzeszów and support many other local initiatives.

knowledge

Virtusity – our training division

We really value the personal growth of our #VLteam. That’s why we have a rich offer of internal courses, workshops etc. (technical, as well as soft skills trainings)

community

Contributing to more than 40 open source initiatives

We contribute to numerous open source projects and enhance software that developers use worldwide. We support both world-famous solutions as well as we create our own.

path

Flexibility in projects

We make room for your autonomy and provide you with mentoring and guidance from the first day.

work

Flexible hybrid work model

You decide where & when to start the work – we have the opportunity to work fully remotely, in the office or in the hybrid model. Just choose what suits you best!

events

Having fun together

Walking and cycling tours, kayaking, fun at rope park, pizza party or a mountain trip with SPA? Yes, we've got it all. It's never boring with #VLteam.

knowledge

Language lessons

We refund 50% of the price of English, German, Russian or Polish lessons you'd like to take.

environment & atmosphere

Modern offices in the centre of Kraków, Kielce and Rzeszów

Our highest quality spaces are well-communicated and adapted to all sanitary standards. A table football, board games, comfy deckchairs, pouffes to relax - we have it all!

extras

Multisport card or MyBenefit points

Do you like sports and want to have free unlimited admission to sports facilities in your city? Or maybe you prefer the cinema, SPA in the mountains, delicious dinner or Allegro shopping? At VirtusLab, you decide whether you choose a sport card or vouchers for countless attractions.

extras

Private healthcare

Our insurer's offer gives you the opportunity to have free health advice, treatments, diagnostic tests etc., in more than 1,500 clinics operating all over the country.

environment & atmosphere

Reimbursement for remote work equipment

You can rent all the necessary equipment from our offices, but also you can get up to 800 PLN refund if you need to buy something.

environment & atmosphere

Stretching

Feeling tense or tired? Take a break from work and join stretching sessions.

knowledge

VirtusLab Training Package

We give you 24 paid hours of work for your development. You can choose the training from our Virtusity portfolio or an external one (we'll cover up to 50% of its cost). If the course’s certificate is aligned with our core specifications - we’ll refund 100% of its price.

Show more benefits Hide benefits

Description

Benefits

Apply now

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.

Necessary

Always Enabled

Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.

Cookie	Duration	Description
cookielawinfo-checbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Marketing

Analytics

Others

Skills required

Available projects

What we expect in general

Application form