Recently our big data team was approached by one of our clients – a company wanting to minimize costs of analytics of its sales data. The client had an analytic team working with the Microstrategy BI tool [1]. The tool fetches data by running SQL queries on an underlying Teradata DB [2]. As Microstrategy can […]

Monolithic application architecture was the go to solution for many years in IT. It is very convenient because there are numerous books with answers to almost every question that pops up during the design phase. On the other hand, there is this new (well, maybe not so new, but still just entering the mainstream), emerging approach called microservices. […]

TL;DR: If you work a with a git rebase flow, git machete will (vastly!) help you manage the jungle of branches stacking on top of each other when you’re, for example, working on a couple of different PRs in parallel. The problem In day-to-day development it is common that for various reasons coders have to work with […]

Introduction Blue-green deployment can be defined as a strategy based on two switchable environments. A new version of an application is introduced to only one environment without changing the other. It allows zero downtime deployment with the option to quickly rollback if there are defects in the newer release. To achieve this, two application stacks […]

Introduction Have you ever wondered what’s happening under the hood of Kubernetes bootstrap process? In this blog post I will take you on a journey through the steps of the Kubernetes cluster provisioning in general without focusing on any particular cloud provider or hardware virtualization. It will give you an overview how the various components […]

New wave of container development Deis (now part of Microsoft) created an interesting three part ecosystem: Helm is a package manager and template engine for Kubernetes applications Draft is the “inner loop” of a developer’s workflow: code, deploy, commit Brigade is an event-based pipeline scripting tool for Kubernetes I’m familiar with Helm and see a […]

In this blog post you will learn about how to implement your first type class, which is fundamental language feature in the icon of the functional programming languages — Haskell. Type Class is a pattern that originates from Haskell and it is its standard way of implementing polymorphism. This type of polymorphism is called ad-hoc polymorphism. Its […]

Advent of Code: Were you nice or naughty this year? What’s going on? Advent of Code is an annual programming challenge which consists of 25 (2-part) puzzles at different difficulty levels. I heard about it on our company’s chat and it’s the first edition I have participated in. The event itself is a great opportunity […]

Mechanics of unboxed union types in Scala Paweł Batko, VirtusLab This post is inspired by Miles Sabin’s “Unboxed union types in Scala via the Curry-Howard isomorphism”. Wouldn’t it be nice to be able to write a type like this: And then use it as follows? I’ll try to show you how you can define such […]

In the previous blog post I explained the basic concepts of data lakes. Some core problems which can occur in data lakes were defined and I gave some hints to avoid them. Most of these pitfalls are caused by the traits of data lakes. Unfortunately, current Hadoop distributions can’t resolve them entirely. Additionally, the architecture […]

Nowadays almost every company wants to have their own Big Data system to analyse client behaviour and optimise operating costs. One of the most popular solutions for implementing such systems is a Data Lake based on the Hadoop ecosystem. If you don’t know what exactly a Data Lake is, you can read about it in […]

Introduction Cats is a library providing abstractions for functional programming in Scala. There are a couple of great posts and courses on Cats out there on the web (like Herding cats and a tutorial on Scala Exercises), but they tend to explore the categories/type classes implemented in the library rather than give practical ready-to-use examples […]

Rapid growth of unstructured data is a serious business challenge for organizations. Data repositories, known as data lakes, have a great chance to play an important role in extracting valuable business information from enormous amounts of data. Storing and processing data on such a scale is a very complex and demanding task. Existing RDBMS-based systems […]

Summary & initial requirements If you use and have a basic understanding of the core concepts of the Apache Spark and Spark SQL (RDDs, DataFrames, Execution Plan, Jobs & Stages & Tasks, Scheduling), then after reading this blog post you should be able to answer the following questions: How do Spark SQL’s Datasets relate to […]

In April I gave a presentation (50 Shades of Scala Compiler) at the ScalaUA conference. In the abstract, I promised to give some hints on how to write compiler-friendly code. The talk lasted only 40 minutes and Scala compilers have many, many shades (I almost forgot about Dotty!), so all the hints didn’t fit into […]

Or how to use custom / database specific operators in Slick 3.x. Couple of days ago we had our company Xmas dinner. As we are a software company we obviously talked mostly about technology (surprise!). One of the conversation we had was about a comparison: Slick vs Quill. It was about difficulties of adding customization. […]

Introduction   Once upon a time, there was complete chaos, where you could draw and instantly share anything like a boss. However, the days of Adobe Flash are long gone, so a new idea was born: make chaos great again! And this is how from chaos arose AkkaPaint The idea was pretty straightforward: create a […]

Problem: Imagine you have to manage hundreds of different projects, that could have several different actions or building blocks. These projects can be grouped into “monitoring groups” – and you have some people assigned to execute them, some others are assigned to monitor them. You have “cost centers”, progress and performance measures and all this […]

Introduction ScalaMock is a powerful mocking library written purely in Scala. It uses macros to create mocks. Macros and compile-time reflection allow to create type safe code or manipulate programs. It is almost mythological tool, the Holy Grail of programming languages. Almost all modern languages have it. All features of ScalaMock are listed on its […]

ScalaSphere: conference focused on technical problems and solutions ScalaSphere was one-of-a-kind conference focused on Scala tooling. Those who expected presentations full of theory or maths, might feel disappointed – conference was dedicated purely to Scala tooling. “I’ve never been on such deeply technical conference” was often heard and I agree with those comments. Quo vadis, […]

TL;DR 1. Here’s a repo: VirtusLab/mesos-on-vagrant, 2. Clone it, make all your RAM free and setup VirtualBox, 3. vagrant up, 4. Congratulations, you’ve 6-node (3 master, 3 slave nodes) “highly-available” cluster running on virtual machines. End notes – to begin with :-) As it’s the only part (despite of short descriptions of each service), I’ll […]

In part I I showed how Kleisli arrows could be used to implement domain modeling. Arrows serve as a foundation for a ‘DSL’ in which one can implement typical scenarios that arise in business-logic code: decoupling flow control from domain code, dealing with errors etc. Much to the spirit of Railway Oriented Programming but implemented […]

During Scala Days Amsterdam I came across a concept of arrow. It was called a general interface to computation and it looked like a technique that could bridge code that is imperative in nature with proper functional programming. Later on I read about Railway Oriented Programming and found it really neat. I was compelled to […]

Immutability in Scala is great. A lot of concerns go away when you use it, but those that remain are not so clear to reason about. Look at the code below: My question is: what is the value of expression (new Bar).b? In this post I will explain why it is neither 1 nor 2. […]

Unlike most of the development boards, Intel Edison uses binary, proprietary blob for delivering tables of devices connected to I2C and SPI buses. They’re loaded via Simple Firmware Interface (SFI) during kernel startup in Intel MID SFI initialization code (located in intel_mid_sfi.c) and compared to board data structure (board.c). “To summarise, in the need of […]

In the previous interview I was talking with Rex Kerr about his work and contribution to Scala. This time I had a pleasure of asking several questions to two well-known (in the community) Scala ecosystem contributors from Karlsruhe. Simon Ochsenreither (soc) studies computer science and works on various Scala-related things in his free time, including […]

People complain a lot about Scala tooling. People complain a lot about weather. But is Scala tooling like weather? Something you can’t do anything about except from moving to different climate (language)? Luckily no. Scala tooling was created for Java (especially metadata handling that is vital for tools) and usually has only the JVM stack […]

During Scala Days in SF, Dick Wall was trying to convince people to spend their spare time helping to make Scala language, compiler and documentation even better. He focused mostly on showing the Scala’s web page and available resources which can help newcomers. Also some time ago Adriaan Moors completely redesigned readme.md on github having […]

For a long time programming embedded systems with limited hardware resources forced you to use native languages and run your firmware bare-metal. That effectively meant that being a Scala software house and doing embedded weren’t an easy job. Fortunately, more and more relatively high-performance platforms have been recently emerging. After playing for some time with […]

Let’s start with small refreshment of what actually the dependent typing is. Using Wikipedia as a source: In computer science and logic, a dependent type is a type that depends on a value. Neat, huh? But what does it mean in practice? If you take a look at the Wikipedia’s list of languages that implement […]

During my career as an administrator I’ve tried multiple well-known and developed a few home-grown ways of dealing with applications created in micro-services architecture. Every time I fell into pitfalls of these approaches: the problem of maintaining them in reactive way. ConductR is a Typesafe’s response to this problem, addressing full life cycle of an […]

Attending (or, even better, organizing) a conference is one of the best opportunities to predict what the future may bring. Knowing what the future may bring is undoubtedly the most important knowledge for any enterprise – especially when you’re in IT. We recognize this at VirtusLab and we’ve always been trying to be on top […]

I’ve been recently asked what are my favorite and least favorite Scala features. Probably few options would be matching here, but I decided to give one answer for both questions: implicits. Why? Implicits are very simple and yet very powerful concept but with great power comes great responsibility. They can be used for implicit parameters, […]