As business needs and market trends change, Hadoop and cloud data platforms will evolve together. Will Hadoop remain a data solution companies rely on?
Spark 3.2.0’s support for Scala 2.13 technically allows Scala 3 Spark jobs—but it remains “an uphill path,” requiring workarounds for encoders and data shapes. Using libraries like Iskra smooths the path, yet production readiness is still experimental.
VirtusLab created the pandas‑stubs library to enhance pandas with type information, enabling stronger type‑safety in pandas‑dependent projects. It emerged from challenges integrating pandas and pyspark, where missing stubs led to API conflicts and unchecked code.
In this post, we describe how we built a pipeline for the type of “incoming data” situation, and how we came up with a good solution in the end.
