Sinbadflow is a simple pipeline creation and execution tool. It was created having Databricks notebooks workflow in mind, however with flexible implementation options the tool can be extended and customized to fit any task. Named after famous cartoon “Sinbad: Legend of the Seven Seas” the library provides ability to create and run agents with specific triggers and conditional functions in parallel or single mode. With the simple, yet intuitive, fully code based syntax we can create elaborative pipelines to solve any data engineering, data science or software development task.

During these uncertain times, as most of us are staying inside, it’s a great time to try and learn new things. Last week we noticed few posts in Linkedin by professionals trying out Databricks and posting their initial opinions. That sparked an idea to summarize and write a short, 20 rule list as a helper for everyone who decides to try using Databricks in production. We are two data engineers (Robertas Sys and me, Eimantas Jazonis) who spend years developing and improving our production level data science platform on Azure. Our current setup is mainly based on Databricks, so feel that we gathered enough experience to share with you all.

In this post I will explain the need and the use case of Data Science platform, which can be deployed to Azure cloud in one lazy afternoon. Many tutorials focus on one element or technology, making you to piece the information together in order to come up with a full architecture. My goal is to provide a quick and very simple setup for testing purposes from the beginning to the end.

This blog post is a summary of my talk at the 4th Vilnius Microsoft Data Platform Meetup, which took place on November 6th @Cognizant Lithuania. In this talk I presented a gentle introduction to the new Delta Lake solution from Databricks foundation, its new features, use cases and our production experience.

Raspberry pi based project that gets upcoming day weather information from OpenWeather API and sends it directly to users as a facebook message every morning.