Decode Analytics

If you are a data engineer, chances are you are dealing with the complexities of data pipeline in daily basis. While Apache Airflow is really popular since it was already available for several years, the learning curve is quite steep, especially if you just starting to build a simple pipeline.

Mage is an open-source tool designed for automating and managing data workflows. It facilitates the scheduling, execution, and monitoring of complex data processes, helping to streamline operations like data extraction, transformation, and loading (ETL).

Mage is gaining popularity due to its user-friendly interface, which simplifies the building and monitoring of workflows without deep technical expertise. It offers flexibility to integrate with a wide range of data tools and provides robust monitoring and alerting features for real-time oversight. Its emphasis on ease of use and scalability makes it a compelling choice for businesses seeking efficient and adaptable data orchestration solutions.

Starting from pure open-source, now Mage also has the managed version with affordable pricing. They start from USD 100 as baseline and then pay-as-you-use based on the compute and memory you consume per month.

I personally try it out myself and this tool is a breeze compared to Airflow code style, despite both tools are supporting Python (in Mage you can use SQL and R too!). You can run each block independently while having the variables passed to another block and the next block can use different programming language. For example, you can create the extraction process from API request in Python and store it as DataFrame variable, then this variable can be called in the next block when you create transformation in SQL. There is also a mini-map so you can combine more complex data pipeline process.

You can try it online to using the demo link. They also post all of their tutorial and latest product update in their YouTube channel. Safe to say I’m pretty bullish for this tool to be widely adopted, especially those who have bad experience in using Airflow.

Hope you find this update useful and subscribe if you don’t want to miss latest advancements in data analytics worth of your time!