Amazon MWAA

You select the size of the cluster, node capacity, and a set of services, and Yandex Data Proc automatically creates and configures Spark and Hadoop clusters and other components. Collaborate by using Zeppelin notebooks and other web apps via a UI proxy. You get full control of your cluster with root permissions for each VM. Install your own applications and libraries on running clusters without having to restart them. Yandex Data Proc uses instance groups to automatically increase or decrease computing resources of compute subclusters based on CPU usage indicators. Data Proc allows you to create managed Hive clusters, which can reduce the probability of failures and losses caused by metadata unavailability. Save time on building ETL pipelines and pipelines for training and developing models, as well as describing other iterative tasks. The Data Proc operator is already built into Apache Airflow.

Managed Service for Apache Airflow is a fully managed workflow orchestration platform from Google Cloud built on the open-source Apache Airflow project. It allows users to author, schedule, and monitor data pipelines using Python-based workflows known as DAGs. The platform eliminates the need to manage infrastructure, enabling teams to focus on building and running pipelines. It integrates seamlessly with Google Cloud services such as BigQuery, Dataflow, and Managed Service for Apache Spark. It also supports hybrid and multi-cloud environments, allowing workflows to span across different systems. Users benefit from built-in monitoring, logging, and troubleshooting tools for reliability. The service is designed to simplify complex data workflows, including ETL, MLOps, and automation tasks. Overall, it provides a scalable and flexible solution for orchestrating modern data pipelines.