Audience
Anyone who wants to to programmatically author, schedule and monitor workflows
About Apache Airflow
Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity. Airflow pipelines are defined in Python, allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically. Easily define your own operators and extend libraries to fit the level of abstraction that suits your environment. Airflow pipelines are lean and explicit. Parametrization is built into its core using the powerful Jinja templating engine. No more command-line or XML black-magic! Use standard Python features to create your workflows, including date time formats for scheduling and loops to dynamically generate tasks. This allows you to maintain full flexibility when building your workflows.
Other Popular Alternatives & Related Software
JS7 JobScheduler is an Open Source workload automation system designed for performance, resilience and security. It provides unlimited performance for parallel execution of jobs and workflows. JS7 offers cross-platform job execution, managed file transfer, complex no-code job dependencies and a real REST API.
Platforms
- Cloud scheduling from Containers for Docker®, Kubernetes®, OpenShift® etc.
- True multi-platform scheduling on premises for Windows®, Linux®, AIX®, Solaris®, macOS® etc.
- Hybrid use for cloud and on premises
User Interface
- Modern, no-code GUI for inventory management, monitoring and control with web browsers
- Near real-time information brings immediate visibility of status changes and log output of jobs and workflows
- Multi-client capability, role based access management
High Availability
- Redundancy and Resilience based on asynchronous design and autonomous Agents
- Clustering for all JS7 products, automatic fail-over and manual switch-over
Learn more
dbt
dbt helps data teams transform raw data into trusted, analysis-ready datasets faster. With dbt, data analysts and data engineers can collaborate on version-controlled SQL models, enforce testing and documentation standards, lean on detailed metadata to troubleshoot and optimize pipelines, and deploy transformations reliably at scale. Built on modern software engineering best practices, dbt brings transparency and governance to every step of the data transformation workflow.
Thousands of companies, from startups to Fortune 500 enterprises, rely on dbt to improve data quality and trust as well as drive efficiencies and reduce costs as they deliver AI-ready data across their organization. Whether you’re scaling data operations or just getting started, dbt empowers your team to move from raw data to actionable analytics with confidence.
Learn more
Union Cloud
Union.ai is an award-winning, Flyte-based data and ML orchestrator for scalable, reproducible ML pipelines. With Union.ai, you can write your code locally and easily deploy pipelines to remote Kubernetes clusters.
“Flyte’s scalability, data lineage, and caching capabilities enable us to train hundreds of models on petabytes of geospatial data, giving us an edge in our business.”
— Arno, CTO at Blackshark.ai
“With Flyte, we want to give the power back to biologists. We want to stand up something that they can play around with different parameters for their models because not every … parameter is fixed. We want to make sure we are giving them the power to run the analyses.”
— Krishna Yeramsetty, Principal Data Scientist at Infinome
“Flyte plays a vital role as a key component of Gojek's ML Platform by providing exactly that."
— Pradithya Aria Pura, Principal Engineer at Goj
Learn more
AWS Step Functions
AWS Step Functions is a serverless function orchestrator that makes it easy to sequence AWS Lambda functions and multiple AWS services into business-critical applications. Through its visual interface, you can create and run a series of checkpointed and event-driven workflows that maintain the application state. The output of one step acts as an input to the next. Each step in your application executes in order, as defined by your business logic. Orchestrating a series of individual serverless applications, managing retries, and debugging failures can be challenging. As your distributed applications become more complex, the complexity of managing them also grows. With its built-in operational controls, Step Functions manages sequencing, error handling, retry logic, and state, removing a significant operational burden from your team. AWS Step Functions lets you build visual workflows that enable fast translation of business requirements into technical requirements.
Learn more
Integrations
Company Information
The Apache Software Foundation
Founded: 1999
United States
airflow.apache.org
Videos and Screen Captures
Other Useful Business Software
Ship Agents Faster
Gemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
Product Details
Platforms Supported
Cloud
