Dagster is an orchestration platform for the development, production, and observation of data assets.
Dagster as a productivity platform: With Dagster, you can focus on running tasks, or you can identify the key assets you need to create using a declarative approach. Embrace CI/CD best practices from the get-go: build reusable components, spot data quality issues, and flag bugs early.
Dagster as a robust orchestration engine: Put your pipelines into production with a robust multi-tenant, multi-tool engine that scales technically and organizationally.
Dagster as a unified control plane: The ‘single plane of glass’ data teams love to use. Rein in the chaos and maintain control over your data as the complexity scales. Centralize your metadata in one tool with built-in observability, diagnostics, cataloging, and lineage. Spot any issues and identify performance improvement opportunities.
Features
- Shared, reusable, configurable data processing components.
- Modularized testing and familiar programming model.
- Elegant, Pythonic APIs.
- Flexible, programmable first class sensors that can be individually monitored.
- Robust, fault-tolerant scheduling gives you all the options you need.
- Advanced partitioning helps optimize pipeline performance and minimize recompute.
- A thriving community and full-time engineering support.
- Per-job package dependencies and Python version. Isolated codebases and environmental requirements, and new processes on each run make Dagster more robust and less prone to failures.
- Dagster’s I/O Manager lets you focus on the business logic, not the data hand-offs.
- Automate your efforts with scriptable alerting, asset-level status reports, and Dagster’s fully extensible alerting framework.
- Integrate your infrastructure and monitoring tools with the most versatile and extensible platform.
Project Samples
