Browse Code
With CueLake, you can use SQL to build ELT (Extract, Load, Transform) pipelines on a data lakehouse.
You write Spark SQL statements in Zeppelin notebooks. You then schedule these notebooks using workflows (DAGs).
To extract and load incremental data, you write simple select statements. CueLake executes these statements against your databases and then merges incremental data into your data lakehouse (powered by Apache Iceberg).
To transform data, you write SQL statements to create views and tables in your data lakehouse.
CueLake uses Celery as the executor and celery-beat as the scheduler. Celery jobs trigger Zeppelin notebooks. Zeppelin auto-starts and stops the Spark cluster for every scheduled run of notebooks.
Features
- Create DAGs. Group notebooks into workflows and create DAGs of these workflows.
- Data Security. Your data always stays within your cloud account.
- Create Views in data lakehouse. CueLake enables you to create views over Iceberg tables.
- Upsert Incremental data. CueLake uses Iceberg’s merge into query to automatically merge incremental data.
- Elastically Scale Cloud Infrastructure. CueLake uses Zeppelin to auto create and delete Kubernetes resources required to run data pipelines.
- In-built Scheduler to schedule your pipelines.
- Monitoring. Get Slack alerts when a pipeline fails.CueLake maintains detailed logs.
- Versioning in Github. Commit and maintain versions of your Zeppelin notebooks in Github.
You Might Also Like
Our Free Plans just got better! | Auth0 by Okta
You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your secuirty. Auth0 now, thank yourself later.
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of Cuelake!