Web Panel
Local-first ETL/ELT studio: a drag-and-drop visual pipeline designer
Brought to you by:
slothflowlabs
Menu
▾
▴
#75 Web Panel
Originally created by: hatamiarash7
Originally owned by: SouravRoy-ETL
Hi
Thank you for this amazing product.
Is there any web panel for Duckle? So I can run it on servers and manage everything remotely. I can't find anything in the documentation.
There is a Server deployment (Build Pipeline) option, but it's very different because we can't modify pipelines there.
Best regards
Originally posted by: SouravRoy-ETL
Currently there is none but we will look into this option. Thanks for the idea.
Originally posted by: SebZbp
Indeed, I work mainly in virtual IDEs (Conveyor IDE mainly) - that's the only way to connect to our data as secrets are only accessible from there. We can't run a desktop app from there, but we can run a web app either locally or inside a docker container
Originally posted by: SouravRoy-ETL
Let us internally check how we can implement this to userbases
Originally posted by: SouravRoy-ETL
Shipped in v0.5.0! There's now a web management console - click the green Open web dashboard button in the top bar and it opens a browser dashboard: pipelines grouped by job with last status, schedule, duration and run history + logs, a runs timeline, and a built-in interval scheduler. No extra binary, no server, no auth.
You can also run it headless from the terminal with the
duckle serve --workspace <path>command for a server box. Make sure Duckle is present in the location from where duckle serve is called.Please download - https://github.com/slothflowlabs/duckle/releases/tag/v0.5.0 and let me know!
Originally posted by: SebZbp
Great! I will test that.
But my point and I think @hatamiarash7 's point was to be able also to be able to edit the pipeline from the web interface. Your local first approach is limitative in my case as I have only access to my data from an headless OS (Web-based IDE with a headless Ubuntu-- console-only). I cannot run the Duckle from there. What I could do is to run a container or even an executable that would expose in another tab of my browser the Duckle devlopment application as a web app as a complement to the web app. This is what is done by Rill Data with their Rill Developer component.
https://github.com/rilldata/rill
https://docs.rilldata.com/developers/get-started/install
By the way, in my company, for security and standardisation reasons, we are moving away from tools that require the installation of software on our laptops:
But the runner web UI is definitely something that would be needed anyway.
Originally posted by: SouravRoy-ETL
That makes sense now. I might have misunderstood the requirement. Great.
This is something I will have to research and see if it fits with Duckle and to what extent the solutioning is possible.
Originally posted by: hatamiarash7
Hi @SouravRoy-ETL
To make it easier to understand, take a look at the Arroyo project.
Ideally, we need Duckle to run standalone on a server (e.g., a Docker container), and then everything can be managed via a web address in the browser.
In fact, this is the current need in this field. Few people go for a local application because the main data for processing is not on our personal systems =))) It is on servers and in the cloud environment.
Originally posted by: SouravRoy-ETL
@hatamiarash7 - Very clear. Will work on it. Keep tabs until then.
Originally posted by: SebZbp
BTW, port 8080 is quite standard and often taken already (it's the case our Conveyor IDEs running code-server in the background on port 8080). It's safer to select another port to avoid structural conflicts.
Originally posted by: hatamiarash7
Don't think about that :))) we can use CLI args or ENV to choose another port number.