#90 Decision: is Airbyte still on the roadmap? (ELT/ETL leaves scope under the multi-tenant direction)

[Anonymous]

Originally created by: grynn-in

Context

The local Airbyte setup uses abctl, which provisions Airbyte into a throwaway kind (k8s-in-docker) cluster on its own Docker network. This is a dev/POC convenience, not a production deployment. Several pain points encountered locally are artifacts of abctl, not the design:

• The Airbyte kind node must be manually bridged to the open_epm Docker network (docker network connect open_epm airbyte-abctl-control-plane), and this bridge drops on every host reboot / Airbyte reinstall — konsol then can't reach the Airbyte API (all calls fail until re-attached). Documented in scripts/setup-airbyte.sh but fragile.
• Airbyte API credentials are random per-install (abctl local credentials) and currently hand-pasted into konsol EPM Settings.
• workspace_id and d365_source_definition_id are found manually via the UI/API.

What production should look like

Two viable shapes:

A. Airbyte Cloud (managed SaaS) — stable API (api.airbyte.com), OAuth application client-id/secret created once, stable workspace/source-definition IDs. The ClickHouse destination must be reachable from Airbyte Cloud (TLS public endpoint + IP allowlist, or tunnel/PrivateLink). D365 source side is already public HTTPS (Azure OData).

B. Self-hosted Airbyte on Kubernetes (Helm) — Airbyte + ClickHouse in the same cluster/VPC, stable service DNS (no docker bridge, no reboot-drop). Custom D365 connector image published to a registry (GHCR/ECR), versioned.

What changes local → prod

Concern	Local (abctl)	Production
Networking	manual docker network connect, drops on reboot	VPC/cluster service DNS (self-host) or secured endpoint/tunnel (Cloud)
Airbyte creds	random per-install, hand-pasted	secret manager (Vault / AWS Secrets Manager / K8s secrets), injected
workspace/source-def IDs	hunt in UI/API	set once via IaC (Terraform / Helm values / bootstrap)
EPM Settings Airbyte block	clicked in UI	provisioned by config/IaC at deploy time
D365 connector	local docker image	published, versioned registry image
Stability	reboot breaks it	restart policies, persistence, monitoring, HA

What stays the same

The konsol layer is prod-ready as-is: provision_connector_airbyte is idempotent (find-or-create), D365 creds stored encrypted in connector password fields, and the config → source → schema-apply → build flow is identical. Productionizing means swapping the Airbyte substrate, not rebuilding konsol.

Multi-tenant (hosted-clients) notes

• One Airbyte workspace per client (or dedicated instance for isolation); connector → workspace_id scopes it.
• Per-tenant D365 creds stay encrypted in each tenant's konsol (never exposed to an LLM/MCP — only a configured: true boolean is ever returned).
• Per-tenant ClickHouse target (separate database/schema, e.g. epm_raw_<tenant>).
• Onboarding = IaC creating a workspace + connection + calling idempotent provision_connector_airbyte, not a human clicking buttons.

Key decision

The single biggest real design decision is how Airbyte reaches ClickHouse securely: in-VPC service DNS (self-host) vs secured endpoint/tunnel (Cloud). Everything konsol-side carries over unchanged.

Acceptance / next steps

• [ ] Decide Cloud vs self-hosted Helm
• [ ] Define ClickHouse-destination networking for the chosen path
• [ ] Move Airbyte + per-tenant secrets into a secret manager
• [ ] Provision EPM Settings Airbyte block + workspace/source-definition via IaC
• [ ] Publish the source-d365-fno custom connector image to a registry

---

Filed from a working session analyzing the D365 → consolidation wiring on the local stack.

[Anonymous]

Originally posted by: grynn-in

Decision (next round, P2). Local abctl/kind works for dev today, so this isn't blocking — but it's required before a real multi-tenant/prod deployment (durable connectors, managed lifecycle, no local kind). Schedule as a P2 infra item; not urgent while dev runs on the local connector. Keeping open.

[Anonymous]

Originally posted by: grynn-in

📋 Decision brief (options + trade-offs + recommendation): docs/developer-guide/decisions/konsolidat-90-airbyte-prod.md — merged in [#133].

[Anonymous]

Originally posted by: grynn-in

Verdict: LIVE but OVERTAKEN by an architectural decision — recommend re-scoping or closing.

Still accurate as written: 9 files across the repo still reference abctl.

But the multi-tenant decision of 15 Sep 2026 is that the ELT/ETL layer stays out of scope (the dbt project moves into the Frappe app; connectors do not come with it). Productionising Airbyte is work on a layer the product is stepping away from. Related: konsolidat#207 (D365 cannot be switched off) is the same boundary seen from the other side.

Worth a decision rather than a fix: is Airbyte still on the roadmap for single-tenant deployments, or does this close?

🤖 Triage against main — Claude Code · https://claude.ai/code/session_01P3Pf9835FeLeXjRrYTTZ1M

[Anonymous]

Originally posted by: grynn-in

Re-scoped — this is now a decision, not a task

Still accurate as written: 9 files across the repo reference abctl, and none of the pain points in the body have been addressed.

But the multi-tenant direction of 15 Sep 2026 leaves the ELT/ETL layer out of scope. The dbt project moves into the Frappe app; the connectors do not come with it. Productionising Airbyte is investment in a layer the product is stepping away from.

The same boundary shows up from the other side in konsolidat#207: eleven bronze models ref() stg_d365_fo__* directly, so D365 cannot be switched off — which is a blocker for a connector-less tenant and has to be resolved by the dbt-into-the-app work regardless.

The question to answer here: is Airbyte still the ingestion story for single-tenant deployments, and if so does it need productionising, or does ingestion move outside the product entirely? Recommend recording the answer and then either re-opening this as a scoped task or closing it.

🤖 Re-scoped 15 Sep 2026 against main — Claude Code · https://claude.ai/code/session_01P3Pf9835FeLeXjRrYTTZ1M

[Anonymous]

Originally posted by: grynn-in

Decision: no. Airbyte is off the roadmap.

Recorded 15 Sep 2026 (user). The ELT/ETL layer leaves scope: the dbt project moves into the Frappe app, the connectors do not come with it. Productionising abctl is investment in a layer the product is stepping away from, so this closes without being done.

What this decision now settles elsewhere (the point of recording it here rather than only in a conversation):

• konsolidat#207 — D365 cannot be switched off because eleven bronze models ref() stg_d365_fo__* directly. The fix is no longer "make erp_sources symmetric"; it is to remove the D365 dependency from bronze outright.
• konsol#200 — the build preflight still gates on Airbyte and connector sync status. That becomes a deletion rather than a redesign.
• konsolidat#57 (cluster mode) is a separate decision and stays open.

The surface this affects, for whoever scopes the retirement: 26 Airbyte/D365 fields on EPM Settings, airbyte_service.py (443 lines), 13 konsol modules referencing Airbyte, 8 Connector/Pipeline doctypes, source-d365-fno and source-erpnext in this repo, and 16 D365 + 7 ERPNext staging models. None of that has to go at once — but nothing new should be built on it.

🤖 Claude Code · https://claude.ai/code/session_01P3Pf9835FeLeXjRrYTTZ1M

[Anonymous]

Ticket changed by: grynn-in

• status: open --> closed

[Anonymous]

Originally posted by: grynn-in

Correction to the decision recorded above

The decision is not that Airbyte disappears. Refined by the user, same day:

we have to come up with a way for Airbyte to read data from different sources and deliver us CSVs in our canonical format.

So Airbyte moves outside the product boundary rather than off the roadmap. It stops being an in-product ELT layer that lands source-shaped rows in the warehouse, and becomes an upstream producer that reads a customer's D365, ERPNext or anything else and emits CSVs in konsol's canonical trial-balance contract — which then enter through the intake that is already the canonical source.

What stands from the original closure:

• Productionising abctl as this repo's concern still closes. Airbyte is no longer deployed as part of the product stack.
• konsolidat#207 still holds, and for the same reason: bronze must stop depending on stg_d365_fo__*. If Airbyte delivers canonical CSVs, the warehouse never sees a D365 shape at all.
• konsol#200 still becomes a deletion: the preflight gates on trial-balance readiness.

What was wrong: the line "Airbyte is off the roadmap" and the implication that the connector surface is simply dead weight. The connector concept survives; its position changes.

I am filing the design question separately rather than reopening this, since it is a different piece of work from productionising abctl.

🤖 Claude Code · https://claude.ai/code/session_01P3Pf9835FeLeXjRrYTTZ1M