Changelog

All notable changes to ClaraVerse.

This project uses semantic versioning. Tags are cut as vMAJOR.MINOR.PATCH.

[0.3.0] — 2026-06-01

Project-scoped knowledge bases (RAG) wired through every surface that touches an LLM. Same retrieval pipeline across Chat, Nexus daemons, and Workflows — the differentiator that closes the biggest remaining gap vs. OpenWebUI per the v0.2.0 retrospective.

RAG stack

• Qdrant + FastEmbed sidecars added to docker-compose.yml and docker-compose.production.yml. Fresh installs now boot the Knowledge feature end-to-end with no extra setup. Embedding models cached in a docker volume so restarts are fast after the first ~140 MB cold download.
• Default stack: bge-small-en-v1.5 (384-dim cosine, dense) + Qdrant/bm25 (sparse) + bge-reranker-base (cross-encoder). All three swappable per-deployment via EMBEDDINGS_* env vars.
• One Qdrant collection per project (kb_<project_oid>) so per-project snapshots, deletes, and embedder choice are trivial.
• Hybrid retrieval (dense + sparse with RRF fusion) runs by default for fresh collections. Phase A's pure-dense fallback is retained for pre-existing collections — Qdrant doesn't support adding a vector kind retroactively, so older collections stay dense-only until reingested.
• Reranker on by default: cross-encoder rerank on top-50 → top-K measurably better than vector order alone (verified live: same query went 5.24 → 6.77 → 8.45 score progression through dense → hybrid → hybrid+rerank).
• Background warmup: embeddings sidecar pre-loads dense + sparse models on boot in daemon threads so the FastAPI port binds immediately and the first ingest doesn't pay the cold-start cost.

Three surfaces, one model

• Chat: multi-project chip picker above the input. Click + Knowledge, check projects, chips appear. Per-chat selection persists to localStorage (survives reloads). When the user sends, search_knowledge is injected as a per-turn tool scoped to the selected projects. Picker selection is the source of truth — drop the chips, drop the tool.
• Nexus: daemons spawned on a project task automatically get search_knowledge when the project has indexed knowledge. The Cortex classifier reads the project's knowledge state and biases the daemon's task summary toward "Search the project knowledge base for X" phrasing. The researcher quality gate accepts search_knowledge calls as a satisfier (alongside search_web / fetch_url / read_artifact).
• Workflows: new knowledge_search block in the agent builder. Full settings panel: project multi-select pulled from REST, templated query field, top_k (1-30), rerank toggle. Outputs {chunks, count, elapsed_ms} for downstream LLM blocks.

Frontend additions

• New per-project Knowledge tab under each project in Nexus. Drag-and-drop upload (PDF / MD / TXT / HTML, up to 50 MB), file list with live ingest progress (polled every 3s while anything is non-terminal), embeddings sidecar warmup banner that shows only when files are actually queued/ingesting.
• Chat sidebar's previously "Coming Soon" Projects entry now active. Click navigates to Nexus for project management; tooltip surfaces how many projects are currently attached to the chat.
• New stores: useChatKnowledgeStore (per-chat persisted picker selection), useNexusStore partial persist for live activity panel survival across navigation.

Data model

• New Mongo collections: nexus_knowledge_files, nexus_knowledge_collections. Files are catalog only — actual chunk text + vectors live in Qdrant. Per-collection record tracks embedder fingerprint so we refuse to ingest a file when the embedder's dim drifts from what the collection was built with.
• New backend models: KnowledgeFile, KnowledgeCollection, KnowledgeSearchHit.
• New KnowledgeProjectIDs + InjectedTools fields on UserConnection for per-turn chat tool wiring.
• NexusOrchestrationState carries ProjectID so a resumed multi-daemon run re-attaches search_knowledge correctly.

Backend services

• New backend/internal/services/rag/ package: parser (PDF/MD/TXT/HTML with page + section preservation), markdown- and code-fence-aware recursive chunker (1000/200), Qdrant + FastEmbed HTTP clients, ingest worker (drains a Mongo queue, batches of 64), search orchestrator (multi-project, hybrid, RRF, dedupe, rerank).
• New RAGSearcher interface in services/ (avoids import cycle with services/rag). NewRAGSearcher wires it into the concrete service. Both CortexService and ChatService accept it via setters.
• daemon_runner.go: per-runner injectedTools map for context-bound tools that shouldn't go through the global registry. executeTool dispatches injected tools first.
• tools/registry.go: GetMCPTools now filters by Source == ToolSourceMCPLocal instead of returning anything in the userTools bucket — fixes a misroute where injected built-in tools were getting routed through the MCP bridge.

Test harnesses

• scripts/rag_e2e.sh: full Nexus path. Provisions a user, creates a project, uploads a synthetic doc with a distinctive marker, waits for ingest, runs REST search, fires a Nexus task that should auto-use search_knowledge, asserts the marker appears in the daemon's reply.
• scripts/rag_chat_e2e.py: same shape for the chat WebSocket path. (Currently blocked by a fiber upgrade quirk in the websockets python client; left in for follow-up debug. Chat wiring is verified by build + boot log + the shared RAGSearcher interface backing both paths.)

Documentation

• docs/RAG.md: full design doc, including the quality-lever ranking (hybrid → rerank → contextual chunking → smart chunking → citations → per-project embedder).
• .env.example: documents all QDRANT_URL, EMBEDDINGS_SERVICE_URL, EMBEDDINGS_*_MODEL, and EMBEDDINGS_PRELOAD_RERANKER knobs.
• README.md: Knowledge bases row added to the feature matrix; the single-container docker run install now flags that RAG needs the Compose setup (no qdrant/embeddings sidecars in the one-container image).

Deferred to next release

• Contextual chunk prefixing (Anthropic-style, ~35% retrieval improvement). Toggleable flag preserved on KnowledgeCollection.ContextualEnabled so this lands without a schema migration.
• Per-project embedder override in admin UI (backend already stores EmbedderID per collection; UI not yet built).
• Reingest button on the file list for migrating pre-Phase-C collections to hybrid.

Fixes for new surfaces (caught during dogfooding)

• Frontend nexusService + knowledgeService list endpoints coerce JSON null → [] (Go's nil-slice serialization). Saved every consumer from a defensive ?? [] guard.
• Embeddings sidecar pre-warms models in a daemon thread at boot, so the "model warming up" banner doesn't get stuck on indefinitely.
• Knowledge tab upload was reading auth_token from localStorage, but the app stores it as access_token. Switched to authClient.getAccessToken() — single source of truth.
• Chat picker checkboxes were unresponsive because the Zustand selector subscribed to s.get (stable function ref). Switched to subscribing to s.selections[chatKey] ?? KNOWLEDGE_EMPTY — toggling now updates the chip row instantly.

[0.2.0] — 2026-05-31

A hardening release focused on making Nexus — the multi-agent orchestration layer — production-grade. Sixteen fixes across the daemon pipeline, the LLM transport layer, the Kanban UI lifecycle, and the multi-user safety boundary. Plus a rebrand revert to the original ClaraVerse name and rose-pink palette.

Nexus — orchestration durability

• Daemon quality gate. Before a daemon can claim "done," we verify it actually did its job. Multi-daemon workers with downstream consumers must have called produce_artifact at least once (without it, downstream daemons re-derive everything from a thin summary). Researcher-role daemons must have called at least one information-gathering tool (search_web, fetch_url, read_artifact, etc.) — finishing without searching means hallucinated output ~95% of the time. Violations inject a corrective reprompt and continue the loop, capped at 2 enforcement cycles so a stubborn model can't be pinned forever.
• Five knobs for long-running tasks. Bumped the orchestrator execution timeout from 10 min → 30 min, daemon max iterations from 40 → 100, and per-user daemon concurrency from 5 → 10. Added exponential backoff (capped at 30 s) for transient LLM errors — 429s, 5xx, network blips no longer kill the run. Added phase-summary recycling: every 25 iterations the daemon compacts its conversation to [system, original task, phase summary] so 100-iteration runs don't blow the context window even after aggressive tool-result trimming.
• Fixed the "queued tasks never start" bug. Three independent root causes were silently swallowing tasks: (1) pending daemons were being dropped from the queue before slot acquisition succeeded; (2) a slot exhaustion deadlock had no break-out — fixed with a stuck-pending detector that marks the daemon failed and exits the loop; (3) zombie daemons surviving a backend restart were never reaped, so they occupied phantom slots forever — fixed with boot-time CleanupStaleDaemons and CleanupStaleTasks.
• Artifact isolation per orchestration. The synthesis step used to pull every artifact the user had ever produced into the final prompt, causing stale Python code to leak into edge-computing answers. Now uses ListSince(parent.CreatedAt) to scope to artifacts produced within this orchestration's lifetime.
• Synthesis resilience. Added structured error publishing (error_code, user_message, hint, internal err) so the UI can show actionable messages instead of raw stack traces. Added 30-min ceilings on resume-from-mongo contexts.

Nexus — LLM transport (Bedrock OpenAI shim)

The Bedrock OpenAI-compatible endpoint is strict about request body shape; four cooperating fixes closed a long-running 400 loop:

• cache_control removal. Bedrock rejects Anthropic-style cache_control blocks on the OpenAI endpoint.
• HTML-escape disabled on JSON encode (the default Go behavior rewrites <>& to <>&, which broke the parser on certain inputs).
• 20 KB request-body guard with aggressive tool-result shrinking when a single message body exceeds the threshold.
• Tool call ID sanitization. Bedrock rejects the functions.X:N-style IDs some providers emit; rewrites them to call_<n>.
• parallel_tool_calls=false forced — Bedrock 400s on assistant messages containing multiple tool_calls in a single turn.
• filterValidToolCalls drops tool calls whose arguments JSON was truncated mid-string by the model — echoing those back triggers "Unterminated string at column N" 400s.
• Anti-loop nudge appended 5 iterations before the cap, telling the model to write its final answer instead of burning the budget on more tool calls.

Multi-agent — what actually makes the multi-daemon path work

• Artifact handoff between daemons. Wired the artifact store + a SubagentRunner adapter onto every daemon path, taught the system prompt to use produce_artifact / list_artifacts / read_artifact for cross-daemon handoff (without this, downstream daemons only got a thin summary of upstream output).
• PDF auto-nudge. The context builder detects tasks containing pdf/report/document and biases the prompt toward an artifact-producing flow.
• Classifier bias. BuildClassificationPrompt now strongly favors MULTI_DAEMON for "X and Y"-shaped requests (research X and write Y was previously misclassified as single-daemon).
• Multi-daemon context. Each daemon now gets a terse Position: daemon N of M / Handoff: block in its system prompt — the model understands it has downstream consumers and uses the artifact tools.

Multi-user safety

• Audited isolation across the shared services. EventBus, task store, daemon pool queries, artifact store, orchestration state, WebSocket handler — all correctly scope by userId/sessionId.
• Closed the one gap that existed. DaemonPool.Cancel(daemonID) had no ownership check; any authenticated user with a daemon ObjectID could cancel another user's daemon. Added CancelForUser(ctx, userID, daemonID) that loads the daemon scoped to the user before cancelling; routed the WebSocket cancel_daemon handler through it.
• Concurrent stress test (scripts/nexus_multiuser_stress.sh): provisions N test users, fires N tasks in parallel with unique marker strings, asserts each user's result contains its OWN marker and none of the others'. Validates the entire shared-service isolation under load. 5/5 clean, ~36 s end-to-end concurrent run, zero cross-user contamination.

Frontend — Nexus UI lifecycle

• Live activity panel survives navigation. useNexusStore now wraps persist with sessionStorage, partialized to conversation (last 200), daemons, classification, missedUpdates. Switching from Chat → Nexus → Workflows → Nexus keeps the panel populated; hard reload within the tab session also restores it.
• REST rehydration on mount. Nexus.tsx fetches active daemons from GET /api/nexus/daemons on mount so even a cold reload (where sessionStorage was empty) repopulates the panel headers instantly, while the WebSocket streams live updates on top.
• Task vanishing fix. Tasks now rehydrate from REST on connect (not just from the WS session_state message, which can lag if events arrived while the WS was disconnected). Three trigger paths: connect/projectId change, document visibilitychange, and mount.
• Daemon panel. expandedColumns includes 'done' by default; extractFileLinks regex matches (api/)?files/<id> and renders green download buttons in TaskDetailPanel.
• Toast on completion. task_completed events now surface a success toast with a truncateForToast helper.

Backend — execution + headless API

• /api/nexus/run and /api/nexus/run/sync endpoints. Headless REST for firing a Nexus task: fire-and-poll vs. block-until-completion. Both honor a 30-min ceiling.
• /api/workflows/templates endpoint. Workflow template gallery backend with 5 built-in templates.
• Backup/restore admin endpoints.
• Integration test suite for nexus_orchestration_state and nexus_artifact_store. Caught a bson tag bug on NexusArtifactSummary (size_bytes / created_at were returning 0).
• CosineSimilarity precision fix. The homemade sqrt with 4 Newton iterations was imprecise; replaced with math.Sqrt — caught by unit test.
• sendUpdate panic guard. Was crashing the whole backend on a send on closed channel; now uses defer recover().

Branding

• Rose pink restored. Design tokens reverted to the original ClaraVerse palette: #e91e63 accent with hover #f06292 / active #c2185b, HSL 340 82% 52% for shadcn primary/ring, matching glow shadows and gradients. The "Emerald Edition" was an experimental detour.
• Name restored. 70 files reverted from DobbyAI → ClaraVerse across user-facing copy, meta tags, comments, log lines, and config defaults. URLs dobbyai.app → claraverse.app. The DOBBY_DF / <</DOBBY_DF>> protocol tokens between the Go runtime and the Python runner were deliberately preserved — those are a wire-format sentinel, not a brand string, and renaming them would silently break DataFrame extraction.

Scripts

• scripts/nexus_e2e.sh — single-task end-to-end tester against /api/nexus/run/sync.
• scripts/nexus_multiuser_stress.sh — concurrent multi-user isolation/contamination test.
• backend/cmd/mint_token — mints a JWT for the first mongo user (used by both scripts).

Documentation

• docs/SYSTEMS.md — ~600-line walkthrough of the Nexus pipeline: Cortex classifier → DaemonPlan DAG → DaemonRunner goroutines → synthesis. Reference for anyone debugging the multi-agent path.

---

For prior history (v0.1.x and earlier), see GitHub Releases.