19-Knowledge-Pack

Mirror of the public Nokido wiki (source of truth: https://github.com/Nokido-labs/nokido/wiki/19-Knowledge-Pack). Edits belong there.

19 — Knowledge Pack

Updated: 2026-10-07

🌐 English · Français

Status (2026-10-07) : the export / import tools work on a local file and the release builder now
produces the pack ; no pack has been published yet (no GitHub Release carries one).

The Nokido Knowledge Pack is an optional, pre-vectorized snapshot of
the Nokido source code chunks (modules forge_*.py, docs, curated
skills) shipped as a single NPZ file with BGE-M3 1024D embeddings already
computed. Fresh installs (and forks) can import it in one command and
get instant semantic search over the Nokido codebase without running
the local embedder on cold chunks.

🎯 Why ship a knowledge pack?

A fresh clone only has the text seeds (seed/*.jsonl, 11 files on 2026-09-29).
Those seeds cover lessons, ADRs, biblio, system
rules — but not the Nokido code itself. The user has two options
out of the box :

• Wait for the local embedder (NokidoLlamaEmbed, :8099 — the old brain_worker :5557 is
  disabled) to vectorize the forge modules : 586 app/forge_*.py + 840 tools/forge_*.py
  on 2026-09-29.
• Run BM25-only semantic search (text keyword match, no embedding
  similarity). Loses 60% of the value of the Nokido anti-duplication
  workflow described in CLAUDE.md §3.

The knowledge pack ships the embeddings pre-computed, so the user gets
(1) zero waiting time, (2) immediate semantic search over the Nokido
codebase, (3) instant ability to run the anti-duplication rule before
creating a new forge_*.py.

Forks especially benefit : they inherit the same vectorized base. If
they add their own modules, those get embedded locally on top of the
pre-shipped pack.

📦 What's in the pack

One rule decides what enters the pack : a chunk is kept only if its text (whitespace
normalized) appears word for word in a file published in the public repository, read at the
promoted commit. Its content is therefore already public : nothing private can leak, even if a
chunk's source label is wrong, and nothing stale gets in. On top of that, a chunk is dropped if it
matches a secret pattern, a private identity, or a RED signal of the egress gate (public profile).
The pack is licensed like the repository, AGPL-3.0-or-later.

Format nokido-knowledge-pack/2 (NPZ archive, no pickled objects : the importer loads it with
allow_pickle=False, so a downloaded pack can never run code) :

Field	Type	Description
embeddings	float16[N, 1024]	BGE-M3 dense vectors (read back as float32)
meta	uint8 (UTF-8 JSON)	manifest, chunk_ids, sources (published path), domains
texts	uint8 (UTF-8 JSON)	the chunk texts, verbatim

Domains read : nokido_doc, doctrine, forge_core, nokido_code, docs, policy_rules,
policy_roles, policy_security. Memory, sessions, third-party documentation and watch material
are never read.

📊 Pack size (measured 2026-10-07)

Against the public repository at 51d10ef : 29,695 chunks, 27.8 MB of text and about
30 MB of float16 vectors before compression. forge_knowledge_pack_export.py --dry-run prints
the real count per domain (read / kept / dropped per reason) before anything is written. The
vectors are BGE-M3 : the pack needs the pinned bge-m3-Q8_0.gguf embedder (pack local-llm) to
answer queries.

🚀 Export workflow (maintainer-side)

# 1. Make sure your local RAG base is populated and embedded
#    (i.e. you've been using Nokido for a while)
python tools/forge_knowledge_pack_export.py --dist <public repo clone> --verbose --dry-run
# Shows stats : read / kept / dropped per reason, per domain

# 2. Generate the pack (the release builder does this with --with-assets)
python tools/forge_knowledge_pack_export.py --dist <public repo clone> --version 0.1.0
# → writes data/nokido_knowledge_pack_v0.1.0.npz
# → prints SHA256 (record it!)

# 3. Attach to GitHub Release
gh release create v0.1.0 \
    --title "Nokido v0.1.0 + Knowledge Pack" \
    --notes-file CHANGELOG.md \
    data/nokido_knowledge_pack_v0.1.0.npz

The exporter is idempotent — re-run it any time you want to refresh the
pack with newer code chunks. Each release ships its own version.

📥 Import workflow (user-side)

Option A — Latest pack from GitHub Releases (one-liner)

python tools/forge_knowledge_pack_import.py --download

Downloads latest release asset, verifies SHA256 (manifest-embedded),
imports into the local RAG base. Its default repository is Nokido-labs/nokido, the
public showcase that carries the releases ; until it is public, use Option C.

Option B — Specific version

python tools/forge_knowledge_pack_import.py --download --version 0.1.0

Option C — Local file (air-gapped or dev)

python tools/forge_knowledge_pack_import.py \
    --from data/nokido_knowledge_pack_v0.1.0.npz

Option D — Verify before importing

python tools/forge_knowledge_pack_import.py --download \
    --expect-sha256 abc123def456...

🧪 What happens on import

Per chunk in the pack :

State in local DB	Action
Chunk doesn't exist	INSERT (text + embedding + metadata)
Chunk exists, embedding NULL	UPDATE embedding only (keep local text changes)
Chunk exists, embedding present	SKIP (don't overwrite local) — use --overwrite to force

The importer preserves any local modifications you've made — your forked
edits to forge_*.py won't be reverted by the pack.

After import, the hub's index needs a reload. The importer calls POST /api/rag/reload,
but the current hub exposes no such route (checked 2026-09-29) : restart the hub
after an import.

🔄 Updating the pack

Maintainers re-export and re-release periodically (suggested cadence :
once per minor version bump). Users re-import to get newer chunks :

python tools/forge_knowledge_pack_import.py --download --overwrite

The --overwrite flag is needed only if you want upstream chunks to
replace your locally-edited versions. Default behavior preserves local
edits.

🛡️ Security

The pack contains :

• ✅ Nokido source code chunks (the same text as the repository).
• ✅ Pre-computed BGE-M3 embeddings (one-way representation, can't
  reconstruct original text exactly).
• ❌ NO conversation logs, NO agent_messages, NO sessions, NO secrets,
  NO PII.

The export sanitization is append-only — chunks containing secrets
are dropped (not redacted) to err on the safe side. The SHA256 in the
manifest lets users verify they got the file the maintainer signed.

❓ FAQ

Q. Why not ship the pack inside the repo (data/ directory)?

A pack weighs several MB and grows with the code : too heavy for git history.
GitHub Releases is the idiomatic distribution channel for "binary blob attached to a
tagged release", and keeps the repo lean for users who only want the source.

Q. Why not host on HuggingFace Datasets?

Both work. GitHub Releases is simpler (no separate account needed,
versioned alongside the code) but HF Datasets has better
discoverability. We may publish a mirror on HF eventually — open an
issue if you want to drive that.

Q. Can I fork and ship my own knowledge pack?

Yes — re-run forge_knowledge_pack_export.py on your fork's RAG,
attach to your fork's GitHub Release. The importer --repo flag lets
users target any fork.

Q. Will the pack ever contain user data?

No. The export filter explicitly drops agent_messages,
conversation_log, shared_prompt_log, and any chunk matching PII
heuristics. The only domains exported are nokido_code,
nokido_docs, curated_skills, policy_rules — all source-code-
equivalent content.

Q. What if the BGE-M3 model version changes upstream?

The manifest pins model: "BAAI/bge-m3". If you use a different
embedder locally, the imported vectors won't align with yours.
There is no one-shot rebuild flag today (forge_embed_auto_trigger.py only takes
--max-passes) : plan a re-embed with your local model before mixing vectors from two
embedders.

Q. Can the pack be malicious?

Only via the text content (Nokido source code chunks). The
embeddings are just numpy float32 arrays — no executable payload
possible. Verify SHA256 against the manifest before importing, and
only download from the project's official release page.