#4 Wiki Indexer and Retriever

[Anonymous]

Originally created by: alirezamshi
Originally owned by: alirezamshi

Description

Given wiki files in data/wikis/, we need an indexer and retriever mechanism to efficiently navigate the wiki content.

Current state:

• Initial code exists at src/knowledge/search/
• Current implementation assumes input is JSON with nodes and edges structure
• Uses Neo4j for graph storage and LLM-guided navigation

Problem:

• Signatures and code structure need to change to work with wiki files (.mediawiki format)
• Need to parse wiki files and extract nodes/edges from semantic links
• Should be packaged as a tool for coding agents

Goal

Build an indexer and retriever that:

1. Reads wiki files from data/wikis/
2. Parses .mediawiki files to extract nodes and semantic relationships
3. Indexes content for efficient retrieval
4. Provides a clean API for LLMs and coding agents

Steps

Step 1: Define Signatures

Define clear interfaces for:

• WikiIndexer - Parses and indexes wiki files
• index_directory(path: str) - Index all wiki files in a directory
• index_file(path: str) - Index a single wiki file

• clear() - Clear the index

• WikiRetriever - Retrieves relevant wiki content

• search(query: str, top_k: int) - Search for relevant pages
• get_page(page_name: str) - Get a specific page
• get_related(page_name: str) - Get related pages via edges

Step 2: Implement Structure

Update src/knowledge/search/ to:

1. Parse .mediawiki files and extract:
2. Page metadata (type, name, repo)
3. Content sections
4. Semantic links ([[step::...]], [[consumes::...]], etc.)
5. Build graph from extracted relationships
6. Index for both keyword and semantic search

Step 3: Package as Tool

Create a tool in src/execution/ that coding agents can call:

• Tool name: wiki_search or knowledge_retrieve
• Input: query string, optional filters (page type, repo)
• Output: relevant wiki content formatted for LLM consumption

Deliverable

1. Updated src/knowledge/search/ with wiki-native indexer/retriever
2. New tool definition in src/knowledge/search/ for agent use
3. Clear documentation

Test

1. Place sample wiki files in data/wikis/
2. Run indexer to build the index
3. Query via retriever and verify relevant results
4. Test tool call from a coding agent

References

• src/knowledge/search/ - Current search implementation
• src/knowledge/search/kg_llm_navigation_search.py - LLM-guided navigation logic
• src/knowledge/search/base.py - Base classes
• src/knowledge/search/archive_product_code/ - Existing product code for these operations
• parser.py - Wiki parsing logic
• indexer.py - Indexing implementation
• search.py - Search functionality
• update.py - Update operations
• editor.py - Editor functionality
• wiki_tools_interface.py - Tool interface for agents
• data/wikis/ - Wiki file storage
• src/execution/ - Coding agent execution

Dependencies

• dependent to the structure of wiki in [#2]

[Anonymous]

• assigned_to: alirezamshi

[Anonymous]

Originally posted by: alirezamshi

building indexer and retriever based on [#31]

[Anonymous]

Originally posted by: alirezamshi

implemented in [#38]

[Anonymous]

Ticket changed by: alirezamshi

• status: open --> closed