Search Results for "token"

...Support Chinese word segmentation (concurrent word segmentation using the sego word segmentation package, speed 27MB/sec). Support to calculate the proximity distance of keywords in the text (token proximity). When a request to add a document to the index comes in, the main coroutine will send the text to be segmented to a word segmentation coroutine through a channel, and the coroutine will segment the text and send it to a word segmentation through another channel. Indexer coroutines. The indexer coroutine builds an inverted index from the search keyword to the document, and the inverted index table is stored in memory for quick recall. ...