Search Results for "token"

This tool performs token-by-token alignment of two versions of a text with differing tokenization by interpreting the results of a file diff (https://docs.python.org/3/library/difflib.html). It is intended for use in the preparation of annotated linguistic corpora, where differences in tokenization may arise (i) following corrections or modifications to the source text or (ii) through the creation of different layers of annotation (part-of-speech, treebank) requiring different tokenization. ...

This library provides an implementation of tables indexed by strings. The binary search is used for names of known length. It is also possible to search a table for names of unknown length, i.e. to parse a string using some table. Table elements can be of any private type. Key- insensitive tables are supported.

THIS PROJECT MIGRATED TO https://gitlab.com/mwetoolkit/mwetoolkit3/ The Multiword Expressions toolkit aids in the automatic identification and extraction of multiword units in running text. These include idioms (kick the bucket), noun compounds (cable car), phrasal verbs (take off, give up), etc. Even though it focuses on multiword expresisons, the framework is quite complete and can also be useful in any corpus-based study in computational linguistics. The mwetoolkit can be...

...After above sentences be segmented by wordseg, the result as follows for each sentence: 张晓晨[PER] 和 付仲恺[PER] 一起 坐 在 家 （ 西坝河东里社区[LOC] ） 里 的 沙发[PDT] 上 看 非 诚 勿扰 。 百度公司[ORG] 的 名字 源于 “ 众 里 寻 他 千百度 ” 这 诗句 。 In above, if a token has some attributes, the attribute result will be appended into the corresponding token within "[]". Since wordseg has introduced statistics model to segment text by context, for same sub string in different context, dif

CFD scientific software. Provides simulation of 3D CFD problem based on Quasi-Gas Dynamics approach with radiation processes token into account. Lots of spectral intervals are considered. Parallel MPI/OpenMP-based implementation is provided.

Token server reference implementation for the TorBank anonymous virtual currency system, useable via TOR onion routing or the WWW. Includes the bank-network backend server, a web frontend for end users, and a library for payment and online transactions.