#15 Add support for grammar based tokenizers

[_duSt_]

Currently files are only tokenized using a simple whitespace/separator tokenizer. In order to provide additional search functionality, source code files should be parsed using grammars and recognized elements of certain types should be indexed separately:

• Type definitions
• Struct definitions
• Class definitions
• Variable definitions
• Member definitions
• Parameter definitions
• Function definitions
• Method definitions
• Namespace definitions
• Class usages
• Variable usages
• Member usages
• Parameter usages
• Function usages
• Method usages
• Literals
• Constants
• Statements
• Keywords
• Expressions
• Operators
• Assignments
• ...

This will enable advanced search capabilities, as search queries may then contain generic placeholders to find certain code constructs.
In addition to this this additional analysis can be used for better syntax highlighting.

In order to implement this functionality the .NET Irony library can be used, which allows construction of parsers at runtime. Grammar based tokenizers, should be configurable similar to highlighters through xml files.