#26 Files containing ascii8 are not indexed (feature/request)

[Anonymous]

Originally created by: kilve...@gmail.com

What steps will reproduce the problem?
1. File containing: PAT and ä (0xE4, a umlaut), ö (0xF6,o umlaut), other high bit ascii chars
2. cindex -reset .
3. csearch -i PAT

What is the expected output? What do you see instead?
Expected to find PAT. Instead no match.

What version of the product are you using? On what operating system?
codesearch-0.01-windows-amd64.zip

Please provide any additional information below.
I looked at the source (write.go) and it seems to expect that files are in UTF-8 only (this is a Go specification?). However it would be nice if csearch could be used with any source files, including those with high bit ascii characters. Or that there would be a command line option for this.

[Anonymous]

Originally posted by: kilve...@gmail.com

But, i don't see a log message about skipping the file on the console. cindex run looks normal.

[Anonymous]

Originally posted by: manpreet...@gmail.com

I encountered all these issues you mentioned and was annoyed enough by them to implement the following changes for myself at https://github.com/junkblocker/codesearch

1) Do not stop at first bad UTF-8 character encountered. Instead allow a percentage of non-UTF-8 characters to be in the document. These are ignored but the rest of the document gets indexed. The option, which I call, -maxinvalidutf8ratio, defaults to 0.1. This combined with considering a document containing a 0x00 byte as binary has been working great for me.

2) Allow custom trigrams size. The current hardcoded limit is at 20000 trigrams but I sadly have to work on code with one important source file beyond that. (-maxtrigrams).

3) Add message and reasoning for every document skipped from indexing.

I would love to get those changes merged or at least considered for alternate implementation here in this official sources but am not sure about the aliveness of project here.

[Anonymous]

Originally posted by: rsc@golang.org

The project is not super alive. Mostly the code just works and we
leave it alone. I think the UTF-8 heuristic works pretty well as does
the trigram size heuristic. It's possible to tune these forever, of
course. How many trigrams does your important file have?

I thought that the indexer already did print about files it skipped if
you run it in verbose mode, but maybe I am misremembering.

[Anonymous]

Originally posted by: manpreet...@gmail.com

All source files being UTF-8 is a pretty big assumption. A lot of files may be latin-1 etc.  which is the most common problem I encountered. Having random european author's name with a diacritic in the source or some cyrillic, for example, loses a whole file from index making codesearch something that can't be depended on at all. When I am changing code based on what codesearch finds in my codebase, I don't wanna miss some files for this reason. codesearch should not be less reliable that a regular grep.

The file I mentioned is around 30K trigrams. It was simple to just add a custom limit flag.

The indexer misses the warning in a couple of places mainly because of the assumptions it makes about the input data. The one example I recall off the top of my head is about quietly ignoring symlinked paths (which I submitted another patch to optionally not ignore for).