inc/CH_Common/CHCharsetNew.h is the header for a new charset class that will be used across the tools in future releases. Implement as needed in src/CH_Common/CHCharsetNew.cpp. The new class should entirely use vectors for passing data about, and with the exception of the legacy functions currently needed, should not use new or delete on anything that will be returned. The vector copy constructors should be used to handle this. The charset class will handle charsets loaded from files as is currently done, with newline-separated data. However, when data is added with the addCharacterAtPosition function, null and non-ASCII characters (including newlines) should be allowed. Add appropriate sanity checks where needed - the charset class should not, for instance, allow the loading of a few million rows. Please specify these values as defines in the header file. Questions can be addressed to bitferret via whatever means are appropriate.
r164 contains the implementation of every method in the CHCharsetNew.h class. It could use some more error checking. I'll handle that later today.