CPD - Tokenizer not initialized with requested properties
A source code analyzer
Brought to you by:
adangel,
juansotuyo
Menu
▾
▴
#1114 CPD - Tokenizer not initialized with requested properties
Executing CPD by command line or ant task does not propagate the tokenizer options (ignoreIdentifiers/ignoreLiterals/ignoreAnnotations). I have tried multiple versions, and it appears to have quit working in 5.0.0 (it works in 5.0.0-alpha).
I took a quick look at the code of 5.0.4. The LanguageFactory for cpd has a method which takes a Properties object (public Language createLanguage(String language, Properties properties)), but it never actually uses the Properties object. For this reason the language tokenizer CPD uses always sets ignoreLiterals/ignoreIdentifiers/ignoreAnnotations to false.
See also the discussion here: http://sourceforge.net/p/pmd/discussion/188193/thread/91553283
I could reproduce this problem. We set the properties too late, after the tokenizer has been already initialized.
I think it's worth a 5.0.5.