Getting a SYNTAX ERROR in recent builds

[Dan Kalowsky]

Hi,

I recently asked about this on the IRC channel but really got not much of a response. We've been using cppcheck for a few years now, and have held on to version 1.81 for a little too long. Moving to update to the latest version, and we're seeing cppcheck consistently reporting back a syntax error in our code. This is odd because the code compiles fine on clang, gcc, and MSVC and in earlier versions of cppcheck. What I'd like to know is, how I can get more details as to what cppcheck thinks is the syntax error?

The exact error looks like:
/test/cppunit_util.h:23:1: warning: syntax error [syntaxError]

^

Which isn't terribly helpful. So, sharing the inside of the file might help:

#ifndef CPPUNIT_UTIL_H_
#define CPPUNIT_UTIL_H_

// Long description here...

template <typename T0, typename T1, typename T2, typename T3>
bool cu_eq(const std::pair<T0, T1>& pair0, const std::pair<T2, T3>& pair1)
{
  // code goes here
}

// more code here

Line 23:1 is template. Which as far as I can tell is okay C++. This can be confirmed in version 1.82 which gives a little more detail on the error line:

/test/cppunit_util.h:23:0: warning: syntax error [syntaxError]
template <typename T0, typename T1, typename T2, typename T3>
^

We run the tool with the following configs:
--std=c++14 --language=c++ --inline-suppr --force --enable=warning --platform=unix64

Having now run a git bisect 1.81 1.87 it's narrowed down to the commit fc1ac180e6bf2207e94e31f119e8f3f2c479b5f4.

All builds were done with a small variation on the config provided on the cppcheck website:
make clean && make SRCDIR=build CFGDIR=cfg HAVE_RULES=yes CXXFLAGS="-O2 -DNDEBUG -Wall -Wno-sign-compare -Wno-unused-function" -j8

Is there any way to get more detail as to what cppcheck does not like about the syntax so that we may move forward in using the latest version? The same issue persists with HEAD as of commit 4939e0c30

[versat]

There is not always someone in the IRC chat, it is good to post it here. Thanks!
I can not reproduce this with the current head. Not even when compiling Cppcheck with -O2.
Can you try to find a reduced but complete code example that throws the syntax error? Ideally g++ -fsyntax-only code_example.cpp would not print any errors.
I just tried it with this code which might not be enough to trigger the syntax error:

#include <utility>

template <typename T0, typename T1, typename T2, typename T3>
bool cu_eq(const std::pair<T0, T1>& pair0, const std::pair<T2, T3>& pair1)
{
  // code goes here
}

Output:

$ g++ -fsyntax-only template_syntax_error1.cpp && ./cppcheck --std=c++14 --language=c++ --inline-suppr --force --enable=warning --platform=unix64 --debug template_syntax_error1.cpp
Checking template_syntax_error1.cpp ...

##file template_syntax_error1.cpp
3: template < typename T0 , typename T1 , typename T2 , typename T3 >
4: bool cu_eq ( const std :: pair < T0 , T1 > & pair0@1 , const std :: pair < T2 , T3 > & pair1@2 )
5: {
6:
7: }

##Value flow

[David Brady]

Just wanted to say that I am seeing the same problem in the latest head. I removed the validation code in lib/tokenize.cpp (under the //garbage templates comment), and it appears to have resolved the problem for us locally (at a glance, it seemed to be just handling the case of invalid templates, but doesn't work properly). This code caused cppcheck to throw a syntax error (but oddly not in the last official release):

template <> void f<std::string>(std::string const& x) {}

[versat]

I have tried to reproduce it (with current head), but i do not see a syntax error with this command:

$ ./cppcheck --enable=all template_syntax_error.cpp
Checking template_syntax_error.cpp ...
[template_syntax_error.cpp:1]: (style) The function 'f < std :: string >' is never used.

Maybe it depends on the parameters? How do you check it?

[Robert Reif]

Can you try with 1.88 or latest head? There are a lot of template fixes in 1.88 and evem more in current head.

[Richard Smith]

I'm getting this as well using d4d9bb4830a9198c161e82ef81dd1d0d6c6cef92

The code it referencing is:

        wsble_to_nordic_cmd_header_t to_nordic_go_connectable_cmd =
        {
            .command            = WSBLE_TO_NORDIC_CMD_SET_CONNECTABLE,
            .command_payload[0] = con_evt->on,
            .command_payload[1] = 0x00u
        };

[Daniel Marjamäki]

Thanks Richard! I can reproduce a "syntax error" for that code and it looks valid as far as I see.

[Dan Kalowsky]

Just tried rebuilding with commit 413a5a486 still receiving the same error. I'm fairly certain the actual line of code being reported is not where the error is occuring, but I haven't a good guess on how to go any further on debugging. More importantly, nor do I know how to create a smaller test case because of that limitation.

[Daniel Marjamäki]

I believe we need to clarify our "syntax error" messages. So users have a chance to understand what we are complaining about and can clearly tell us if that is a bug in Cppcheck or not.

[Daniel Marjamäki]

but I haven't a good guess on how to go any further on debugging.

could you compile latest cppcheck and debug it? If you set a breakpoint in Tokenizer::syntaxError. Where is that called from?

[Daniel Marjamäki]

Richard smith: I fixed that syntax error with 64ef879ebfe098b811ccb2eba1e92521212bf040

[Richard Smith]

I can confirm that latest HEAD fixes my syntax error.

However, the removal of the -P option from the misra command line completely breaks our setup.

[Dan Kalowsky]

@danielmarjamaki I'd agree on the clarity on "syntax error" messages. With that said, I did try a little of the debugging as request. Alright let's start with a little of the background data here. Directory layout looks like:

README.md
cppunit_util.h
pch.cpp
pch.hpp
p_tests.cpp
p_tests.h
s_tests.cpp
s_tests.h
v_tests.cpp
v_tests.h

I'll list the full backtrace first, with a little of my own brief dive into the debugging that I got to do on this today. All code was built in debug configuration from hash 421a8da6a.

First, the backtrace. Draw your own conclusions from it here....

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000100631c19 cppcheck`Tokenizer::syntaxError(this=0x00007ffeefbf4d98, tok=0x0000000103245f30, code="") const at tokenize.cpp:8582
    frame #1: 0x000000010069e019 cppcheck`Tokenizer::findGarbageCode(this=0x00007ffeefbf4d98) const at tokenize.cpp:9308
    frame #2: 0x0000000100660cbe cppcheck`Tokenizer::simplifyTokenList1(this=0x00007ffeefbf4d98, FileName="p_tests.cpp") at tokenize.cpp:4181
    frame #3: 0x000000010065ee52 cppcheck`Tokenizer::simplifyTokens1(this=0x00007ffeefbf4d98, configuration="") at tokenize.cpp:2284
    frame #4: 0x0000000100343195 cppcheck`CppCheck::checkFile(this=0x00007ffeefbfd6f8, filename="p_tests.cpp", cfgname="", fileStream=0x00007ffeefbfb138) at cppcheck.cpp:506
    frame #5: 0x0000000100336da0 cppcheck`CppCheck::check(this=0x00007ffeefbfd6f8, path="p_tests.cpp") at cppcheck.cpp:190
    frame #6: 0x000000010002c34f cppcheck`CppCheckExecutor::check_internal(this=0x00007ffeefbfe1f8, cppcheck=0x00007ffeefbfd6f8, (null)=8, argv=0x00007ffeefbfe4b0) at cppcheckexecutor.cpp:884
    frame #7: 0x0000000100028d36 cppcheck`CppCheckExecutor::check(this=0x00007ffeefbfe1f8, argc=8, argv=0x00007ffeefbfe4b0) at cppcheckexecutor.cpp:198
    frame #8: 0x00000001000012bf cppcheck`main(argc=8, argv=0x00007ffeefbfe4b0) at main.cpp:95
    frame #9: 0x00007fff6f5483d5 libdyld.dylib`start + 1

Now the minor investigations of someone not at all familiar with the source code. Take any comments/conclusions as probably wrong. The parsing of the pch.hpp file seems to go correctly, as nothing gets triggered there. It's when moving into the second file listed p_tests.cpp that the error occurs. The p_tests.cpp file begins with:

// bunch of comments here
#include "pch.hpp"

#include <cppunit_util.h>
#include <p_tests.h>
// other includes and source goes here

Which leads to the eventual syntax error from cppunit_util.h. First thing to stand out, looking at frame 3, is the print out of list.getFiles() :

(lldb) p list.getFiles()
(const std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >) $5 = size=5 {
  [0] = "p_tests.cpp"
  [1] = "pch.hpp"
  [2] = ""
  [3] = "cppunit_util.h"
  [4] = "p_tests.h"
}

Not sure if the blank line is supposed to be there or not. The main issue seems to be coming from the following path. Condensed a bit for brevity and posting.

Process 93972 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x000000010069e3d6 cppcheck`Tokenizer::findGarbageCode(this=0x00007ffeefbf4d98) const at tokenize.cpp:9304
   9301 
   9302     // Garbage templates..
   9303     if (isCPP()) {
-> 9304         for (const Token *tok = tokens(); tok; tok = tok->next()) {
   9305             if (!Token::simpleMatch(tok, "template <"))
   9306                 continue;
   9307             if (tok->previous() && !Token::Match(tok->previous(), "[:;{})>]"))
Target 0: (cppcheck) stopped.

(lldb) p *tok
(const Token) $190 = {
  mTokensFrontBack = 0x00007ffeefbf4d98
  mStr = "END_ALL_WARNINGS_OFF"
  mNext = 0x0000000103d02e50
  mPrevious = 0x0000000103d0a060
  mLink = 0x0000000000000000
  mTokType = eName
  mFlags = 1048576
  mImpl = 0x0000000103894b70
}

In this case the mStr is showing a preprocessor macro defined for use from the pch.hpp, which is actually defined in some other header file for us (in this case clang.h, as it's a cross-platform product). The path taken is !simpleMatch evaluates true, and the continue is selected. Which brings the process to the next token...

Process 93972 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step over
    frame #0: 0x000000010069de75 cppcheck`Tokenizer::findGarbageCode(this=0x00007ffeefbf4d98) const at tokenize.cpp:9305
   9302     // Garbage templates..
   9303     if (isCPP()) {
   9304         for (const Token *tok = tokens(); tok; tok = tok->next()) {
-> 9305             if (!Token::simpleMatch(tok, "template <"))
   9306                 continue;
   9307             if (tok->previous() && !Token::Match(tok->previous(), "[:;{})>]"))
   9308                 syntaxError(tok);
Target 0: (cppcheck) stopped.

(lldb) p *tok
(const Token) $191 = {
  mTokensFrontBack = 0x00007ffeefbf4d98
  mStr = "template"
  mNext = 0x0000000103cd07f0
  mPrevious = 0x0000000103d0e600
  mLink = 0x0000000000000000
  mTokType = eName
  mFlags = 1048576
  mImpl = 0x000000010388a5d0
}

At this point the !simpleMatch evaluates false, and moves on to Token::Match. The tok->previous() is the same as $190, which again is a macro. This eventually comes to fail at the following section:

frame #0: 0x000000010061329a cppcheck`Token::Match(tok=0x0000000103d0e600, pattern="[:;{})>]", varid=0) at token.cpp:654
   651          // [.. => search for a one-character token..
   652          if (p[0] == '[' && chrInFirstWord(p, ']')) {
   653              if (tok->str().length() != 1)
   654                  return false;
   655  
   656              const char *temp = p+1;
   657              bool chrFound = false;
(lldb) 

The tok->str().length() != 1, as the value is "END_ALL_WARNINGS_OFF" and has nothing to do with a template.

EDIT: forgot this part. The END_ALL_WARNINGS_OFF macro evaluates out to the following:

#define END_ALL_WARNINGS_OFF _Pragma("clang diagnostic pop")

Hope that helps some.