#82 CQP can hang without output when executing queries ending in a comment

[Timm Weber]

Executing the following malformed query in CQP causes its prompt to hang without output: [pos="JJ.?]? ("#?brexit"%c) To get CQP out of this stuck state, the user has to input an additional ;\n to get an error message.

This is unexpected behaviour and has the potential to break software using CQP in child process mode, since a deadlock situation arises (both cqp and the host program get stuck on reading stdin and stdout, respectively). See also this issue (and subsequent workaround) for cwb-cc.

The underlying issue seems to be that any query containing a # character outside a string hangs indefinitely, since CQP will treat the rest of the line following the hash symbol (including the ; and newline) as a comment. Just executing # or #;is enough to replicate the issue and will also hang cqp in the same way.

Interestingly, this only seems to occur when not using input line editing (the -e flag), i.e. in cqp started without any flag or in child process mode.

[Timm Weber]

Proposed fix: Change the comment rule in the lexer to always return a semicolon character to the parser. This way comments will always act as terminators and cqp_parse_file() will no longer hang on yyparse() in the edge case described above. This comes at the cost of introducing additional semicolons and thus empty query expressions when parsing. Initial testing suggests that this does not affect CQP functionality in the slightest. In lines containing both a valid query expression and a comment (e.g. "hello"; #world) this has the minor side effect that the following CQP prompt will be printed multiple times (in non line-editing, non-child mode). This will also introduce extra virtual semicolons in macro files on lines only containing a comment.

@schtepf @andrewhardie can you think of a good reason against implementing this proposed fix?

Index: cqp/parser.l
===================================================================
--- cqp/parser.l        (revision 1915)
+++ cqp/parser.l        (working copy)
@@ -314,7 +314,13 @@
                           return(FLAG);
                         }
 {whitespaces}           ;
-#.*                     ; /* single line comment */
+#.*                     {

+                          /* single line comment */
+                          /* this rule returns a ';' so comments always act as terminators.
+                          this avoids an edge case where malformed queries could produce a comment
+                          that swallows a final ';', which leads to hangs in CQP interactive mode.*/
+                          return ';';
+                        }
 {string}                {
                           /* cut off the leading "/' and the trailing "/' */
                           yylval.strval = strip_quotes(yytext);

[Stephanie Evert]

This is not a bug, CQP behaves as intended (I refrain from writing “documented” since we don't have a precise specification of the query language). In normal mode, a CQP command can span multiple lines until the terminating semicolon is encountered, and each of these lines can include a comment. So the example [pos="JJ.?]? ("#?brexit"%c) is parsed as [pos="JJ.?]? (", i.e. an incomplete command. It is correct behaviour for CQP to wait for additional input until a semicolon or parse error is encountered. In readline mode (-e), CQP requires each command to be on a single line, so it will signal an error for an incomplete command.

For this reason, this should be treated as an enhancement request rather than a bug. I'll comment separately on why the proposed solution doesn't work.

[Stephanie Evert]

As for the proposed enhancement (not bugfix!), this would break backwards compatibility. Comments in mutliline CQP commands have been allowed for decades, so it would be unacceptable to disable them now. As a start, this would break the first example in Sec 6.4 of the CQP User Manual.

I would argue that no such enhancement is needed:

• End users can simply complete the command on the following line (but would hopefully use readline mode anyway). It's the same as the continuation prompt for an incomplete Python command.
• Programming interfaces using CQP in child mode should take care to execute commands in a safe manner. I believe that CWB/Perl does this correctly by sending the string "$cmd;\n .EOL.;\n to CQP. Though another \n might be needed to safeguard against all possibilities.
• A programming interface that doesn't handle edge cases properly is just as likely to get confused by the extra blank line printed due to the stray semicolon.

If there is strong demand for such an enhancement, the best solution seems to be to introduce an option that requires commands to be on single lines (i.e. make CQP child mode behave as if in readline mode). This would be off by default but could be activated by a programming interface, which then has to preprocess all CQP commands to remove linebreaks.

[Timm Weber]

Thanks for the feedback, I didn't take into consideration that multi-line queries with interspersed comments are a valid use-case in interactive mode, although I've already gotten a hunch about the special nature of -e-mode after poking around in the code earlier. Guess I still have to learn more about CQP's arcana.

I agree that this is not a bug then and just some (annoying) expected behaviour instead. My proposed fix will definitely not work in this case.

Considering this new info I think it's enough to document the edge case/add info about best practices to the manual for child mode users (maybe a recommended query epilogue that guarantees return?). I've already implemented a fix containing a similar epilogue in cwb-ccc that successfully got rid of this edge case.

I think this ticket can be closed and marked as "won't fix".

[Stephanie Evert]

Admittedly, I just had to fix this in CWB/Perl, which did indeed hang, too … The proper string to send to CQP is "$cmd\n;\n.EOL.\n"