#1338 Bash lexer does not properly handle single quoted-strings

[Anonymous]

Problem:
If a singly-quoted string in a shell script ends with a backslash, everything is highlighted as a quote until another single quote is encountered.

Cause:
Syntax highlighter is interpreting escaped characters inside of single quotes when it should not.

To reproduce:
Creat a new file "test.sh", with an editor that uses Scintilla.
Add the following line to test.sh:
echo 'test \' this is bad!

"this is bad!" will be highlighted, but it should not be.

Solution:
Handle singly-quoted state uniquely, ignoring all characters except for ' and dropping state upon encountering it.

Bug exists on CVS repo as of 3/28.
I have attached a patch that fixes this bug. The patch should be applied from the top directory of the 'scintilla' module.

[Anonymous]

patch generated by cvs diff -u

[Anonymous]

I first submitted this bug to geany, but found that this was the more appropriate location.

[Anonymous]

picture of the bug in action

[Anonymous]

the same example after patching

[Marko Njezic]

The provided patch breaks ANSI C quoting in Bash ($'string'), which does support backslash-escaped characters.

Also, CVS is not used anymore. Current code resides in Mercurial repository.

[Kein-Hong Man]

Yes, this is a highlighting bug. I'm the fella who missed it; bad, bad test cases... :-)

Appears we also need to switch the highlighting for $'string' strings to SCE_SH_STRING. This preserves c-style escapes for $'string' strings.

line 516-517 should be set to:
} else if (sc.ch == '\'') {
sc.ChangeState(SCE_SH_STRING);

I have tested your patch and it works for me; LexBash hasn't changed much since the move to hg. If you could update the patch (pull from hg if possible), then Neil can check it in. Thanks for the report.

[Manbeardo]

Okay, I pulled the hg repo, applied my patch with khman's fix and made a patch with hg. I also rediscovered my old SF account and appear to have broken the one I submitted the bug with, so I put the patch on pastebin:
http://pastebin.com/XWZAyKdj

[Manbeardo]

Found another lexing bug. The following BASH outputs a closed paren:

TEST="$(echo "echo )")"; echo $($TEST)

But this code traps the lexer in the SCE_SH_BACKTICKS state.
This problem and a few others could be resolved by using a quote stack instead of tracking a single quote. I'd be interested in reworking LexBash.cxx to do that. Where should I ask for a go-ahead before doing that?

[Kein-Hong Man]

The latest patch looks okay to me.

I am to blame for a lot of the more recent stuff in LexBash.cxx, so I'm interested in helping to test your code. Send it to me or we can discuss it in the scintilla-interest list.

Nested quoted pairs is a known bug (SF ticket #1515556). So far I have avoided implementing it. I went as far as test a screenful of test cases on bash, then gave up. :-) I recall being terribly confused by backquote behaviour.

[Kein-Hong Man]

Sorry, make the last sentence: "I recall being terribly confused by backslash
behaviour". Probably it also has to do with the exact way the parsing and evaluation is done. I guess I could have checked how the bash maintainers did it in the bash sources, but by then my patience had already run out. Too much of jury-rigged syntax for me to swallow... :-)

[Neil Hodgson]

Committed the patch on pastebin. The patch isn't in the correct format for easy importing with metadata as is normally produced with the 'hg export' command. So the import was edited a bit.

Here's an example patch with working metadata:
http://pastebin.com/bd90DQPK

[Kein-Hong Man]

The nested delimiter pair issue can be continued on the scintilla-interest list or the SF ticket at:
https://sourceforge.net/tracker/?func=detail&aid=1515556&group_id=2439&atid=102439
so that the original issue can be closed as fixed later.
I do think the nested thingy is worth doing, even a partial implementation that covers most reasonable code people write.

[Kein-Hong Man]

Posting this here because SF did not have a valid e-mail address for Manbeardo.

I am currently testing the behaviour of bash delimiter pair nesting. Only $(cmd) and `cmd` allow multi-level nesting. Still need to study the details, because for example, if $(cmd) has a 'literal' string within it, parentheses need not balance, and so on.

Will probably start coding this weekend. If you already have code written, let me know, I can help test. If not, then I'll get it sorted out. Any new developments on this can be continued on the other tickets or a new ticket. Thanks.