This looks a lot like [#3732] but I'm running 5.1.0 so that's fixed according to the changelog.
Edit mode=shellscript, enter the following text:
echo escaped quotes don\'t count echo do they
and it will look like the ' opens a string, even though the backslash-as-escape does escape it on the command line.
Help|About says: jEdit 5.1.0 server mode, using Sun Microsystems inc Java 1.6.0_27.