#79 Macro Template is not working when used after UTF-8 two byte characters

[David Yanagaki Bonafé]

Macro Templates is not working correctly when used on a position after a UTF-8 two byte character on the source code. It was working OK on latest version published on gexperts.org site (1.38), but not anymore on 1.3.11 version from Thomas.

To reproduce:
1) Create a new VCL Forms Project.
2) Change source-code to UTF-8.
3) Double click on the form to add a FormCreate handler.
4) Paste the following declarations before the begin keyword:

const
  x = 'áéíóúýçäëïöüÿãõñàèìòùâêîôû';
  y = 'ÁÉÍÓÚÝÇÄËÏÖÜŸÃÕÑÀÈÌÒÙÂÊÎÔÛ';

5) If you try to expand any macro (eg. type begin then hit Expand Macro Template hotkey, I think the default is Alt+Shift+T) before those declarations, it works OK. If you try to expand de macro any place after those declarations, it cuts the wrong part of the code.

Expanding begin before x resulted in this:

const
  begin

  end;
  x = 'áéíóúýçäëïöüÿãõñàèìòùâêîôû';
  y = 'ÁÉÍÓÚÝÇÄËÏÖÜŸÃÕÑÀÈÌÒÙÂÊÎÔÛ';

Expanding after x, resulted in this:

const
  x = 'áéíóúýçäëïöüÿãõñàèìòùâêîôû';
  beginbegin

  end;
 'ÁÉÍÓÚÝÇÄËÏÖÜŸÃÕÑÀÈÌÒÙÂÊÎÔÛ';

[Thomas Mueller]

Works fine for me in Delphi 2007 and 10.2. Which Delphi version are you using?

(Maybe it has already been fixed in the current source code, I have not tried the release version.)

[David Yanagaki Bonafé]

It happens on Tokyo update 3 (10.2.3) with version 1.3.11 2018-08-05, which seems to be the latest version. Version 1.3.8.50 works fine.

Did you change the file format to UTF-8 (Right click on blank area > File Format > UTF8)?

My default ANSI encoding is Cp1252.

With regards.

[Thomas Mueller]

It seems to depend on the code page, that's why I could not reproduce it.
CP1252 should have been the default for my Windows installation. I could reproduce the problem only afer switching to "Japanese" (whatever that actually means), which is odd given that you expliclitly mention Cp1252.

Never mind, I can reproduce it, so I might be able to solve it.

[Thomas Mueller]

It's enough to have the following in a unit to reproduce this problem:

ÎÔÛ
b
// <- place cursor here and insert a simple macro template
e

(Yes that' s not valid code, but for the test this doesn' t matter. And of course, the comment doesn't need to be there.)

The macro template can be something really simple like "bla" -> "blabla". It will be placed behind the "b" in the same line (probably after the CR but before the LF), rather than in its own line.

If you put another character in the line between b and e, place the cursor after that character invoke the macro, it will be placed before that character rather than behind it. This proves that there is an off by one index problem.

The bug seems to be in GX_OTAUtils.GxOtaGetActiveEditorTextAsStringBytes. It uses a TStringStream to read the editor buffer and then assigns TStringStream.DataString to a (Unicode-) string. This should probably be done differently.

(But I am too tired right now to solve this.)

As stated above: I have switched Windows 8 to use Japanese, which is done in the Control Panel -> (search for) Region -> Adminstrative -> Change system locale.
(Needs a reboot)

After the change chcp on in the console window says the code page is 932. (Before it said 850 on my computer.)

[David Yanagaki Bonafé]

CP1252 are usually for locales that uses latin characters, so English and other Western European should be CP1252. In my case I use Brazilian Portuguese. To figure which CP you're using, you can call GetACP Windows API which returns the current thread CP.

Anyway, not sure if the fastest way, but this is the safest way I use to convert between Unicode and different Ansi encodings:

function UnicodeToAnsi(const Source: String; ACodePage: Cardinal): RawByteString;
var
  strLen: Integer;
  {$IFDEF CLR}
  AnsiData: TBytes;
  i: Integer;
  {$ENDIF}
begin
  Result := '';

  if ACodePage = 0 then
    ACodePage := GetACP;

  {$IFNDEF CLR}
  strLen := LocaleCharsFromUnicode(ACodePage, 0, PWideChar(Source), Length(Source), nil, 0, nil, nil);
  if strLen > 0 then
  begin
    SetLength(Result, strLen);
    LocaleCharsFromUnicode(ACodePage, 0, PWideChar(Source), Length(Source), PAnsiChar(Result), strLen, nil, nil);
    SetCodePage(Result, ACodePage, False);
  end;
  {$ELSE}
  AnsiData := Encoding.GetEncoding(ACodePage).GetBytes(Source);
  strLen := Length(AnsiData);
  SetLength(Result, strLen);

  for i := 1 to strLen do
    Result[i] := AnsiChar(AnsiData[i - 1]);
  {$ENDIF}
end;

[Ulrich Kobsa]

As it seems to be the same problem, another way to reproduce this:

Delphi 10.2.3 (German), Windows 10m, German, GE 1.3.14.80

• on IDE create a new VCL project
• add a new method 'Test' to TForm1
• place cursor on beginning of line above implementation of 'Test'
• add comment: '// Über'
• place cursor in implemntation of 'test' after Begin (see GE_bug_1)
• type 'ph' or 'xdoc' (one of them should be defined as macro template
• invoke the template
• only part of template shortcut is removed, beginning 'p' or 'x' is still there (see GE_bug_2)

[Ulrich Kobsa]

Any chance that this issue get a somewhat higher priority? At the moment this bug prevents us from using any newer GExpert version.
I tried to find a fix my myself but I'm not familiar with debugging IDE Extensions.

[Ulrich Kobsa]

Hi,

I've done some more testing and logging:

It seems that the problem is slightly different:

start with TMacroTemplatesExpert.ExpandTemplate:

here at the very beginning the template and it's positions are calculated byte-based:

GxOtaGetCurrentIdentEx(TemplateName, IdentOffset, IdentPos, CaretPos, AfterLen, True);

later when adding the template to editor with InsertTemplateIntoEditor, the template is added right, but deleting the template text fails as it tries to do that on character postitions:
GxOtaDeleteTextFromPos(ATemplateOffset, CodeLen);
But ATemplateOffset is the byte based position, not the characterbased.
Using GxOtaDeleteByteFromPos(ATemplateOffset, CodeLen); instead seems to work.
So GX_MacroTemplatesExpert.InsertTemplateIntoEditor should be changed this way:

procedure InsertTemplateIntoEditor(const ATemplateName, ATemplateCode: string;
  ATemplateOffset, AInsertOffset: Integer);
var
  CodeLen: Integer;
  InsertOff: Integer;
begin
  CodeLen := Length(ATemplateName);
  if ATemplateOffset = AInsertOffset then
  begin
    InsertOff := AInsertOffset + CodeLen;
    GxOtaInsertTextIntoEditorAtBufferPos(ATemplateCode, InsertOff);
   // Delete template code (done after the insert to avoid problems)
    if CodeLen > 0 then
      GxOtaDeleteByteFromPos(ATemplateOffset, CodeLen);  // Position is already byte based! UKO
      //GxOtaDeleteTextFromPos(ATemplateOffset, CodeLen);
  end
  else
  begin
    InsertOff := AInsertOffset;
    if CodeLen > 0 then
      GxOtaDeleteTextFromPos(ATemplateOffset, CodeLen);  // here too? UKO
    GxOtaInsertTextIntoEditorAtBufferPos(ATemplateCode, InsertOff);
  end;
end;

As I'm not very familiar with GEXperts code, can someone with more experience check this?

[Thomas Mueller]

Thanks a lot. This seems to solve the problem.

[Thomas Mueller]

fixed in revision #2837