FlightGear doesn't understand/respect UTF-8.

FlightGear Flight Simulator: free open-source multiplatform flight sim

Brought to you by: curt, jmturner, stuartbuchanan, torstendreyer

#2512 FlightGear doesn't understand/respect UTF-8.

Milestone: 2020.4

Status: Fixed

Owner: James Turner

Labels: None

Priority: Medium

Updated: 2021-02-07

Created: 2021-01-11

Creator: Megaf

Private: No

Hi all, So, I was trying to help the dev of the new 787-family model.
https://github.com/IskenderWang/787-family/blob/master/Models/Instruments/PNEU.xml

Hi all, I'm trying to help @skenderWang with an encoding issue here for the 787. And basically I learned that FG doesnt seem to support UTF-8 nor ISO-8859-1. Even if I specify <?xml version="1.0" encoding="ISO-8859-1"?> or <?xml version="1.0" encoding="UTF-8"?>. I know the header is parsed and interpreted correctly, but the encoding still broken, showing Â°C when it should show °C. This is the xml line in question. <format type="string">%3.0f°C</format> In the screenshot we see the actual artifact.

1 Attachments

fgfs-20210111013048.png

Discussion

James Turner - 2021-01-11

At least text animations and Canvas text should be able to do full UTF-8, if the font contains suitable glyphs. PUI and 2D panels are ASCII only since they use TXF / bitmap fonts. Do you know what kind of text is being used in the 787?

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

James Turner - 2021-01-11

And if it's Canvas or OSG text: do you know which font is being used? Eg Sidi Liang added Chinese TTF font and we correclty show Chinese startup hints using OSG text, so I guess some things do work. (Look at Chinese translation files, maybe)

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Megaf - 2021-01-20

The author said he is using B612 font from fgdata and it is in ttf format.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

James Turner - 2021-01-20

Suggestion, make a hack to use LiberationSans and see if anything changes: that's the default font which should have special characters for sure. (It doesn't contain Chinese ideograms, but Swedish/Polish work: maybe even Cyrillic and Greek). This would tell us if we haev a font bug or a data bug.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- IskenderWang - 2021-02-01
  
  Hello James! First off, big up for all the great work you've done/are doing for FG. Despite Megaf's message, it isn't that we've lost interest – just that we've been busy with some things irl lately. I can confirm to you that using LiberationSans instead of B612 as we were using earlier (even though really I aim to use Nunito but I wasn't able to get it to work through a custom font directory path (Aircraft/787-family/Fonts) , only through FGdata for some reason so I might propose for it to be added since like many of the other fonts it's OFL) made no difference, therefore this issue must be some sort of data bug:
  
  Last edit: IskenderWang 2021-02-01
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Megaf - 2021-02-01

The devs of the 787 seem to have lost interest on this bug.
We may close this report in my opinion since they are no longer interested.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

James Turner - 2021-02-02

Okay, can you post your test setup, so I can try to reproduce it? I'd prefer something using the UFO, but I guess that won't work here. It's better if you can strip it down to as little as possible: if I have to load the full aircraft, it might take a day just to get to the problem string. Making a dummy Canvas dialog with just the affected items in their own Canvas is ideal, but not always possibe.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- IskenderWang - 2021-02-02
  
  I don't think this is even on Canvas, if I'm not mistaken, though one of our goals is to port over all the cockpit instruments to Canvas at some point. As mentioned earlier, this is most likely an issue that lies internally in FG with not adopting/recognizing UTF-8 only. I'm really not sure how to go about isolating only the parts of the aircraft where the issue lies? It could (and probably does) go far beyond the scope of just that anyway, if it is like Megaf said and the symbols like ° have the same problem in chat. That said, I can lead you to the directory where all our affected files are! That would be right here :)
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Megaf - 2021-02-02

@jmturner I think a dummy canvas window could be used indeed, it would display some symbols and then we see if that work or not.

As Iskender said in the post above, swapping the font didn't fix the issue.

And I do agree with him, it must be some bug in FG, since you can't use any symbols in the chat either.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

James Turner - 2021-02-03

Okay, I'm still a bit stuck, for what to do to reproduce the problem then. Can you suggest an existing aircraft in FGaddon I could modify to show the issue, maybe?

I need some small change I can make to see the problem ocurring in front of me, so I can start to investigate why it's going wrong.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Megaf - 2021-02-03
  
  You can try the 787-8 from FGADDON.
  Though it will show a slightly different behaviour, it simply wont show the degree symbol or any accents at all.
  In the file 787-8/Models/Instruments/HEAT.xml search for lines <format type="string"></format> Replace with anything you want.
  
  In this image I replaced ENG 1 TEMP with 32°C and ENG 2 TEMP with ñ, it simply wont show them.
  
  Same will happen if you type "°" or "ñ" in chat, it wont show anything.
  Hence, me stating, FG doesn't support UTF-8.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

James Turner - 2021-02-03

Okay, again I was hoping for something simpler than 'use the 787', but I'll see if I can work through it.

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
- Megaf - 2021-02-03
  
  Well, you can use anything with canvas, or create a dummy canvas and test then, I don't think it's 787 specific.
  
  I don't know how NASAL nor Canvas, so can't help much, sorry.
  
  If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

James Turner - 2021-02-07

Figured it out; Canvas text works; it's text animations which were missing a flag to indicate they should use UTF-8. Will make the fix on next for wider testing, and we can decide if this is safe to back-port to 2020.3

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

James Turner - 2021-02-07

SimGear/next commit 9561895335ad22ef24cee30f571326b23245e113

If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

James Turner - 2021-02-07

status: New --> Fixed

assigned_to: James Turner

Milestone: None --> 2020.4
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:

Log in to post a comment.