gnuplot development

On Tuesday, November 01, 2011 05:25:41 pm Shigeharu TAKENO wrote:
> shige 11/02 2011
> ----------------
> 
> Recently, variable degree_sign with encoding is introduced. But, 
> I think it has a problem for Japanese environment.
> 
> There are 3 encoding systems we use normaly in Japan, Shift_JIS
> for most of MS-Win user, EUC-JP for many unix user, and UTF-8.
> 
> Current version of gnuplot has support for Shift_JIS encoding 
> because the Shift_JIS Japanese characters have a singular 
> feature that the 2nd byte of them may be 7bit code (0x40-0x7E).

According to the code table on Wikipedia, the SJIS code for 
degree sign is not the same as any of the other encodings.
There is a comment in the new code that gives the SJIS code,
but right now it returns an empty string.  However, for SJIS
it works anyway in the gd and cairo terminals because iconv()
is used on output.

> We have not need care of EUC encoding in gnuplot because all 
> bytes of EUC-JP Japanese characters are 8bit code (0x8E,0x8F,
> and 0xA1-0xFE).
> 
> However, newly introduced degree_sign has a default value of 
> 8bit code 0260 (0xB0), which is a proper code of iso-8859-1 
> encoding, but is not recognized in EUC-JP encoding.

Right now the degree sign is not being used anywhere, so it
cannot be a problem yet.  The intention is to use the degree
sign to create a default format for writing geographic coordinates
(patch #3428074 on SourceForge). Even so, this would only be a
default format.  You could still specify a different format if
it is needed.

> To solve the problem, make the new encoding support for 
> EUC-JP, or to set the default value of degree_sign to 7bit
> character strings, I think.

If you tell me the correct character sequence for EUC-JP I
can at least put it in a comment, as it is now for SJIS and
CP950.

Or the program could use iconv() to convert the symbol into the 
locale setting of the program environment.  

	Ethan


-- 
Ethan A Merritt
Biomolecular Structure Center,  K-428 Health Sciences Bldg
University of Washington, Seattle 98195-7742

shige 11/02 2011
----------------

Thank you for your reply.

Ethan Merritt <merritt@...> wrote:
| Right now the degree sign is not being used anywhere, so it
| cannot be a problem yet.  The intention is to use the degree
| sign to create a default format for writing geographic coordinates
| (patch #3428074 on SourceForge). Even so, this would only be a
| default format.  You could still specify a different format if
| it is needed.

I see.

| If you tell me the correct character sequence for EUC-JP I
| can at least put it in a comment, as it is now for SJIS and
| CP950.
| 
| Or the program could use iconv() to convert the symbol into the 
| locale setting of the program environment.  

I think both are not need now. I will consider them when the 
degree_sign is used at somewhere in fact.

+========================================================+
 Shigeharu TAKENO     NIigata Institute of Technology
                       kashiwazaki,Niigata 945-1195 JAPAN
 shige@...   TEL(&FAX): +81-257-22-8161
+========================================================+

On Wednesday, November 02, 2011 02:40:47 am Shigeharu TAKENO wrote:
> | 
> | Or the program could use iconv() to convert the symbol into the 
> | locale setting of the program environment.  
> 
> I think both are not need now. I will consider them when the 
> degree_sign is used at somewhere in fact.

I have revised the new code to use iconv() if it is available.
I tested under linux by setting the locale to ja_JP.EUC-JP,
which causes it to emit the character sequence \241 \353 (octal)
which is 0xA1EB (hexidecimal).  But I don't have any EUC-JP fonts
installed, so I cannot confirm if this is correct.

If you build the new code, you can test it by saying
   set encoding locale
   show decimal

If your locale is EUC-JP and you see a degree sign, then it is working.
If you get an error message then there is a problem with recognizing the
locale.

	Ethan

shige 11/04 2011
----------------

Ethan A Merritt <sfeam@...> wrote:
| I have revised the new code to use iconv() if it is available.
| I tested under linux by setting the locale to ja_JP.EUC-JP,
| which causes it to emit the character sequence \241 \353 (octal)
| which is 0xA1EB (hexidecimal).  But I don't have any EUC-JP fonts
| installed, so I cannot confirm if this is correct.

The code 0xA1EB is correct for EUC-JP.

Well, I think the idea

+    if (locale) { 
+       /* This should work even if gnuplot doesn't understand the encoding */
+       char *encoding = strchr(locale, '.');
+       if (encoding) {
+           encoding++; /* Step past the dot in, e.g., ja_JP.EUC-JP */

of src/set.c is not so good, since substrings of locale name 
after '.' may not proper name of the character encoding. For 
example, in Solaris 9,

  EUC-JP enconding locale name = "ja", "ja_JP.EUC", "ja_JP.eucJP"
  Shift_JIS encoding locale name = "ja_JP.PCK"
  UTF-8 encoding locale name = "ja_JP.UTF-8"

To obtain encoding codeset name of current locale, there seems to
be two methods:

1) Use nl_langinfo(CODESET) (#include <langinfo.h>)
2) Use locale_charset() (#include <localcharset.h>) in libcharset,
  which is included in GNU libiconv.

cf. http://www.haible.de/bruno/packages-libcharset.html

On Solaris 9, nl_langinfo(CODESET) of system library returns:

  locale = ja, ja_JP.eucJP => "eucJP"
  locale = ja_JP.PCK => "PCK"

and locale_charset() of GNU libcharset returns:

  locale = ja, ja_JP.eucJP => "EUC-JP"
  locale = ja_JP.PCK => "SHIFT_JIS"

I think that names returned by locale_charset() may be better,
but libcharset may not be installed in default. 


| If you build the new code, you can test it by saying
|    set encoding locale
|    show decimal

Though it puts a incorrect string on "ja" locale, it works fine
on "ja_JP.eucJP" locale since iconv library admits "eucJP".

+========================================================+
 Shigeharu TAKENO     NIigata Institute of Technology
                       kashiwazaki,Niigata 945-1195 JAPAN
 shige@...   TEL(&FAX): +81-257-22-8161
+========================================================+

For the currently active locale you could also simply use wcrtomb() to 
convert.

   Bastian

Am 04.11.2011 08:54, schrieb Shigeharu TAKENO:
> shige 11/04 2011
> ----------------
>
> Ethan A Merritt<sfeam@...>  wrote:
> | I have revised the new code to use iconv() if it is available.
> | I tested under linux by setting the locale to ja_JP.EUC-JP,
> | which causes it to emit the character sequence \241 \353 (octal)
> | which is 0xA1EB (hexidecimal).  But I don't have any EUC-JP fonts
> | installed, so I cannot confirm if this is correct.
>
> The code 0xA1EB is correct for EUC-JP.
>
> Well, I think the idea
>
> +    if (locale) {
> +       /* This should work even if gnuplot doesn't understand the encoding */
> +       char *encoding = strchr(locale, '.');
> +       if (encoding) {
> +           encoding++; /* Step past the dot in, e.g., ja_JP.EUC-JP */
>
> of src/set.c is not so good, since substrings of locale name
> after '.' may not proper name of the character encoding. For
> example, in Solaris 9,
>
>    EUC-JP enconding locale name = "ja", "ja_JP.EUC", "ja_JP.eucJP"
>    Shift_JIS encoding locale name = "ja_JP.PCK"
>    UTF-8 encoding locale name = "ja_JP.UTF-8"
>
> To obtain encoding codeset name of current locale, there seems to
> be two methods:
>
> 1) Use nl_langinfo(CODESET) (#include<langinfo.h>)
> 2) Use locale_charset() (#include<localcharset.h>) in libcharset,
>    which is included in GNU libiconv.
>
> cf. http://www.haible.de/bruno/packages-libcharset.html
>
> On Solaris 9, nl_langinfo(CODESET) of system library returns:
>
>    locale = ja, ja_JP.eucJP =>  "eucJP"
>    locale = ja_JP.PCK =>  "PCK"
>
> and locale_charset() of GNU libcharset returns:
>
>    locale = ja, ja_JP.eucJP =>  "EUC-JP"
>    locale = ja_JP.PCK =>  "SHIFT_JIS"
>
> I think that names returned by locale_charset() may be better,
> but libcharset may not be installed in default.
>
>
> | If you build the new code, you can test it by saying
> |    set encoding locale
> |    show decimal
>
> Though it puts a incorrect string on "ja" locale, it works fine
> on "ja_JP.eucJP" locale since iconv library admits "eucJP".
>
> +========================================================+
>   Shigeharu TAKENO     NIigata Institute of Technology
>                         kashiwazaki,Niigata 945-1195 JAPAN
>   shige@...   TEL(&FAX): +81-257-22-8161
> +========================================================+

Shigeharu TAKENO <shige@...> wrote

> On Solaris 9, nl_langinfo(CODESET) of system library returns:
>   locale = ja, ja_JP.eucJP => "eucJP"
>   locale = ja_JP.PCK => "PCK"

Thank you for mentioning nl_langinfo().  I did not know about that routine.

On Solaris 9 does iconv_open() accept "PCK" as an encoding name?
We don't care what the name is, we only care that iconv() can use it.

So if nl_langinfo(CODESET) returns "PCK"
and iconv_open( "PCK", "ISO-8859-1" ) works correctly
then this method seems OK.

We will have to check for nl_lanfinfo() in the configure script,
but that's not a problem.

	Ethan



> Ethan A Merritt <sfeam@...> wrote:
> | I have revised the new code to use iconv() if it is available.
> | I tested under linux by setting the locale to ja_JP.EUC-JP,
> | which causes it to emit the character sequence \241 \353 (octal)
> | which is 0xA1EB (hexidecimal).  But I don't have any EUC-JP fonts
> | installed, so I cannot confirm if this is correct.
> 
> The code 0xA1EB is correct for EUC-JP.
> 
> Well, I think the idea
> 
> +    if (locale) { 
> +       /* This should work even if gnuplot doesn't understand the encoding */
> +       char *encoding = strchr(locale, '.');
> +       if (encoding) {
> +           encoding++; /* Step past the dot in, e.g., ja_JP.EUC-JP */
> 
> of src/set.c is not so good, since substrings of locale name 
> after '.' may not proper name of the character encoding. For 
> example, in Solaris 9,
> 
>   EUC-JP enconding locale name = "ja", "ja_JP.EUC", "ja_JP.eucJP"
>   Shift_JIS encoding locale name = "ja_JP.PCK"
>   UTF-8 encoding locale name = "ja_JP.UTF-8"
> 
> To obtain encoding codeset name of current locale, there seems to
> be two methods:
> 
> 1) Use nl_langinfo(CODESET) (#include <langinfo.h>)
> 2) Use locale_charset() (#include <localcharset.h>) in libcharset,
>   which is included in GNU libiconv.
> 
> cf. http://www.haible.de/bruno/packages-libcharset.html
> 
> On Solaris 9, nl_langinfo(CODESET) of system library returns:
> 
>   locale = ja, ja_JP.eucJP => "eucJP"
>   locale = ja_JP.PCK => "PCK"
> 
> and locale_charset() of GNU libcharset returns:
> 
>   locale = ja, ja_JP.eucJP => "EUC-JP"
>   locale = ja_JP.PCK => "SHIFT_JIS"
> 
> I think that names returned by locale_charset() may be better,
> but libcharset may not be installed in default. 
> 
> 
> | If you build the new code, you can test it by saying
> |    set encoding locale
> |    show decimal
> 
> Though it puts a incorrect string on "ja" locale, it works fine
> on "ja_JP.eucJP" locale since iconv library admits "eucJP".
> 
> +========================================================+
>  Shigeharu TAKENO     NIigata Institute of Technology
>                        kashiwazaki,Niigata 945-1195 JAPAN
>  shige@...   TEL(&FAX): +81-257-22-8161
> +========================================================+
>

2011/11/4 Bastian Märkisch <bmaerkisch@...>:
> For the currently active locale you could also simply use wcrtomb() to
> convert.

If Solaris is reporting "PCK" instead of "Shift-JIS" in LC_CTYPE then
I would guess that wcrtomb and iconv either both fail or both work..
One would hope it would accept its own name convention!

I will try Shige's suggestion to use nl_langinfo().

         Ethan


>
>  Bastian
>
> Am 04.11.2011 08:54, schrieb Shigeharu TAKENO:
>>
>> shige 11/04 2011
>> ----------------
>>
>> Ethan A Merritt<sfeam@...>  wrote:
>> | I have revised the new code to use iconv() if it is available.
>> | I tested under linux by setting the locale to ja_JP.EUC-JP,
>> | which causes it to emit the character sequence \241 \353 (octal)
>> | which is 0xA1EB (hexidecimal).  But I don't have any EUC-JP fonts
>> | installed, so I cannot confirm if this is correct.
>>
>> The code 0xA1EB is correct for EUC-JP.
>>
>> Well, I think the idea
>>
>> +    if (locale) {
>> +       /* This should work even if gnuplot doesn't understand the
>> encoding */
>> +       char *encoding = strchr(locale, '.');
>> +       if (encoding) {
>> +           encoding++; /* Step past the dot in, e.g., ja_JP.EUC-JP */
>>
>> of src/set.c is not so good, since substrings of locale name
>> after '.' may not proper name of the character encoding. For
>> example, in Solaris 9,
>>
>>   EUC-JP enconding locale name = "ja", "ja_JP.EUC", "ja_JP.eucJP"
>>   Shift_JIS encoding locale name = "ja_JP.PCK"
>>   UTF-8 encoding locale name = "ja_JP.UTF-8"
>>
>> To obtain encoding codeset name of current locale, there seems to
>> be two methods:
>>
>> 1) Use nl_langinfo(CODESET) (#include<langinfo.h>)
>> 2) Use locale_charset() (#include<localcharset.h>) in libcharset,
>>   which is included in GNU libiconv.
>>
>> cf. http://www.haible.de/bruno/packages-libcharset.html
>>
>> On Solaris 9, nl_langinfo(CODESET) of system library returns:
>>
>>   locale = ja, ja_JP.eucJP =>  "eucJP"
>>   locale = ja_JP.PCK =>  "PCK"
>>
>> and locale_charset() of GNU libcharset returns:
>>
>>   locale = ja, ja_JP.eucJP =>  "EUC-JP"
>>   locale = ja_JP.PCK =>  "SHIFT_JIS"
>>
>> I think that names returned by locale_charset() may be better,
>> but libcharset may not be installed in default.
>>
>>
>> | If you build the new code, you can test it by saying
>> |    set encoding locale
>> |    show decimal
>>
>> Though it puts a incorrect string on "ja" locale, it works fine
>> on "ja_JP.eucJP" locale since iconv library admits "eucJP".
>>
>> +========================================================+
>>  Shigeharu TAKENO     NIigata Institute of Technology
>>                        kashiwazaki,Niigata 945-1195 JAPAN
>>  shige@...   TEL(&FAX): +81-257-22-8161
>> +========================================================+
>
>

shige 11/05 2011
----------------

Ethan A Merritt <sfeam@...> wrote:
| On Solaris 9 does iconv_open() accept "PCK" as an encoding name?
| We don't care what the name is, we only care that iconv() can use it.

Though Solaris's iconv accepts "PCK", it does not seem to admit
the translation from "ISO-8859-1" to "PCK" unfortunately. 
iconv_open() fails the translation.

GNU iconv may accept "PCK" if it is compiled with "-DUSE_SOLARIS_ALIAS".
It translates 0xb0 to 0x818b correctly.

+========================================================+
 Shigeharu TAKENO     NIigata Institute of Technology
                       kashiwazaki,Niigata 945-1195 JAPAN
 shige@...   TEL(&FAX): +81-257-22-8161
+========================================================+