rlib-devel

[Rlib-devel] Locale problem in RLIB

[Zoltan B. <zb...@du...>]

Hi Bob,

Hungarian problems again. :-)

Now I reached a phase where my program actually compiles and works
under Windows, and links to RLIB. But the resulting PDF is unusable.
Of course I saved it in an O_BINARY file. The problem is that printf(),
et.al. are conforming to the locale settings, so controlling numbers
in the PDFs are written as e.g.

/Frpdf0 9,000 Tf

instead of

/Frpdf0 9.000 Tf

since the decimal separator is a comma, not a dot here in Hungary.

A simple trick will do it (I fixed unixODBC this way in 2000):

-------------------------------------------
char *saved_locale;

saved_locale =3D strdup(setlocale(LC_ALL, NULL));

/* national decimal separator here */
setlocale(LC_ALL, "C");
/* decimal dots strictly required here */
setlocale(LC_ALL, saved_locale);
/* national decimal separator again */
free(saved_locale);
-------------------------------------------

Fortunately setlocale() exists under MingW, too.

I haven't made a patch against RPDF yet, but you know better
where to fix it.

Best regards,
Zolt=E1n B=F6sz=F6rm=E9nyi

Re: [Rlib-devel] Locale problem in RLIB

[Bob D. <bd...@si...>]

Hello,

> Hungarian problems again. :-)

;)

What you want to do seems Fine.  I know Chet @ SICOM did something in
libsrc/pdf.c which is **supposed** to not allow this to happen.  Have a
look in there.  It should really be fixed in RPDF though.

Feel free to make the patch :)

If you do fix it @ the RPDF level perhaps you can biff the libsrc/pdf.c
stuff also since RPDF is really the place to fix it


> Now I reached a phase where my program actually compiles and works
> under Windows, and links to RLIB. But the resulting PDF is unusable.
> Of course I saved it in an O_BINARY file. The problem is that printf(),
> et.al. are conforming to the locale settings, so controlling numbers
> in the PDFs are written as e.g.
> 
> /Frpdf0 9,000 Tf
> 
> instead of
> 
> /Frpdf0 9.000 Tf
> 
> since the decimal separator is a comma, not a dot here in Hungary.
> 
> A simple trick will do it (I fixed unixODBC this way in 2000):
> 
> -------------------------------------------
> char *saved_locale;
> 
> saved_locale = strdup(setlocale(LC_ALL, NULL));
> 
> /* national decimal separator here */
> setlocale(LC_ALL, "C");
> /* decimal dots strictly required here */
> setlocale(LC_ALL, saved_locale);
> /* national decimal separator again */
> free(saved_locale);
> -------------------------------------------
> 
> Fortunately setlocale() exists under MingW, too.
> 
> I haven't made a patch against RPDF yet, but you know better
> where to fix it.

Re: [Rlib-devel] Locale problem in RLIB

[Zoltan B. <zb...@du...>]

Bob Doan =EDrta:
> Hello,
>=20
>=20
>>Hungarian problems again. :-)
>=20
>=20
> ;)
>=20
> What you want to do seems Fine.  I know Chet @ SICOM did something in
> libsrc/pdf.c which is **supposed** to not allow this to happen.  Have a
> look in there.  It should really be fixed in RPDF though.
>=20
> Feel free to make the patch :)

Here it is. :-) It doesn't change the situation under Linux
e.g. the PDFs are still correctly readable with AcroRead, GV and XPDF
but RLIB didn't link under MingW, I must have been messed it
up somehow. Not the patch, my MingW installation.
I rechecked rlib-1.3.3 with my MingW installation's present state
and it's also doesn't link. I will reinstall it this evening.
In the meantime, would you please try to compile the present
CVS + this patch for me? ;-)

> If you do fix it @ the RPDF level perhaps you can biff the libsrc/pdf.c
> stuff also since RPDF is really the place to fix it

I haven't touched that yet, the patch is for rpdf/rpdf.c only.

Best regards,
Zolt=E1n B=F6sz=F6rm=E9nyi

Re: [Rlib-devel] Locale problem in RLIB

[Bob D. <bd...@si...>]

> Here it is. :-) It doesn't change the situation under Linux
> e.g. the PDFs are still correctly readable with AcroRead, GV and XPDF
> but RLIB didn't link under MingW, I must have been messed it
> up somehow. Not the patch, my MingW installation.
> I rechecked rlib-1.3.3 with my MingW installation's present state
> and it's also doesn't link. I will reinstall it this evening.
> In the meantime, would you please try to compile the present
> CVS + this patch for me? ;-)
> 
> > If you do fix it @ the RPDF level perhaps you can biff the libsrc/pdf.c
> > stuff also since RPDF is really the place to fix it
> 
> I haven't touched that yet, the patch is for rpdf/rpdf.c only.
> 

I believe the patch looks right.  in libsrc/pdf.c you could make
USEPDFLOCALE 0 and give it a try.  Let me know if it still works for you
in Windows and Linux.  This way we can clean libsrc/pdf.c

Re: [Rlib-devel] Locale problem in RLIB

[Zoltan B. <zb...@du...>]

Bob Doan =EDrta:
>>Here it is. :-) It doesn't change the situation under Linux
>>e.g. the PDFs are still correctly readable with AcroRead, GV and XPDF
>>but RLIB didn't link under MingW, I must have been messed it
>>up somehow. Not the patch, my MingW installation.
>>I rechecked rlib-1.3.3 with my MingW installation's present state
>>and it's also doesn't link. I will reinstall it this evening.
>>In the meantime, would you please try to compile the present
>>CVS + this patch for me? ;-)
>>
>>
>>>If you do fix it @ the RPDF level perhaps you can biff the libsrc/pdf.=
c
>>>stuff also since RPDF is really the place to fix it
>>
>>I haven't touched that yet, the patch is for rpdf/rpdf.c only.
>>
>=20
>=20
> I believe the patch looks right.  in libsrc/pdf.c you could make
> USEPDFLOCALE 0 and give it a try.  Let me know if it still works for yo=
u
> in Windows and Linux.  This way we can clean libsrc/pdf.c

I tried that and I had to fix that up since a setlocale()
call references a local variable in two functions that only declared
when USEPDFLOCALE=3D1. After that, I recreated my PDFs and found that
there is a change in the output. Now all the floating point numbers
are printed with the locale's decimal separator and the summing
variable that was declared in the XML as

<Variables>
   <Variable name=3D"ossz_sum" value=3D"val(mainquery.osszeg)" type=3D"su=
m"=20
resetonbreak=3D""/>
   </Variables>

still works correctly, e.g. it sums the field.  I was afraid that val()
may break. But it doesn't conform to the locale settings, it prints
decimal dots. I am investigating it.

Also, I found that I can save one setlocale() call from my previous
patch, as it always returns the current settings. That wasn't crystal
clear from the manpage. And there was a bug in there too, I used
strdup() only in one of the two functions but freed in both. :-(

Here's the updated patch that deletes setlocale() calls from
libsrc/pdf.c, too, and fixes my bug.

Best regards,
Zolt=E1n B=F6sz=F6rm=E9nyi

Re: [Rlib-devel] Locale problem in RLIB

[Zoltan B. <zb...@du...>]

Zoltan Boszormenyi =EDrta:
> [val() (and other functions?)] doesn't conform to the locale settings, =
it prints
> decimal dots. I am investigating it.

How about changing rlib's locale behaviour?

I found that rlib_set_locale() changes the current locale
but doesn't save the current setting.

I would expect the following behaviour:

- If rlib_set_locale() was called, convert the given setting to UTF-8
   with make_utf8_locale() and save it. Also notice in struct rlib that
   we have set the locale.

- On every RLIB API call entry, remember the current locale locally.

   - If rlib_set_locale() was called, make the saved/converted setting
     current.

   - If rlib_set_locale() wasn't called, convert the currently set
     locale to utf8, make it current. (*)

- On exiting from the API call, restore the old locale we saved
   on entry.

(*) I know it makes RLIB slower but I don't know how much. I would have
to benchmark setlocale(). It may also be a NOP if the current locale is
already UTF-8 but you cannot know that

The advantage of the above is that RLIB doesn't suppose anything about
the current locale, e.g. one can play games with setlocale() between
RLIB API calls, so the locale setting may be different from the one
before rlib_init().

Call chains like this:

/************************************************/
   rlib_function1() {
     /* save locale */
     /* do something */
     /* restore locale */
   }
   rlib_function2() {
     /* save locale */
     /* do something that involves calling rlib_function1() */
     /* restore locale */
   }
/************************************************/

should be modified to this:

/************************************************/
   rlib_function1_internals() {
     /* do something */
   }
   rlib_function1() {
     /* save locale */
     rlib_function1_internals();
     /* restore locale */
   }
   rlib_function2() {
     /* save locale */
     /* do something that involves calling rlib_function1_internals() */
     /* restore locale */
   }
/************************************************/

so it doesn't make it even slower. It wouldn't cause bugs,
follow my proof:

Suppose the current locale is hu_HU in my application,
rlib_set_locale() is not called. The scenario would be:

1. In function2(): save current locale (hu_HU),
    convert it to UTF-8 -> hu_HU.utf8, call function1().

2. In function1(): save current locale (hu_HU.utf8),
    convert it to UTF-8, that's a NOP but extra work.
    On exit, restore the saved hu_HU.utf8.

3. On exit from function2(), restore hu_HU.

The other scenario is when I call rlib_set_locale() with e.g. en_US.
That seems to be converted to UTF-8 internally.

1. In function2(): save current locale (hu_HU), set the saved one
    (en_US.utf8), call function1().

2. In function1(): save the current locale (en_US.utf8) and set
    the saved one (en_US.utf8), again a NOP with extra work done.
    On exit, restore en_US.utf-8.

3. On exit from function2(), restore hu_HU.

Best regards,
Zolt=E1n B=F6sz=F6rm=E9nyi

Re: [Rlib-devel] Locale problem in RLIB

[Zoltan B. <zb...@du...>]

Zoltan Boszormenyi =EDrta:
> Zoltan Boszormenyi =EDrta:
>=20
>> [val() (and other functions?)] doesn't conform to the locale settings,=
=20
>> it prints
>> decimal dots. I am investigating it.

I found that my original modification was correct. My "extra"
setlocale() call is needed. If you give a new locale to setlocale(),
then on error it returns NULL, on success it returns the newly set
locale. setlocale(LC_ALL, NULL) is needed to return the original locale.
But I saved some setlocale() calls anyway. I didn't recognize at first
that rpdf_make_page_stream() is called by rpdf_finalize(), so it's
enough to modify the locale in rpdf_finalize().

But it didn't fix the sum, its still displayed with decimal dot
instead of a comma. All the other numbers above it contains comma.
They come directly from the SQL query. In RLIB's datasources,
the values are actually strings, aren't they? The actual values aren't
converted to numbers and back to strings, I suppose.
Only the report variables are treated differently.

Besides that, my original proposal still stands.

Best regards,
Zolt=E1n B=F6sz=F6rm=E9nyi

Re: [Rlib-devel] Locale problem in RLIB

[Zoltan B. <zb...@du...>]

Zoltan Boszormenyi =EDrta:
> But it didn't fix the sum, its still displayed with decimal dot
> instead of a comma. All the other numbers above it contains comma.
> They come directly from the SQL query. In RLIB's datasources,
> the values are actually strings, aren't they? The actual values aren't
> converted to numbers and back to strings, I suppose.
> Only the report variables are treated differently.

Only a small fix was needed for
libsrc/formatstring.c::rlib_number_sprintf()
to fix it, and now the report variables also displayed with
correct decimal separator according to the locale settings.

Best regards,
Zolt=E1n B=F6sz=F6rm=E9nyi

Re: [Rlib-devel] Locale problem in RLIB

[Bob D. <bd...@si...>]

Applied!

I'm still thinking about your proposal for the setlocale stuff but since
this makes things more better We'll go with this for today

Thanks!

On Sun, 2005-05-15 at 21:51 +0200, Zoltan Boszormenyi wrote:
> Zoltan Boszormenyi írta:
> > But it didn't fix the sum, its still displayed with decimal dot
> > instead of a comma. All the other numbers above it contains comma.
> > They come directly from the SQL query. In RLIB's datasources,
> > the values are actually strings, aren't they? The actual values aren't
> > converted to numbers and back to strings, I suppose.
> > Only the report variables are treated differently.
> 
> Only a small fix was needed for
> libsrc/formatstring.c::rlib_number_sprintf()
> to fix it, and now the report variables also displayed with
> correct decimal separator according to the locale settings.
> 
> Best regards,
> Zoltán Böszörményi
-- 
Bob Doan <bd...@si...>