Re: I18N in OI2 (was: [Openinteract-help] project alive?)

[Chris W. <ch...@cw...>]

Okay, I've thought about this a little bit. Keep in mind that I'm still 
open to different approaches as you read this, and the rambling nature 
of the email reflects it. (Lots of thinking out loud...)

On Feb 13, 2004, at 6:30 PM, Teemu Arina wrote:
...
> because once you translated the string from english to some other 
> language,
> you don't see the original english string anywhere unless you put it in
> comments or look for it in the orginial english translation file. My 
> question
> for opaque IDs is: how do you easily improve a translation if the keys 
> are
> not based on the base language (comparing original to current, making
> modifications...)?
>
> Another thing was the self-documenting aspects of having the base 
> language key
> in the code. Consider for example:
>
> $lh->maketext( 'Hello [_1], today is [_2]', $username, $date );
>
> vs:
>
> $lh->maketext( 'desktop.welcome', $username, $date );

I agree with your points, but my three main objections to using the 
base language as the key are:

  * What do you do with long strings? (One or more sentences)
  * What happens when you change the base language text?
  * Many of the base language keys will be quite long.

For the first objection I guess you could just use the first part of it 
the long string. This kind of eliminates the benefit of using the base 
language as key, it's not used as often.

The second objection seems like a big problem. How often does it 
happen? Not sure. But like I mention in the manual doc it seems very 
brittle. (Using your suggestion from later in the email -- syncing 
things up with a script -- seems like an interesting idea. But still 
work...)

The third one seems like it would be very annoying for authors. You 
have more experience with this, granted. I'm really trying to make it 
easy to do the right thing for people who want to distribute their 
applications.

> Now another point: It is very common that some projects just never 
> have 100%
> complete translations and it is common that translation files lacking 
> latest
> improvements have some required keys missing. In that case I would 
> suggest
> the logic:
>
> 1. look for requested key from the requested translation (e.g. spanish)
> 2. If not found, look for requested key from the default translation 
> (english)
> 3. If not found, produce error

Locale::Maketext takes care of this. In fact it will do language 
differentiation (or whatever the term is) as well. For instance if I'm 
creating customizations for Mexican Spanish I can create a file:

  mymessages-es_mx.msg

Any messages not found there will look in:

  mymessages-es.msg

And any messages not found there will look in the default language, 
which is normally English:

  mymessages-en.msg

But you can also present to L::M a series of languages to try out. OI2 
will pull the browser preferences ('Accept-Language' header) if they're 
available so if your preference is Spanish then Portuguese it will try 
those before the default.

> If you consider using base language as keys, consider working around 
> the
> global key problem (if I understood correctly, the keys have to be 
> unique,
> even over package boundaries). For example, Maketext objects could be
> initialized for each package, so that the key name-space is unique for 
> every
> package. Very useful, in my opinion.. you never know what strings or 
> keys
> some other project will use.

This is true but... (thinking out loud here)

Well, my initial impression is that this is getting too complicated. 
But perhaps not. If we think of a package as a distributable 
application then there's really no reason that a package would use the 
keys of another package, right? In fact we could probably assume that 
the only other keys a package needs are some global keys for text used 
all over the place (standard button text, copyright, disclaimers, 
etc.).

Okay, I think I could go along with this. I would like the option of 
putting a package's keys in the global namespace for applications using 
multiple packages. But the default should be that when requesting a 
message key you only look in your package's space and the global space 
if it's not found there.

The only tricky part with this is now we have to let OI2 know what 
package we're currently in. This isn't so hard for templates or even 
actions. But normal perl classes will probably be more difficult -- 
that is, if I want to emit an error message from my class 
OpenInteract2::User:

  return $lh->maketext( 'error.password_mismatch' );

Then how will OI know to look into the OI2::I18N::base_user package for 
the key? ... Maybe at server startup we can do a mapping of all keys to 
the package they were read from and in cases of ambiguity refer to it. 
We can also create a wrapper for the maketext call that would allow 
another parameter to nail down the right package. (This wrapper idea 
might come in handy to use different message storage schemes, more 
below...)

>> these use a slightly modified Java message bundle syntax (easy to
>> write!)
>
> In my opinion you should use gettext (PO files), because there are easy
> translation project management tools available for programs using 
> gettext.
> I for example find Kbabel brilliant for that job:
> http://i18n.kde.org/tools/kbabel/
... (lots more interesting stuff about kbabel clipped)

This is a good idea -- I would like to make it possible to use tools 
like this. But I don't want to get rid of the benefits of using 
plaintext files as well.

After a little investigation it looks like the 
Locale::Maketext::Lexicon module (from another superhero, Autrijus) 
solves this problem for us. (Hooray for CPAN!) It seems to allow you to 
read in .po/.mo files for the Locale::Maketext lexicon. It also looks 
like it would be no problem to adapt it to our OI2::I18N::mypackage 
naming scheme from above. (That is, keeping every package separate.) It 
seems to even translate the gettext functions %1 %2 to [_1] [_2] used 
by maketext. Sweet!

> Another point why to use base language strings as keys here: you could 
> easily
> write a script that goes through your script, extracts strings from 
> commands
> like $lh->maketext( 'Hello [_1], today is [_2]', $username, $weekday ) 
> and
> generate a translation file for you. This way you could just improve 
> the
> english translation in your code and then generate an up to date 
> version of
> the translation file.

True, but I think most uses of the messages will be in templates where 
it's much more difficult to parse groups of text out into individual 
entries.

> Also, gnu gettext utils also contain some really useful tools for 
> development
> like msgfmt. It is helpful for checking translation file syntax, 
> produce some
> statistics etc. msgcmp compares two .po files finding differencies like
> missing msgid strings.

Very useful.

> Hope this helps,

Immensely. Thanks for the great ideas and time it took to put them 
together.

Chris

--
Chris Winters
Creating enterprise-capable snack systems since 1988