translate-pootle

[translate-pootle] Merging of Pootle very resource intensive

[Sayamindu D. <say...@gm...>]

Hi,
I'm trying to update the PO files (using the Update from templates)
for some pretty large PO files[1], and Pootle seems to be taking
forever (and taking up 100% CPU) to update the files. I'm using Pootle
1.0.2-2 (Ubuntu) along with translate-toolkit 1.0.1-1.
Is this normal - or am I doing something stupid?

Btw - we are using Pootle for all of our translations at the OLPC, and
we are really happy with it. Great work guys :-)

Thanks,
Sayamindu



[1] https://dev.laptop.org/translate/projects/etoys/


-- 
Sayamindu Dasgupta
[http://sayamindu.randomink.org/ramblings]

Re: [translate-pootle] Merging of Pootle very resource intensive

[F W. <fr...@tr...>]

Op Saterdag 2008-01-26 skryf Sayamindu Dasgupta:
> Hi,
> I'm trying to update the PO files (using the Update from templates)
> for some pretty large PO files[1], and Pootle seems to be taking
> forever (and taking up 100% CPU) to update the files. I'm using Pootle
> 1.0.2-2 (Ubuntu) along with translate-toolkit 1.0.1-1.
> Is this normal - or am I doing something stupid?
> 
> Btw - we are using Pootle for all of our translations at the OLPC, and
> we are really happy with it. Great work guys :-)
> 
> Thanks,
> Sayamindu
> 
> 
> 
> [1] https://dev.laptop.org/translate/projects/etoys/
> 
> 

Hallo, and thank you for the great feedback. There are a few issues here
to be considered.

 * updating from templates uses pot2po in the Translate Toolkit, which
does fuzzy matching using the translations in the existing files. The
larger the file, the longer it takes. The default python implementation
works, but can be quite slow. Solution: install python-Levenshtein,
which should provide a speedup. If you restart Pootle it should
automatically be used.
http://sourceforge.net/project/showfiles.php?group_id=91920&package_id=260161

 * Many tools in the Translate Toolkit can benefit from installing
psyco. It should be packaged for your distribution (something like
'psyco' or 'python-psyco'), or you can get it here:
http://psyco.sourceforge.net/
I don't have good benchmarks for the difference this brings, so you
might want to experiment and report back on your success on this. Also
watch the memory usage on the server if you are constrained in this
area, as psyco increases memory consumption to some extent. For
information on the psyco options, see this page:
http://translate.sourceforge.net/wiki/toolkit/psyco_mode

 * You might want to have a server administrator update the files from
the commandline on the server. Here performance might simply not be an
issue, since the Pootle server is not involved. You can finetune the
parameters of pot2po (the type of fuzzy matching, or to use a
compendium / Translation Memory), or use msgmerge if you prefer. This
way migrations can carefully be reviewed before copying them into the
tree. As long as files are not added or removed, but only updated, you
should be able to simply copy it over the existing files without
restarting Pootle. More information on pot2po here:
http://translate.sourceforge.net/wiki/toolkit/pot2po
I often do this anyway, as I like to review updates before putting them
live.

 * In your specific case (the URL you give) many of the translations are
still empty. You can simply recreate the files afresh on the command
line without having to update from templates. I'm not sure how much of
an improvement this will give.

 * If you decide to use the web front end for updating, you can try to
only do one language at a time, instead of all of them at once. 

 * There is a bug in Pootle versions up to 1.0.2 that causes some
unnecessary processing during the updating of templates. I fixed this,
but this fix is so far only available in the release candidate for
Pootle 1.1.0.  This is available here:
http://translate.sourceforge.net/snapshots/
Note that this version also needs the newest snapshot of the Translate
Toolkit that is available in the same directory (currently version
1.1.1rc1)
I'm not sure how big difference this will make, but it should be an
improvement. New versions should be released soon, and I will definitely
recommend that for you.

 * In these new versions, you could try to run Pootle with the
environment variable USECPO=1  which will use the new experimental PO
handling code. This should provide some speedup, but is perhaps not
ideal yet for production use. It might be worthwhile to try it out for
the future, though. We definitely need some feedback on this for our
next releases.

I hope this gives you some insight into some solutions. You should at
least install python-Levenshtein if you haven't done so yet.

By the way, I see you don't have the translations files for the project
called "Pootle". These files are distributed with Pootle and is
installed in the default po directory. This provides the interface
translations of Pootle itself, so that people can use Pootle in their
language of preference, if available. Perhaps you want to consider
adding those files.

Keep well
Friedel

Re: [translate-pootle] Merging of Pootle very resource intensive

[Clytie S. <cl...@ri...>]

On 27/01/2008, at 4:27 AM, Sayamindu Dasgupta wrote:

> Btw - we are using Pootle for all of our translations at the OLPC, and
> we are really happy with it. Great work guys :-)

I didn't know about this translation project. If you intend to target  
Vietnamese users, please add Vietnamese to your Pootle and add me as  
admin. I have just registered and activated.

Also, which items on your Pootle have higher priority for translation?

Please do load the Pootle interface translation: you have a much  
higher chance of getting translators then. It's so much more  
comfortable for them to work in a native-language interface,  
regardless of their level of English proficiency.

Good luck with your project. :)

from Clytie

Vietnamese Free Software Translation Team
http://vnoss.net/dokuwiki/doku.php?id=projects:l10n

Re: [translate-pootle] Merging of Pootle very resource intensive

[Sayamindu D. <say...@gm...>]

Hello,

On Feb 2, 2008 1:15 PM, Clytie Siddall <cl...@ri...> wrote:
>
> On 27/01/2008, at 4:27 AM, Sayamindu Dasgupta wrote:
>
> > Btw - we are using Pootle for all of our translations at the OLPC, and
> > we are really happy with it. Great work guys :-)
>
> I didn't know about this translation project. If you intend to target
> Vietnamese users, please add Vietnamese to your Pootle and add me as
> admin. I have just registered and activated.
>

Wow - thanks :-) I have created the projects and added you as the admin.

> Also, which items on your Pootle have higher priority for translation?
>

XO Core, then XO Bundled, then Etoys

> Please do load the Pootle interface translation: you have a much
> higher chance of getting translators then. It's so much more
> comfortable for them to work in a native-language interface,
> regardless of their level of English proficiency.
>

I thought it was there - thanks for for pointing out. It was a issue
with the symlinks that we have - I have fixed it now.

> Good luck with your project. :)
>

Thank you :-)

Warm regards,
Sayamindu

-- 
Sayamindu Dasgupta
[http://sayamindu.randomink.org/ramblings]