After long nights in the last weeks, I have finally a first *TEST* solution to generate language resource files from PO files. I wrote some VBScripts (located at the "Languages" folder) for this:
1. CreateMasterPotFile.vbs
This script creates the master POT file (English.pot) and must run after every change in the Merge.rc file. (Need 4-5 seconds on my computer.)
2. CreateRcFilesFromPoFiles.vbs
This script creates the language RC files from the language PO files and must run before you want to compile the language dlls. (Need unfortunately 660 seconds on my computer for all lang files.)
3. (ConvertRcFilesToPoFiles.vbs)
This script converts the language RC files to the language PO files and must only run ONCE.
4. At the moment I have no script to update the language PO files, but this is something what the translator can do with his poeditor. He can update his PO file from the master POT file.
* The master POT file called "English.pot" and is in the "Languages" folder.
* The language PO files are called after the language (for example "German.po") and are in the language subfolders.
* The created RC files are "Merge<Language>Test.rc" for testing.
Known Problems
--------------
* At the moment I can't open the "ChineseSimplified", "ChineseTraditional" and "Korean" PO files with poEdit. He says, that he can't convert the file to unicode (which he use internally). But I don't know why! :-(
* We need some changes at the Merge.rc file:
- Change LANGUAGE 9, 1 to LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_US
- Change SUBLANG_DEFAULT to SUBLANG_ENGLISH_US
- Include manifest file only by LANG_ENGLISH
- Move strings like IDS_SUGGEST_IGNOREEOL or IDS_NO_ZIP_SUPPORT to one line
Is would be cool, if you can test and comment this stuff! :)
Greetings,
Tim
First patch for Merge.rc
------------------------
Index: Merge.rc
===================================================================
--- Merge.rc (revision 4411)
+++ Merge.rc (working copy)
@@ -48,7 +48,7 @@
"\r\n"
"#if !defined(AFX_RESOURCE_DLL) || defined(AFX_TARG_ENU)\r\n"
"#ifdef _WIN32\r\n"
- "LANGUAGE 9, 1\r\n"
+ "LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_US\r\n"
"#pragma code_page(1252)\r\n"
"#endif\r\n"
"#include ""res\\Merge.rc2"" // non-Microsoft Visual C++ edited resources\r\n"
@@ -61,7 +61,7 @@
// Include the manifest file for VC6 and VS2003.Net
-#if _MSC_VER < 1400
+#if _MSC_VER < 1400 && LANGUAGE == LANG_ENGLISH
#ifdef UNICODE
CREATEPROCESS_MANIFEST_RESOURCE_ID RT_MANIFEST "res\\WinMergeU.exe.manifest"
#else
@@ -2953,7 +2953,7 @@
#if !defined(AFX_RESOURCE_DLL) || defined(AFX_TARG_ENU)
#ifdef _WIN32
-LANGUAGE LANG_ENGLISH, SUBLANG_DEFAULT
+LANGUAGE LANG_ENGLISH, SUBLANG_ENGLISH_US
#pragma code_page(1252)
#endif
#include "res\Merge.rc2" // non-Microsoft Visual C++ edited resources
See also:
---------
RFE: [ 1729161 ] Generate language resource files from PO files?
http://winmerge.org/rfe/1729161
Developers Forum: Generate lang resource files from PO files?
http://sourceforge.net/forum/forum.php?thread_id=1745640&forum_id=41639
first public version from the scripts
Logged In: YES
user_id=652377
Originator: YES
> (Need unfortunately 660 seconds on my computer for all lang files.)
I optimize my local version from CreateRcFilesFromPoFiles.vbs a little bit and need now only 215 seconds. Maybe there is more! ;)
But now it is time for my bed...
Logged In: YES
user_id=631874
Originator: NO
Very, very, very cool!!
I ran the first (create .pot file) and third (create .po files) scripts. And quickly looked at generated files. Looks ok to me.
Are there perhaps some parts I could write with C/C++ to make it faster?
I think only way to really test this is we create one experimental with these scripts, and see what happens (I doubt nobody else than you and me run these even after they are committed to SVN).
This looks like a giant step forward with translations. (I just *have* to set resolution to 'Accepted'! :)
This easily saves me like one to two hours of dull merging job with resource changes, per patch.
second public version from the scripts (faster!)
Logged In: YES
user_id=652377
Originator: YES
Ok, I added new versions from the scripts. They are now a little bit faster, more user-friendly and fix some small issues. I also added two batch files for the "Tools\Scripts" folder.
> * At the moment I can't open the "ChineseSimplified", "ChineseTraditional"
> and "Korean" PO files with poEdit. He says, that he can't convert the file
> to unicode (which he use internally). But I don't know why! :-(
Problem is the "©" char from the StringFileInfo.LegalCopyright. If we would use "(c)" in Merge.rc we would have no problems!
> Are there perhaps some parts I could write with C/C++ to make it faster?
The main work from the scripts are parsing text files and comparing strings. Of course could be a real program faster then a slow script language.
The new version from "CreateRcFilesFromPoFiles.vbs" needs 160-170 seconds on my computer for all lanugages. I think this is ok for the beginning.
If we have many "free" time, we can still write a own tool for this! ;)
> I think only way to really test this is we create one experimental with
> these scripts, and see what happens
By the way, what is with "FileVersion" and "ProductVersion" from the "StringFileInfo" block? Currently they are putted into the PO(T) files. Should I exclude them?
Should we hard-coded the version number to Merge.rc? In this case we could drop the SetResourceVersions.bat script.
> This looks like a giant step forward with translations.
I hope so! :)
Greetings,
Tim
p.s.:
What must I do, that SVN replace me the $Id$ at the beginning of the file? It hasn't worked for the file "CreateReviewPoFiles.vbs". Next time I will do it better! *fg*
File Added: CreateRcFilesFromPoFiles_v2.zip
Logged In: YES
user_id=631874
Originator: NO
> If we would use "(c)" in Merge.rc we would have no problems!
This was discussed (maybe in devel forum or in some tracker items) couple of years ago. That symbol is problematic for many translators and e.g. Japanese translation uses (c). (c) is perfectly valid and if that solves our problems, I'd say we use it.
> The main work from the scripts are parsing text files and
> comparing strings.
..which is tricky to do in C/C++, that's why there are script languages... The safest way is to just do it with scripts and see if we can later improve it.
> By the way, what is with "FileVersion" and "ProductVersion" from the
> "StringFileInfo" block? Currently they are putted into the PO(T) files.
> Should I exclude them?
> Should we hard-coded the version number to Merge.rc? In this case we could
> drop the SetResourceVersions.bat script.
Good question! We need version info in every .rc file in compile time. If the version number can be read just from Merge.rc to all other files, then that is fine. But I still prefer simple script that puts version numbers to even Merge.rc. It has happened few times with ShellExtension.rc I've forgot to update some of the four numbers, or mistyped it. That can cause an update to fail!
> What must I do, that SVN replace me the $Id$ at the beginning of the file?
You need to set Subversion 'keywords' property for the file. And preferably other properties also. I just added short story about keywords here:
http://winmerge.org/Wiki/Commit_Patch
Yes, I've been forgetting to set keywords also... :( I think there are still many files which miss them.
Logged In: YES
user_id=652377
Originator: YES
> But I still prefer simple script that puts
> version numbers to even Merge.rc.
Ok, I will exclude the version numbers from the PO(T) files and we will still use SetResourceVersions.bat for this. Maybe later we could modify the script to change only Merge.rc and use CreateRcFilesFromPoFiles.vbs after this step. The language RC files use always the numbers from Merge.rc.
> Yes, I've been forgetting to set keywords also... :( I think there are
> still many files which miss them.
Are there no global setting for this?
Logged In: YES
user_id=631874
Originator: NO
[About SVN keywords..]
> Are there no global setting for this?
No, its per-file property. Remember there are binary files also, and we don't wan any conversions done for those.
Logged In: YES
user_id=652377
Originator: YES
Ok, I added version 3 from my scripts:
* Exclude the version numbers from the PO(T) files
* Add "POT-Creation-Date" to generated POT file
* Fix warnung message
File Added: CreateRcFilesFromPoFiles_v3.zip
third public version from the scripts
Logged In: YES
user_id=652377
Originator: YES
Ok, I commited the "CreateMasterPotFile" and "CreateRcFilesFromPoFiles" script to SVN (In Revision 4426).
Next step will be optimize Merge.rc and then create the PO(T) files. For the moment I will leave the patch open...
Logged In: YES
user_id=652377
Originator: YES
I commited optimized Merge.rc to SVN (In Revision 4427).
Logged In: YES
user_id=631874
Originator: NO
Lets NOT do the conversion in trunk. Lets have a branch for it. So DON'T convert trunk lang files, not without coordinating with me.
The plan:
- create a branch from current trunk, name it something like "PO_Translations_test"
- do the conversion in that branch
- we'll release next experimental from that branch
- we don't do translation-related changes in trunk until we see if the new branch is good or not
- if we determine it is good, we'll commit it (with some fixes I expect) to trunk
Like I said earlier, its best to see how this works in practice, and separate branch is the safe way to do it.
And if/when this is applied to trunk, we'll need to update bunch of documentation, and send proper notices so people know what to do with translations.
Logged In: YES
user_id=652377
Originator: YES
> Lets NOT do the conversion in trunk. Lets have a branch for it. So
> DON'T convert trunk lang files, not without coordinating with me
No problem, but can you do the branch? I am not so familiar with SVN. :(
Logged In: YES
user_id=631874
Originator: NO
Ok, the branch is there:
https://winmerge.svn.sourceforge.net/svnroot/winmerge/branches/PO_translations_test
- use that for checkout dialog and you get the branch.
Branching isn't so bad, there is own item for it in TSVN menu, just give a branch url when it asks about it. .../branches/branch-name-you-invented.
I myself messed it up couple of first times, but luckily it was easy to fix with SVN. :)
Logged In: YES
user_id=652377
Originator: YES
Ok, I worked this evening with the branch and I think we are ready for a try run! I commited all PO(T) files and deleted the old lang *.rc files. I also speeded up the scripts a lot! The script "CreateRcFilesFromPoFiles.vbs" needs now only 35-40 seconds for all lang files on my computer! *jiiiiiha* ;)
Logged In: YES
user_id=631874
Originator: NO
I could make a experimental release today. Just to be sure, can you tell what I now need to do to compile languages from the branch? Like I earlier said, I'll do this release from the test branch..
Logged In: YES
user_id=652377
Originator: YES
> what I now need to do to compile languages from the branch?
Calling the script "Tools\Scripts\CreateRcFilesFromPoFiles.bat" is enough. After 35-40 seconds you will have the language *.rc files on the old known place. Then you must set the version number with "SetResourceVersions.bat" to all *.rc files. After that you can compile the DLL like normal.
You need the script "CreateMasterPotFile.bat" only after changes on the Merge.rc file.
Logged In: YES
user_id=631874
Originator: NO
I see you have been working to fix translation issues in the branch. That is great!
I hope you don't mind keep working in the branch until most issues (differences between translations) are fixed? It is easier to merge whole system in one commit than start going back and forth between trunk and branch.
And just in case we for some reason need to do (temporarily) backout, its still one patch. I'm just realistic. This is maybe the biggest change we've done to WinMerge since unicode support, so it is only realistic that there will be some problems.
By the way I realized one more huge positive effect for this patch - strings can't anymore get into different order in .rc files. It was quite a pain when somebody edited .rc file with Visual Studio and lots of strings were reordered..
Logged In: YES
user_id=652377
Originator: YES
> I hope you don't mind keep working in the branch until
> most issues (differences between translations) are fixed?
That is no problem, since I hope one day all the changes go to trunk! ;)
I will say, if I have fix all items from "WinMerge_Resource_Discrepencies.txt".
> I realized one more huge positive effect for this patch
And I work at the moment on a script to get the status of the translations. So we can have later a page like this:
http://tortoisesvn.net/translation_devel_gui
I think this is a big positive effect too! :)
Logged In: YES
user_id=652377
Originator: YES
Ok, this is the current "WinMerge_Resource_Discrepencies.txt" from PO_translations_test branch:
-------------------------8<-------------------------
IDD_EDIT_REPLACE
Catalan widens buttons 50->63
IDD_PREFERENCES
widened in Swedish 352->376
tree widened in Swedish 235->259
Spanish similarly 352->371, tree 235->254
Slovak 352->364 and tree 235->247
Norwegian 352->358 and tree 235->243
Dutch 352->367 and tree 235->250
------------------------->8-------------------------
I widens the buttons in IDD_EDIT_REPLACE to 60, that is enough for all languages. Only Catalan have no space before and after in 1 button text. Is this a problem?
I think the IDD_PREFERENCES changes are needless. They didn't change the tree but the IDC_TREEOPT_HOSTFRAME control. And this fits also with the current wide for all languages.
I think after bug #1785215 the main problems should fixed and maybe ready for Trunk. Moving meta data (like codepage, sub_lang, ...) from PO file to separate text file could also be done in Trunk.
Logged In: YES
user_id=631874
Originator: NO
Yes, forgot IDD_PREFERENCES changes - it was just Perry's misunderstanding (or dislike of that dialog).
This would be a big change in trunk, so we'll need some preparations. I definitely want to test at least compile with all visual studio versions. Which probably takes few days as I'm busy few next days with other activities.
And I'd rather do the merge as first commit after experimental release. So that we have one experimental with old-and-good system and then next experimental with new system. It makes it a lot easier for people to determine if bug/behavior change is caused by new system. So, I'll be doing one experimental from current trunk + couple of patches. Probably at Thursday. After that you can merge this + possible needed fixes.
Logged In: YES
user_id=631874
Originator: NO
Sorry, didn't have time for doing the release. And that ShellExtension compile problem caused me some headache too. But I'll see if I can do release in next few days..
Logged In: YES
user_id=652377
Originator: YES
> But I'll see if I can do release in next few days..
No problem!
Btw: Is there later a easy way to merge trunk with the branch? I think about the SVN properties (eol-style, keywords) of the new files.