omegat-development — OmegaT development list

Re: [OmTdev] AI plugin dltranslator

[Jean-Christophe H. <jea...@tr...>]

I think it’s acknowledged that for professional translators, working from MT "drafts" is actually a waste of time.

I’m fine with adding plugins that do this and that, but Omegat development should really focus on features that make it easier for professional translators to deliver good work.

Jean-Christophe

> On Oct 17, 2024, at 22:03, Martin Engelke via OmegaT-development <ome...@li...> wrote:
> 
> Thats exactly what I thought.
> Why call it ‚AI', if it is still a 'Machine Translation‘?
> 
> I mean you could give the model some ‚heat‘ parameters so that the model is for some types of texts more free in his translation and it may suggests you different opportunities, which would be indeed an ‚AI‘ feature then.
> 
> 3rd cent.
> 
> Anyway, good decision to head for this direction, since we should be up to date.
> 
> :-)
> 
>> Am 17.10.2024 um 13:29 schrieb Peter Sandrini via OmegaT-development <ome...@li...>:
>> 
>> Hi,
>> 
>> 
>> the dltranslator llm tool is a wonderful extension to OmegaT.
>> However, it could be optimized to take full advantage of AI. I am thinking of two things right now:
>> 
>> 1)
>> the main advantage of LLMs is that I can write prompts where I can define the whole context of my translation, i.e. target audience, subject area, text type, style, etc.
>> Now, if there is no such possibility (as of now in dltranslator), the LLM works with the simple prompt "translate this form SL to TL.
>> So I can imagine that an interface in OmegaT which allows specification of additional information would be a real advantage. It could be something like:
>> 
>> translate (Variable Source segment) from (Var source language) into (Var target language) taking into account the text from (Variable open text input field)
>> 
>> The user gets presented an input field where he could optionally input some context for the translation before sending the extended prompt to the dlt server. Alternatively he could let this input window empty and send the standard prompt to the server.
>> 
>> If not, the AI extension works just as another machine translation plugin but at a high cost in terms of hardware, speed and disk space.
>> 
>> 
>> 2)
>> Downloading the 2 LL models by default means consuming hard disk space without need because I can use only one at a time in the preferences. Maybe an option for choosing a preferred LLM to start with, and an option to download the second if I want, or maybe other models later on would be better.
>> 
>> These are my 2 cents. What do you think?
>> 
>> Peter
>> 
>> 
>> 
>> 
>> _______________________________________________
>> OmegaT-development mailing list
>> Ome...@li...
>> https://lists.sourceforge.net/lists/listinfo/omegat-development
> 
> 
> 
> _______________________________________________
> OmegaT-development mailing list
> Ome...@li...
> https://lists.sourceforge.net/lists/listinfo/omegat-development

-- 
Jean-Christophe Helary
@jch...@em...
https://sr.ht/~brandelune/

Re: [OmTdev] AI plugin dltranslator

[Martin E. <mar...@ic...>]

Thats exactly what I thought.
Why call it ‚AI', if it is still a 'Machine Translation‘? 

I mean you could give the model some ‚heat‘ parameters so that the model is for some types of texts more free in his translation and it may suggests you different opportunities, which would be indeed an ‚AI‘ feature then.

3rd cent.

Anyway, good decision to head for this direction, since we should be up to date.

:-)

> Am 17.10.2024 um 13:29 schrieb Peter Sandrini via OmegaT-development <ome...@li...>:
> 
> Hi,
> 
> 
> the dltranslator llm tool is a wonderful extension to OmegaT.
> However, it could be optimized to take full advantage of AI. I am thinking of two things right now:
> 
> 1)
> the main advantage of LLMs is that I can write prompts where I can define the whole context of my translation, i.e. target audience, subject area, text type, style, etc.
> Now, if there is no such possibility (as of now in dltranslator), the LLM works with the simple prompt "translate this form SL to TL.
> So I can imagine that an interface in OmegaT which allows specification of additional information would be a real advantage. It could be something like:
> 
> translate (Variable Source segment) from (Var source language) into (Var target language) taking into account the text from (Variable open text input field)
> 
> The user gets presented an input field where he could optionally input some context for the translation before sending the extended prompt to the dlt server. Alternatively he could let this input window empty and send the standard prompt to the server.
> 
> If not, the AI extension works just as another machine translation plugin but at a high cost in terms of hardware, speed and disk space.
> 
> 
> 2)
> Downloading the 2 LL models by default means consuming hard disk space without need because I can use only one at a time in the preferences. Maybe an option for choosing a preferred LLM to start with, and an option to download the second if I want, or maybe other models later on would be better.
> 
> These are my 2 cents. What do you think?
> 
> Peter
> 
> 
> 
> 
> _______________________________________________
> OmegaT-development mailing list
> Ome...@li...
> https://lists.sourceforge.net/lists/listinfo/omegat-development

[OmTdev] AI plugin dltranslator

[Peter S. <pet...@ui...>]

Hi,


the dltranslator llm tool is a wonderful extension to OmegaT.
However, it could be optimized to take full advantage of AI. I am 
thinking of two things right now:

1)
the main advantage of LLMs is that I can write prompts where I can 
define the whole context of my translation, i.e. target audience, 
subject area, text type, style, etc.
Now, if there is no such possibility (as of now in dltranslator), the 
LLM works with the simple prompt "translate this form SL to TL.
So I can imagine that an interface in OmegaT which allows specification 
of additional information would be a real advantage. It could be 
something like:

translate (Variable Source segment) from (Var source language) into (Var 
target language) taking into account the text from (Variable open text 
input field)

The user gets presented an input field where he could optionally input 
some context for the translation before sending the extended prompt to 
the dlt server. Alternatively he could let this input window empty and 
send the standard prompt to the server.

If not, the AI extension works just as another machine translation 
plugin but at a high cost in terms of hardware, speed and disk space.


2)
Downloading the 2 LL models by default means consuming hard disk space 
without need because I can use only one at a time in the preferences. 
Maybe an option for choosing a preferred LLM to start with, and an 
option to download the second if I want, or maybe other models later on 
would be better.

These are my 2 cents. What do you think?

Peter

Re: [OmTdev] AI reviewer debut in LanguageTools project

[Hiroshi M. <mi...@no...>]

LanguageTool team operates open-core/closed-proprietary strategy with LGPL-ed core library.
They provided AI-powered premium service as commercial.

It looks natural for them to utilize OpenAI GPT based technology in their project.

In contrast with it, we run free software project that promise our users all the source
is free as in freedom, and guarantee no collection of users data.

What is our philosophy against AI generated contents?

As you know, Open Source Initiatives working on Open Source AI Definition.
https://opensource.org/ai

I think it is good chance to improve our understandings for AI trends.

There is an idea that we are prioritizing an Open Source AI than others.

On 2024/10/06 11:04, Jean-Christophe Helary wrote:

>> On Oct 6, 2024, at 10:27, Hiroshi Miura via OmegaT-development
>> [<ome...@li...>](mailto:ome...@li...)
>> wrote:
>> When I collaborated on the LanguageTool project,
>> CodeRabbit AI reviewer of OSS project for free give me a very precise, helpful code review, and modification suggestions.
>> You can find the live example at
>> https://github.com/languagetool-org/languagetool/pull/10810
>
> Interesting.
> It looks like your proposals are going to be accepted. Am I understanding correctly ?

--
Hiroshi Miura from north-side in tokyo

[OmTdev] problems converting 3.6-style project with custom language codes

[Stephan P. <ste...@zo...>]

hello.

(long time heavy user, first time wannabe-contributor. thx y'all & keep up
the good work!)

i have been forced to finally upgrade from old style team projects because
the signed legacy mac version
https://sourceforge.net/projects/omegat/files/OmegaT%20-%20Legacy/OmegaT%203.6.0%20update%2011/
does no longer run under the latest macOS Sequoia.

my translation projects use self-chosen fictitious target language codes
similar to this "DE-ABCD" and "DE-BANANANA". after converting to new style
projects, those intentional codes get converted to the unintended "de-Abcd"
and "de-Bananana" which seems to break the matching of source segments with
tmx-entries. i am still trying to analyze what exactly is going wrong while
trying to get to know the codebase.

i suspect that i might need some sort of command line switch when running
Ωt to tell it to not be so strict with enforcing the format of language
codes as specified by standards. any idea if there is already such a thing?
or where i might look to get started with an implementation of command line
argument to achieve a more lax and lenient language code handling. e.g.
--language-codes-lenient vs. --language-codes-strict

ciao a tutti

-- 

Mit freundlichen Grüßen,
Stephan Pakebusch
Software-Entwickler

*zollsoft GmbH*, Ernst-Haeckel-Platz 5/6, 07745 Jena, Germany
Geschäftsführer: Dr. Andreas Zollmann, Johannes Zollmann
Registergericht: Amtsgericht Jena, HRB 507075
www.zollsoft.de | www.tomedo.de


Vertrieb: 03641 - 269 41  <//03641%20-%20269%2041%2062>62
<//03641%20-%20269%2041%2062>
Support: 03641 - 268 41 51 <//03641%20-%20268%2041%2051>
Fax: 03641 - 268 71 83 <//03641%20268%20718%203>

Re: [OmTdev] AI reviewer debut in LanguageTools project

[Jean-Christophe H. <jea...@tr...>]

> On Oct 6, 2024, at 10:27, Hiroshi Miura via OmegaT-development <ome...@li...> wrote:
> 
> When I collaborated on the LanguageTool project,
> CodeRabbit AI reviewer of OSS project for free give me a very precise, helpful code review, and modification suggestions.
> It also gives a summary of changes for the project human developer.
> 
> My proposal does not get enough dense of a code review, because the project run with very small number of core developers as same as OmegaT. 
> My proposals, Java 9 full support, bump Lucene versions and dynamic language modules, are relatively big and deep changes that require thorough review.
> I frustrated with no response in several months for it. Now rabbit give me feedback rapidly, and feels better (than nothing). 
> 
> You can find the live example at
> https://github.com/languagetool-org/languagetool/pull/10810

Interesting.

It looks like your proposals are going to be accepted. Am I understanding correctly ?

-- 
Jean-Christophe Helary
@jch...@em...
https://sr.ht/~brandelune/

[OmTdev] AI reviewer debut in LanguageTools project

[Hiroshi M. <mi...@no...>]

When I collaborated on the LanguageTool project,
CodeRabbit AI reviewer of OSS project for free give me a very precise, helpful code review, and modification suggestions.
It also gives a summary of changes for the project human developer.

My proposal does not get enough dense of a code review, because the project run with very small number of core developers as same as OmegaT.
My proposals, Java 9 full support, bump Lucene versions and dynamic language modules, are relatively big and deep changes that require thorough review.
I frustrated with no response in several months for it. Now rabbit give me feedback rapidly, and feels better (than nothing).

You can find the live example at
https://github.com/languagetool-org/languagetool/pull/10810

How do you feel?

--
Hiroshi Miura from north-side in tokyo

Re: [OmTdev] Backport the fix for BUGS#1164 into 6.0.2

[Hiroshi M. <mi...@no...>]

Now merged.

On 2024/10/05 19:52, Jean-Christophe Helary wrote:

> Thank you Hiroshi.
>
> Jean-Christophe
>
>> On Oct 5, 2024, at 19:07, Hiroshi Miura via OmegaT-development
>> [<ome...@li...>](mailto:ome...@li...)
>> wrote:
>>
>> Hi,
>>
>> I plan to backport the fix for BUGS#1162 into releases/6.0 branch that will be published as version 6.0.2
>> https://github.com/omegat-org/omegat/pull/1060

--
Hiroshi Miura from north-side in tokyo

Re: [OmTdev] Backport the fix for BUGS#1164 into 6.0.2

[Jean-Christophe H. <jea...@tr...>]

Thank you Hiroshi.

Jean-Christophe 

> On Oct 5, 2024, at 19:07, Hiroshi Miura via OmegaT-development <ome...@li...> wrote:
> 
> Hi,
> 
> I plan to backport the fix for BUGS#1162 into releases/6.0 branch that will be published as version 6.0.2
> 
> https://github.com/omegat-org/omegat/pull/1060
> 
> 
> Please review it.
> 
> --
> Hiroshi Miura from north-side in tokyo
> 

-- 
Jean-Christophe Helary
@jch...@em...
https://sr.ht/~brandelune/

Re: [OmTdev] JGit 7.0.0

[Jean-Christophe H. <jea...@tr...>]

Thank you Hiroshi.

Jean-Christophe 

> On Oct 5, 2024, at 19:04, Hiroshi Miura via OmegaT-development <ome...@li...> wrote:
> 
> Hi,
> 
> OmegaT team feature depends on JGit library to handle git repository.
> OmegaT 6.1.0 development (master branch) uses JGit 6.10.0.
> 
> As you know, JGit major version up v7.0.0 has been released in Sept. 2024.
> It bumps minimum required Java version to 17, and there are many API breaking changes. 
> 
> Now a bot propose a version up, and I modified it to catch API changes.
> 
> https://github.com/omegat-org/omegat/pull/1139
> 
> It looks working with Java 17 and JGit 7.0.0.
> 
> 
> The proposal will be merged when the team decided to bump target Java to 17 after OmegaT 6.1.0 released.
> 
> --
> Hiroshi Miura from north-side in tokyo
> 

-- 
Jean-Christophe Helary
@jch...@em...
https://sr.ht/~brandelune/

[OmTdev] Backport the fix for BUGS#1164 into 6.0.2

[Hiroshi M. <mi...@no...>]

Hi,

I plan to backport the fix for BUGS#1162 into releases/6.0 branch that will be published as version 6.0.2

https://github.com/omegat-org/omegat/pull/1060

Please review it.

--
Hiroshi Miura from north-side in tokyo

[OmTdev] JGit 7.0.0

[Hiroshi M. <mi...@no...>]

Hi,

OmegaT team feature depends on JGit library to handle git repository.
OmegaT 6.1.0 development (master branch) uses JGit 6.10.0.

As you know, JGit major version up v7.0.0 has been released in Sept. 2024.
It bumps minimum required Java version to 17, and there are many API breaking changes.

Now a bot propose a version up, and I modified it to catch API changes.

https://github.com/omegat-org/omegat/pull/1139

It looks working with Java 17 and JGit 7.0.0.

The proposal will be merged when the team decided to bump target Java to 17 after OmegaT 6.1.0 released.

--
Hiroshi Miura from north-side in tokyo

Re: [OmTdev] localize standard dialogs, open/save to be localized in various languages

[Jean-Christophe H. <jea...@tr...>]

> On Oct 4, 2024, at 21:29, Hiroshi Miura via OmegaT-development <ome...@li...> wrote:
> 
> Hi,
> 
> We provide a UI localization for >40 langauges. Java runtime only provides 11 localization of UI parts.
> 
> It is great if OmegaT show localized title for Open dialog. Now I realized it. Please find a Pull-Request.
> 
>     https://github.com/omegat-org/omegat/pull/1143
> 
> It localizes save open dialogs and preference color chooser in
> 
> - Arabian
> - Catalan
> - Ukrainian
> - Russian
> 
> I am very happy if L10N team is interested in localize it for languages.
> 
> Please check the library https://github.com/omegat-org/swing-extra-locales
> and go to resource bundles at
> 
>  https://github.com/omegat-org/swing-extra-locales/tree/main/src/main/resources/org/omegat/swing/extra
> 
> These translations are come from machine translation, so there may be spaces to improve. 

We can’t use MT to localize OmegaT. We need to put those strings into OmegaT’s l10n package.


-- 
Jean-Christophe Helary
@jch...@em...
https://sr.ht/~brandelune/

[OmTdev] improve test robustness

[Hiroshi M. <mi...@no...>]

Hi,

There are some unit test cases that become failure when internet connection is not available.
Here is a PR to solve it

https://github.com/omegat-org/omegat/pull/1156

https://github.com/omegat-org/omegat/pull/1155

--
Hiroshi Miura from north-side in tokyo

[OmTdev] localize standard dialogs, open/save to be localized in various languages

[Hiroshi M. <mi...@no...>]

Hi,

We provide a UI localization for >40 langauges. Java runtime only provides 11 localization of UI parts.

It is great if OmegaT show localized title for Open dialog. Now I realized it. Please find a Pull-Request.

https://github.com/omegat-org/omegat/pull/1143

It localizes save open dialogs and preference color chooser in

- Arabian
- Catalan
- Ukrainian
- Russian

I am very happy if L10N team is interested in localize it for languages.

Please check the library https://github.com/omegat-org/swing-extra-locales
and go to resource bundles at

https://github.com/omegat-org/swing-extra-locales/tree/main/src/main/resources/org/omegat/swing/extra

These translations are come from machine translation, so there may be spaces to improve.

--
Hiroshi Miura from north-side in tokyo

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Julian R. <jul...@gm...>]

Hi Thomas

I'll make up a list with a selection of each type of source term.
It will probably be sometime over the weekend.

Julian

On Thu, 3 Oct 2024 at 15:49, Thomas CORDONNIER via OmegaT-development <
ome...@li...> wrote:

>
> > In your version, the order seems to be the same, except that hiragana
> and katakana are mixed rather than sorted separately.
>
> Interesting, in my own tests I had the impression that nothing changed but
> maybe my small samples of japanese words were not relevant.
> Can you send us a small set of examples (only a small list of terms, no
> need for a file), showing this difference? I would like to try to reproduce
> it in my side, to check whenever it is dependent from the OS (my OS is not
> configured for japanese at all)
> It would also be useful to check if it is conform to JIS X 4061 mentioned
> by Hiroshi, but unfortunately it seems that the book is expansive and
> available only in japanese language.
>
> And of course, the good question is whenever you (and japanese users like
> Hiroshi) consider this behaviour as better or not.
>
> Thomas
>
> Le jeudi 3 octobre 2024 à 16:19:35 UTC+2, Julian R. <
> jul...@gm...> a écrit :
>
>
> Hi Thomas
>
> > When the alphabetic sort is done with
> > Japanese, I have noticed that single-byte punctuation and numbers seem
> > to come first, then hiragana, then katakana, and finally Kanji.
> > Other single-byte characters would probably come at the beginning too,
> although I haven't checked for that.
>
>
> Interesting. Is it the same with my branch (in case you lost it:
> https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order)? For
> the moment I could not really see any effect of my changes but I admit that
> I don't have lot of samples in Japanese.
>
>
> In your version, the order seems to be the same, except that hiragana and
> katakana are mixed rather than sorted separately.
>
> Julian
>
>
>
> On Thu, 3 Oct 2024 at 07:12, Thomas CORDONNIER via OmegaT-development <
> ome...@li...> wrote:
>
> HI all
>
> Since there were a lot of messages, I try to group my answers
>
>
>
> Le mercredi 2 octobre 2024 à 10:49:53 UTC+2, Hiroshi Miura via
> OmegaT-development <ome...@li...> a écrit :
>
> > It uses `String.contains` method that create strange behavior for users.
>
> > There are two options to fix it.
> > 1. remove two "contains" conditions.  When enabled the option sort by source length, it sort it just with the length.
>
> Yes, that is what OmegaT < 5.8 did
>
> > 2. change "contains" to "startsWith". It does sort  the terms with length when  terms are like "トヨタ" and "トヨタ自動車"  but does not the terms  "Claim" and "Aim"
>
> Yes, this variant seems to be transitive so it would be ok for me.
>
>
> Le mercredi 2 octobre 2024 à 17:45:01 UTC+2, Julian R. <
> jul...@gm...> a écrit :
>
> > When the alphabetic sort is done with > Japanese, I have noticed that
> single-byte punctuation and numbers seem > to come first, then hiragana,
> then katakana, and finally Kanji.
> > Other single-byte characters would probably come at the beginning too,
> although I haven't checked for that.
>
>
> Interesting. Is it the same with my branch (in case you lost it:
> https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order)? For
> the moment I could not really see any effect of my changes but I admit that
> I don't have lot of samples in Japanese.
>
>
> Le jeudi 3 octobre 2024 à 01:31:41 UTC+2, Hiroshi Miura via
> OmegaT-development <ome...@li...> a écrit :
>
> > The topic for sort order of languages. It depends on locale as Thomas
> pointed out.
>
> > There is an international standard for it. In Japanese, there is a sort
> order standard JIS X 4061
>
> > Java has a Collator class for the purpose. I want to propose it for a
> glossary sorter.
>
> Yes, this was exactly the purpose of my branch: using the
> language-dependent sort order as directly supported by Java, even if I did
> not know in detail what the collator for Japanese does.
>
>
>
> Le jeudi 3 octobre 2024 à 05:00:01 UTC+2, Hiroshi Miura
> <mi...@no...kyo> a écrit :
>
> > Unfortunately your branch does not pass unit test.
> > The branch has a change like
>
>  >            if (c == 0) {
> > -                c =
> o1.getLocText().compareToIgnoreCase(o2.getLocText());
> > +                c =
> compareLanguageDependent(Core.getProject().getProjectProperties().getTargetLanguage(),
> o1.getSrcText(), o2.getSrcText());
>              }
>
> > Should it be o1.getLocText() instead of o1.getSrcText() ?
>
> You are right, I just pushed the correction.
>
> Hiroshi, I will study your other correction proposals later. In the
> meantime, could anybody do functional tests on this branch (i.e. with the
> editor, not only unit tests!)? I would really be interested to know if the
> language-dependent collator has an effect on Japanese words (in the few
> examples I had it seemed to give similar results as String.compare but I
> had only a few). This can be done without your proposals, which are mostly
> technical.
>
>
> Regards
> Thomas
>
> _______________________________________________
> OmegaT-development mailing list
> Ome...@li...
> https://lists.sourceforge.net/lists/listinfo/omegat-development
>
> _______________________________________________
> OmegaT-development mailing list
> Ome...@li...
> https://lists.sourceforge.net/lists/listinfo/omegat-development
> _______________________________________________
> OmegaT-development mailing list
> Ome...@li...
> https://lists.sourceforge.net/lists/listinfo/omegat-development
>

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Hiroshi M. <mi...@no...>]

Hi,

On 2024/10/03 17:16, Thomas CORDONNIER wrote:

> Hi Hiroshi
>
>> - make sortGlossaryEntries to be package-private class method
>> - use lang field (with renaming srcLang) for collation object creation
>
>> - extend GlossarySearcher class ctor with src and target languages
>
> I have an alternative proposal: detach sorting from searching class.
> Create a class org.omegat.util.EntryComparator extends Comparator<ITranslationEntry>
> where you implement the current algorithm
> and which has, indeed, the two languages in the constructor.
>
> That would open this feature to other parts of OmegaT (or to scripts and plugins) which would like to use the same sort criteria.

I like your idea providing the separate class.

>> - improve GlossarySearcherTest test cases with several locales
>
> If you want to write unit tests, you need samples for which you are sure about the expected behaviour and which may cause problems in case of wrong implementation. And, if possible, whose result should differ from unicode comparison (what String.compareTo does)
>
> For french language I think I can find them.
> For Japanese, I suggest that you read the document you quoted in another message (thanks for the link, but I did not have time to read): it would be interesting to find samples where you can test that the Japanese collator has a different behaviour than String.compare

I think I can find them without buying the expensive standard book :-)

--
Hiroshi Miura from north-side in tokyo

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Thomas C. <t_c...@ya...>]

 
> In your version, the order seems to be the same, except that hiragana and katakana are mixed rather than sorted separately.
Interesting, in my own tests I had the impression that nothing changed but maybe my small samples of japanese words were not relevant.
Can you send us a small set of examples (only a small list of terms, no need for a file), showing this difference? I would like to try to reproduce it in my side, to check whenever it is dependent from the OS (my OS is not configured for japanese at all)It would also be useful to check if it is conform to JIS X 4061 mentioned by Hiroshi, but unfortunately it seems that the book is expansive and available only in japanese language.
And of course, the good question is whenever you (and japanese users like Hiroshi) consider this behaviour as better or not.
Thomas

    Le jeudi 3 octobre 2024 à 16:19:35 UTC+2, Julian R. <jul...@gm...> a écrit :  
 
 Hi Thomas

> When the alphabetic sort is done with
 > Japanese, I have noticed that single-byte punctuation and numbers seem
 > to come first, then hiragana, then katakana, and finally Kanji.
> Other single-byte characters would probably come at the beginning too, although I haven't checked for that.


Interesting. Is it the same with my branch (in case you lost it: https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order)? For the moment I could not really see any effect of my changes but I admit that I don't have lot of samples in Japanese.

In your version, the order seems to be the same, except that hiragana and katakana are mixed rather than sorted separately.
Julian



On Thu, 3 Oct 2024 at 07:12, Thomas CORDONNIER via OmegaT-development <ome...@li...> wrote:

 HI all
Since there were a lot of messages, I try to group my answers


    Le mercredi 2 octobre 2024 à 10:49:53 UTC+2, Hiroshi Miura via OmegaT-development <ome...@li...> a écrit :  
 > It uses `String.contains` method that create strange behavior for users.> There are two options to fix it.
> 1. remove two "contains" conditions.  When enabled the option sort by source length, it sort it just with the length.

Yes, that is what OmegaT < 5.8 did

> 2. change "contains" to "startsWith". It does sort  the terms with length when  terms are like "トヨタ" and "トヨタ自動車"  but does not the terms  "Claim" and "Aim"

Yes, this variant seems to be transitive so it would be ok for me.


Le mercredi 2 octobre 2024 à 17:45:01 UTC+2, Julian R. <jul...@gm...> a écrit : 
> When the alphabetic sort is done with > Japanese, I have noticed that single-byte punctuation and numbers seem > to come first, then hiragana, then katakana, and finally Kanji.> Other single-byte characters would probably come at the beginning too, although I haven't checked for that.

Interesting. Is it the same with my branch (in case you lost it: https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order)? For the moment I could not really see any effect of my changes but I admit that I don't have lot of samples in Japanese.


  Le jeudi 3 octobre 2024 à 01:31:41 UTC+2, Hiroshi Miura via OmegaT-development <ome...@li...> a écrit :  
 > The topic for sort order of languages. It depends on locale as Thomas pointed out.
 > There is an international standard for it. In Japanese, there is a sort order standard JIS X 4061
 
> Java has a Collator class for the purpose. I want to propose it for a glossary sorter.

Yes, this was exactly the purpose of my branch: using the language-dependent sort order as directly supported by Java, even if I did not know in detail what the collator for Japanese does.


  Le jeudi 3 octobre 2024 à 05:00:01 UTC+2, Hiroshi Miura <mi...@no...kyo> a écrit :  

> Unfortunately your branch does not pass unit test.
 > The branch has a change like
 
 
 >            if (c == 0) {
 > -                c = o1.getLocText().compareToIgnoreCase(o2.getLocText());
> +                c =compareLanguageDependent(Core.getProject().getProjectProperties().getTargetLanguage(), o1.getSrcText(), o2.getSrcText());
              }
 
 
> Should it be o1.getLocText() instead of o1.getSrcText() ?

You are right, I just pushed the correction.
Hiroshi, I will study your other correction proposals later. In the meantime, could anybody do functional tests on this branch (i.e. with the editor, not only unit tests!)? I would really be interested to know if the language-dependent collator has an effect on Japanese words (in the few examples I had it seemed to give similar results as String.compare but I had only a few). This can be done without your proposals, which are mostly technical.

RegardsThomas

  _______________________________________________
OmegaT-development mailing list
Ome...@li...
https://lists.sourceforge.net/lists/listinfo/omegat-development

_______________________________________________
OmegaT-development mailing list
Ome...@li...
https://lists.sourceforge.net/lists/listinfo/omegat-development
  

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Julian R. <jul...@gm...>]

Hi Thomas

> When the alphabetic sort is done with
> > Japanese, I have noticed that single-byte punctuation and numbers seem
> > to come first, then hiragana, then katakana, and finally Kanji.
> > Other single-byte characters would probably come at the beginning too,
> although I haven't checked for that.


Interesting. Is it the same with my branch (in case you lost it:
> https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order)? For
> the moment I could not really see any effect of my changes but I admit that
> I don't have lot of samples in Japanese.


In your version, the order seems to be the same, except that hiragana and
katakana are mixed rather than sorted separately.

Julian



On Thu, 3 Oct 2024 at 07:12, Thomas CORDONNIER via OmegaT-development <
ome...@li...> wrote:

> HI all
>
> Since there were a lot of messages, I try to group my answers
>
>
>
> Le mercredi 2 octobre 2024 à 10:49:53 UTC+2, Hiroshi Miura via
> OmegaT-development <ome...@li...> a écrit :
>
> > It uses `String.contains` method that create strange behavior for users.
>
> > There are two options to fix it.
> > 1. remove two "contains" conditions.  When enabled the option sort by source length, it sort it just with the length.
>
> Yes, that is what OmegaT < 5.8 did
>
> > 2. change "contains" to "startsWith". It does sort  the terms with length when  terms are like "トヨタ" and "トヨタ自動車"  but does not the terms  "Claim" and "Aim"
>
> Yes, this variant seems to be transitive so it would be ok for me.
>
>
> Le mercredi 2 octobre 2024 à 17:45:01 UTC+2, Julian R. <jul...@gm...> a écrit :
>
> > When the alphabetic sort is done with
> > Japanese, I have noticed that single-byte punctuation and numbers seem
> > to come first, then hiragana, then katakana, and finally Kanji.
> > Other single-byte characters would probably come at the beginning too, although I haven't checked for that.
>
>
> Interesting. Is it the same with my branch (in case you lost it: https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order)? For the moment I could not really see any effect of my changes but I admit that I don't have lot of samples in Japanese.
>
>
>
> Le jeudi 3 octobre 2024 à 01:31:41 UTC+2, Hiroshi Miura via
> OmegaT-development <ome...@li...> a écrit :
>
> > The topic for sort order of languages. It depends on locale as Thomas
> pointed out.
>
> > There is an international standard for it. In Japanese, there is a sort
> order standard JIS X 4061
>
> > Java has a Collator class for the purpose. I want to propose it for a
> glossary sorter.
>
> Yes, this was exactly the purpose of my branch: using the
> language-dependent sort order as directly supported by Java, even if I did
> not know in detail what the collator for Japanese does.
>
>
>
> Le jeudi 3 octobre 2024 à 05:00:01 UTC+2, Hiroshi Miura
> <mi...@no...kyo> a écrit :
>
> > Unfortunately your branch does not pass unit test.
> > The branch has a change like
>
>  >            if (c == 0) {
> > -                c =
> o1.getLocText().compareToIgnoreCase(o2.getLocText());
> > +                c =
> compareLanguageDependent(Core.getProject().getProjectProperties().getTargetLanguage(),
> o1.getSrcText(), o2.getSrcText());
>              }
>
> > Should it be o1.getLocText() instead of o1.getSrcText() ?
>
> You are right, I just pushed the correction.
>
> Hiroshi, I will study your other correction proposals later. In the
> meantime, could anybody do functional tests on this branch (i.e. with the
> editor, not only unit tests!)? I would really be interested to know if the
> language-dependent collator has an effect on Japanese words (in the few
> examples I had it seemed to give similar results as String.compare but I
> had only a few). This can be done without your proposals, which are mostly
> technical.
>
>
> Regards
> Thomas
>
> _______________________________________________
> OmegaT-development mailing list
> Ome...@li...
> https://lists.sourceforge.net/lists/listinfo/omegat-development
>

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Thomas C. <t_c...@ya...>]

 Hi Hiroshi
 
> - make sortGlossaryEntries to be package-private class method
 > - use lang field (with renaming srcLang) for collation object creation

 > - extend GlossarySearcher class ctor with src and target languages
I have an alternative proposal: detach sorting from searching class.Create a class org.omegat.util.EntryComparator extends Comparator<ITranslationEntry>where you implement the current algorithmand which has, indeed, the two languages in the constructor.
That would open this feature to other parts of OmegaT (or to scripts and plugins) which would like to use the same sort criteria.

> - improve GlossarySearcherTest test cases with several locales
If you want to write unit tests, you need samples for which you are sure about the expected behaviour and which may cause problems in case of wrong implementation. And, if possible, whose result should differ from unicode comparison (what String.compareTo does)

For french language I think I can find them.For Japanese, I suggest that you read the document you quoted in another message (thanks for the link, but I did not have time to read): it would be interesting to find samples where you can test that the Japanese collator has a different behaviour than String.compare
RegardsThomas

    Le jeudi 3 octobre 2024 à 01:53:25 UTC+2, Hiroshi Miura <mi...@no...kyo> a écrit :  
 
  
Hi Thomas,
 
 On 2024/09/27 23:17, Thomas CORDONNIER wrote:
  
 
    Hi all 
  If I correctly follow the history of various branches, to Julian's question 
  > > >>> Is there any way that the option of sorting the full list by the length of the source term, as was the case  previously (eg 5.7.1), could be included as an option? Of course, with priority given to the writable glossary. 
  it seems to me that option to use source term length as primary criteria has been set as an option in 6.1 and 6.0.1 (but not in 5.8)
  
  If the option is activated, we use 1) priority 2) source length 3) source alphabetical order If the option is unactivated, we use 1) priority 2) source alphabetical order 
 The point I want to correct is the last one (source alphabetical order). So I implemented my change in a branch based on main (6.1): if you inactivate sort by length, you can test what a real language-dependent order does, which should be better than actually implemented Unicode order (which gives good result only for English without accented letters at all), and tell us if what Java proposes for Japanese is correct or not.
   
  See this branch : https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order 
   
     
Your branch looks good direction, and we can make it better.
 
A function sortGlossaryEntries is now static to test it easier currently.
 GlossarySearcher class has a field Language lang which is source language.
 
 Here is my proposal;
 
- make sortGlossaryEntries to be package-private class method
 - use lang field (with renaming srcLang) for collation object creation
 - extend GlossarySearcher class ctor with src and target languages
 - update GlossarySearcherTest to use updated ctor
 - improve GlossarySearcherTest test cases with several locales

  -- 
Hiroshi Miura from north-side in tokyo   

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Thomas C. <t_c...@ya...>]

 HI all
Since there were a lot of messages, I try to group my answers


    Le mercredi 2 octobre 2024 à 10:49:53 UTC+2, Hiroshi Miura via OmegaT-development <ome...@li...> a écrit :  
 > It uses `String.contains` method that create strange behavior for users.> There are two options to fix it.
> 1. remove two "contains" conditions.  When enabled the option sort by source length, it sort it just with the length.

Yes, that is what OmegaT < 5.8 did

> 2. change "contains" to "startsWith". It does sort  the terms with length when  terms are like "トヨタ" and "トヨタ自動車"  but does not the terms  "Claim" and "Aim"

Yes, this variant seems to be transitive so it would be ok for me.


Le mercredi 2 octobre 2024 à 17:45:01 UTC+2, Julian R. <jul...@gm...> a écrit : 
> When the alphabetic sort is done with > Japanese, I have noticed that single-byte punctuation and numbers seem > to come first, then hiragana, then katakana, and finally Kanji.> Other single-byte characters would probably come at the beginning too, although I haven't checked for that.

Interesting. Is it the same with my branch (in case you lost it: https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order)? For the moment I could not really see any effect of my changes but I admit that I don't have lot of samples in Japanese.


  Le jeudi 3 octobre 2024 à 01:31:41 UTC+2, Hiroshi Miura via OmegaT-development <ome...@li...> a écrit :  
 > The topic for sort order of languages. It depends on locale as Thomas pointed out.
 > There is an international standard for it. In Japanese, there is a sort order standard JIS X 4061
 
> Java has a Collator class for the purpose. I want to propose it for a glossary sorter.

Yes, this was exactly the purpose of my branch: using the language-dependent sort order as directly supported by Java, even if I did not know in detail what the collator for Japanese does.


  Le jeudi 3 octobre 2024 à 05:00:01 UTC+2, Hiroshi Miura <mi...@no...kyo> a écrit :  

> Unfortunately your branch does not pass unit test.
 > The branch has a change like
 
 
 >            if (c == 0) {
 > -                c = o1.getLocText().compareToIgnoreCase(o2.getLocText());
> +                c =compareLanguageDependent(Core.getProject().getProjectProperties().getTargetLanguage(), o1.getSrcText(), o2.getSrcText());
              }
 
 
> Should it be o1.getLocText() instead of o1.getSrcText() ?

You are right, I just pushed the correction.
Hiroshi, I will study your other correction proposals later. In the meantime, could anybody do functional tests on this branch (i.e. with the editor, not only unit tests!)? I would really be interested to know if the language-dependent collator has an effect on Japanese words (in the few examples I had it seemed to give similar results as String.compare but I had only a few). This can be done without your proposals, which are mostly technical.

RegardsThomas

  

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Hiroshi M. <mi...@no...>]

Hi,

Unfortunately your branch does not pass unit test.
The branch has a change like

if (c == 0) {
- c = o1.getLocText().compareToIgnoreCase(o2.getLocText());
+ c = compareLanguageDependent(Core.getProject().getProjectProperties().getTargetLanguage(), o1.getSrcText(), o2.getSrcText());
}

Should it be o1.getLocText() instead of o1.getSrcText() ?

Hiroshi

On 2024/10/03 8:53, Hiroshi Miura wrote:

> Hi Thomas,
>
> On 2024/09/27 23:17, Thomas CORDONNIER wrote:
>
>> Hi all
>>
>> If I correctly follow the history of various branches, to Julian's question
>>
>>> > >>> Is there any way that the option of sorting the full list by the length of the source term, as was the case previously (eg 5.7.1), could be included as an option? Of course, with priority given to the writable glossary.
>>
>> it seems to me that option to use source term length as primary criteria has been set as an option in 6.1 and 6.0.1 (but not in 5.8)
>>
>> If the option is activated, we use 1) priority 2) source length 3) source alphabetical order
>> If the option is unactivated, we use 1) priority 2) source alphabetical order
>>
>> The point I want to correct is the last one (source alphabetical order). So I implemented my change in a branch based on main (6.1): if you inactivate sort by length, you can test what a real language-dependent order does, which should be better than actually implemented Unicode order (which gives good result only for English without accented letters at all), and tell us if what Java proposes for Japanese is correct or not.
>>
>> See this branch : https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order
>
> Your branch looks good direction, and we can make it better.
>
> A function sortGlossaryEntries is now static to test it easier currently.
> GlossarySearcher class has a field Language lang which is source language.
>
> Here is my proposal;
>
> - make sortGlossaryEntries to be package-private class method
> - use lang field (with renaming srcLang) for collation object creation
> - extend GlossarySearcher class ctor with src and target languages
> - update GlossarySearcherTest to use updated ctor
> - improve GlossarySearcherTest test cases with several locales
>
> --
> Hiroshi Miura from north-side in tokyo

--
Hiroshi Miura from north-side in tokyo

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Hiroshi M. <mi...@no...>]

Hi Thomas,

On 2024/09/27 23:17, Thomas CORDONNIER wrote:

> Hi all
>
> If I correctly follow the history of various branches, to Julian's question
>
>> > >>> Is there any way that the option of sorting the full list by the length of the source term, as was the case previously (eg 5.7.1), could be included as an option? Of course, with priority given to the writable glossary.
>
> it seems to me that option to use source term length as primary criteria has been set as an option in 6.1 and 6.0.1 (but not in 5.8)
>
> If the option is activated, we use 1) priority 2) source length 3) source alphabetical order
> If the option is unactivated, we use 1) priority 2) source alphabetical order
>
> The point I want to correct is the last one (source alphabetical order). So I implemented my change in a branch based on main (6.1): if you inactivate sort by length, you can test what a real language-dependent order does, which should be better than actually implemented Unicode order (which gives good result only for English without accented letters at all), and tell us if what Java proposes for Japanese is correct or not.
>
> See this branch : https://github.com/t-cordonnier/omegat/tree/Glossaries-Sort-Order

Your branch looks good direction, and we can make it better.

A function sortGlossaryEntries is now static to test it easier currently.
GlossarySearcher class has a field Language lang which is source language.

Here is my proposal;

- make sortGlossaryEntries to be package-private class method
- use lang field (with renaming srcLang) for collation object creation
- extend GlossarySearcher class ctor with src and target languages
- update GlossarySearcherTest to use updated ctor
- improve GlossarySearcherTest test cases with several locales

--
Hiroshi Miura from north-side in tokyo

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Hiroshi M. <mi...@no...>]

Hi,

The topic for sort order of languages. It depends on locale as Thomas pointed out.
There is an international standard for it. In Japanese, there is a sort order standard JIS X 4061

Java has a Collator class for the purpose. I want to propose it for a glossary sorter.

As you know, OmegaT project has a sort order unit test.
test/src/org/omegat/gui/glossary/FindGlossaryThreadTest.java

Current test uses quite simple entries. We can improve quality by adding test cases and using more entries.
Because we discuss the sort order which depends on a locale, the test cases should be aware of locales.

Currently, only Latin alphabet and kanji are tested.

entries

.add(

new

GlossaryEntry(

"dog"

,

"doggy"

,

"cdog"

,

false

,

null

));

entries

.add(

new

GlossaryEntry(

"cat"

,

"catty"

,

"ccat"

,

false

,

null

));

entries

.add(

new

GlossaryEntry(

"cat"

,

"mikeneko"

,

"ccat"

,

false

,

null

));

entries

.add(

new

GlossaryEntry(

"zzz"

,

"zzz"

,

"czzz"

,

true

,

null

));

entries

.add(

new

GlossaryEntry(

"horse"

,

"catty"

,

"chorse"

,

false

,

null

));

entries

.add(

new

GlossaryEntry(

"

向上

"

,

"enhance"

,

""

,

false

,

null

));

entries

.add(

new

GlossaryEntry(

"

向

"

,

"direct"

,

""

,

false

,

null

));

entries

.add(

new

GlossaryEntry(

"

上

"

,

"up"

,

""

,

false

,

null

));

Could you suggest additional entries that should be tested?

On 2024/10/03 0:44, Julian R. wrote:

> Thanks Hiroshi.
>
> It would be good to see these two options in action, but my initial feeling is that 2 would be an improvement. It would of course mean some related terms would be omitted from the group, but it would also remove quite a lot of the shorter unrelated terms, which would be a benefit.
>
> When the alphabetic sort is done with Japanese, I have noticed that single-byte punctuation and numbers seem to come first, then hiragana, then katakana, and finally Kanji.
> Other single-byte characters would probably come at the beginning too, although I haven't checked for that.
> Would it be reasonably straightforward to put single-byte punctuation and numbers at the end, after Kanji?

Hiroshi

--
Hiroshi Miura from north-side in tokyo

Re: [OmTdev] [Omegat-users] Testers for OmegaT 6.0.1 ?

[Julian R. <jul...@gm...>]

Thanks Hiroshi.

It would be good to see these two options in action, but my initial feeling
is that 2 would be an improvement. It would of course mean some related
terms would be omitted from the group, but it would also remove quite a lot
of the shorter unrelated terms, which would be a benefit.

When the alphabetic sort is done with Japanese, I have noticed that
single-byte punctuation and numbers seem to come first, then hiragana, then
katakana, and finally Kanji.
Other single-byte characters would probably come at the beginning too,
although I haven't checked for that.
Would it be reasonably straightforward to put single-byte punctuation and
numbers at the end, after Kanji?

One other complication I have noticed with grouping similar terms together
is in a case in which the main source term of a group appears both in the
priority glossary and another glossary, but sub-terms that would be grouped
together with the main term appear only in a glossary other than the
priority glossary.
In this case, if "Merge alternate definitions of the same term" is checked,
the main source term is included in the priority glossary list, but the
sub-terms only appear further down in the main part of the list, but
without the main source term. This can be particularly confusing if the
sub-terms don't start with the same character as the main source term, as
they appear out of order in relation to other terms in the list.
When grouping of similar source terms is used, it might make more sense for
the sub-terms in this case to be listed together with the main term in the
priority glossary.

Thanks.

Julian

On Wed, 2 Oct 2024 at 09:50, Hiroshi Miura via OmegaT-development <
ome...@li...> wrote:

> Hi all,
>
> I feel there is still under consideration the sort order OmegaT _should_
> do.
>
> Here is a code snippet from `org.omegat.gui.glossary.softGlossaryEntries`
>
> ```static void sortGlossaryEntries(List<GlossaryEntry> entries) {
>     entries.sort((o1, o2) -> {
>         int p1 = o1.getPriority() ? 1 : 2;
>         int p2 = o2.getPriority() ? 1 : 2;
>         int c = p1 - p2;
>         if (c == 0 && Preferences.isPreferenceDefault(Preferences.GLOSSARY_SORT_BY_SRC_LENGTH, true)
>                 && (o2.getSrcText().contains(o1.getSrcText())
>                         || o1.getSrcText().contains(o2.getSrcText()))) {
>             // longer is better if one contains another            c = o2.getSrcText().length() - o1.getSrcText().length();
>         }
>         // sort source text alphabetically, first ignore a case, then        // consider a case  // ... snip
> }
> ```
>
> It uses `String.contains` method that create strange behavior for users.
>
> There are two options to fix it.
> 1. remove two "contains" conditions.  When enabled the option sort by source length, it sort it just with the length.
> 2. change "contains" to "startsWith". It does sort  the terms with length when  terms are like "トヨタ" and "トヨタ自動車"  but does not the terms  "Claim" and "Aim"
>
> No1 will produce a result of the example
> ---
> ファミリ ー なし = family: none
> トヨタ自動車=toyota moters
> ファミリ ー = family
> ファミリ = family
> トヨタ=toyota
> ミリ = milli-
>  ー = long vowel mark (used in katakana)
> し = (obsc) 10%%24 (kanji is JIS X 0212 kuten 4906)/septillion (American)/quadrillion (British)/
> ---
>
> No2 may become like
>  ---
> トヨタ自動車=toyota moters
> トヨタ=toyota
> ファミリ ー なし = family: none
> ファミリ ー = family
> ファミリ = family
> ミリ = milli-
>   ー = long vowel mark (used in katakana)
>   し = (obsc) 10%%24 (kanji is JIS X 0212 kuten 4906)/septillion (American)/quadrillion (British)/
> ---
>
> I would be better to write unit test to check regression of the sort order with the proposed strings.
>
> Hiroshi
>
> On 2024/10/02 8:45, Jean-Christophe Helary via OmegaT-development wrote:
>
> On Oct 1, 2024, at 22:48, Julian R. <jul...@gm...> <jul...@gm...> wrote:
>
> Hi Thomas
>
> Many thanks for the clarification. It will be interesting to hear what thoughts others have on this issue.
>
> As long as the adopted solution is not buggy and offers a satisfying compromise to you, I’m fine.
>
> Jean-Christophe
>
> --
> Hiroshi Miura from north-side in tokyo
>
> _______________________________________________
> OmegaT-development mailing list
> Ome...@li...
> https://lists.sourceforge.net/lists/listinfo/omegat-development
>