Collator problem: Object #<Object> has no method '_init'

[Serkan Durusoy]

This code:

var ilib = require("ilib").ilib;
var collator = ilib.Collator({locale: "tr"});
var key = collator.sortKey("Ömer");

console.log(key);

throws the following exception:

TypeError: Object #<Object> has no method '_init'
    at Object.$ilib$Collator$.ilib.loadData.callback (/usr/lib/node_modules/ilib/ilib-dyn-full-compiled.js:600:501)
    at Object.<anonymous> (/usr/lib/node_modules/ilib/ilib-dyn-full-compiled.js:91:73)
    at nodeLoader.loadFiles (/usr/lib/node_modules/ilib/nodeglue.js:92:4)
    at Object.$ilib$_callLoadData$ [as _callLoadData] (/usr/lib/node_modules/ilib/ilib-dyn-full-compiled.js:85:267)
    at Object.$ilib$loadData$ [as loadData] (/usr/lib/node_modules/ilib/ilib-dyn-full-compiled.js:90:232)
    at Object.$ilib$Collator$ [as Collator] (/usr/lib/node_modules/ilib/ilib-dyn-full-compiled.js:600:181)
    at Object.<anonymous> (/home/serkandurusoy/Desktop/ilibtest.js:8:21)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)

My node version is 0.10.38 if that matters.

Installed ilib version is 10.0.6

[Serkan Durusoy]

I fired up node 0.12.2 and this time the error shows as:

/root/node_modules/ilib/ilib-dyn-full-compiled.js:600
().replace(/-/g,"_");ilib.Collator.cache[$spec$$]=$collation$$}this._init($col
                                                                ^
TypeError: undefined is not a function
    at Object.<anonymous> (/root/node_modules/ilib/ilib-dyn-full-compiled.js:600:501)
    at Object.<anonymous> (/root/node_modules/ilib/ilib-dyn-full-compiled.js:91:73)
    at nodeLoader.loadFiles (/root/node_modules/ilib/nodeglue.js:92:4)
    at Object.$ilib$_callLoadData$ [as _callLoadData] (/root/node_modules/ilib/ilib-dyn-full-compiled.js:85:267)
    at Object.$ilib$loadData$ [as loadData] (/root/node_modules/ilib/ilib-dyn-full-compiled.js:90:232)
    at Object.$ilib$Collator$ [as Collator] (/root/node_modules/ilib/ilib-dyn-full-compiled.js:600:181)
    at Object.<anonymous> (/root/ilibtest.js:2:21)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)

[Edwin H]

Hi Serkan, I see the same problem when I run it in node interactively with the npm module. But when I add the code to the unit tests so I can debug it, it works fine. There is something odd going on there! I am looking into this and get back to you shortly.

[Edwin H]

Okay, I see the problem now. You need to put the "new" keyword to create a new instance of the ilib.Collator class, like this:

var ilib = require("ilib").ilib;
var collator = new ilib.Collator({locale: "tr"});
var key = collator.sortKey("Ömer");

When you do that, it does not crash any more.

Unfortunately, though, Turkish is not one of the languages for which we have collation data yet so you will get the default English rules. Let me know if you need me to add it.

[Serkan Durusoy]

Ah, that makes perfect sense and I should have seen it. But perhaps you an update the examples since that's misleading.

In the meantime, I have two questions:

1) Is there a complete list of locales available to sortKey? The getLocales() returns an empty array in my case.

2) Is there a way I can help out with Turkish since I need it very badly! :)

[Edwin H]

Ah yes, good catch. I've updated one spot in the inline docs for the Collator class where I forgot to put the "new" keyword. (http://sourceforge.net/p/i18nlib/code/1816/) This will be in the next build 11.0.002. Did you find any others?

1) In the builds, the collator class only supports the following languages:

af (Afrikaans)
ca (Catalan)
de (German)
en (English)
es (Spanish)
et (Estonian)
eu (Basque)
gl (Galician)
it (Italian)
id (Indonesian)
lt (Lithuanian)
lv (Latvian)
ms (Malaysian)
nl (Dutch)
pt (Portuguese)
sw (Swahili)

In the "collation3" branch, which is still in development, the following languages are added:

cs (Czech)
da (Danish)
el (Greek)
fi (Finnish)
fo (Faroese)
he (Hebrew)
ja (Japanese including the kanas)
ko (Korean)
nb (Norwegian Bokmal)
nn (Norwegian Nynorsk)
no (Norwegian)
ru (Russian)
sv (Swedish)
zh (Chinese pinyin + bopomofo)
zh-Hans (Chinese Simplified)
zh-Hant (Chinese Traditional)

I can finish off the work in that branch and merge it to trunk in the next few weeks.

2) Turkish is not too difficult to do, as it is Latin-based and there are only a few modifications from English. I'll work on it. Send an email to marketing@jedlsoft.com and I can let you know when it is ready. It would be great if you could test it and let me know if you find any problems with it.

[Serkan Durusoy]

There are two examples without new at http://docs.jedlsoft.com/ilib/jsdoc/symbols/ilib.Collator.html (sorry that I've given the jsdoc link, I'm having a hard time browsing the source because I'm not used to sourceforge)

Thanks for the news.

Is there anything I can do to get the work on Turkish going, apart from testing?

[Edwin H]

Hi Serkan, I have finished off the Turkish support and merged the collation3 branch to trunk. It will be in the next build, 11.0.002. Please take a look at the Turkish collation unit tests and let me know if they look like they are testing the correct thing: https://sourceforge.net/p/i18nlib/code/HEAD/tree/trunk/js/test/collate/test/testcollation_tr.js

[Serkan Durusoy]

Hi Edwin,

This is great news!

The tests look alright. I think Turkish default sensitivity is "primary" where sorting is case insensitive. I've frankly never seen case sensitive sorting in use.

I have two follow up questions:

1) When do you think you'll be able to release this on npm?
2) Is there a function that returns an array of available locales/sensitivities? console.log(ilib.Locale.getAvailableLocales()) prints out an empty array.

[Serkan Durusoy]

OK, I see 11.0.1 on npm and I'm a happy developer now, thanks!

And I'll be even happier if I can get the locale list programmatically ;)

[Serkan Durusoy]

Upon further experimenting, I find that I always get the same sortkeys whatever locale I try to use. I'm suspecting it cannot find the locales and defaults to something else entirely.

For example:

var ilib = require("ilib").ilib;
var collator = new ilib.Collator({locale: XXX});
var key = collator.sortKey("ö");

yields 1d6 whatever I use for XXX, even "XXX" results in the same sortkey.

[Edwin H]

If you are using build 11.0.001, then it does not have the latest collation changes in it yet. Most collators will fall back to the English/generic collator, so that is why you are seeing the same sort key for every locale you have tried. If you check out Latvian (lv) or Lithuanian (lt) in that build, you will see the difference in sort keys then because those locales are already supported.

The static function Locale.getAvailableLocales() was only documented to return the list of locales that the copy of ilib was pre-assembled with. That is, when the locale data was put right into the JS file directly instead of being loaded dynamically from a json file on disk. This pre-assembled data was intended for use in web pages.

Anyways, I have fixed getAvailableLocales() to return the list of locales that are available on disk in a dynamic data load situation as well. However, I'm not sure what it is you need from that. You can form any locale specifier by choosing a language tag and a region tag and putting a dash in between them. iLib can load information about any arbitrary combination language-region because the locale information for languages is separated from the information about the regions and it can load each and mix them together.

I will do build 002 today so that you can see the latest collation changes and put it up on npm.

[Edwin H]

Hi Serkan, build 002 is up on sourceforge and npm now. Please take a look.

[Serkan Durusoy]

Hi Edwin,

I do confirm that I get different sortkeys with different locales on 11.0.2 from npm.

Thank you very much.

But I still cannot seem to get the list of locales. I've tried:

console.log(ilib.Locale.getAvailableLocales())

which throws

TypeError: Object function (language, region, variant, script){ilib.Locale=require("../lib/Locale.js");return new ilib.Locale(language, region, variant, script);} has no method 'getAvailableLocales'

and

var Locale = new ilib.Locale();
console.log(Locale.getAvailableLocales());

which throws

Object en-US has no method 'getAvailableLocales'

The reason I need this is I want to validate the new ilib.Collator({locale: XXX}) initialization so that if XXX is not among the available locales, I can log a message to the console, informing the developer that the locale is not available and is falling back to en-US.

Also, regarding your comments on dynamic/preassembled, I'm just requiring ilib from npm and assuming all the locale data is already bundled and available.

I'll also comment on client side loading on the other forum thread.

Thanks again.

[Edwin H]

Ah yes, the big change in 11.0 of ilib is the "modularization". That is, instead of hanging everything off of the "ilib" namespace, now you have to require() the individual classes directly that your code is using. This way, you only load in the code you need, which saves both in terms of loading time and some memory footprint. It is also in preparation for the modularization that will be in ES6 whenever that gets approved.

Now, having said that, existing iLib users probably wouldn't upgrade to 11.0 until they are ready because they have to change their code to use it. So to ease the transition, I made a "stub" file which recreates the old ilib namespace dynamically using the new modules. The way the stubs work is that they use require() to load in the "real" code, decorate the ilib namespace with that code (thereby replacing themselves), and then call the code directly. The next time you call a class or function, the "real" code is already loaded and it is called directly.

What you did is call a static method on the ilib.Collator stub, which doesn't exist. I can add it though -- it just needs to require() the Collator class and then call the static method of that. It was an edge case.

The new way to run the code is like this:

// Loads in and runs the initialization code. This must be done before requiring
// in any other classes.
var ilib = require("ilib");  

// require() the collator class directly.
// This sort of relative path syntax works only in nodejs at the moment, but 
// will work in all the other supported environments after build 003 as well:
var Collator=require("ilib/lib/Collator.js");

// This code is basically the same as before but without the "ilib" namespace
var col = new Collator({locale: "tr"});
var key = col.sortKey("ö");

Please see the wiki article I wrote for more detailed info on these changes: [New Modularization Support]

[Serkan Durusoy]

Oh, thanks for the pointer.

It's a shame though since this now breaks the ilib package I created for Meteor.

I need to either create a separate package now for each class, or create a big monolithic package that loads and exports all available classes.

I guess at this point, it would be great if you could publish the meteor package officially since it can be integrated into your build workflow.

If I continue to do it, I'll need to update it manually for each class.

[Edwin H]

I'm not really familiar with Meteor, so I think it would be foolish for me to sign up for the responsibility for that plugin package.

What I can do is create a wrapper file that exports all the classes:

// Loads in and runs the initialization code. The real code for all classes 
// is already loaded underneath here with no stubs, so you don't need any
// additional require(), but it uses up lots of memory. Data will still be
// loaded dynamically.
var ilib = require("ilib/lib/ilib-all.js");  

var supportedLocales = ilib.Locale.getAvailableLocales(); // not a stub
var col = new ilib.Collator({locale: "tr"}); // not a stub
var key = col.sortKey("ö");

Does that work?

[Serkan Durusoy]

Wow! That would be great and should put me right back on track.

I've unpublished the meteor package, but as soon as you push this through to npm, I can republish the updated version.

[Edwin H]

Okay, I have added "ilib-node-all.js" which will be in the next build (003).

[Serkan Durusoy]

Thanks!

[Edwin H]

Build 11.0.003 is published now on this site and npm.

[Serkan Durusoy]

I've published it on atmosphere at https://atmospherejs.com/serkandurusoy/ilib

thanks.