Re: [Bibdesk-develop] Japanese encodings

[Adam R. M. <ama...@ma...>]

On Nov 24, 2006, at 05:47, Christiaan Hofman wrote:

> OK, I understand this part now. The parser reads from the file, not  
> from the data that it gets. So the problem here is with btparse,  
> which seems to have trouble with Japanese encodings.

Are there particular characters it chokes on?  I'm not familiar with  
the Japanese encodings at all, but btparse also chokes on UniChars,  
presumably because of size differences.  That caused some problems  
when we removed the NSString-based parser, since some people had saved  
data as Unicode.

> I can only think of one solution: always hand UTF-8 to the parser  
> (or perhaps for some encodings). This has a problem though. First,  
> the parser should always parse the data instead of the file. This  
> means that the filepath for errors is wrong.

And that's a useful feature I'd hate to lose.  We could do this just  
for certain encodings, though.

> Also we then can't save groups in UTF-8 anymore, because we need to  
> convert the whole file content. This could give problems with  
> compatibility and when saving in ASCII.

I don't see any way around that, other than saving the groups in a  
file wrapper plist or as xattrs.

Adam

>
>
> On 11/24/06, Christiaan Hofman <cmh...@gm...> wrote:
> I really don't understand anymore what goes wrong with (Japanese)  
> string encodings. We are unable to read several characters in those  
> encodings. However saving and reading in UTF-8 works OK. What is  
> really weird is that it doesn't even work if I first convert the  
> data to UTF-8 using
>
> NSString *string = [[[NSString alloc] initWithData:data  
> encoding:encoding] autorelease];
> data = [string dataUsingEncoding:NSUTF8StringEncoding];
> [self setDocumentStringEncoding:NSUTF8StringEncoding];
>
> How would this be different from converting to UTF-8 encoding  
> externally (e.g. using TextEdit) and reading using UTF-8 encoding?
>
> The strings I get (both in Japanese encoding and UTF-8 encoding)  
> look OK. Just the value in the parser is nil.
>
> Christiaan
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to  
> share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV_______________________________________________
> Bibdesk-develop mailing list
> Bib...@li...
> https://lists.sourceforge.net/lists/listinfo/bibdesk-develop