afpfs-ng-devel — Discussion about various technical issues with afpfs-ng.

[Afpfs-ng-devel] afpcmd

[<ra...@rs...>]

Alex,

On Tue, 17 Jun 2008 09:09:25 -0400, Alex deVries <ale...@gm...>
wrote:
> On 17-Jun-08, at 1:28 AM, Ralph Böhme wrote:
>> Am 16.06.2008 um 23:19 schrieb Alex deVries:
>>> The unix permissions situation in netatalk is a bit odd, and it is  
>>> likely true that afpcmd is broken in that it expects netatalk to  
>>> have unix permissions enabled.  I see this as a bug in netatalk,  
>>> but afpfs-ng should be able to handle a broken netatalk.
>>
>> Imo it's not a bug and it's not broken, it's just *different*.
> 
> It depends on what the objective of netatalk is, and there may be  
> different opinions on that.  I had thought that the objective was to  
> make an AFP server that would mimic Apple's AFP servers.  If that were  
> the case, then netatalk would provide unix privs for any AFP 3.x  
> connection.

Let me reitarate: imo netatalk indeed needs to change it's semantics,
that's why I am working on it. But: it need not only support upriv's by
default in most cases, it *must* also provide a mechanism which some call
"smart permissions" (Helios) others call "inherited permissions" (OS X).
Both are ways where the server happily advertises UNIX perms and silently
ignores any FPSetFil[Dir]Parms call to change mode.

Got the point?

I guess the netatalk folks haven't looked closely at what OS X afpd does
when serving a "inherited permissions" volumes and the semantics it uses.
So when you tell them "hell, you gotto use upriv per default", they do not
consider that it is possible to signalize and hand out UNIX privs, but
discard any attempt to change them, and therefore preserver the *very*
important semantics of privilege inheritence.
As said: I am working on it, still in early stages, so it's not time jet to
start discussing in on netatalk-devel. I will start that discussion within
a few weeks -- stay tuned... ;o)

Regards,
-Ralph

Re: [Afpfs-ng-devel] afpcmd

[Alex d. <ale...@gm...>]

On 17-Jun-08, at 1:28 AM, Ralph Böhme wrote:
>
> Am 16.06.2008 um 23:19 schrieb Alex deVries:
>> The unix permissions situation in netatalk is a bit odd, and it is  
>> likely true that afpcmd is broken in that it expects netatalk to  
>> have unix permissions enabled.  I see this as a bug in netatalk,  
>> but afpfs-ng should be able to handle a broken netatalk.
>
> Imo it's not a bug and it's not broken, it's just *different*.

It depends on what the objective of netatalk is, and there may be  
different opinions on that.  I had thought that the objective was to  
make an AFP server that would mimic Apple's AFP servers.  If that were  
the case, then netatalk would provide unix privs for any AFP 3.x  
connection.

But I don't maintain (or contribute to) netatalk, so my opinion  
shouldn't count for much.  I just end up dealing with afpfs-ng bugs  
which are really netatalk config problems.

> What happens is (afaikt):
> - OS X and Helios signalize to clients "I support UNIX perms", and  
> handle all the difference between classic "inherited perms" vs. UNIX  
> perms _internally_, you won't see any difference on the wire in the  
> packets the server generates.
> - netatalk when running withouth "upriv" doesn't set "support UNIX  
> perms"

Oh, of course.  And afpfs-ng detects that when it opens the volume.  
There's just bugs with how it handles that :)

- A

Re: [Afpfs-ng-devel] afpcmd cores after status

[Alex d. <ale...@gm...>]

Hm, I couldn't reproduce this easily.

Could you run it through gdb and make sure that the variable called  
text actually has a \0 somewhere?

- A



On 17-Jun-08, at 5:58 AM, <ra...@rs...> <ra...@rs...> wrote:

> Hi list,
>
> afpcmd cores after in "status" after connecting to a netalk upriv  
> volume:
>
> ralph@inti:~$ afpcmd
> afpcmd: connect afp://ralph:PASS@localhost:10548/UNIXperms
> Connected to server inti using UAM "DHCAST128"
> Connected to volume UNIXperms
> afpcmd: status
> AFPFS Version: 0.8.1
> UAMs compiled in: Cleartxt Passwrd, No User Authent, Randnum Exchange,
> 2-Way Randnum Exchange, DHCAST128, DHX2
> Segmentation fault
>
> SBT is attached.
>
> Regards
> -Ralph<gdb  
> core 
> .txt 
> > 
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php_______________________________________________
> Afpfs-ng-devel mailing list
> Afp...@li...
> https://lists.sourceforge.net/lists/listinfo/afpfs-ng-devel

[Afpfs-ng-devel] afpcmd cores after status

[<ra...@rs...>]

Hi list,

afpcmd cores after in "status" after connecting to a netalk upriv volume:

ralph@inti:~$ afpcmd
afpcmd: connect afp://ralph:PASS@localhost:10548/UNIXperms
Connected to server inti using UAM "DHCAST128"
Connected to volume UNIXperms
afpcmd: status
AFPFS Version: 0.8.1
UAMs compiled in: Cleartxt Passwrd, No User Authent, Randnum Exchange,
2-Way Randnum Exchange, DHCAST128, DHX2
Segmentation fault

SBT is attached.

Regards
-Ralph

[Afpfs-ng-devel] afpcmd

[Ralph B. <ra...@rs...>]

Am 17.06.2008 um 02:03 schrieb Alex deVries:

> Here's a patch to fix this problem, against 0.8.1.  I tested it
> against netatalk 2.0.3 with Fedora patches, with a volume without unix
> privs enabled.

Thanks!

> Sorry about this... and thanks for reporting it.

Thanks for providing the diff insanely quick! I was prepared to do it  
myself... ;o)

-Ralph

Re: [Afpfs-ng-devel] afpcmd

[Ralph B. <ra...@rs...>]

Alex,

Am 16.06.2008 um 23:19 schrieb Alex deVries:
> The unix permissions situation in netatalk is a bit odd, and it is  
> likely true that afpcmd is broken in that it expects netatalk to  
> have unix permissions enabled.  I see this as a bug in netatalk, but  
> afpfs-ng should be able to handle a broken netatalk.

Imo it's not a bug and it's not broken, it's just *different*.

> I haven't understood why netatalk doesn't have unix privs enabled by  
> default, and my discussions with the netatalk folks haven't cleared  
> that up.

What happens is (afaikt):
- OS X and Helios signalize to clients "I support UNIX perms", and  
handle all the difference between classic "inherited perms" vs. UNIX  
perms _internally_, you won't see any difference on the wire in the  
packets the server generates.
- netatalk when running withouth "upriv" doesn't set "support UNIX  
perms"

>  Any Apple AFP server released since Mac OS 10.1 (except the airport  
> extreme) has that turned on by default.

See above: the difference is not just what is enabled by default, but  
how it's implemented.

> There's another odd netatalk bug which doesn't let you set certain  
> bits with chmod (like the x bit).  That's in raw 2.0.3 (now three  
> years old), but has been fixed in CVS.

I know. I'm working from CSV Base.

> I try to detect these situations with unix privs in afpfs-ng; you  
> can see this by doing a 'status' command.

That gave me a coredump... ;o) If I have the time I'll look into it,  
but afair it coredumped whe working on a volume without upriv.

> Thanks for using afpfs-ng...

Thanks for writing!

Regards,
-Ralph

Re: [Afpfs-ng-devel] afpcmd

[Alex d. <ale...@gm...>]

Here's a patch to fix this problem, against 0.8.1.  I tested it
against netatalk 2.0.3 with Fedora patches, with a volume without unix
privs enabled.

Sorry about this... and thanks for reporting it.


- Alex

Re: [Afpfs-ng-devel] afpcmd

[Alex d. <ale...@gm...>]

Ralph,

The unix permissions situation in netatalk is a bit odd, and it is  
likely true that afpcmd is broken in that it expects netatalk to have  
unix permissions enabled.  I see this as a bug in netatalk, but afpfs- 
ng should be able to handle a broken netatalk.

I haven't understood why netatalk doesn't have unix privs enabled by  
default, and my discussions with the netatalk folks haven't cleared  
that up.  Any Apple AFP server released since Mac OS 10.1 (except the  
airport extreme) has that turned on by default.

There's another odd netatalk bug which doesn't let you set certain  
bits with chmod (like the x bit).  That's in raw 2.0.3 (now three  
years old), but has been fixed in CVS.  Some netatalk versions in some  
distributions fix that also.

I try to detect these situations with unix privs in afpfs-ng; you can  
see this by doing a 'status' command.

In my environment, I have probably only ever tested afpcmd it with  
"options:upriv" configured.  I'm in Japan on business this week, but  
if I have a few mins today I'll see if I can fix this.

Thanks for using afpfs-ng...

- Alex


On 16-Jun-08, at 10:35 AM, <ra...@rs...> <ra...@rs...> wrote:

> Update:
> it seems as though afpcmd requires UNIX perms for cd to work? At  
> least when
> using afpcmd againt my patched netatalk this only works with volumes  
> that
> announce UNIX perms. Therefor I will build an unpatched netatalk  
> HEAD and
> see if behaviour is the same and post results.
>
> Regards
> -Ralph
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> Afpfs-ng-devel mailing list
> Afp...@li...
> https://lists.sourceforge.net/lists/listinfo/afpfs-ng-devel

Re: [Afpfs-ng-devel] afpcmd

[<ra...@rs...>]

Update:
it seems as though afpcmd requires UNIX perms for cd to work? At least when
using afpcmd againt my patched netatalk this only works with volumes that
announce UNIX perms. Therefor I will build an unpatched netatalk HEAD and
see if behaviour is the same and post results.

Regards
-Ralph

[Afpfs-ng-devel] afpcmd

[<ra...@rs...>]

Hello afpfs-ng-devel!

I just began some testing whether I could use afpfs-ng's afpcmd to debug
some netatalk hacking I've been doing. Turns out I'm debugging afpcmd
now... ;o)

I can connect and ls just find, but when I try to cd I get this error:

   afpcmd: cd permtest
   /permtest is not a directory, mode is 00

Is this supposed to work?

Looking at the wire I see afpcmd asking for FPGetFileDirParms("permtest")
to which netatalk is responding as expected -- at least afaikt: I'm in the
middle of rehacking the way netatalk handles/publishes UNIX perms vs.
inherited perms. So I might just be shooting my own foot here, but I'm
almost sure I haven't broken things so badly... Finder.app mounting,
listing etc. still works.

Regards
-Ralph

Re: [Afpfs-ng-devel] Attempting AFP mount of an iPod Touch

[Alex d. <ale...@gm...>]

Zed,

After the mount, what's the output of 'afp_client status'?

- Alex

On Wed, May 7, 2008 at 3:03 PM, Zed A. Shaw <ze...@ze...> wrote:
> Hi Everyone,
>
>  So I got this iPod Touch, and I'm a Linux user with a good collection
>  of music who doesn't much like the snooping and other nasty things
>  Apple does with iTunes.  Well, I managed to get my iPod Touch to run
>  afpd and can mount it very nicely on a mac.  Works amazingly well, from
>  a mac.
>
>  Now I've got afpfs-ng 0.8.1 and fuse 2.7.3-1 for ArchLinux.
>
>  I have /etc/fuse.conf with user_allow_other.
>
>  I have confirmed all the networking is accessible, pings all arounds,
>  afp working from the mac.
>
>  Everything *almost* works except the mount point gets hosed during the
>  mount, and mount_afs prints out weird messages about localhost and such:
>
>  # mkdir /tmp/afptest
>  # chown zedshaw.users /tmp/afptest
>  # ls /tmp/afptest
>  # ls -l /tmp/afptest
>  total 0
>  # ls -ld /tmp/afptest
>  drwxr-xr-x 2 zedshaw users 1 May  7 17:48 /tmp/afptest/
>  # ls -ld /tmp/afptest
>  drwxr-xr-x 2 zedshaw users 1 May  7 17:48 /tmp/afptest/
>
>  # mount_afp -o user=zedshaw,group=users,rw
>  "afp://root:SECRET@192.168.11.4/iPhone Root FileSystem" /tmp/afptest
>
>  Mounting 192.168.11.4 from iPhone Root FileSystem on /tmp/afptest
>  Volume iPhone Root FileSystem changes the server's encoding Mounting of
>  volume iPhone Root FileSystem of server localhost succeeded.
>
>  # ls /tmp/afptest
>  ls: cannot access /tmp/afptest: Input/output error
>  # ls -l /tmp/ | grep afptest
>  ls: cannot access /tmp/afptest: Input/output error
>  ?????????? ? ?       ?       ?            ? afptest
>
>  Now, I can also mount the ipod via curlftpfs (although that don't work
>  as well), and will be trying sshfs, but ideally afpfs would be the best.
>
>  Any pointers about why the above directory comes out all garbage?
>
>  Thanks in advance.
>
>  --
>  Zed A. Shaw
>  - Hate: http://savingtheinternetwithhate.com/
>  - Good: http://www.zedshaw.com/
>  - Evil: http://yearofevil.com/
>
>  -------------------------------------------------------------------------
>  This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>  Don't miss this year's exciting event. There's still time to save $100.
>  Use priority code J8TL2D2.
>  http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
>  _______________________________________________
>  Afpfs-ng-devel mailing list
>  Afp...@li...
>  https://lists.sourceforge.net/lists/listinfo/afpfs-ng-devel
>

[Afpfs-ng-devel] Attempting AFP mount of an iPod Touch

[Zed A. S. <ze...@ze...>]

Hi Everyone,

So I got this iPod Touch, and I'm a Linux user with a good collection
of music who doesn't much like the snooping and other nasty things
Apple does with iTunes.  Well, I managed to get my iPod Touch to run
afpd and can mount it very nicely on a mac.  Works amazingly well, from
a mac.

Now I've got afpfs-ng 0.8.1 and fuse 2.7.3-1 for ArchLinux.

I have /etc/fuse.conf with user_allow_other.

I have confirmed all the networking is accessible, pings all arounds,
afp working from the mac.

Everything *almost* works except the mount point gets hosed during the
mount, and mount_afs prints out weird messages about localhost and such:

# mkdir /tmp/afptest
# chown zedshaw.users /tmp/afptest
# ls /tmp/afptest
# ls -l /tmp/afptest
total 0
# ls -ld /tmp/afptest
drwxr-xr-x 2 zedshaw users 1 May  7 17:48 /tmp/afptest/
# ls -ld /tmp/afptest
drwxr-xr-x 2 zedshaw users 1 May  7 17:48 /tmp/afptest/

# mount_afp -o user=zedshaw,group=users,rw
"afp://root:SECRET@192.168.11.4/iPhone Root FileSystem" /tmp/afptest

Mounting 192.168.11.4 from iPhone Root FileSystem on /tmp/afptest
Volume iPhone Root FileSystem changes the server's encoding Mounting of
volume iPhone Root FileSystem of server localhost succeeded.

# ls /tmp/afptest
ls: cannot access /tmp/afptest: Input/output error
# ls -l /tmp/ | grep afptest
ls: cannot access /tmp/afptest: Input/output error
?????????? ? ?       ?       ?            ? afptest

Now, I can also mount the ipod via curlftpfs (although that don't work
as well), and will be trying sshfs, but ideally afpfs would be the best.

Any pointers about why the above directory comes out all garbage?

Thanks in advance.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

Re: [Afpfs-ng-devel] precompose and decompose

[HAT <ha...@fa...>]

Hi.

I am checking new source written by Michael Ulbrich.

I wrote a perl script for this test.
This script make the directories with names U+00000080 - U+0002FFFF.
In case of running on Linux, the precomposed directory names are made.
In case of running on Mac OS X 10.5.2, the imperfect decomposed directory
names are made.
If the application don't use the CoreFoundation, correct name are
not made.
When these directories are copied to another HFS+ by the Finder,
these become correct NFD filename.

I find that Apple tell lie.

> Mac OS X 10.5           Unicode 4  (compose-table is same as 3.2)

Mac OS 10.5.2 is based on Unicode 5.
The following table is used.

> Change Unicode 4.x -> 5.0 -----------------------------------------
> added the following table.
> 
> {0x00001B06, 0x00001B0500001B35},   /* BALINESE LETTER AKARA TEDUNG */
> {0x00001B08, 0x00001B0700001B35},   /* BALINESE LETTER IKARA TEDUNG */
> {0x00001B0A, 0x00001B0900001B35},   /* BALINESE LETTER UKARA TEDUNG */
> {0x00001B0C, 0x00001B0B00001B35},   /* BALINESE LETTER RA REPA TEDUNG */
> {0x00001B0E, 0x00001B0D00001B35},   /* BALINESE LETTER LA LENGA TEDUNG */
> {0x00001B12, 0x00001B1100001B35},   /* BALINESE LETTER OKARA TEDUNG */
> {0x00001B3B, 0x00001B3A00001B35},   /* BALINESE VOWEL SIGN RA REPA TEDUNG */
> {0x00001B3D, 0x00001B3C00001B35},   /* BALINESE VOWEL SIGN LA LENGA TEDUNG */
> {0x00001B40, 0x00001B3E00001B35},   /* BALINESE VOWEL SIGN TALING TEDUNG */
> {0x00001B41, 0x00001B3F00001B35},   /* BALINESE VOWEL SIGN TALING REPA TEDUNG */
> {0x00001B43, 0x00001B4200001B35},   /* BALINESE VOWEL SIGN PEPET TEDUNG */


> However, U2000 to U2FFF, UFE30 to UFE4F, and U2F800 to U2FA1F are not
> decomposed. It is for compatibilty.

CJK COMPATIBILITY IDEOGRAPH (U+2F800 - U+2FA1D) are decomposed.

After checking Michael's source, I will write a patch for Hangul.

-- 
HAT

Re: [Afpfs-ng-devel] precompose and decompose

[Michael U. <mu...@re...>]

HAT wrote:

<SNIP>

Hi HAT,

thanks for your in depth explanation!

Unfortunately, I currently do not have the time to fully comprehend this ...

> -----------------------------------------------------------------------
>
> At first, afpfs-ng should implement bare UCS4precompose() that having
> perfect table[] for Mac OS X 10.2 and later.

Yes, ok! To get things going, I will try to adapt the routines from
unicode.c to use the new (complete/perfect) table[] and UCS4 internally
tomorrow. Please stay tuned!

> The Mac OS X 10.0/10.1 support is the next step.

Best regards ... Michael

Re: [Afpfs-ng-devel] precompose and decompose

[HAT <ha...@fa...>]

Mac OS X don't use bare precomopsition and bare decomosition.
NFC and NFD are implemented. These are complete normalization.
The processes of NFC and NFD are as follows.

    NFC
    decompose(including singleton) -> ordering -> precompose

    NFD
    decompose(including singleton) -> ordering

NFC and NFD are irreversible conversions.
If the table of decomposition have a term "X -> Y",

   NFC
   X -> Y
   Y -> Y

   NFD
   X -> Y
   Y -> Y

"X" doesn't exist in HFS+. It is saved as "Y".

Maybe, the problem will not occur even if afpfs-ng imprement or not
because Mac OS X 10.2 and later implement NFC and NFD perfectly.

Netatalk don't implement Singleton. I think that it's ideal.
Netatalk use not HFS+ but general UNIX filesystem.
"X" and "Y" might exist in one directory.
It is not ideal that netatalk report to the clients, "I have Y and Y".
It's better reporting "I have X and Y".
This is not a perfect method. But I think it's better.

If afpfs-ng implement singleton and netatalk have X only...
   Netatalk says "I have X"
   Afpfs-ng convert from X to Y.
   Afpfs-ng says "Please give me Y"
   Netatalk says "I have no Y"

Therefore, I think that afpfs-ng should not implement Singleton.

AFP server might have to act like a HFS+ as much as possible.
But afpfs-ng should act like a UNIX filesystem.
Afpfs-ng need not perfectly implement NFC and NFD.

-----------------------------------------------------------------------

Mac OS X 10.0/10.1's NFC and NFD are imprefect because it's besed on
Unicode 2.x.
The decompose() and ordering() are necessary for Mac OS X 10.0/10.1's
imperfect filenames.

	  decompose(excluding singleton) -> ordering -> precompose

However, A complete Mac OS X 10.0/10.1 support will be impossible maybe.
It is unstable that Mac OS X 10.2/10.3/10.4 connect to 10.0/10.1.

-----------------------------------------------------------------------

At first, afpfs-ng should implement bare UCS4precompose() that having
perfect table[] for Mac OS X 10.2 and later.

The Mac OS X 10.0/10.1 support is the next step.

-- 
HAT

Re: [Afpfs-ng-devel] precompose and decompose

[HAT <ha...@fa...>]

> > Summary.
> > Unicode 3.2 (same as 4.0) is needed for Mac OS X 10.2 -10.5.
> > Unicode 5 will be needed for future Mac OS X maybe.
> > Canonical ordering is needed for Mac OS X 10.0 - 10.1.
> > Singleton is needed??? We should discuss it.
> 
> Before I start digging up the Unicode standard and try to understand
> what 'singleton' means, could you please give me a short intro?

Decomposition from one character to one character.

See.
http://www.unicode.org/unicode/reports/tr15/index.html
http://www.unicode.org/Public/UNIDATA/CompositionExclusions.txt

This is irreversible conversion.

See also attach files.

-- 
HAT

Re: [Afpfs-ng-devel] precompose and decompose

[HAT <ha...@fa...>]

Oh, Composition is difficult.

>> Apple adopts the latest Unicode.
>> However, U2000 to U2FFF, UFE30 to UFE4F, and U2F800 to U2FA1F are not
>> decomposed. It is for compatibilty.
>
>There are still some decompositions from the range 2000 - 2FFF listed in
>our table. I've checked with 2260, 226E, 226F and 219A and they are
>_not_ decomposed in HFS+ on 10.4. From my POV these entries should be
>removed from the list. What's your opinion? (I don't oversee, whether
>keeping them in the list might cause any harm ...)

Because Apple promise these range positively, 
operation is the same even if it is supported or not.

When afpfs-ng connect to Apple's AFP server,
the problem doesn't occur at all.

Other *stupid* AFP server might decompose these characters...
I don't know whether these are *stupid* AFP servers.

Apple say "Don't DEcompose these range."
Apple don't say "Don't PREcompose these range."

Therefore, I think that afpfs-ng should precompose these range.

-- 
HAT

Re: [Afpfs-ng-devel] precompose and decompose

[Michael U. <mu...@re...>]

HAT wrote:
> Hmm...
> It is difficult for me to explain this problem because I am not good
> at English.
> 
> Mac OS 8                Unicode 1.x
> Mac OS 9 - X 10.1       Unicode 2.x
> Mac OS X 10.2 - 10.4    Unicode 3.2
> Mac OS X 10.5           Unicode 4  (compose-table is same as 3.2)
> 
> When MacOS is upgraded from old version to newer version,
> the installer check the filesystem and rewrite filenames.
> 
> Change Unicode 1.x -> 2.x -----------------------------------------
> Hangul is re-defined. (imcompatible)
> When upgrading MacOS8 to MacOS9/10.0/10.1, Hangul code is changed.
> 
> Change Unicode 2.x -> 3.x -----------------------------------------
> Composition of Unicode 2.x is buggy.
> Unicode 3.x defines Canonical composision, Canonical ordering and Singleton.
> Unicode 3.x has "upper-compatibility" with Unicode 2.x by canonical
> normalization.
> When upgrading MacOSX10.0/10.1 to 10.2, the following filenames is decomposed.
> 
> {0x0001D15E, 0x0001D1570001D165},   /* MUSICAL SYMBOL HALF NOTE */
> {0x0001D15F, 0x0001D1580001D165},   /* MUSICAL SYMBOL QUARTER NOTE */
> {0x0001D160, 0x0001D15F0001D16E},   /* MUSICAL SYMBOL EIGHTH NOTE */
> {0x0001D161, 0x0001D15F0001D16F},   /* MUSICAL SYMBOL SIXTEENTH NOTE */
> {0x0001D162, 0x0001D15F0001D170},   /* MUSICAL SYMBOL THIRTY-SECOND NOTE */
> {0x0001D163, 0x0001D15F0001D171},   /* MUSICAL SYMBOL SIXTY-FOURTH NOTE */
> {0x0001D164, 0x0001D15F0001D172},   /* MUSICAL SYMBOL ONE HUNDRED TWENTY-EIGHTH NOTE */
> {0x0001D1BB, 0x0001D1B90001D165},   /* MUSICAL SYMBOL MINIMA */
> {0x0001D1BC, 0x0001D1BA0001D165},   /* MUSICAL SYMBOL MINIMA BLACK */
> {0x0001D1BD, 0x0001D1BB0001D16E},   /* MUSICAL SYMBOL SEMIMINIMA WHITE */
> {0x0001D1BF, 0x0001D1BB0001D16F},   /* MUSICAL SYMBOL FUSA WHITE */
> {0x0001D1BE, 0x0001D1BC0001D16E},   /* MUSICAL SYMBOL SEMIMINIMA BLACK */
> {0x0001D1C0, 0x0001D1BC0001D16F},   /* MUSICAL SYMBOL FUSA BLACK */

Hi HAT,

yes, now I see the light ;-)

I've tried this on my MAC according to the example (1D15E) you gave
yesterday and it is actually decomposed (1D157, 1D165) when written to
the file system. Hence I agree to include these decompositions in the table.

> There are some changes, too.
> 
> Change Unicode 3.x -> 4.x -----------------------------------------
> Composition is same.
> 
> Change Unicode 4.x -> 5.0 -----------------------------------------
> added the following table.
> 
> {0x00001B06, 0x00001B0500001B35},   /* BALINESE LETTER AKARA TEDUNG */
> {0x00001B08, 0x00001B0700001B35},   /* BALINESE LETTER IKARA TEDUNG */
> {0x00001B0A, 0x00001B0900001B35},   /* BALINESE LETTER UKARA TEDUNG */
> {0x00001B0C, 0x00001B0B00001B35},   /* BALINESE LETTER RA REPA TEDUNG */
> {0x00001B0E, 0x00001B0D00001B35},   /* BALINESE LETTER LA LENGA TEDUNG */
> {0x00001B12, 0x00001B1100001B35},   /* BALINESE LETTER OKARA TEDUNG */
> {0x00001B3B, 0x00001B3A00001B35},   /* BALINESE VOWEL SIGN RA REPA TEDUNG */
> {0x00001B3D, 0x00001B3C00001B35},   /* BALINESE VOWEL SIGN LA LENGA TEDUNG */
> {0x00001B40, 0x00001B3E00001B35},   /* BALINESE VOWEL SIGN TALING TEDUNG */
> {0x00001B41, 0x00001B3F00001B35},   /* BALINESE VOWEL SIGN TALING REPA TEDUNG */
> {0x00001B43, 0x00001B4200001B35},   /* BALINESE VOWEL SIGN PEPET TEDUNG */

These decompositions are already included in the table.

> --------------------------------------------------------------------------
> 
> I tested all of characters about MacOSX 10.1/10.2/10.4 before.
> These strictly observe the Unicode Standard.
> 
>>> Do the tables need to be AFP version specific?
> 
> It is not compatible between Unicode 1.x and 2.x.
> Mac OS 8 is based on Unicode 1.x. It's no problem because AFP2 don't
> use Unicode.
> Unicode 2.x and later have upper-compatibility.
> Therefore, newest Unicode should be used.
> 
>> well, from my understanding the decomposition table does not depend on
>> the version of AFP but on the _filesystem_ used by the server OS.
>> According to document tn1150 the decomposition table (tn1150table) is
>> specified for HFS plus which was introduced with MAC OS 8.1.
> 
> Never trust Apple's documentation.
> Try to check your machine.
> 
>> tn1150 further states under "Unicode subtleties" that:
>>
>> --------------------------
>>
>> IMPORTANT:
>> An implementation must not use the Unicode utilities implemented by its
>> native platform (for decomposition
>> and comparison), unless those algorithms are equivalent to the HFS Plus
>> algorithms defined here, and are
>> guaranteed to be so forever. This is rarely the case. Platform
>> algorithms tend to evolve with the Unicode
>> standard. The HFS Plus algorithms cannot evolve because such evolution
>> would invalidate existing HFS Plus
>> volumes.
>>
>> --------------------------
> 
> Do not believe this.
> Apple doesn't say the lie.
> Because the documents have no been renewed, it is not suitable for
> the current state. 
> 
> Apple adopts the latest Unicode.
> However, U2000 to U2FFF, UFE30 to UFE4F, and U2F800 to U2FA1F are not
> decomposed. It is for compatibilty.

There are still some decompositions from the range 2000 - 2FFF listed in
our table. I've checked with 2260, 226E, 226F and 219A and they are
_not_ decomposed in HFS+ on 10.4. From my POV these entries should be
removed from the list. What's your opinion? (I don't oversee, whether
keeping them in the list might cause any harm ...)

<SNIP>

>>>>> Will you be able (and willing ... ;-)  ) to do the rewrite? I might do
>>>>> it as well, but am not sure when I will find the time to actually do s
>> o.
>>>> It's possible.
>>>> But I do not understand all of source.
>>>> Is it the following files that use UCS2 as internal code?
>>>>
>>>> codepage.c
>>>> unicode.c
>>>> unicode.h
>>>>
>>>>> If you have a patch, I will help with the testing, though.
>>>> First of all, I wrote a sample header file.
>>>> This is based on Unicode 5.0.0.
>> Yeah, it looks good, but it again contains all the decompositions, which
>> were deleted from our current table because they were not in tn1150table.
>>
>> Let's try to find a consensus on what should be in the table before
>> going on with the code here.

So, we probably reached that agreement now ...

>> Another question to HAT:
>>
>> Reading tn1150 page 34 I found the following sentence:
>>
>> In addition, the Korean Hangul characters with codes in the range u+AC00
>> through u+D7A3 are illegal and must be replaced with the
>> equivalent sequence of conjoining jamos, as described in the Unicode 2.0
>> book, section 3.10.
>>
>> Probably we should add these conversions to make Korean Hangul work -
>> what do you think?
> 
> It's not necessary
> This change is from Unicode 1.x to 2.x.
> Korean Mac OS 8/9 use not Unicode 1.x but MacKorean via AFP2.
> Mac OS X is based on 2.x and later via AFP3.

Ok.

> Summary.
> Unicode 3.2 (same as 4.0) is needed for Mac OS X 10.2 -10.5.
> Unicode 5 will be needed for future Mac OS X maybe.
> Canonical ordering is needed for Mac OS X 10.0 - 10.1.
> Singleton is needed??? We should discuss it.

Before I start digging up the Unicode standard and try to understand
what 'singleton' means, could you please give me a short intro?

I will take a look at the rest of the code and come back with a UCS4
based solution.

Thanks + Best regards ... Michael

Re: [Afpfs-ng-devel] precompose and decompose

[HAT <ha...@fa...>]

Compose test.

/* MUSICAL SYMBOL HALF NOTE */

UCS4 composition
1D15E = 1D157 + 1D165

UTF16 (surrogate pair)
1D15E = D834 + DD5E

UTF8 composition
F0 9D 85 9E = F0 9D 85 97 + F0 9D 85 A5

1) Using Finder, make directory
2) Using command line shell, dump stdout of ls command

hat@imacchan:~/compose >"ls" -w1 | od -x
0000000      9df0    9785    9df0    a585    000a                        
0000011

This is little endian because my mac is intel.

-- 
HAT

Re: [Afpfs-ng-devel] precompose and decompose

[HAT <ha...@fa...>]

Hmm...
It is difficult for me to explain this problem because I am not good
at English.

Mac OS 8                Unicode 1.x
Mac OS 9 - X 10.1       Unicode 2.x
Mac OS X 10.2 - 10.4    Unicode 3.2
Mac OS X 10.5           Unicode 4  (compose-table is same as 3.2)

When MacOS is upgraded from old version to newer version,
the installer check the filesystem and rewrite filenames.

Change Unicode 1.x -> 2.x -----------------------------------------
Hangul is re-defined. (imcompatible)
When upgrading MacOS8 to MacOS9/10.0/10.1, Hangul code is changed.

Change Unicode 2.x -> 3.x -----------------------------------------
Composition of Unicode 2.x is buggy.
Unicode 3.x defines Canonical composision, Canonical ordering and Singleton.
Unicode 3.x has "upper-compatibility" with Unicode 2.x by canonical
normalization.
When upgrading MacOSX10.0/10.1 to 10.2, the following filenames is decomposed.

{0x0001D15E, 0x0001D1570001D165},   /* MUSICAL SYMBOL HALF NOTE */
{0x0001D15F, 0x0001D1580001D165},   /* MUSICAL SYMBOL QUARTER NOTE */
{0x0001D160, 0x0001D15F0001D16E},   /* MUSICAL SYMBOL EIGHTH NOTE */
{0x0001D161, 0x0001D15F0001D16F},   /* MUSICAL SYMBOL SIXTEENTH NOTE */
{0x0001D162, 0x0001D15F0001D170},   /* MUSICAL SYMBOL THIRTY-SECOND NOTE */
{0x0001D163, 0x0001D15F0001D171},   /* MUSICAL SYMBOL SIXTY-FOURTH NOTE */
{0x0001D164, 0x0001D15F0001D172},   /* MUSICAL SYMBOL ONE HUNDRED TWENTY-EIGHTH NOTE */
{0x0001D1BB, 0x0001D1B90001D165},   /* MUSICAL SYMBOL MINIMA */
{0x0001D1BC, 0x0001D1BA0001D165},   /* MUSICAL SYMBOL MINIMA BLACK */
{0x0001D1BD, 0x0001D1BB0001D16E},   /* MUSICAL SYMBOL SEMIMINIMA WHITE */
{0x0001D1BF, 0x0001D1BB0001D16F},   /* MUSICAL SYMBOL FUSA WHITE */
{0x0001D1BE, 0x0001D1BC0001D16E},   /* MUSICAL SYMBOL SEMIMINIMA BLACK */
{0x0001D1C0, 0x0001D1BC0001D16F},   /* MUSICAL SYMBOL FUSA BLACK */

There are some changes, too.

Change Unicode 3.x -> 4.x -----------------------------------------
Composition is same.

Change Unicode 4.x -> 5.0 -----------------------------------------
added the following table.

{0x00001B06, 0x00001B0500001B35},   /* BALINESE LETTER AKARA TEDUNG */
{0x00001B08, 0x00001B0700001B35},   /* BALINESE LETTER IKARA TEDUNG */
{0x00001B0A, 0x00001B0900001B35},   /* BALINESE LETTER UKARA TEDUNG */
{0x00001B0C, 0x00001B0B00001B35},   /* BALINESE LETTER RA REPA TEDUNG */
{0x00001B0E, 0x00001B0D00001B35},   /* BALINESE LETTER LA LENGA TEDUNG */
{0x00001B12, 0x00001B1100001B35},   /* BALINESE LETTER OKARA TEDUNG */
{0x00001B3B, 0x00001B3A00001B35},   /* BALINESE VOWEL SIGN RA REPA TEDUNG */
{0x00001B3D, 0x00001B3C00001B35},   /* BALINESE VOWEL SIGN LA LENGA TEDUNG */
{0x00001B40, 0x00001B3E00001B35},   /* BALINESE VOWEL SIGN TALING TEDUNG */
{0x00001B41, 0x00001B3F00001B35},   /* BALINESE VOWEL SIGN TALING REPA TEDUNG */
{0x00001B43, 0x00001B4200001B35},   /* BALINESE VOWEL SIGN PEPET TEDUNG */

--------------------------------------------------------------------------

I tested all of characters about MacOSX 10.1/10.2/10.4 before.
These strictly observe the Unicode Standard.

>> Do the tables need to be AFP version specific?

It is not compatible between Unicode 1.x and 2.x.
Mac OS 8 is based on Unicode 1.x. It's no problem because AFP2 don't
use Unicode.
Unicode 2.x and later have upper-compatibility.
Therefore, newest Unicode should be used.

>well, from my understanding the decomposition table does not depend on
>the version of AFP but on the _filesystem_ used by the server OS.
>According to document tn1150 the decomposition table (tn1150table) is
>specified for HFS plus which was introduced with MAC OS 8.1.

Never trust Apple's documentation.
Try to check your machine.

>tn1150 further states under "Unicode subtleties" that:
>
>--------------------------
>
>IMPORTANT:
>An implementation must not use the Unicode utilities implemented by its
>native platform (for decomposition
>and comparison), unless those algorithms are equivalent to the HFS Plus
>algorithms defined here, and are
>guaranteed to be so forever. This is rarely the case. Platform
>algorithms tend to evolve with the Unicode
>standard. The HFS Plus algorithms cannot evolve because such evolution
>would invalidate existing HFS Plus
>volumes.
>
>--------------------------

Do not believe this.
Apple doesn't say the lie.
Because the documents have no been renewed, it is not suitable for
the current state. 

Apple adopts the latest Unicode.
However, U2000 to U2FFF, UFE30 to UFE4F, and U2F800 to U2FA1F are not
decomposed. It is for compatibilty.

>Given that HFS Plus is still the preferred filesystem for OSX (even for
>the latest version - please correct me if I'm wrong here) the
>decomposition table _may_ not be changed for the reasons given above. If
>the encoding of HFS Plus names would change with every release of OSX
>the volumes would no longer be compatible.
>
>> On 30-Mar-08, at 12:59 PM, HAT wrote:
>> 
>>> Hi.
>>>
>>>> The resulting table was further checked against:
>>>> http://developer.apple.com/technotes/tn/tn1150table.html
>>>>
>>>> and decompositions not marked "illegal" in tn1150table were removed fr
>om
>>>> the result.
>>>
>>> The tn1150table is based on Unicode 2.x (Mac OS X 10.1 and earlier).
>>> Unicode 2.x has no concept of "ordering".
>>> Mac OS X 10.2 and later are based on Unicode 3.2.
>>> Recent Leopard is based on Unicode 4.
>>> The character has increased.
>>> Maybe, Mac OS X 10.6 will be based on Unicode 5.x.
>>> I think that afpfs-ng should use newest Unicode.
>>> At present, latest version is 5.0.0.
>
>HAT, please take a look at the arguments given above. IMHO the
>conversion table is (and _must_ stay) independent of any changes in
>Unicode support for the different OSX releases.
>
>>>> Please let me know, if there is a newer version of the unicode
>>>> decomposition table for HFS Plus.
>>>
>>> Yes, newest version is 5.0.0.
>>> However, it's not necesary to compare it with the tn1150table.
>>>
>>>> Your suggestion (from your first message) to change the internal
>>>> character representation in lib/unicode.c from UCS2 to UCS4 absolutely
>>>> makes sense
>>>
>>> Ok.
>>>
>>>> although there are currently no decompositions in the range
>>>>> U+010000 used by HFS Plus that I'm aware of.
>>>
>>> Mac OS 10.5.2 Leopard support the following decomposition.
>>>
>>> 0x1D15E, 0x1D157 0x1D165  MUSICAL SYMBOL HALF NOTE
>>> 0x1D15F, 0x1D158 0x1D165  MUSICAL SYMBOL QUARTER NOTE
>>> 0x1D160, 0x1D15F 0x1D16E  MUSICAL SYMBOL EIGHTH NOTE
>>> 0x1D161, 0x1D15F 0x1D16F  MUSICAL SYMBOL SIXTEENTH NOTE
>>> 0x1D162, 0x1D15F 0x1D170  MUSICAL SYMBOL THIRTY-SECOND NOTE
>>> 0x1D163, 0x1D15F 0x1D171  MUSICAL SYMBOL SIXTY-FOURTH NOTE
>>> 0x1D164, 0x1D15F 0x1D172  MUSICAL SYMBOL ONE HUNDRED TWENTY-EIGHTH NOTE
>>> 0x1D1BB, 0x1D1B9 0x1D165  MUSICAL SYMBOL MINIMA
>>> 0x1D1BC, 0x1D1BA 0x1D165  MUSICAL SYMBOL MINIMA BLACK
>>> 0x1D1BD, 0x1D1BB 0x1D16E  MUSICAL SYMBOL SEMIMINIMA WHITE
>>> 0x1D1BF, 0x1D1BB 0x1D16F  MUSICAL SYMBOL FUSA WHITE
>>> 0x1D1BE, 0x1D1BC 0x1D16E  MUSICAL SYMBOL SEMIMINIMA BLACK
>>> 0x1D1C0, 0x1D1BC 0x1D16F  MUSICAL SYMBOL FUSA BLACK
>
>Where do you have this information from?

Try to check your HFS+.

>>> At present, the glyphs of these characters do not exist.
>>> However, I think that it is necessary to support for the future.
>>>
>>>> Will you be able (and willing ... ;-)  ) to do the rewrite? I might do
>>>> it as well, but am not sure when I will find the time to actually do s
>o.
>>>
>>> It's possible.
>>> But I do not understand all of source.
>>> Is it the following files that use UCS2 as internal code?
>>>
>>> codepage.c
>>> unicode.c
>>> unicode.h
>>>
>>>> If you have a patch, I will help with the testing, though.
>>>
>>> First of all, I wrote a sample header file.
>>> This is based on Unicode 5.0.0.
>
>Yeah, it looks good, but it again contains all the decompositions, which
>were deleted from our current table because they were not in tn1150table.
>
>Let's try to find a consensus on what should be in the table before
>going on with the code here.
>
>---------
>
>Another question to HAT:
>
>Reading tn1150 page 34 I found the following sentence:
>
>In addition, the Korean Hangul characters with codes in the range u+AC00
>through u+D7A3 are illegal and must be replaced with the
>equivalent sequence of conjoining jamos, as described in the Unicode 2.0
>book, section 3.10.
>
>Probably we should add these conversions to make Korean Hangul work -
>what do you think?

It's not necessary
This change is from Unicode 1.x to 2.x.
Korean Mac OS 8/9 use not Unicode 1.x but MacKorean via AFP2.
Mac OS X is based on 2.x and later via AFP3.

Summary.
Unicode 3.2 (same as 4.0) is needed for Mac OS X 10.2 -10.5.
Unicode 5 will be needed for future Mac OS X maybe.
Canonical ordering is needed for Mac OS X 10.0 - 10.1.
Singleton is needed??? We should discuss it.

-- 
HAT

Re: [Afpfs-ng-devel] precompose and decompose

[Michael U. <mu...@re...>]

Alex deVries wrote:
> Of course, I don't understand enough about internationalization to
> follow all of this.
> 
> Do the tables need to be AFP version specific?

Alex, HAT,

well, from my understanding the decomposition table does not depend on
the version of AFP but on the _filesystem_ used by the server OS.
According to document tn1150 the decomposition table (tn1150table) is
specified for HFS plus which was introduced with MAC OS 8.1.

tn1150 further states under "Unicode subtleties" that:

--------------------------

IMPORTANT:
An implementation must not use the Unicode utilities implemented by its
native platform (for decomposition
and comparison), unless those algorithms are equivalent to the HFS Plus
algorithms defined here, and are
guaranteed to be so forever. This is rarely the case. Platform
algorithms tend to evolve with the Unicode
standard. The HFS Plus algorithms cannot evolve because such evolution
would invalidate existing HFS Plus
volumes.

--------------------------

Given that HFS Plus is still the preferred filesystem for OSX (even for
the latest version - please correct me if I'm wrong here) the
decomposition table _may_ not be changed for the reasons given above. If
the encoding of HFS Plus names would change with every release of OSX
the volumes would no longer be compatible.

> On 30-Mar-08, at 12:59 PM, HAT wrote:
> 
>> Hi.
>>
>>> The resulting table was further checked against:
>>> http://developer.apple.com/technotes/tn/tn1150table.html
>>>
>>> and decompositions not marked "illegal" in tn1150table were removed from
>>> the result.
>>
>> The tn1150table is based on Unicode 2.x (Mac OS X 10.1 and earlier).
>> Unicode 2.x has no concept of "ordering".
>> Mac OS X 10.2 and later are based on Unicode 3.2.
>> Recent Leopard is based on Unicode 4.
>> The character has increased.
>> Maybe, Mac OS X 10.6 will be based on Unicode 5.x.
>> I think that afpfs-ng should use newest Unicode.
>> At present, latest version is 5.0.0.

HAT, please take a look at the arguments given above. IMHO the
conversion table is (and _must_ stay) independent of any changes in
Unicode support for the different OSX releases.

>>> Please let me know, if there is a newer version of the unicode
>>> decomposition table for HFS Plus.
>>
>> Yes, newest version is 5.0.0.
>> However, it's not necesary to compare it with the tn1150table.
>>
>>> Your suggestion (from your first message) to change the internal
>>> character representation in lib/unicode.c from UCS2 to UCS4 absolutely
>>> makes sense
>>
>> Ok.
>>
>>> although there are currently no decompositions in the range
>>>> U+010000 used by HFS Plus that I'm aware of.
>>
>> Mac OS 10.5.2 Leopard support the following decomposition.
>>
>> 0x1D15E, 0x1D157 0x1D165  MUSICAL SYMBOL HALF NOTE
>> 0x1D15F, 0x1D158 0x1D165  MUSICAL SYMBOL QUARTER NOTE
>> 0x1D160, 0x1D15F 0x1D16E  MUSICAL SYMBOL EIGHTH NOTE
>> 0x1D161, 0x1D15F 0x1D16F  MUSICAL SYMBOL SIXTEENTH NOTE
>> 0x1D162, 0x1D15F 0x1D170  MUSICAL SYMBOL THIRTY-SECOND NOTE
>> 0x1D163, 0x1D15F 0x1D171  MUSICAL SYMBOL SIXTY-FOURTH NOTE
>> 0x1D164, 0x1D15F 0x1D172  MUSICAL SYMBOL ONE HUNDRED TWENTY-EIGHTH NOTE
>> 0x1D1BB, 0x1D1B9 0x1D165  MUSICAL SYMBOL MINIMA
>> 0x1D1BC, 0x1D1BA 0x1D165  MUSICAL SYMBOL MINIMA BLACK
>> 0x1D1BD, 0x1D1BB 0x1D16E  MUSICAL SYMBOL SEMIMINIMA WHITE
>> 0x1D1BF, 0x1D1BB 0x1D16F  MUSICAL SYMBOL FUSA WHITE
>> 0x1D1BE, 0x1D1BC 0x1D16E  MUSICAL SYMBOL SEMIMINIMA BLACK
>> 0x1D1C0, 0x1D1BC 0x1D16F  MUSICAL SYMBOL FUSA BLACK

Where do you have this information from?

>> At present, the glyphs of these characters do not exist.
>> However, I think that it is necessary to support for the future.
>>
>>> Will you be able (and willing ... ;-)  ) to do the rewrite? I might do
>>> it as well, but am not sure when I will find the time to actually do so.
>>
>> It's possible.
>> But I do not understand all of source.
>> Is it the following files that use UCS2 as internal code?
>>
>> codepage.c
>> unicode.c
>> unicode.h
>>
>>> If you have a patch, I will help with the testing, though.
>>
>> First of all, I wrote a sample header file.
>> This is based on Unicode 5.0.0.

Yeah, it looks good, but it again contains all the decompositions, which
were deleted from our current table because they were not in tn1150table.

Let's try to find a consensus on what should be in the table before
going on with the code here.

---------

Another question to HAT:

Reading tn1150 page 34 I found the following sentence:

In addition, the Korean Hangul characters with codes in the range u+AC00
through u+D7A3 are illegal and must be replaced with the
equivalent sequence of conjoining jamos, as described in the Unicode 2.0
book, section 3.10.

Probably we should add these conversions to make Korean Hangul work -
what do you think?

>> --HAT

Thanks + Best regards ... Michael

Re: [Afpfs-ng-devel] precompose and decompose

[Alex d. <ale...@gm...>]

Of course, I don't understand enough about internationalization to  
follow all of this.

Do the tables need to be AFP version specific?

- Alex

On 30-Mar-08, at 12:59 PM, HAT wrote:

> Hi.
>
>> The resulting table was further checked against:
>> http://developer.apple.com/technotes/tn/tn1150table.html
>>
>> and decompositions not marked "illegal" in tn1150table were removed  
>> from
>> the result.
>
> The tn1150table is based on Unicode 2.x (Mac OS X 10.1 and earlier).
> Unicode 2.x has no concept of "ordering".
> Mac OS X 10.2 and later are based on Unicode 3.2.
> Recent Leopard is based on Unicode 4.
> The character has increased.
> Maybe, Mac OS X 10.6 will be based on Unicode 5.x.
> I think that afpfs-ng should use newest Unicode.
> At present, latest version is 5.0.0.
>
>> Please let me know, if there is a newer version of the unicode
>> decomposition table for HFS Plus.
>
> Yes, newest version is 5.0.0.
> However, it's not necesary to compare it with the tn1150table.
>
>> Your suggestion (from your first message) to change the internal
>> character representation in lib/unicode.c from UCS2 to UCS4  
>> absolutely
>> makes sense
>
> Ok.
>
>> although there are currently no decompositions in the range
>>> U+010000 used by HFS Plus that I'm aware of.
>
> Mac OS 10.5.2 Leopard support the following decomposition.
>
> 0x1D15E, 0x1D157 0x1D165  MUSICAL SYMBOL HALF NOTE
> 0x1D15F, 0x1D158 0x1D165  MUSICAL SYMBOL QUARTER NOTE
> 0x1D160, 0x1D15F 0x1D16E  MUSICAL SYMBOL EIGHTH NOTE
> 0x1D161, 0x1D15F 0x1D16F  MUSICAL SYMBOL SIXTEENTH NOTE
> 0x1D162, 0x1D15F 0x1D170  MUSICAL SYMBOL THIRTY-SECOND NOTE
> 0x1D163, 0x1D15F 0x1D171  MUSICAL SYMBOL SIXTY-FOURTH NOTE
> 0x1D164, 0x1D15F 0x1D172  MUSICAL SYMBOL ONE HUNDRED TWENTY-EIGHTH  
> NOTE
> 0x1D1BB, 0x1D1B9 0x1D165  MUSICAL SYMBOL MINIMA
> 0x1D1BC, 0x1D1BA 0x1D165  MUSICAL SYMBOL MINIMA BLACK
> 0x1D1BD, 0x1D1BB 0x1D16E  MUSICAL SYMBOL SEMIMINIMA WHITE
> 0x1D1BF, 0x1D1BB 0x1D16F  MUSICAL SYMBOL FUSA WHITE
> 0x1D1BE, 0x1D1BC 0x1D16E  MUSICAL SYMBOL SEMIMINIMA BLACK
> 0x1D1C0, 0x1D1BC 0x1D16F  MUSICAL SYMBOL FUSA BLACK
>
> At present, the glyphs of these characters do not exist.
> However, I think that it is necessary to support for the future.
>
>> Will you be able (and willing ... ;-)  ) to do the rewrite? I might  
>> do
>> it as well, but am not sure when I will find the time to actually  
>> do so.
>
> It's possible.
> But I do not understand all of source.
> Is it the following files that use UCS2 as internal code?
>
> codepage.c
> unicode.c
> unicode.h
>
>> If you have a patch, I will help with the testing, though.
>
> First of all, I wrote a sample header file.
> This is based on Unicode 5.0.0.
>
> -- 
> HAT
> <compose.h.sample.gz><make- 
> compose 
> .tar 
> .gz 
> > 
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace_______________________________________________
> Afpfs-ng-devel mailing list
> Afp...@li...
> https://lists.sourceforge.net/lists/listinfo/afpfs-ng-devel

Re: [Afpfs-ng-devel] precompose and decompose

[HAT <ha...@fa...>]

Hi.

> The resulting table was further checked against:
> http://developer.apple.com/technotes/tn/tn1150table.html
> 
> and decompositions not marked "illegal" in tn1150table were removed from
> the result.

The tn1150table is based on Unicode 2.x (Mac OS X 10.1 and earlier).
Unicode 2.x has no concept of "ordering".
Mac OS X 10.2 and later are based on Unicode 3.2.
Recent Leopard is based on Unicode 4.
The character has increased.
Maybe, Mac OS X 10.6 will be based on Unicode 5.x.
I think that afpfs-ng should use newest Unicode.
At present, latest version is 5.0.0.

> Please let me know, if there is a newer version of the unicode
> decomposition table for HFS Plus.

Yes, newest version is 5.0.0.
However, it's not necesary to compare it with the tn1150table.

> Your suggestion (from your first message) to change the internal
> character representation in lib/unicode.c from UCS2 to UCS4 absolutely
> makes sense

Ok.

> although there are currently no decompositions in the range
> > U+010000 used by HFS Plus that I'm aware of.

Mac OS 10.5.2 Leopard support the following decomposition.

0x1D15E, 0x1D157 0x1D165  MUSICAL SYMBOL HALF NOTE
0x1D15F, 0x1D158 0x1D165  MUSICAL SYMBOL QUARTER NOTE
0x1D160, 0x1D15F 0x1D16E  MUSICAL SYMBOL EIGHTH NOTE
0x1D161, 0x1D15F 0x1D16F  MUSICAL SYMBOL SIXTEENTH NOTE
0x1D162, 0x1D15F 0x1D170  MUSICAL SYMBOL THIRTY-SECOND NOTE
0x1D163, 0x1D15F 0x1D171  MUSICAL SYMBOL SIXTY-FOURTH NOTE
0x1D164, 0x1D15F 0x1D172  MUSICAL SYMBOL ONE HUNDRED TWENTY-EIGHTH NOTE
0x1D1BB, 0x1D1B9 0x1D165  MUSICAL SYMBOL MINIMA
0x1D1BC, 0x1D1BA 0x1D165  MUSICAL SYMBOL MINIMA BLACK
0x1D1BD, 0x1D1BB 0x1D16E  MUSICAL SYMBOL SEMIMINIMA WHITE
0x1D1BF, 0x1D1BB 0x1D16F  MUSICAL SYMBOL FUSA WHITE
0x1D1BE, 0x1D1BC 0x1D16E  MUSICAL SYMBOL SEMIMINIMA BLACK
0x1D1C0, 0x1D1BC 0x1D16F  MUSICAL SYMBOL FUSA BLACK

At present, the glyphs of these characters do not exist.
However, I think that it is necessary to support for the future.

> Will you be able (and willing ... ;-)  ) to do the rewrite? I might do
> it as well, but am not sure when I will find the time to actually do so.

It's possible.
But I do not understand all of source.
Is it the following files that use UCS2 as internal code?

codepage.c
unicode.c
unicode.h

> If you have a patch, I will help with the testing, though.

First of all, I wrote a sample header file.
This is based on Unicode 5.0.0.

-- 
HAT

Re: [Afpfs-ng-devel] precompose and decompose

[Michael U. <mu...@re...>]

HAT wrote:

>>> 3) decompose: sample only
>> There is no decomposition, since the MAC filesystem does the appropriate
>> conversions before writing. I was not aware of any problems related to
>> the missing functionality so far ...
> 
> Yes.
> It is a problem which standard or implementation is important.

Hi,

hmm, I'm not sure what you mean here ...

> How is a table[] in lib/unicode.c generated?

The data is generated from

ftp://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt

taking the values for "canonical decompositions" from column #5.

I've compared this file with

http://www.unicode.org/Public/5.0.0/ucd/UnicodeData.txt

and they are identical. This would IMHO mean that we are up to date at
least with respect to the set of "officially" defined decompositions.

The resulting table was further checked against:

http://developer.apple.com/technotes/tn/tn1150table.html

and decompositions not marked "illegal" in tn1150table were removed from
the result.

Please let me know, if there is a newer version of the unicode
decomposition table for HFS Plus.

> This is not at least based on Unicode 3.2.
> We should discuss which version to adopt.
> I think that we should use the newest version 5.0.0.

Your suggestion (from your first message) to change the internal
character representation in lib/unicode.c from UCS2 to UCS4 absolutely
makes sense, although there are currently no decompositions in the range
> U+010000 used by HFS Plus that I'm aware of.

Of course it is necessary to handle this character range cleanly when
sending the strings through precompose().

Will you be able (and willing ... ;-)  ) to do the rewrite? I might do
it as well, but am not sure when I will find the time to actually do so.

If you have a patch, I will help with the testing, though.

Thanks + Best regards ... Michael

Re: [Afpfs-ng-devel] precompose and decompose

[HAT <ha...@fa...>]

>Mac OS X 10.5.2 Leopard use newer Unicode.

Oh, Leopard is based on Unicode 4.
http://www.apple.com/macosx/techspecs/